multithreading - How check if a task is already in python Queue? -

-

I am writing a simple crawler in Python using threading and Q modules. I get a page, check the link and place them in a queue, when a certain thread process ends the page, it grabs the next one from the queue, I use an array for those pages I am looking to filter the links that I have already added to the queue, but if there are multiple threads and they get the same link on different pages Qatar put in duplicate links I've already exists in the URL line that I can find out how to do it or not, to avoid putting it again?

If you do not care about the order in which the items are processed, then I I try a sub-class of which internally uses set : 

  class setQueue (queue): def _init (self , Maxsize): self.maxsize = maxsize self.queue = set () def_put (self, item): self.queue.add (item) def _get (self): Paul McGuire told that to add this "duplicate item" The latter has been removed from the "to-b-processed" set and has not yet been added, To resolve this on your own KQPop ()

set of "processed", you can store both sets in the line example , But when you are checking in a large number that the item has been processed, you can also go back to the queue which will order the requests properly. 

  class setQueue (line): def _init (auto, max): QE (self, max) self.all_items = set () def _put (auto, item): if The item is not in the self. All_items: Qi._put (self, items) self.all_items.add (item)

Using a set aside, unlike its profit, is that line < / Code> methods are thread-safe, so that you do not need extra locking to check the other set.


Comments

Post a Comment

Popular posts from this blog

asp.net - Javascript/DOM Why is does my form not support submit()? -

-
This is my first time working with asp.net and javascript, so I do not know very good web resources, or Much about javascript debugging I have to find out that on line oFormObject.submit (); Microsoft JScript Runtime Error: Object does not support this property or method I am using the link to work as a button because I am terrible in aesthetics and I think That link will look more professional than a table of 50 buttons. I have read that this solution can cause problems with browsers due to pre-rendering of my links and many additional traffic and problems, but CSS gives me a button like a link, which has issues of alignment and difference seemed to. & lt ;! DOCTYPE HTML PUBLIC "- // W3C // DTD HTML 4.01 // N" "http://www.w3.org/TR/html4/strict.dtd"> & Lt; Html xmlns = "http://www.w3.org/1999/xhtml" & gt; & Lt; Head & gt; & Lt; Link rel = "stylesheet" type = "text / css" href = "defect differen
Read more

sockets - Delphi: TTcpServer, connection reset when reading -

-
I am trying to implement one for Delphi, but there are some problems with communication. Fitnesse will start my process, and give me a portmember as a command line argument. Then I should make a socket on the port number given to me, and fitness will connect to that port I am using a TCPCP server for the job: < Pre> TcpServer1.LocalPort: = ParamStr (ParamCount); TcpServer1 Active: = true; On OnEXEPT () -Event, I send the protocol version to use as specified in the protocol. Process TForm1.TcpServer1Accept (From: TOBject; ClientSocket: TCustomIpClient); Var s: ansistring; Start clientSocket. Syndrome ('slim - v 0.0', # 10); SetLength (S, 6); ClientSketts Receavebook (S, 6); End; When I call ReceiveBuf (), the process ends, and fitness throws an exception: java.net.SocketException: connection Reset I have seen what sent and received. It shows that my code sends the protocol version, fitness sends back a message, and when I try to get this message
Read more

javascript - Classic ASP "ExecuteGlobal" statement acting differently on two servers -

-
I have a strange problem that I really do not know. In addition to developing a new ASP.NET site, I have to support the old "classic ASP" site. It is written in VBScript with a batch of JavaScript functions. Many of the JavaScript functions include 'files' Everything is fine on our production system (which security is controlled by another group - this is a military installation) Debugging them once did not work I). However, I have a fix for a vague bug that needs to be fully tested and some functions do not run under the "local" copy of the website that runs on my PC. For the record, I am running IIS 5.1 on XP. The problem is that some Javascript functions are running "OK" on the production server, but when they use it as a server, they are not defined on my PC what is happening that last Programmer, for some reason, has decided to put some functions in a series of files, which according to the observations, imitate the "need_once
Read more