multithreading - How check if a task is already in python Queue? -

-

I am writing a simple crawler in Python using threading and Q modules. I get a page, check the link and place them in a queue, when a certain thread process ends the page, it grabs the next one from the queue, I use an array for those pages I am looking to filter the links that I have already added to the queue, but if there are multiple threads and they get the same link on different pages Qatar put in duplicate links I've already exists in the URL line that I can find out how to do it or not, to avoid putting it again?

If you do not care about the order in which the items are processed, then I I try a sub-class of which internally uses set : 

  class setQueue (queue): def _init (self , Maxsize): self.maxsize = maxsize self.queue = set () def_put (self, item): self.queue.add (item) def _get (self): Paul McGuire told that to add this "duplicate item" The latter has been removed from the "to-b-processed" set and has not yet been added, To resolve this on your own KQPop ()

set of "processed", you can store both sets in the line example , But when you are checking in a large number that the item has been processed, you can also go back to the queue which will order the requests properly. 

  class setQueue (line): def _init (auto, max): QE (self, max) self.all_items = set () def _put (auto, item): if The item is not in the self. All_items: Qi._put (self, items) self.all_items.add (item)

Using a set aside, unlike its profit, is that line < / Code> methods are thread-safe, so that you do not need extra locking to check the other set.


Comments

Post a Comment

Popular posts from this blog

MySql variables and php -

-
I am getting an error with php in order to format mysql_query () in php to format this string What is the right way? SELECT SELECT SELECT (*) Agents @AgentCount; User_Agent_English, User_Angent_Oursal, US.___Gent_Aginal, COUNT (User_Agent_Percade) ACC, COUNT (*) / (AgentCount) Percentage by agents user_agent_preser order DEC limit limit 50; In PHP, how am i setting $ query $ query = "SELECT count (*) agents @ agentcount; SELECT user_agent_parsed, User_agent_original , COUNT (user_agent_parsed) ascount, COUNT (*) / (@AgentCount) as a percentage by user_agent_parced by URI date DESC limit 50 "; This exact query will work properly if I put it in MySQL directly through a command line session. Do I need to release two separate php calls from mysql_query () and store the results first? I get the following error: There is an error in your SQL syntax; Instead of using a sub-selection and instead of using the 'SELECT user_agent_parsed, user_agent_original, ne
Read more

c# - How can I find out if there are windows above a control? -

-
If I have Winforms controls, can it be possible to tell whether the window above it (from any application)? Actually, I have to know that some parts of my control are actually appearing on the screen. If you are happy to p / call, the EnumWindows function provides all top-level windows HWND on the screen is; This allows you to get a Non-Client Rectangle (GetWindowRect) to compare the limits of your form. This non-rectangular opaque areas (media player and fancy skins for choice), so you can be false positive for such a control window.
Read more

javascript - Classic ASP "ExecuteGlobal" statement acting differently on two servers -

-
I have a strange problem that I really do not know. In addition to developing a new ASP.NET site, I have to support the old "classic ASP" site. It is written in VBScript with a batch of JavaScript functions. Many of the JavaScript functions include 'files' Everything is fine on our production system (which security is controlled by another group - this is a military installation) Debugging them once did not work I). However, I have a fix for a vague bug that needs to be fully tested and some functions do not run under the "local" copy of the website that runs on my PC. For the record, I am running IIS 5.1 on XP. The problem is that some Javascript functions are running "OK" on the production server, but when they use it as a server, they are not defined on my PC what is happening that last Programmer, for some reason, has decided to put some functions in a series of files, which according to the observations, imitate the "need_once
Read more