caching - Race-condition creating folder in Python -

-

I have a urllib2 caching module, which crashes oddly due to the following code: 

  If not os.path.exists (self.cache_location): os.mkdir (self.cache_location)

The problem is that until the second row is executed, the folder exists Maybe, and the error will be: 

 File "... / cache.py", in line 103, __init__ os.mkdir (self.cache_location) OSError: [Error 17] File exists: ' / Tmp / examplecachedir / '

This is because the script was launched several times Or has been, there is no control of my third-party code.

The code (before trying to fix the bug) can be found

I can not use it, because it can be used by a randomly named directory () Resolve the situation, which will defeat the purpose of the cash.

T to simply abandon the error, as error error Errno 17 error should be raised if the name of the folder exists as a file (a different error), for example:

< Ex> $ touch blah $ python >>> imported OS >>> os.mkdir ("blah") traceback (most recent call final): file "" in OSERR, line1,: [error 17] file exists : 'Bla' >>> << pre>

I can not use the threading. Rollock As the code is called from several processes.

So, I tried writing a simple file-based lock (), but there is a problem in it: This lockfile creates a level above, so /tmp/example.lock For / tmp / example / , if you use / tmp / as a cache directory (such as /tmp.lock Tries to create.)

In short, I need to cache the urllib2 response to the disk. To do this, I have a known directory The need to reach (it is possible, if necessary), a Bhuproses secure it OS X, Linux and need to work on Windows.

Thoughts? The only alternative way I can think, instead of files, is to rewrite the cache module using the SQLite3 storage.

instead 

  if not os.path.exists (self .cache_location): os.mkdir (self.cache_location)

you can 

  try: osirr: osmcards (save cash_lion): pass

As you end up with the same functionality

Disclaimer: I do not know how this can be. By using

SQLite3 ,

If you choose "many" "To do the work, then , concurrently inserting and filtering, it is a good idea to use SQLite3 , because it does not add much complexity on simple files (argued It can be that it removes complexity).

Reading your questions (and comments) I can understand your problem better.

What is the probability that file can create a similar race situation?

If it is too small, then osource: pass 

  if not os.path.isfile (self.cache_location): try Do: os.makedirs (self.cache_location) P> In addition, reading your code, I will change  
  else: # Our target dior is already a file, or different error, #default Relay! OSERR (E)  
 
 to 
 
  else: # Our goal is already a file from Dior, or increase different error, increase # relay error! Raise  
 
 As it really is what you want, the dragon rereise the exact same exception  (just nitpicking) . 
 
 
 One more thing can be used for you (just like Unix).

Comments

Post a Comment

Popular posts from this blog

MySql variables and php -

-
I am getting an error with php in order to format mysql_query () in php to format this string What is the right way? SELECT SELECT SELECT (*) Agents @AgentCount; User_Agent_English, User_Angent_Oursal, US.___Gent_Aginal, COUNT (User_Agent_Percade) ACC, COUNT (*) / (AgentCount) Percentage by agents user_agent_preser order DEC limit limit 50; In PHP, how am i setting $ query $ query = "SELECT count (*) agents @ agentcount; SELECT user_agent_parsed, User_agent_original , COUNT (user_agent_parsed) ascount, COUNT (*) / (@AgentCount) as a percentage by user_agent_parced by URI date DESC limit 50 "; This exact query will work properly if I put it in MySQL directly through a command line session. Do I need to release two separate php calls from mysql_query () and store the results first? I get the following error: There is an error in your SQL syntax; Instead of using a sub-selection and instead of using the 'SELECT user_agent_parsed, user_agent_original, ne...
Read more

url rewriting - How to implement the returnurl like SO in PHP? -

-
This URL should work even after the rewrite. Is this possible? Otherwise, I can only hardcoded everything If you are trying to create a login system with redirection You need to use the HTTP_REFERER header to find out where the user came from so that you can send it back there. The best way to do this is to use something like this in your login form: // - Login php & Lt; Input type = "password" name = "password" /> & Lt; Input type = "hidden" name = "returnrelale" value = "& lt ;? php echo $ redirectURL ;? & gt;" / & Gt; & Lt; / Form & gt; Please note that some ISP, browser or proxy will delete the referrer header, so you can not rely on each user to work fully. If you use a statement, to check whether the header exists or not, you can send those users back to the homepage or whatever. Edit If you are looking to use $ _SERVER ['REQUEST_URI'], if you are running Apache It sho...
Read more

Which Python client library should I use for CouchdB? -

-
I am starting to experiment with CouchDB because it looks like we have the right solution to certain problems That all the work will be on a new project, which does not have any legacy dependency, which the client library will suggest you use, and why? If we overlap any of the OS. Freeband is available only in the collection of its ports, but asks to use the library's project website. None of those people are Ubuntu, which is only with ships, since those two OSAs are not common in libraries, I am probably installing something from source (and hopefully if I have time, then Ubuntu and FreeBSD People pack the package). For those interests, I want to use CouchDB as a convenient intermediate storage space for data passed between different services - think of a message bus system, but less formalities with. For example, we have daemons that download and pars the web page, then send an interesting bit to other daemons for further processing. Many of them are ill-defined until the...
Read more