Check for files (robots.txt, favicon.ico) to a website php -

I would like to check on a remote website if there are some files in it for example. robots.txt , or favicon.ico must have files accessible (read mode).

So if the website is: http://www.example.com/ I have to check whether the HTTP://www.example.com/robots .txt .

I tried to fetch the URL like http://www.example.com/robots.txt . And sometimes you can see that the file is there because you did not get the header error in the page.

But some websites handle this error and you call some HTML code that the page is not found.

You get the header with status code 200.

So how will anyone know if the file is actually or not or not?

Thanax, Gannit

If they serve an error page with HTTP 200 I suspect you have a reliable way to find out that it is unnecessary to say that it is stupid to serve like a page with error ...

You can try:

1. Issuing a head request that only gives you the header for the request processing. You may find a more reliable status code.
2. Check the content-type header if it's text / html , then you can assume that it's a Robots.txt instead of a custom error page (which should be served as text / plain ). Similar to favicon but I think checking the text / html is the most reliable way.