Robots.txt isn't available when I try accessing it

Hello!
I tried adding a “robots.txt” to the “htdocs” folder of my website, so Google can crawl it. However, when I try accessing it, like domain.cf/robots.txt it keeps redirecting me to my 404 page.

This should be the URL to my robots.txt file: https://strangertime.cf/robots.txt
This is my website URL: https://strangertime.cf/

I am trying to index my website into Google and it kept rejecting it, so I thought it might be because of the robots.txt.
Not sure if I am doing something wrong or it is an error.

  1. It’s accessible just fine
  2. Google crawler should work here…
    Try resubmitting?
1 Like

Alright, maybe it is my browser only that cannot access that file.
I am going to try to see why Google won’t index it. Thank you!

Try clearing your cache. You can also add “meta:noindex” tags on your pages if you do not feel that the robots.txt is working.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.