My website is katvik.info . Google bot is successfully indexing the homepage, but does not crawl through the links in the homepage - such as navigation or tiles with individual animal pages.
There is nothing I can see in Google Search Console to explain the issue, which leads me to believe it might be because of the bot protection here. Is there anyway to bypass this issue?
Make sure to add a robots.txt file and a sitemap.xml.
This to files should be present, to ensure that Google fully crawls your site.
Please also note that it can take a couple Days for google to fully crawl your site.
Thanks, is there a sitemap generator that works? All the ones I tried fail at crawling as well.
If you use WordPress or another content management system, the best way to generate a sitemap is to use an option or plugin of your website system to generate it. The integrated module can automatically keep your sitemap up to date when you add and remove content, and isn’t reliant on links on your site to discover posts and pages.
And in my opinion, there is not much point in using a crawler to generate a sitemap, just for the purpose of helping an even more sophisticated crawler index your site. Search engines recommend sitemaps, but they can index your site perfectly fine without one.
Thanks for the suggestion, I’ll try a plugin.
I know that sitemaps are not necessary, especially for small website like mine, but unfortunately the website is not being crawled by google correctly as mentioned in the original post.
How do you know exactly the website is not being crawled? Pages being crawled does not automatically mean they are also presented in search results.
I am using Google Search Console, and it shows which pages have been crawled and from where.
Thanks for the elaboration. To go back to your original message:
Google decides which pages to crawl at what time at their own discretion. If they only want to index your home page and no other pages, that’s all up to Google to decide.
If there was anything which is blocking Google from accessing your website, they would say so.
The bot protection system has been tested to not affect search engine crawlers. Search index crawlers do execute Javascript code, allowing them to complete the browser validation challenge. Most sitemap generators don’t do that, and just do a simple HTML code fetch, which is blocked.
@tagrendy If you want it to be indexed, you must use the url inspection tool. It will then say whether it can be indexed. If not, change it so it meets google criteria. Then click 'Request Indexing`. Then if you want the site itself, type in ‘site:your.domain’. I have done this before. If you need help, feel free to ask. You may message me.
My site is irishluckyman.com.
I’ve had very few hits over the 6 months since it was launched. But that hasn’t mattered as I’ve been building and changing it slowly over that time.
Today I wanted to know if the google bot had visited, crawled and indexed my site.
Google advised that I needed to add the following text to infinity’s DNS address to effect a google crawl and that my site needed to be verified.
google-site-verification=E6_R9pUPnzPNdwHDW6QV5VVUoos1w7dtJY2WAXnYub0.
OK so far.
But when I went to my infinity control panel and found the entry for my DNS server address it became unclear how or if I should substitute (or add) to what is already there.
Your guidance please.
You can’t change DNS, that is domain method. Verify by URL. You’ll have to upload as many files as how many URLs you want to be indexed.
Here are steps with screenshots. I will be using one of my URLs in a real domain(it is a dummy one, it is named dummy).
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
