It only crawls 2 pages.
https://onlinebankifsc.com/
https://onlinebankifsc.com/aes.js
What is aes.js. I didn’t add it.
To answer your question, aes.js is part of a browser validation system we use to protect your website. We added it, and it cannot be turned off.
But why would you use a site crawler to generate a sitemap? The point of a sitemap is to provide information a crawler cannot figure out on it’s own. And Google’s crawler is a lot more sophisticated than whatever free online sitemap generator you found and can do a much better job of mapping your site.
1 Like
I do NOT trust Google to have access to my information and crawl my page, hence I want to crawl it myself and exclude THEM from it. Capiche?
1 Like
Google takes time to crawl and index each and every page and hence I want to generate complete sitemap and submit it in Search Console to speed up the proces.
You can use this service. I tested it with my IF website and it works.
Yes but it only crawls the given webpage and generates only 1 sitemap link in XML.
I want to crawl my complete website which has approximately 2.5 lakh links. Due to the above security (aes.js) it does not crawls it completely.
If you want a sitemap, you should generate it from your website software. Many CMS can generate sitemaps for you based on their contents. If you wrote your own site by hand, you can also write the XML sitemap in the same way too.
Perhaps sitemaps can help a bit, but it’s one of the many different things that can influence crawling speed. Signing up to Google Search Console and requesting a reindex there also helps a lot.
Yes I have paid Screaming Frog software which generates sitemap. But it is only generating sitemap for below URL’s
https://onlinebankifsc.com/
https://onlinebankifsc.com/aes.js
Rest of the links are not crawled by the spider. Any idea ?
I guess this is because aes.js
Hi
Yes, it is due to the security system - anything unable to store cookies or execute JS cannot pass on
Since you have your own domain you can use Cloudflare and in that way our security system will no longer play a role,
so you can then use some online sitemaps tools.
2 Likes
So if you setup Cloudflare on your domain, you will be able to bypass the security system that is preventing this. Here is a guide to get your started.
@Oxy, I wasn’t sure if Cloudflare would get around this, so I desided not to say anything until I did a test (Witch I haven’t done yet), so thanks for clarifying.
2 Likes
Sounds good. Will use Cloudflare and I am familiar with it. Thanks !
1 Like
I have setup Cloudflare and it was working fine but today it shows
My website IP is down: 185.27.134.60. Please check.
Yes working fine now. Thanks !!
2 Likes
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.