Make sure none of your important pages are blocked from indexing If you're not sure whether you have a robots.txt file or not, check the status of the Robots.txt factor in Site Audit.Дў. The main aspects to pay attention to are proper indexing instructions in your robots.txt file and proper HTTTP response codes.ДЎ.Дњheck if your robots.txt file is in place. If your site is hard to crawl and index, you're probably missing out on lots of ranking opportunities on the other hand, you may well be willing to hide certain parts of your site from search engines (say, pages with duplicate content). It's crucial that users and search engines can easily reach all the important pages and resources on your site, including JavaScript and CSS. Step 2: Check crawlability and indexing issues

Finally, hit Next to proceed with the crawling. Select the bot you'd like to crawl your site as if you'd like to discard robots.txt during the crawl, simply uncheck the Follow robots.txt instructions box.

At Step 2, click on the drop-down menu next to the Follow robots.txt instructions option. At Step 1, enter your site's URL and check the Enable expert options box. To do this, create a WebSite Auditor project (or rebuild an existing one). вЂ“ or discard robots instructions altogether and collect all pages of your site, even the ones disallowed in your robots.txt. You may want to tweak this setting to crawl the site as Google, Bing, Yahoo, etc. By default, WebSite Auditor crawls your site using a spider called SEO-PowerSuite-bot, which means it will obey robots instructions for all bots (user agent: *). Crawl your site as Googlebot (or any other bot).

Website auditor not saving