Google is crawling the web differently. That’s clear from both its official updates and the traffic changes sites are seeing.
The shift is simple: Google wants to crawl less and crawl better.
It’s part of a wider move to prioritise useful, unique, and worthwhile content for users. Sites that don’t meet this standard are seeing less visibility and, in some cases, much less traffic.
GET THE INTEL
What’s Changed?
In March 2024, Google rolled out a significant core update. It promised to reduce low-quality and unoriginal content in search results by 40%.
That wasn’t just a tweak. It caused a substantial drop in performance for many websites, which the BBC highlighted in a detailed analysis. It showed how entire sites with weak or repetitive content have fallen down the rankings. And so publishers that rely on quantity over quality felt the impact and had a helluva hill to climb.
And it’s not just the algorithm. Google’s crawling behaviour has changed, too.
Their own documentation on crawl budget explains it clearly: Google won’t waste time crawling everything. It now prioritises sites that are fast, well-structured, and updated consistently with valuable content.
The message is clear. If your site is slow, full of duplicate content, or only occasionally posts a random blog, Googlebot will visit less often. It has become a more selective, resource-conscious system.
Real-World Proof
Evidence From the Industry
- Consolidating pages
- Adding no-follow no-indexing need-to-have-but-do-not-need-to-be-crawled pages
- Cleaning up old or thin content
- Removing unnecessary redirects
- Deleting unnecessary categories, tags, filters and more
Search Engine Journal points out that the crawl budget is affected by server performance. If your site is slow to respond, Googlebot may simply stop trying.
These shifts aren’t just technical – they’re strategic. They reflect Google’s aim to serve better content and reduce spam. Their SpamBrain system continues to evolve, detecting low-value content with increasing accuracy.
Key Sources Evidencing Google's Increased Crawling Selectivity
- Google's March 2024 Core Update: Google's March 2024 Core Update aimed to reduce low-quality, unoriginal content in search results by approximately 40%. - Search Engine Journal
- BBC Analysis on Google's Algorithm Changes: An analysis by the BBC highlighted that Google's recent updates have significantly impacted sites with low-quality content, leading to substantial traffic losses. - BBC
- Google's Official Documentation on Crawl Budget Management: Emphasises the importance of managing crawl budgets effectively, especially for large sites, to ensure that important pages are crawled and indexed. - Google Developers
- Insights from Vizion on Googlebot's Behaviour in 2024: How Googlebot's crawling frequency varies based on page importance and update frequency, indicating a more selective crawling approach. - Vizion Interactive
- Search Engine Land's Guide on Crawl Budget in 2025: Only certain websites, such as large sites with over 1 million unique pages, must actively manage their crawl budget, reflecting Google's selective crawling practices.
Plan of Action
What Should Businesses Do?
This isn’t just a concern for enterprise websites. All businesses should pay attention.
We’ve put together three actions that you can take now to help Google crawl your site efficiently and put your most valuable content front and centre.
Click below to expand
If you want to be found, you need to be worth finding.
Google isn’t crawling less content because it wants to be difficult. It’s crawling less because it wants to surface better results.
If you need a little assistance getting the above steps actioned, we’d really love to help you beat the bloat!
Grab our free audit and we’ll go from there.
