cheapestrugsonline.com.au
Crawl Budget Recovery
for a Large Catalogue
Implemented Generative SEO and entity SEO across a large home décor catalogue — combined with a GPT-based automation workflow for product SEO and a structured data playbook rolled out at scale across all listings.
A large catalogue bleeding crawl budget with no AI search presence.
Cheapestrugsonline.com.au had thousands of product listings but was burning its entire crawl budget on low-value pages — while simultaneously missing out on the growing wave of AI-powered search results for home décor queries.
Thousands of faceted filter URLs, colour/size variants, and out-of-stock product pages were being crawled repeatedly — leaving key collection pages under-indexed and slow to rank.
As Google's AI Overviews and other AI search tools began dominating home décor queries, the site had no structured data, no entity optimisation, and no content formatted for AI citation.
Product pages were templated with minimal unique content — no entity-rich descriptions, no buying guides, no contextual signals that would help Google understand the topical relevance of each listing.
With thousands of SKUs, manually improving each product page was not feasible. There was no automation in place to apply SEO improvements at catalogue scale.
Automation-first SEO for a catalogue that couldn't be touched manually.
The challenge demanded a programmatic solution — not page-by-page editing, but a systematic workflow that could improve thousands of pages simultaneously.
Used log file analysis and Screaming Frog to map exactly which URL types were consuming crawl budget. Built a priority matrix separating high-value indexable pages from low-value crawl traps.
Implemented robots.txt rules, noindex tags, and canonical directives across all faceted navigation, filter combinations, and variant pages — concentrating crawl budget on collection and best-seller pages.
Built a GPT-powered workflow to generate entity-rich, unique product descriptions at scale — feeding category context, material attributes, and use-case signals into each description to improve topical relevance.
Deployed a structured data playbook covering Product, Offer, and BreadcrumbList schema across all listings — templated for scale and validated through Google's Rich Results Test.
Optimised category pages and key product clusters for entity recognition — using semantic markup, co-occurrence signals, and FAQ content to improve visibility in AI-generated search summaries.
Crawl efficiency restored. AI search visibility unlocked.
Results compounded as crawl budget improvements accelerated indexation of the newly optimised product content.
Overall organic impressions grew 15% as previously under-crawled category and product pages gained indexation and began ranking for long-tail queries.
Key product categories began appearing in AI-generated search overviews for home décor queries — a direct result of structured data implementation and entity SEO.
Thousands of product pages received unique, entity-rich content through the GPT automation workflow — without manual editing of individual pages.
What this project taught me.
Crawl budget is more valuable than most SEOs realise. For large catalogues, fixing crawl efficiency often has a bigger ranking impact than creating new content. Googlebot's time on your site is finite — every wasted crawl is a missed indexation opportunity.
GPT automation unlocks SEO at catalogue scale. Manual content improvement doesn't scale past ~200 pages. A well-designed GPT workflow with proper SEO prompting can improve thousands of pages in days — and the quality is good enough to move rankings.
AI search is already here — structured data is your entry ticket. AI Overviews are now present on a significant portion of product-intent queries. Without Product schema and entity signals, you're invisible to this growing traffic source.
Large catalogue with an SEO problem?
I specialise in scalable SEO solutions for large eCommerce catalogues — from crawl recovery to AI-powered content automation.