Publishing a page and having it found are two different events, separated on a large catalogue by weeks. This article looks at the stage nobody reports on: how a crawler spends its time on your site, how sitemaps steer it, and what the Indexing Hub in the Semalt panel actually controls.
Plenty of sites in Metro Vancouver have no fixed number of pages. A tour operator adds departure dates in February and retires them in October. An outdoor retailer leaves a spring and a fall range live at once. A North Shore brokerage lists eleven properties this week and removes seven.
That movement is the problem. Nothing tells a search engine the inventory changed, so it finds out on its own schedule — perhaps six weeks after the booking window closed.
A page that exists is not a page that is found
The CMS will give you a page total to the digit. Nobody can give you the number Google holds, and the gap between those two figures is where seasonal businesses lose quarters.
Three things happen in order, each failing on its own terms. The engine learns the address exists. A bot requests it and gets a usable response. Only then does the engine decide whether to keep the page. Teams collapse all three into one word and argue about the wrong stage.
- Discovery is an information problem. Nothing links to the URL and it sits behind a date picker no crawler will operate. The engine is not ignoring the page — it has never heard of it.
- Crawling is a resource problem. The bot has a finite appetite for your domain each day, and can spend all of it on filter combinations leading nowhere.
- Acceptance is a judgement. Thin availability stubs and near-identical variants lose it routinely.
- The remedies do not transfer. Pushing harder does nothing for a page already fetched and set aside. Rewriting does nothing for one no bot has requested.
One case makes the split obvious. A Whistler shuttle operator publishes 340 date-specific availability pages in November; by January, 90 are indexed. No logged visit against the missing 250 points to discovery, where a push helps. A clean fetch on all of them means pushing achieves nothing.
Crawl budget, and what actually consumes it
Crawl budget is not an allocation you can buy or request. It emerges from two measurements the engine keeps continuously: how fast and how reliably your server answers, and what the domain has earned. Return 503s during a Saturday booking rush and you are crawled carefully for weeks, because a crawler throttles itself rather than topple a struggling host.
The uncomfortable part is that most of it goes to URLs nobody wanted crawled. Software generates them, not a content plan, and they outnumber the real pages tenfold.
| What eats the budget | How it appears on a Lower Mainland catalogue | Scale | Where the correction belongs |
|---|---|---|---|
| Faceted navigation | Size, colour, brand and price filters, each combination answering 200 | Tens of thousands of URLs from a few hundred products | Robots rules and canonicals on filter states |
| Date and session parameters | Booking calendars minting a URL per date pair a visitor tries | Effectively unbounded | Parameter handling at the application layer |
| Expired inventory left live | Sold listings and last season's departures returning 200 | Grows every week the site runs | A retirement rule: 410, 301 or replacement |
| Slow responses under seasonal load | Response times tripling when demand peaks and new pages appear | Depresses the rate for weeks | Capacity planned for the season, not the average |
The arithmetic is stark. Four thousand products across six filter dimensions present well over 100,000 crawlable addresses, so Tuesday's new arrivals wait behind a queue of colour-and-size permutations. Nothing breaks, nothing is logged, and the range surfaces a month late.
Catalogues that never hold still
The businesses feeling this most sharply share one trait: their page set follows inventory rather than an editorial calendar, and availability changes weekly.
Seasonal availability pages
Departures, sailings and room types make a page per date range, most alive for months.
- Indexed too late to sell anything
- Last year's dates never removed
Large outdoor and lifestyle ranges
Several thousand SKUs with variants, plus filter states the storefront makes on its own.
- New ranges queue behind old permutations
- Discontinued lines linger indefinitely
Listings with a two-week life
Listings appear, sell and vanish faster than a normal crawl interval.
- Indexed after the property is gone
- Demand spikes on policy news
A .ca and a .com in parallel
One catalogue, two storefronts, two currencies, largely the same words.
- Every product exists twice
- Signals contradict each other
- US results outrank the Canadian store
All four invert the same way: the page that matters most has the shortest useful life, while the pages that never expire absorb the crawler's attention. That inversion is the quiet bottleneck, and publishing more does not touch it.
Seasonality adds a layer. Month-on-month comparisons tell you little, because the inventory itself moved. Year-on-year is the honest baseline: how much of last February's availability was indexed by mid-January, against this year.
Sitemaps do a job nothing else does
The sitemap is usually a formality produced by a plugin nobody has opened in three years. On a site whose inventory turns over weekly it is the only reliable way to say what changed, and its structure decides whether you can read the answer.
The Hub takes a sitemap as an uploaded file or a URL and parses it recursively to three levels — an index pointing at further indexes pointing at the files that list addresses. One job covers up to 1,000 sitemaps, which makes a split structure practical rather than theoretical.
A sitemap tree that survives a season
Split by the way your inventory behaves, so a lagging section is visible without further investigation.
- Level one is a single index. One file the engine and the Hub both point at, listing the section indexes below it.
- Level two follows turnover, not menus. Separate indexes for evergreen pages, current-season inventory, the catalogue and the French tree.
- Level three holds the addresses. Files of manageable size, so a failure lands in one place rather than across the tree.
- Timestamps must mean something. A lastmod that changes with every nightly rebuild trains a crawler to ignore your dates altogether.
- Only live, canonical URLs belong. No redirects, no 404s, nothing noindexed, nothing whose canonical points elsewhere.
- Retirement is part of publishing. Removing a sold listing is the same task as adding it; most sites automate only the first half.
Structured that way, the sitemap becomes a measurement: a hundred submitted and thirty found in the current-season index tells you which part of the site is stalled.
Pages that change rarely
Home, categories, guides, contact and location pages.
- Stable lastmod values
- Re-submitted only after real edits
Inventory with an expiry date
Availability, departures, listings, this season's range.
- Rebuilt daily from live inventory
- Entries removed the day they expire
The ceilings inside the Indexing Hub
In the panel, the Indexing Hub sits beside the campaign, analytics and rank-tracking modules, and it handles everything preceding a ranking. Four numbers bound it.
What the module will and will not process
Throughput, batch size, depth and concurrency — what any submission calendar must respect.
- One thousand a day, per account. The tracker's throughput ceiling, and it does not roll over: a quiet Sunday buys nothing on Monday.
- Ten thousand in one batch. Batch size is a hand-off, not a rate. Fill one and you have queued ten days of work.
- Sitemaps read three levels down. Upload a file or give an address; nested indexes are followed recursively, to a limit of 1,000 sitemaps in a job.
- Two jobs run, twenty wait. Concurrency is fixed, so a large migration is sequenced rather than parallelised.
Pushes go out through IndexNow, the API that tells crawlers signed up to it — GoogleBot and BingBot included — that something at an address is new or edited. Rather than waiting for a bot to wander back into a neglected corner of the catalogue, you announce the change as it happens: worth most on the day a listing goes live, and again on the day it sells. The module overview for the rebuilt Semalt panel puts this next to the analytics side.
The middle stage, though, the Hub can prove. Every address carries its own record: the visiting bot, the time it called, the status it received, and the detail behind a failure. Three counters run live above all that — submitted, found, failed.
How to read the status of a batch
A batch leaves you three numbers and a record. The numbers mean something only against each other, and the record settles the argument over whose problem this is.
| What the counters show | The likely reading | What to check next |
|---|---|---|
| Submitted high, found low, few failures | Notified, but no bot has arrived yet | Internal linking — orphans are fetched slowly |
| Found near submitted, indexed count flat | Fetching is fine; pages were read and set aside | Duplication — availability stubs and variants first |
| Failures clustered in one section | A routing or template fault in one branch | Error detail, then that section's sitemap file |
| Failures scattered thinly everywhere | Server capacity during the crawl window | Response times at the logged timestamps |
| Timestamped 200s on pages that never appear | The engine read the page and declined it | Nothing here — this moves to content |
That last row is why the record earns its keep. A logged 200 proves the bot called and your server answered; it says nothing about storage. Holding those two apart converts "traffic is down" into a question somebody can be assigned.
Read it weekly rather than daily, since crawling moves in days, and read the detail behind a failure rather than its count: forty errors is a number, forty errors sharing one status on one path is a bug report.
The cross-border duplicate, and the season that moves it
A .ca and a .com running side by side is normal here and rarely planned: the Canadian storefront comes first, a US version follows because a third of the orders ship south, and the catalogue is duplicated with the currency swapped. To an engine that is one catalogue at two addresses.
The result is not a penalty but a choice made for you — one version preferred, the other mostly ignored, and the winner is not always the one carrying your revenue.
| Arrangement | The engine's reading | What it costs | A workable response |
|---|---|---|---|
| Identical catalogue on .ca and .com | One set of pages at two addresses | Capacity spent twice on one catalogue | Differentiate pricing, shipping and availability, and target regions consistently |
| US competitors on Canadian queries | Stronger domains answering the same intent | No direct cost, but a higher bar | Specifics a Seattle retailer cannot match: local stock, pickup, conditions |
| A French tree translated by machine | A second, weaker copy of every page | Daily allowance spent on pages nobody keeps | Rewrite the pages carrying revenue, drop the rest |
Seasonality decides when the work pays. Pushing next summer's availability in April is late; the window opens the moment the pages exist. Property is the exception — those queries follow policy announcements, so listings want continuous submission rather than a seasonal burst.
What to submit is an evidence question. The Search Console views in the same workspace show which pages earn clicks and impressions today, with keyword dynamics following what enters and leaves the top ten, and an export runs to 10,000 rows of CSV or JSON. Sort it by what already earns and you have your order of work; Stream, the assistant beside My SEO, will take the list a batch at a time.
When the engineering queue is the real constraint
Parameter handling and retirement rules need developer time. The keyword and link side need not wait for it.
- Keyword and link work runs in parallel. AutoSEO, 149 USD a month for one domain, does keyword discovery, prioritisation and link building without touching the release process.
- Manual control where it matters. FullSEO, at 500 USD, hands the controls back: keyword choice with an automatic fallback, link placement aimed at a domain rating, and human review before on-site edits ship.
- Four to eight weeks. That is the usual wait for the first measurable movement — an argument for starting a season ahead of it.
Questions that come up
Does a URL pushed through the Hub end up indexed?
No, and a supplier who says otherwise is overselling. A push tells subscribing crawlers that an address is new or edited. The fetch, and the decision to store what was fetched, belong to the engine. You are buying the removal of a delay, not a verdict.
Why 1,000 a day if a batch holds 10,000?
The two measure different things. Batch size is how much you hand over at once; the daily figure is how much moves. Hand over ten thousand and you have scheduled roughly ten days, so plan against the daily number.
Should seasonal availability pages be submitted every year?
Submit them once, as soon as they are published, and retire last year's set at the same time. Re-submitting an unchanged URL achieves nothing and consumes allowance. New dates on new URLs are new pages and belong in the queue immediately, months before demand.
Our property listings sell in two weeks. Is indexing them worth it?
For the individual listing, often not — it outlives its usefulness before a crawler settles on it. The value sits above: neighbourhood, price band and property type pages that persist, kept current by the listings beneath them. Submit the durable pages and let the listings feed them.
The closing calculation, and what to do first
Take a realistic case. An outdoor retailer replatforms: 6,200 products, variants, category and seasonal landing pages, a guide archive and a US storefront. On new paths, the migration leaves roughly 60,000 addresses to be found again.
At the daily ceiling that is 60 days of processing — arithmetic, not a shortcoming of the tool, and worth saying out loud before anyone promises a recovery in three weeks. Six batches carry the volume; the daily number sets the speed.
Sixty days is only a problem if the pages that pay the bills land on day fifty-five. Ordered by measured value rather than by whatever the generator emitted, the commercial pages clear inside two weeks.
| Days | Contents | Volume | Reasoning |
|---|---|---|---|
| 1–3 | Home, top categories, the 200 products that carried last year | 2,800 | Searched for by name; they convert at once |
| 4–9 | The rest of the current-season range and its categories | 6,000 | The season is running; a day of delay is unrecoverable |
| 10–25 | Remaining products with impressions, plus the guides | 15,000 | Proven earners, restored ahead of guesswork |
| 26–60 | Variants, tag pages, paginated archives, discontinued lines | 36,000 | Little measured value — most of it should be deleted |
That final row deserves a hard look before it consumes five weeks of allowance. If 36,000 addresses produced no impressions all year, the treatment is consolidation or deletion, not submission.
Start small and start this week. Clean the sitemap down to live canonical addresses, split it by how fast each section turns over, pull the export of pages that already earn, and work through it in that order at the pace the tracker allows. Let each week's failure clusters choose the next task. To run indexing, analytics and campaigns from one place, sign in to the Semalt dashboard and add a property. Where an outside team takes over is set out on our services page, and more on search in the Lower Mainland sits on the blog.