How often can software read an independent shop's catalogue?

Partle tested 205 OpenStreetMap-derived storefront websites in Illinois and Pennsylvania. A homepage-markup probe found 3 sites at its extraction threshold (1.5%). Platform-specific catalogue readers found products at 21 sites in the original 2026-08-09 run (10.2%). A complete replication on 2026-08-31 found accepted products at 26 sites (12.7%).

The result is about extraction methods, not the prevalence of online shopping in the United States. The sample covers two states, was filtered for Partle's catalogue work, and is not a probability sample. It shows that reading only homepage product markup misses catalogues that a platform-aware reader can retrieve.

Results

Measurement	Date	Stores	Share of 205	What it means
Markup-probe threshold	2026-08-09	3	1.5%	The homepage probe found enough product signals to reach its score threshold.
Platform-and-catalogue prefilter	2026-08-09	39	19.0%	The site appeared to use a supported platform and to expose catalogue signals. This did not prove that products could be returned.
Products returned by the original extractors	2026-08-09	21	10.2%	The original report says the completed extractors returned products. Its row-level output was not preserved.
Platform-and-catalogue prefilter	2026-08-31	39	19.0%	A new complete 205-row run reproduced the prefilter count.
Accepted product-bearing result	2026-08-31	26	12.7%	The current importer returned at least one product and its quality gate accepted the batch.

The 2026-08-31 importer completed normally for 40 stores, but 14 of those returned zero products. Three more stores returned only site-builder demonstration products and were rejected. Ninety-one had no supported platform, 69 failed during fetching, and two were skipped because their earlier robots responses had denied this crawler.

The replication capped each store at five returned products. Its total of 138 fetched products is therefore a test of whether the catalogue reader works, not an estimate of catalogue size.

Method

The sampling frame contains 88 Illinois rows and 117 Pennsylvania rows. us_osm_to_registry.py generated the registries from OpenStreetMap data, then applied Partle's chain blocklist, domain-frequency filter, redirect check, and store-locator check. Every row has a unique key, homepage, state, and coordinates in the source registry.

The historical markup probe inspected homepage JSON-LD, Open Graph product tags, and product-link patterns. Its saved output contains positive results rather than a full 205-row ledger, so the 1.5% figure is retained as a dated aggregate result, not presented as independently reproducible from the packet alone.

The 2026-08-31 replication ran two programs against the unchanged 205-row frame on Partle's y540 scraping host:

python us_platform_survey.py \
  --input us_store_registry_il.json --input us_store_registry_pa.json \
  --workers 10 --timeout 20 \
  --output us_platform_survey_2026-08-31.json

python run_platform_import.py \
  --input us_store_registry_il.json --input us_store_registry_pa.json \
  --dry-run --workers 10 --timeout 20 --limit-per-store 5 \
  --output import_outcomes_2026-08-31.json

The dry run opened no database connection and wrote no products. The importer applied the current robots denylist and batch quality gate. A result counted as product-bearing only when its status was ok and it returned at least one product. Platform detection without returned products did not count.

Download and reproduce

• Sampling frame (CSV)
• Merged store outcomes (CSV)
• Raw platform survey (JSON)
• Raw controlled-import outcomes (JSON)
• Aggregate summary (JSON)
• Source commits, commands, hashes, and run settings (JSON)
• Dataset notes and licence
• Independent spot-check
• Verification script

The source manifest pins the registry, survey, and controlled-importer commits and gives a SHA-256 hash for each published file. The raw files contain store-level status and error evidence, but no product descriptions, images, or catalogue dump.

Limits

• Illinois and Pennsylvania do not represent every United States retailer.
• The registries deliberately try to remove chains, but the 2026-08-09 manual review found some chains, wholesalers, manufacturers, and off-topic businesses still present.
• A website can change, block a request, expire, or move its shop between runs. The 2026-08-31 files are a dated observation.
• has_catalog is a prefilter. Installed commerce software, a shop button, or a demonstration catalogue can produce a positive signal without a usable product feed.
• A returned product is not proof that its price, stock, or technical suitability is correct.
• The original 21-store row output was not preserved. The later 45-store full-run note used "reachable catalogue", which included stores that returned zero products, so it is not the same denominator as the 21-store or 26-store product-bearing measures.

The sampling frame is derived from OpenStreetMap data, © OpenStreetMap contributors, available under the Open Data Commons Open Database License 1.0.