AppWispr

Find what to build

Schema‑Driven Competitor Extraction: A 60‑Minute Workflow to Turn Comparison Pages into Automated Shortlist Signals

AW

Written by AppWispr editorial

Return to blog
MR
IJ
AW

SCHEMA‑DRIVEN COMPETITOR EXTRACTION: A 60‑MINUTE WORKFLOW TO TURN COMPARISON PAGES INTO AUTOMATED SHORTLIST SIGNALS

Market ResearchNovember 14, 20266 min read1,103 words

Founders and product teams rarely have time for sprawling headless scraping projects. This article gives a repeatable 60‑minute workflow that turns publicly available comparison pages into automated shortlist signals you can use for content, competitive monitoring, or product positioning. The approach relies on three reliable primitives: ItemList/Product JSON‑LD as the canonical source, normalized pricing and availability fields, and a compact crawler that produces a cloneable “X vs Y” seed page and structured evidence blocks.

schema-driven-competitor-extractionItemList JSON-LDcomparison pagesproduct schemacompetitor extractionmarket researchpricing normalizationAppWispr

Section 1

Why JSON‑LD ItemList and Product markup is your fastest, highest‑value signal

Link section

Comparison pages and product listing pages often already include machine‑readable schema.org JSON‑LD. The ItemList type identifies the sequence of items, while Product nodes (and nested Offer/AggregateOffer) expose canonical fields such as name, sku, price and priceCurrency. Targeting these blocks avoids brittle HTML scraping and makes evidence provenance straightforward.

Using JSON‑LD reduces noise: you extract the intended canonical product names and the explicit offers that site owners publish for search engines, rather than parsing visual tables or heuristics. That gives you cleaner input for building shortlist signals that are defensible and easy to audit.

  • ItemList reveals what a page considers the ‘shortlist’ or ranked candidates. (schema.org ItemList).
  • Product -> Offer provides price, priceCurrency and availability in a consistent shape.
  • JSON‑LD frequently appears inside <script type="application/ld+json"> blocks, which can be parsed reliably without DOM heuristics.

Section 2

60‑minute step‑by‑step: crawler, extractor, normalizer

Link section

Goal: in one hour produce a CSV/JSON of candidate rows with fields: canonicalName, site, url, price (decimal), priceCurrency (ISO 4217), availability (normalized token), evidenceScriptPointer, and itemListPosition. Start the minute timer and work through these discrete tasks.

High‑level steps: 1) seed the crawler with a handful of comparison URLs; 2) fetch pages and collect every <script type="application/ld+json">; 3) parse JSON‑LD and extract ItemList and Product/Offer nodes; 4) normalize pricing and availability tokens; 5) emit the shortlist table and generate the first draft of an “X vs Y” seed page with evidence blocks that stitch back to the JSON‑LD.

  • Minute 0–10: gather 5–20 comparison pages (Google, niche review sites, category landing pages).
  • Minute 10–25: run a single‑thread crawler (requests/Playwright) to save page HTML and collect JSON‑LD scripts.
  • Minute 25–40: parse JSON into an in‑memory graph, extract ItemList.itemListElement and Product/@graph Offer fields.
  • Minute 40–55: normalize prices to decimal + ISO currency and map availability to tokens (in_stock, out_of_stock, preorder, unknown).
  • Minute 55–60: produce CSV/JSON export and scaffold a lightweight ‘X vs Y’ seed page with evidence blocks.

Section 3

Normalization rules that actually make shortlist signals comparable

Link section

Price normalization is the most important engineering step: convert price strings into numeric decimals, strip thousand separators, ensure a single decimal point, and require ISO 4217 priceCurrency. When pages publish AggregateOffer, use lowPrice/highPrice to represent ranges and select the most relevant value for your use case (lowPrice for bargain‑leading shortlist; highPrice for enterprise positioning).

Availability and price validity are second‑order filters. Map schema.org availability URIs to short tokens (https://schema.org/InStock -> in_stock). Record the original JSON pointer (e.g., script index + JSON Pointer) so every normalized value has an unambiguous provenance link back to the page.

  • Price: parse as a decimal string; remove currency symbols; fail gracefully if missing priceCurrency.
  • Range: when AggregateOffer exists, keep lowPrice and highPrice fields and choose per your ranking logic.
  • Availability: canonicalize to in_stock / out_of_stock / preorder / limited / unknown.
  • Evidence: store script index and JSON pointer (e.g., $script[2].@graph[1].offers.price).

Section 4

Canonical decisions and evidence blocks for the seed 'X vs Y' page

Link section

Comparison pages need defensible canonical choices to avoid duplicate content and to signal authority. For each extracted product row choose one canonical URL — prefer the merchant’s product landing page when available in the Offer.url property, otherwise keep the comparison site URL but mark it as secondary. Record the selection rule: (1) Offer.url from Product/Offer; (2) first related sameAs/@id; (3) page URL as fallback.

Build evidence blocks under each product on the seed page that show the values you extracted plus provenance. An evidence block contains: source site name, published price with currency, availability, the JSON‑LD snippet (small excerpt), and a link to the saved raw HTML or stored script index. These blocks make your comparison defensible for editorial review and help with later automated re‑checks.

  • Canonical decision order: Offer.url -> Product.@id/sameAs -> page URL.
  • Evidence block contents: source, extracted fields, JSON‑LD pointer, timestamp, link to archived HTML.
  • Keep evidence compact and machine‑readable — a short JSON snippet and a link are usually enough.

Section 5

Turning the output into rankable content and a repeatable pipeline

Link section

From the normalized rows you can programmatically render a lightweight seed comparison page for SEO: a short intro, a ranked ItemList (with structured data you control), and per‑product evidence blocks with canonical links. Use server‑side rendered HTML (or a static site generator) to ensure search engines see the canonicalized JSON‑LD you want indexed.

Operationalize the pipeline: schedule the crawler on a cadence (daily/weekly depending on volatility), keep a changelog of price/availability diffs, and wire alerts for new entrants that appear on multiple comparison pages. AppWispr teams use such signals as shortlist triggers for content updates, affiliate checks, product positioning changes and sales ops.

  • Render the seed page with your own ItemList/Product JSON‑LD to own the canonical listing.
  • Log diffs by product id/sku to spot new entrants or price shifts quickly.
  • Automate checks that validate the JSON‑LD still exists on source pages before trusting a recurring data point.

FAQ

Common follow-up questions

Which schema.org types should I prioritize for competitor extraction?

Prioritize ItemList for capturing shortlist order and Product nodes for canonical product facts. Within Product, extract Offer or AggregateOffer for price, priceCurrency and availability, and record the JSON‑LD pointer for provenance.

What if the site doesn't publish JSON‑LD?

Fall back to extracting microdata/OpenGraph or structured HTML tables, but treat those as lower‑confidence sources. Keep provenance and prefer pages that publish JSON‑LD; they save parsing effort and give clearer canonical names and offers.

How do I handle different currencies and price formats?

Require priceCurrency as an ISO 4217 token and convert formatted price strings to a normalized decimal. If a page omits priceCurrency, infer from page locale only as a last resort and mark the record as ‘inferred_currency’ so downstream logic can handle it cautiously.

Is it okay to re‑publish extracted facts on my comparison page?

Yes, if you attribute sources and provide clear evidence blocks that link back to the original pages or saved snapshots. Also choose canonical URLs carefully and avoid verbatim copying of editorial content; your page should add value (ranking, sorting, context).

Sources

Research used in this article

Each generated article keeps its own linked source list so the underlying reporting is visible and easy to verify.

Next step

Turn the idea into a build-ready plan.

AppWispr takes the research and packages it into a product brief, mockups, screenshots, and launch copy you can use right away.