PDP parity validator

What it is

A nightly parity check for a product-page replatform. During a migration each catalogue page exists twice, in the current site and in its replacement, and the two have to be shown to match on everything that is not a deliberate change. The tool renders both, compares them across seven dimensions, and turns the result into a triaged queue with a traffic-light verdict per page. A Streamlit dashboard is where that queue gets worked.

I built the tool. The parity scope it implements came from a proposal written by someone else on the team, which I then engineered against.

The problem

A replatform breaks things quietly. A meta tag drops, a price rounds differently, a configurator option stops appearing, a delivery estimate shifts by a day. Each one is invisible on a page-by-page look and expensive in aggregate, and page-by-page is the only manual option available.

The second problem mattered more: a comparison tool that cries wolf gets switched off. Anything reporting cosmetic difference as regression is worse than nothing, because it teaches people to ignore it.

How it works

Each page is rendered in a headless browser, because a plain fetch does not produce the final markup. The two sides are then read asymmetrically, which is the part worth knowing: one side’s configurator, pricing and turnaround are normalised from a product API response, while the other side’s come from the rendered page, its inline JSON with a captured pricing request folded in. The comparison is therefore not like-for-like plumbing, and the matchers have to absorb the difference in route.

Pairs are built by crawling the sitemaps per locale. The scheduled run covers a curated subset of pages; a full-sitemap run is a manual trigger.

Every dimension has its own matcher: metadata and search-facing tags, visual assets, configurator structure, pricing, turnaround and delivery, content and copy, and HTTP behaviour. Each emits discrete records carrying a field, its old and new value, and a severity, thresholded by a tolerance file kept in version control. A page’s verdict is its worst dimension’s. An allowlist mutes known-acceptable differences without deleting the evidence.

A fetch is retried up to three attempts. When a value still cannot be obtained, the run emits an explicit unavailability record rather than passing the page in silence.

An LLM attaches a hypothesis to each finding: a one-line summary, the most plausible cause, and one next step a triager can actually run. Output is structured, and a parse failure is shown as a failure rather than dressed up as an answer. It is advisory only, and severity is computed without it.

What I’d change

Those unavailability records are emitted at critical severity, and the rollup takes the worst severity per page with no bucket for “could not tell”. So an unreachable page and a genuine regression look the same on the dashboard, which is exactly the cry-wolf failure the tool was built to avoid. The verdict needs a third state, and I would add it before adding an eighth dimension.

I would also argue for grading the LLM. Advisory is the right setting while nothing measures whether its hypotheses hold up, but that is a reason to measure them, not a resting state.