catalog_data_platform

What it is

A standalone ELT pipeline for promotional-goods catalogue data, built end to end on my own time so that every layer of it is mine: extraction, storage, modelling, testing, orchestration and a Streamlit front end. It handles two supplier feeds of quite different shapes, one XML over HTTP and one spreadsheet files behind authenticated URLs. Both land as date-partitioned Parquet on S3, are modelled in dbt against DuckDB, and are browsed through the Streamlit app. Airflow runs the whole thing daily.

The problem

Professionally I work on one part of a pipeline at a time. Building a whole one alone is the only way to find out which decisions you have been inheriting rather than making, and the awkward parts turned out to be exactly the parts I would otherwise have delegated: retention, partition scoping, testing a composite grain.

The problem inside it is genuine too. Two supplier feeds agree on almost nothing: not the transport, not the format, not the grain. One of them carries duplicate rows at the level I wanted as a primary key, which I found only by querying the landed Parquet rather than trusting a source’s own documentation.

How it works

Each supplier sits behind one extractor interface, which is the seam that matters when the shapes differ this much. It is not the only place a third supplier would touch, though: the runner’s extractor registry and the Airflow DAG both name their suppliers explicitly, so a new feed is a class plus a handful of wiring edits.

Extraction parses each feed just far enough to write flat Parquet, partitioned by supplier, entity and date. DuckDB then reads that Parquet straight out of S3 over httpfs, so there is no local copy of the raw data to drift.

dbt runs in three layers. Staging views cast types, rename, and do the light cleaning a single feed needs on its own, including deduplication and unpivoting the one feed that ships six price tiers across the row. Intermediate views normalise each supplier into a shared shape and stamp a supplier column. Marts materialise the unions as tables. The real grain is supplier, product and variant together, and dbt’s built-in uniqueness test only covers a single column, so composite keys get hand-written singular tests instead.

Each run is scoped to one date partition. Retention deletes old partitions after the upload succeeds, never before, so a failed extraction cannot destroy the last good day.

What I’d change

The run date is today’s date, computed inside the task rather than taken from Airflow’s own interval, and the DAG does not catch up. A backfill would therefore quietly read today’s partition instead of the one it was asked for. It has not bitten me because I have never needed a backfill, which is the worst reason to leave something.

The print-options mart also resolves at product level rather than variant level. That gap is recorded in the model’s own schema file rather than quietly tolerated, but it is still a real modelling shortcoming. And CI is honest about being shallow: it unit-tests extraction with I/O mocked and then parses the dbt project. It never runs a model against data, so a change that is syntactically fine and semantically wrong passes.