Marketplace catalog ingestion

Catalog ingestion for a marketplace product

The client needed permitted Amazon and eBay product data represented in a catalog on its own resource. Available bulk interfaces did not cover every required field or operating pattern, while source responses, rate limits, and page structures could change without notice. The client identity and collection volumes remain confidential.

Four months to a stable feed: per-source adapters, queues and backoff, extraction into a mapped catalog schema, alerts when a selector or a block rate jumps, and a review queue when a row cannot be trusted. The engagement is complete.

Adapters, mapping, and data quality

Source-specific adapters isolate collection logic from the shared product model. Python and PHP workers place records onto queues, normalise identifiers and fields, map categories and attributes, detect changes, and persist validated data in PostgreSQL. Redis coordinates background work and backoff.

Rows that cannot be mapped with sufficient confidence enter a review queue instead of silently degrading the customer catalog. Change detection limits unnecessary processing, while monitoring tracks source health, extraction errors, schema drift, queue depth, and unusual blocking behaviour.

A pipeline designed for changing sources

The completed system is an operational data product rather than a one-time import. Docker-based services can be deployed and observed independently, and source failures do not stop the rest of the catalog from being served.

The engagement combined marketplace product engineering, software engineering, and data engineering. No collection volumes or proprietary source methods are published; the case study focuses on resilient ingestion, schema mapping, review, and monitoring.