Re-architecting item matching to scale with the platform, not the haystack
The client's item-identification workflow had to compare each incoming item against an ever-growing set of enrolled items. As volumes climbed, wait times grew with the data. We split the monolithic pipeline into a distributed, event-driven architecture — and broke the link between load and latency.
When wait time scales with your data, not your traffic
Every identification request compares a "needle" item against a "haystack" of enrolled items. The original pipeline ran the whole workflow — the I/O-bound work of calculating the haystack (database lookups, attribute filtering) and the CPU- and vision-bound work of matching (ML inference, shape comparison) — inside a single Celery worker queue.
That coupling was the problem. Cheap, seconds-long haystack calculations competed for the same workers as expensive, minute-long ML comparisons. Under load, queue time correlated strongly with haystack size — analysis put the correlation at −0.93. With larger volumes arriving on the platform in bursts, participants increasingly waited behind one another, and the wait grew as the dataset grew.
The hard constraint: none of this could move the needle on correctness. Match rates were trusted and acted upon, so any re-architecture had to be provably as accurate as what it replaced.
Separate the concerns, then size with the maths
Diagnose the bottleneck
Established statistically that queue wait was driven by request velocity per participant and by I/O work blocking compute — not by raw traffic. This pointed the redesign at the constraint, not a symptom.
Split I/O from compute
Re-architected the workflow into two stages — item_enrolment (I/O: haystack calculation) and item_matching (CPU/CV: ML matching) — so independent compute could be parallelised safely.
Size workers with Little's Law
Used queueing theory to derive worker counts for each stage against real arrival rates and processing times, with headroom added for outlier traffic — turning capacity planning into an explicit, defensible model.
Ship in shadow, then verify
Built v2 alongside production to de-risk breaking changes, then load-tested with representative haystacks and confirmed match rates were unchanged before any cutover.
Load goes up, wait time stays flat
The new two-stage architecture was benchmarked against the original on two datasets: a moderate 497-item batch and a 2,000-item stress test. The gains grew exactly where they were needed most — under heavy load.
| Metric | Baseline (v1) | V2 (distributed) | Improvement |
|---|---|---|---|
| Total workflow time — 2,000 items | 43.2 min | 26.6 min | −38% |
| Mean time in queue — 497 items | 7.98 min | 1.54 min | −81% |
| Mean time in queue — 2,000 items | 16.84 min | 3.47 min | −79% |
| Queue time vs haystack correlation | −0.93 | 0.37 | Decorrelated |
| Match rate (correctness) | Baseline | Identical | No regression |