Teaching the recommender to stop punishing the right answer
The client's Two-Tower recommender learned from in-batch negatives — but roughly one in ten training rows was being penalised for a product the user genuinely engaged with. We measured the problem, masked these false negatives, and turned cleaner gradients into measurable retrieval gains.
The model was being trained to contradict itself
The Two-Tower retrieval model uses in-batch negative sampling: for each positive user–item pair, every other item in the mini-batch is treated as a negative. It's efficient — but it breaks down when the same user or the same product appears twice in a batch. A popular item (a trending dress) shows up across many users' histories, so when two of those users land in the same 4,096-row batch, the loss penalises the model for an association that is actually correct.
These "accidental hits" are false negatives, and they create contradictory gradients on the same embedding — noise that slows convergence and caps embedding separation. Worse, the problem is self-reinforcing: as the model improves, those duplicate items become genuinely high-scoring, so hard-negative mining selects them more often. The better the model gets, the harder it's punished for being right.
Before fixing anything, we needed to know how big the problem actually was — and where it was concentrated.
Measure first, then mask
Instrument the collisions
Built a collision tracker and per-entity registry to log how often duplicate user/product IDs appeared per batch and after hard-negative mining — quantifying the false-negative rate at ~10% of rows.
Mask accidental hits in the loss
Added diagonalisation to the retrieval loss: positions where a "negative" shares a user or product ID with the positive are set to −1e9 before softmax — and crucially, before hard-negative mining, so the miner only sees genuinely hard negatives.
Trial the variants head-to-head
Trained baseline, product-only, user-only and combined variants on identical data and model, isolating the effect of each masking strategy on retrieval metrics.
Evaluate stratified by severity
Compared recall and embedding alignment/uniformity globally and for high-collision entities, confirming the lift came from cleaner gradients — not from simply shrinking the loss.
The counter-intuitive winner
The expectation was that masking both collision types would help most. It didn't. Masking product collisions alone delivered the cleanest lift — user-level masking over-trimmed the negative pool and added noise, and combining both was counter-productive. Removing the highest-impact false negatives while preserving enough valid contrastive signal was the sweet spot.
| Variant | Recall@2400 (Views) | Recall@2400 (Overall) | NDCG@2400 (Overall) |
|---|---|---|---|
| Baseline (no masking) | 0.357 | 0.341 | 0.114 |
| Product diagonalisation | 0.362 | 0.347 | 0.116 |
| User diagonalisation | 0.357 | 0.341 | 0.114 |
| User + product | 0.358 | 0.342 | 0.114 |