search›ranker.py
ranker.py
Hints
Picture a shared spreadsheet of daily temperatures, kept by two people. One of them always enters degrees Celsius. The other always enters Fahrenheit. Neither column says which. Nobody thought it needed to, they each just typed in "the temperature."
Someone builds a "find the closest match" feature on top of it: for any given day, find whichever other day has the nearest number in that column. It runs beautifully. It also matches a scorching 35 degree summer day with a freezing 35 degree winter day, because 35 is close to 35. The feature isn't broken. It's doing exactly what it was told: find the closest number. Nobody told it the two columns don't speak the same language.
That's what happens when a search system's document index quietly mixes embeddings from two different model versions. An embedding model turns text into a list of numbers, a position in a very high-dimensional space, meant to be compared only against other positions from that same model.
Upgrade to a newer, better embedding model and re-embed your incoming documents, but leave the old, already-indexed documents untouched (re-embedding an entire historical archive is slow and expensive), and the index now holds two populations of numbers that look identical, same length, same format, and mean completely different things.
similarity = cosine_similarity(query_vector, doc_vector) # returns 0.61, a real number, a real computation, and completely meaningless
No error thrown. No warning logged. Just a confidently ranked result built from two numbers that were never meant to be compared.
So far, that's the pipeline working as intended: query and documents alike get embedded once and compared in one shared space. This concept starts one step later, at the moment "the same space" quietly turns out to mean two different models.
An embedding model defines its own private coordinate system. Two different models, or even two versions of the same model, don't just produce slightly different numbers for the same text. They produce numbers from fundamentally unrelated spaces. Position 47 in the vector might encode something about tone in one model and something about verb tense in another. There's no shared meaning to line them up, even when both vectors happen to have the exact same length.
Cosine similarity doesn't know or care where its inputs came from. It's pure arithmetic: multiply, sum, divide, entirely blind to which model produced either vector. Feed it two vectors from the same model and the number reflects real semantic closeness. Feed it two vectors from different models and the number is computed just as confidently, and means just as little. The function has no way to notice, let alone refuse, a mismatched pair.
Re-embedding an entire historical archive is expensive, so most teams don't do it all at once. Upgrading to a better embedding model for new content is an easy, obviously correct call. Re-processing years of existing documents through it immediately is a real cost, in API calls and in engineering time, so it usually gets scheduled for "later." The index now silently holds two incompatible populations of vectors, indistinguishable by type or shape, only distinguishable by which model produced them, information the vector database usually isn't even asked to store.
The trap: everything about the migration looks careful and correct. New documents get embedded fine. Old documents keep answering queries that happen to still land near where they always did. The corruption only shows up as certain searches quietly returning oddly ranked, subtly wrong results, no error, no crash, nothing that looks like the obvious signature of a bug, unless someone thinks to check which model actually embedded which document.
Don't take our word for it. Watch a mixed index rank the wrong document first.
Run the search. Then re-embed documents one at a time and watch the ranking change as they move from the old model version to the new one.
Run the search, then re-embed the actual best-matching document into v2 and watch it become the top result, to unlock the problem stage.