Embeddings and Vector Search

Vector search finds passages whose meaning is close to the query rather than whose words match. For a policy library that is worth real money: a buyer writes "it turned up smashed", the document that settles the case is headed "damage in transit", and the two sentences have almost no vocabulary in common. Support tickets are written by people describing a problem, and policy documents are written by people describing a rule.

The same mechanism is worse than a plain keyword index at the lookups a support agent does constantly: order SU-88421, a seller called Ashcombe Furniture, clause 4.2, the difference between 14 days and 30. Any design that picks one of the two engines has chosen which half of its queries to be bad at, which is why the answer at the end of this page is both.

What an Embedding Is, Operationally

Text goes into a model, a fixed-length vector of numbers comes out, and texts with similar meaning land near each other in that space. Search becomes arithmetic: embed the query, find the nearest stored vectors, return the passages they came from. "Near" is a distance function over the vectors, and nothing in the process reads a word. Machine Learning from Zero covers why training produces that property; the operational summary above is all this chapter needs.

Three consequences fall straight out of it. The index has to be built ahead of time, because every passage must be embedded before it can be found. Every query pays one embedding call before the search runs. And the space belongs to the model that created it — a vector produced by one embedding model is meaningless in an index built by another, which is the single fact behind half the operational advice on this page.

Where It Wins and Loses

The wins are all one shape: the query and the document say the same thing in different words. The losses are also one shape, and it is a shape a marketplace produces constantly — an exact string that must match exactly, or a small difference in wording that reverses the meaning.

Read the second row twice, because it is the one that produces the worst failure. A keyword index that cannot find SU-88421 returns nothing, and nothing is an honest answer the agent can act on. A vector index returns the three most similar things it has, which for an unfamiliar identifier is three passages about other orders — a wrong answer wearing the costume of a right one.

Negation and numbers are the other standing weakness. "Damage must be reported within 48 hours of delivery" and "damage reported more than 48 hours after delivery is not eligible" sit close together in vector space, because they are about the same subject in nearly the same words, and one states the rule while the other states the exception that denies the claim. Swapping 14 for 30 in a sentence about return windows barely moves the vector at all. Meaning-similarity is not logic, and no amount of tuning makes it into logic.

Hybrid Search Is the Default

Run both indexes and merge the results. Pure vector search is a demo choice: it survives until the first ticket that quotes an order id, a SKU or a seller name, which at Sundry is most of them. Pure keyword search is the older mistake and fails on every buyer who describes a problem in their own words instead of the policy team's.

The merge rule matters more than either engine and gets written down. Sundry's numbers on its own retrieval eval, measuring how often the correct document appears in the top three: keyword alone 61%, vector alone 77%, the two merged 89%. That last figure is the number Topic 32's citation work was built on, and it did not come from a better embedding model. It came from running two ordinary searches and combining them on purpose.

The Operational Facts

The embedding model used for querying must be the model used for indexing. Changing it is a reindex of everything, which makes it a migration with a build window, a dual-read period and a rollback plan — not a config change somebody makes on a Thursday. Sundry's library is about 48,000 chunks and a full rebuild takes 40 minutes, during which two indexes disagree about what the policy says. Pin the model in configuration, and treat a version bump as a schema change to a table with 48,000 rows.

Latency and size sit inside the loop's budget rather than beside it. An embedding call plus a nearest-neighbour search over 48,000 vectors is tens of milliseconds, which is nothing against a model turn — until the agent searches three times on one ticket and the searches are serialized behind three separate turns. At this size the hard part is not the vectors; it is keeping the metadata alongside them correct, because a scope field that did not get updated is a wrong answer with no symptom.

Similarity scores are not confidence. A score of 0.83 does not mean the passage is 83% likely to be relevant, and it is not comparable across two different queries, let alone across two corpora. Thresholds are set by measurement on your own documents — take the labelled retrieval set, sweep the cutoff, look at what each value keeps and drops — and they are re-measured after any change to chunking, to the model, or to the merge weights. A constant copied out of a blog post is a guess about somebody else's corpus.

Chunking Decides Quality

Chunk boundaries are chosen at index time, before any query has ever run, and they decide what can be retrieved at all. A clause split down the middle retrieves as two half-answers, neither of which states the rule. The refund matrix — a table mapping marketplace-versus-own-stock against the reason for the return — cut on a fixed 512-token window produces a chunk containing a row's first two columns and no outcome, which reads as a policy statement and answers nothing.

Chunk along the document's own structure: one clause, one table row, one procedure step. Carry the document title and the heading path into each chunk's text, so a passage retrieved on its own still says what it is part of. Topic 34 covers how to diagnose a chunking problem after the fact; the reason it appears here is that fixing one costs a full reindex, so it is worth getting right the first time.

What Sundry Indexes

Sundry indexes the library twice. The vector index holds one chunk per clause, each prefixed with its document title and heading path. The keyword index holds identifiers, seller names, clause numbers and document titles. Both carry the same metadata — document id, clause, scope, effective date, seller id where the document is seller-scoped — because the scope filter has to run before ranking rather than after it.

def search_policy(query, seller_id=None, top_k=3):
    # scope first: another seller's supplement can never be relevant here
    scope = scope_filter(seller_id)          # sundry-wide OR this seller

    kw  = keyword_index.search(query, scope, n=20)        # ids, names, clauses
    vec = vector_index.search(embed(query), scope, n=20)  # paraphrase, wording

    merged = rank_fusion(kw, vec, w_kw=1.0, w_vec=1.0)
    merged = promote_exact_identifier_matches(query, merged)
    return [with_provenance(p) for p in merged[:top_k]]

Four decisions are visible in those seven lines. The scope filter runs before either search, so a supplement belonging to a different seller is never a candidate no matter how well it scores. Both engines return twenty candidates and the fusion step decides the order, rather than one engine getting the first three places. An exact identifier match is promoted explicitly, because that is the case vector similarity handles worst. And the weights are named constants that were set by measurement, so changing them is a diff somebody can review rather than an archaeology exercise in six months.