What does this solve?
Word search does not find a document in which the same idea is written in different words. The query “how to return money for an order” and the instruction “refund procedure” do not overlap lexically almost anywhere, and the usual index is empty. Exactly the same problem kills the answers of the language model in the knowledge base: the model responds according to what was given to it, but there is nothing to put. Vector search compares texts based on the proximity of their meaning and pulls out the desired fragment, even when the wording is completely different. As a side benefit, it closes the search for similar requests, the search for duplicates, and the selection of products by description.
How it works
The pipeline consists of six steps, and the quality of the result is determined by the first, not the last.
Slicing. The document is cut into fragments - there is no point in looking for the entire article of forty thousand characters, the answer will be everything and nothing. They cut along semantic boundaries: headings, sections, paragraphs, with slight overlap so that the sentence at the junction does not lose its context. Metadata is attached to each fragment: source, section, date, access rights, link to the source document.
Embedding. The fragment is given to the model, which returns a vector of fixed dimension - an array of numbers. Texts that are similar in meaning give similar vectors. The dimension is set by the model and directly affects the storage volume and search speed.
Storage. Vectors are placed next to the metadata. B PostgreSQL is a pgvector extension with type vector; There are also specialized storage facilities, but a separate system is not always justified - keeping embeddings in the same database where business tables are located is easier for filters and rights.
Proximity metric. Cosine distance, dot product, Euclidean. In pgvector they correspond to the operators <=>, <#> i <->. The metric is chosen for the model: if the model was trained on cosine proximity and produces normalized vectors, there is no point in taking the Euclidean distance.
Index. Full search is accurate, but linear in the number of fragments. On large corpora, an approximate index is used: HNSW builds a multilayer neighbor graph (configured by construction parameters and search width when querying), IVFFlat divides the space into lists and traverses only the closest ones. Both indices are approximate - part of the correct neighbors is lost in exchange for speed, and this loss is configurable.
Query. The question text is run through the same embedding model, the resulting vector searches for the closest ones. Example on pgvector:
select id, source, chunk
from documents
where tenant_id = $1
order by embedding <=> $2
limit 10;Two more layers are usually placed on top. Hybrid search launches a parallel full-text query by words and a vector one, and merges the results by rank - this is how synonyms and exact terms, articles, and surnames are found. Reranking runs the top two to three dozen candidates through a separate model, which evaluates the entire question-fragment pair and rearranges them more accurately than the distance between the vectors.
Useful scripts
- Knowledge base support answers: the bot finds the required section of the instructions on a question formulated in real language.
- Search for similar requests and duplicate tickets: a new request is compared with closed ones, and the operator is immediately shown a solution.
- Selection of code and documentation for the agent: before editing, the agent is given relevant files instead of the entire repository.
- Recommendations for product descriptions where there is no purchase history and collaborative filtering does not work.
- Feedback clustering: thousands of reviews are grouped by topic without a predefined directory of categories.
Limitations
Exact search vector always loses. Article number, order number, error code, last name, version name - a lexical index is needed here. Embedding dilutes rare tokens, and EPERM-1073 will be next to EPERM-1075, because these are similar strings for the model.
Negations and quantitative conditions do not work. “Everything except the Basic tariff”, “contracts signed after September”, “how many clients are from Rostov” - vector distance can neither be denied, nor counted, nor filtered by attribute. Such conditions are taken into account as usual where, and if the filter cuts off most of the corpus, the approximate index begins to return garbage: it bypasses neighbors before filtering and may not collect the required number of suitable rows. It can be treated with partial indexes on a section or a complete search within a narrow sample.
Search always returns results. Even to a question for which there is no answer in the corpus, the ten closest fragments with seemingly decent distances will come up. The absolute value of the distance does not transfer well between requests, so the “no response” cutoff threshold is selected empirically and still flows.
Quality depends on the cut more than on the model. A fragment cut in the middle of a table or torn from a section heading will not be saved by any re-ranking. The main work to improve search almost always lies in the preparation of the text, and not in the selection of the model.
Changing the embedding model means re-indexing the entire body. Vectors from different models are incomparable, you cannot mix them in one table, and “let’s try a new model on some documents” does not work. On a large case this is a noticeable amount of money and hours of work.
Building an index requires memory and time, and updating documents forces embeddings to be recalculated. For rapidly changing data - rates, balances, order statuses - vector search is not suitable at all; a regular query to the current table is needed.
The index is not needed on a small case. Several thousand fragments are completely searched in milliseconds, accurately and without settings. An approximate index on such a volume will only add to the loss of accuracy.
How to check the result
The golden set is assembled by hand before any implementation: several dozen real questions, each with a fragment identifier that must be found. The questions are taken from support correspondence and not made up, otherwise they will be formulated in the same words as the documents and show false well-being.
On this set, recall@k is calculated - the proportion of questions for which the desired fragment was in the first k results. It is this metric that limits the quality of the final answer: what is not found, the model will not quote.
Vector search is implemented only after the full-text search was tried on the same set and it lost. Baseline - regular search using words using the database. It often turns out that it gives a comparable recall that is an order of magnitude cheaper, and it is not the vector itself that wins, but the hybrid.
You need to look at the fragments themselves, and not just at the final response of the model. Each request is logged along with a list of what was found and distances; ten minutes of reading these logs explains more about search quality than any summary metric. A regular source of surprises are fragments cut off in the middle of a list or table.
Index usage is checked by the query plan: explain analyze should show index access, not brute force with sorting. At the same time, the latency is measured according to the upper percentiles under real load - the average time is useless here.
Separate rights check: a user of one organization should not receive fragments of another. It is checked by making a request on behalf of test accounts, and not by reading the filter code - with an approximate index, the filter and search interact in a non-obvious way.
After any editing of cutting, model or index parameters, the gold set is run again. Without this measurement, every change remains a matter of faith.