Isham Faizal posted a language model attention visualizer. A page with ready-made examples: hover over any generated token and see the highlighting of past tokens from which the model pulled information for it. Works directly in the browser - React plus Transformers.js, a model with 600 million parameters. Some prompts are pre-generated and open instantly, because the model weighs hundreds of megabytes and no one will wait for it to load for the first frame.
The author immediately stipulates simplification. The weight of attention multiplied by the value vector, aggregated over all heads and summed over all layers is highlighted; the resulting one number per token is converted to transparency, the maximum is opaque, the rest is interpolated. A lot of information was thrown out, and he honestly writes that he doubted whether the result would be readable at all.
To get these internal values, I had to go into the model. Transformers.js works with an .onnx file, calculations go to wasm, only pre-declared outputs are sent out. The author edited the onnx file itself with a small script, setting the necessary internal values in the outputs, and uploaded the instrumented model to his repository on Hugging Face. The standard generation cycle is not suitable for this; he rewrote it manually.
What can be seen in the examples
In the example of an office relocation note, hovering over the verbatim copied address and dates sharply highlights the original data in the prompt. The author explains his long-standing question with this: why does a model that predicts the next token probabilistically copy so reliably and not fall apart due to a random error. It doesn't restore a string from a limited internal state - it has access to all past tokens, and when copying it pulls directly from there, so the chance of error is low.
In the function debugging example, a model with 600 million parameters reproduces the entire JS function, changing exactly what was asked. She couldn’t find the problem in the code; the author gave hints. Another observation: the word “remain” in the phrase about passes and telephones simultaneously pulls up “work” from one formulation and “stay the same” from another, gluing both together.
The application output from here is direct. If an exact line is needed in the answer - account number, date, article, piece of code - it must be in the context verbatim. Then the model copies, and its copying is reliable. Retelling instead of a quotation, summarizing a chunk before submitting it to the context, “let’s compress the document so that it fits in the window” - all these techniques take away from the model what it is copying from, and transfer the answer to the recovery mode by weights, where inventions live. The same principle explains why in a knowledge base it is better to store the original paragraph next to its paraphrase, and put both in the context.
The boundaries of this picture are rigid. 600 million parameters is a small model, large layers have more and they behave differently, it is impossible to transfer observations one to another. A single number per token does not explain why the model responded the way it did - it is an illustration of the mechanism, not a prompt debugger.
Open your search pipeline and see what reaches the context: the original lines of the documents or their paraphrase. If paraphrasing, start with the fields where precision is needed and present them verbatim.