Why do answers in a long chat start slowly?

Before a model answers, it reads the whole chat so far. On a chat's later turns, it can skip the part it already read, kept in its prompt cache, and read only your new message. A model that can't skip reads a long chat from the start every turn, so its answers start later as the chat grows.

Where to look

Open Diagnostics. Reusing the chat so far lists each model with the share of its later turns' prompts it skipped, and how many turns and tokens that covers.

Only turns after a chat's first are counted, and only when the model was already running. A model that just started has nothing to skip. Models on paired computers and in other apps, such as Ollama, don't report it, so they aren't listed.

Tip: Toskar's weekly tests hold a small model to 80% or more, and check that an answer from the cache matches one read from scratch, word for word.

Troubleshooting

When a model is marked low:

1. Switch between chats less while you wait for answers. Each chat you send to the same model can push the others out of its cache.
2. Try a different model for long chats. Some models, such as those with sliding-window attention, can't skip as much.
3. Let a long chat be shortened. Auto-compact summarizes older turns, so there's less to read again.
4. If a model stays low, report it from Diagnostics, so Toskar can tune how it runs that model.