Long chats and auto-compact

A model can read only so much at once: its context window. When a chat grows long, Toskar's auto-compact sends the older messages to the model as a summary, and the newest ones word for word. Every message stays saved, and the summary is rebuilt from the original messages, not from an earlier summary.

How big the window is depends on the model and your computer. A token is about three quarters of a word. Toskar starts each model with 8,192 tokens, or with 16,384 or 32,768 when it still fits your computer's memory with plenty to spare and stays on the graphics card, and never more than the model was made for. So a small model on a computer with lots of memory reads about four times as much at once, and a large model that only just fits keeps 8,192, the most it had in Toskar 1.11 and earlier. A bigger window holds memory while the model runs, and Models counts it in each model's memory estimate. The ring beside Send shows the window of the model that's running.

Auto-compact starts when a chat fills about half the model's window, after an answer, without delaying it. Settings › Chat sets when: Early (40%), Normal (50%), or Late (70%). When one long message or a big lookup wouldn't fit, Toskar compacts first and says "Compacting earlier messages…", instead of leaving the oldest messages out.

The context gauge under a chat says when its older messages are a summary. From there you can read the summary and correct it (later summaries keep your corrections), compact now, for example before pasting something long, start a new chat that begins with the summary, or keep the chat verbatim, which turns auto-compact off for that chat only.