Context Window
What is a context window?
A context window is the total amount of information (measured in tokens) that an LLM can process at one time. This includes everything: the input prompt, any uploaded files or supporting documents, conversation history, and system instructions.
Think of it as the LLM's working memory—how much it can "hold in mind" at once when generating a response.
What happens when you hit the context window limit?
When too much information accumulates in a conversation or task, the LLM may start losing track of earlier details. You might notice it suggesting things you've already rejected or forgetting preferences you mentioned earlier.
For example, if you're having a long conversation with ChatGPT about movie recommendations, eventually it might start suggesting movies you've already told it you didn't like. That's the context window limit at work—earlier parts of the conversation are getting pushed out of memory.
For processing very long documents (like 30-40 minute interview transcripts that can be 30,000+ tokens), you may need to use chunking strategies to break the content into smaller pieces the LLM can handle effectively.
How can you work around context window limits?
There are several strategies:
For ongoing conversations: Use tools like Claude Projects or ChatGPT Projects that maintain persistent context separately. Instead of storing everything in the conversation history, you can upload documents to the Project—like a list of your preferences or relevant background information. The LLM accesses these documents as needed without filling up the context window with conversation history.
For long documents: Break content into chunks and process each chunk separately. Which evals need the full transcript and which can use just part depends on what you're evaluating.
While LLMs have increasingly large context windows, their performance across that window tends to degrade as it gets fuller. Understanding these limits helps you design better AI workflows.
Learn more:
- Behind the Scenes: Building the Product Talk Interview Coach
- Building My First AI Product: 6 Lessons from My 90-Day Deep Dive
- 15 Ways to Use AI at Home (and Fill Your AI Product Toolbox)
Related terms: