Key facts
| Definition | Models that accept very large prompts in a single request |
| Why it matters | Fewer retrieval pipelines for tasks that need the whole document set |
| Typical use | Repository Q&A, contract review, long meeting histories and multi-document analysis |
| Trade-offs | Cost and latency scale with tokens; recall can weaken in the middle of long inputs |
| Plugsky support | Long-context models are selectable through the OpenAI-compatible API |
| Comparison tooling | Compare published context windows before choosing a model |
| Related patterns | RAG remains better for large, changing corpora with citations |
| Status | Chat, streaming and function calling are live across the catalogue |
TL;DR
- Long context means fitting more in one request, not remembering forever.
- Use it for whole-artifact tasks; use RAG for large changing corpora.
- Every extra token is processed on every call — cost scales with length.
- Position matters: models can miss details buried mid-prompt.
- Check the window per model before designing a workflow around it.
How it works, step by step
- Decide whether the task needs the whole artifact or just relevant passages.
- Check the context window of candidate models before designing the prompt.
- Place the most important instructions and evidence at the start and end of long inputs.
- Measure quality at increasing input sizes to find where recall drops.
- Trim boilerplate and deduplicate content before sending long prompts.
- Fall back to retrieval when the corpus grows beyond the practical window.
Try it yourself
Open the context window comparison →
What long context changes
A standard context window fits a conversation and a few documents. A long-context model fits a repository, a contract bundle or weeks of history. That removes a whole class of engineering work: no chunking strategy, no embedding index, no retrieval ranking for tasks where the entire artifact is the input. It also shifts the problem — from retrieval quality to how well a model attends across a very large prompt.
When long context beats RAG
- Single artifacts: one large file, contract or specification that must be read whole.
- Cross-references: questions requiring connections across distant sections.
- Prototyping: skip index construction while validating a use case.
- Small, stable corpora: fits entirely and changes rarely.
- Traceability: when you need the model to see exact original ordering.
RAG still wins for large, frequently updated collections, strict citation requirements and cost-sensitive high-volume traffic.
Common mistakes
- Assuming a bigger window means perfect recall — details in the middle can be missed.
- Sending the same huge prompt on every turn instead of maintaining a compact state.
- Ignoring cost: long inputs multiply token processing on each request.
- Choosing a long-context model for a task a small model with retrieval handles better.
- Forgetting that other models in a routing chain may have smaller windows.
How Plugsky implements it
Long-context-capable models are available in the catalogue and reachable through the same OpenAI-compatible endpoint, so selecting one is a model-name change. Windows and capabilities are documented per model, which makes it straightforward to pair a long-context alias for repository or document analysis with cheaper aliases for routine turns. Embeddings and RAG remain live for corpora that outgrow the practical window, and function calling lets an agent retrieve additional context on demand instead of carrying everything.
Honest comparison
| Approach | Long context | RAG retrieval | Chunk-and-summarize |
|---|---|---|---|
| Corpus size | Fits one window | Effectively unlimited | Unlimited, lossy |
| Freshness | Whatever you paste | Reindex as data changes | Summary drift |
| Citations | Exact positions in prompt | Passage ids from index | Weak |
| Cost per call | High — all tokens processed | Lower — top passages only | Moderate |
| Best for | Whole-artifact reasoning | Large changing knowledge bases | Very long histories |
Frequently asked questions
What is long-context AI?
Models that accept very large prompts — hundreds of thousands to millions of tokens — so entire codebases, contracts or long histories can be reasoned over in a single request.
Does long context replace RAG?
Not entirely. Long context suits whole-artifact tasks and prototyping; RAG remains better for large, frequently changing corpora where citations and cost matter.
Why do models miss information in long prompts?
Attention degrades across very long inputs, especially for content positioned in the middle. Important instructions and evidence should sit near the start or end of the prompt.
How does long context affect cost?
All input tokens are processed on every request, so cost and latency grow with prompt length. That is why trimming and retrieval still matter.
Which Plugsky models have long context?
Several models in the catalogue support extended windows. Compare published windows in the docs before choosing one for a long-context workflow.
Can I combine long context with RAG?
Yes, and many production systems do: retrieve a focused candidate set, then pass it with the full artifact when cross-references matter.