Capabilities and Limits: Variability, Hallucination, Knowledge and Context
8 min read
Lesson objectives
- Distinguish ability to produce an answer from reliability of relying on it.
- Explain confabulation and other error sources such as missing information and conflicting context.
- Select a verification method appropriate to the claim type and consequence.
Broad capability does not mean equal reliability across tasks
Generative language models can perform a wide range of text tasks: drafting, rewriting, summarizing, classifying, extracting, comparing, generating examples, and assisting with code. But “can produce an answer” is not the same as “the answer can be relied on.” Reliability changes with task clarity, source quality, similarity to cases the model can handle, and the availability of verification. Capability and reliability must therefore be evaluated separately. A model may be strong at producing a first draft and weak at making a final judgment that depends on facts outside its context.
| Task | Potential strength | Condition for reliance | Risk signal |
|---|---|---|---|
| Rewrite known text | Speed and expressive variety | Compare meaning with the source | A constraint is dropped or tone changes materially |
| Summarize a document | Compress and organize information | Check points against the source | Information appears that is not in the source |
| Propose ideas | Broaden the option set | Review fit and rights | Biased or unsuitable patterns are repeated |
| Answer a current fact | Explain after the source is retrieved | Current source that can be checked | Confident answer without actual source access |
Hallucination or confabulation: when a statement sounds right but is not
NIST uses the term confabulation for confidently stated content that is erroneous or false and may mislead users. In everyday discussion this is often called a “hallucination.” It can affect a name, date, number, quotation, link, causal explanation, or connection among facts. Fluency makes the error harder to notice. The problem is therefore not solved simply by saying “be accurate.” Depending on the task, we need a checkable source, a tool that retrieves evidence, or an independent test of the claim.
Other limits exist beyond confabulation. A model may not have current or organization-specific information unless it is passed in context or provided through a tool. Inputs may themselves be incomplete, biased, or contradictory. The model may fail to follow a long constraint or misread a pronoun or table. Before generation, ask: what does the system actually know from this request? What does it not know? Which source will be authoritative if sources disagree? These questions prevent missing information from being silently converted into generated claims.
Context can constrain an answer without guaranteeing that every piece is used correctly. If three paragraphs are supplied and one is obsolete, the model may combine them without knowing which has been superseded. If the instruction says “use the latest policy” without a date or version identity, the request may lack an executable condition. Quality therefore includes information engineering: label sources, identify versions, remove conflicting duplication when possible, and require traceable evidence locations when provenance matters.
Lesson summary
Fluency is not reliability. A model can confabulate, operate on missing or conflicting context, or misread a constraint. Make knowledge limits visible, connect claims to appropriate evidence or tests, and do not treat “be accurate” as a guarantee.