Week 3: How Language Models Produce Answers

Welcome to Week 3 of the AI Literacy Course.

In Week 2, we learned about machine learning, models, training and generative AI. This week, we look more closely at one important type of generative AI: the language model.

The goal isn’t to study advanced mathematics. It’s to understand why language models can produce useful, fluent answers that may still be incomplete, unsupported or wrong.

Learning goals

By the end of this lesson, learners should be able to:

• Explain what tokens and next-token prediction are.

• Describe how prompts, instructions and context influence an answer.

• Explain why the same prompt may produce different outputs.

• Describe why hallucinations can occur.

• Distinguish model-generated output from tool-grounded information.

• Identify claims that require outside verification.

• Explain why human review and responsibility remain necessary.

1. What a Language Model Receives

When we communicate with a language model, it can feel like a conversation with another person. However, the underlying mechanism is different.

A language model may receive:

• The user’s prompt

• The visible conversation

• System instructions and settings

• Information retrieved from tools, when tools are available

The model uses this available input to produce candidate language.

It doesn’t automatically know everything on a user’s computer, inside an account or in the surrounding world. It can only use information available through its training, current context and any connected tools.

2. Tokens and Learned Patterns

Language models process text as smaller units called tokens.

A token may be:

• A complete word

• Part of a word

• Punctuation

• Another small language unit

Tokenisation can differ between languages. A long Swedish compound word, for example, may be divided into several tokens.

During training, a language model learns statistical relationships among large numbers of tokens. It learns that certain words, phrases, facts, styles and structures often appear together.

These patterns can support explanations, summaries, translations and drafts. However, statistical relationships aren’t the same as human experience or verified knowledge.

3. Context and Attention

The meaning of a word often depends on the words around it.

Compare:

The bank was crowded with customers.

and:

The fisherman sat on the river bank.

The surrounding context helps the model estimate which meaning of “bank” is relevant.

An attention mechanism helps the model weigh relationships among tokens in the available context. This allows it to connect names, descriptions, instructions and ideas across a piece of text.

A context window is the amount of text available to the model while it produces an answer. Information outside that window may no longer be directly available. In a very long conversation, important instructions may also become harder for the model to use consistently.

Important requirements may therefore need to be stated clearly or repeated.

4. Next-Token Generation

A language model generates an answer step by step. At each step, it estimates which tokens could plausibly come next.

For example, after:

The capital of Sweden is...

“Stockholm” would probably receive a high probability.

The selected token becomes part of the context for predicting the following token. This process continues until the answer is complete.

Next-token prediction can produce fluent language because the model has learned many relationships and structures from training data. However, high probability isn’t the same as verified truth.

A plausible continuation may still be wrong, especially when the topic is specialised, recent, unclear or poorly represented in the training data.

5. Variation and Generation Settings

A language model doesn’t always select the single most probable next token. It may choose among several plausible possibilities. This process is called sampling.

Sampling helps explain why the same prompt may produce different answers.

Variation can be useful for:

• Brainstorming

• Creative writing

• Producing alternative explanations

• Exploring different perspectives

Variation can be risky when an answer must be exact, stable or evidence-based.

Some systems also allow settings such as temperature to influence generation. Lower temperature generally produces more predictable output, while higher temperature generally allows more variation. Not every AI service gives users access to this setting.

A lower temperature doesn’t guarantee accuracy. It may make an incorrect answer more consistent without making it true.

6. Why Hallucinations Occur

A hallucination is information that sounds plausible but is false, unsupported or fabricated.

A language model can produce a hallucination because it generates plausible continuations rather than independently checking every claim against reality.

It may invent or misrepresent:

• A source

• A quotation

• A date

• A study

• A legal rule

• A confident explanation

Fluency, detail and confidence don’t prove that a claim is correct. This is why important factual claims require outside verification.

We’ll examine hallucinations in greater depth in Week 11.

7. Model Output and Tool-Grounded Information

A language model may generate an answer using learned patterns and the information available in its current context. Some AI systems can also use external tools.

These tools may include:

• Web search

• Calculators

• Databases

• Document retrieval

• Code execution

It’s important to distinguish between two kinds of information:

Model-generated output:
Language produced from learned patterns and the current context.

Tool-grounded information:
Information supported by a search result, document, calculation, database or another external tool.

Tools can improve accuracy and provide current information. However, tool access doesn’t remove the need for source criticism.

Ask:

• Was a tool actually used?

• Which source supplied the information?

• Is the source reliable and current?

• Does the source support the claim?

• Which parts came from the source, and which parts were generated by the model?

A citation is useful only when it leads to a real source that supports the statement.

8. Prompt Quality and Human Review

The wording, order, examples and context in a prompt can influence the answer.

A clearer prompt may improve relevance and reduce unnecessary guessing. Asking the model to identify assumptions or separate sourced information from interpretation may also make an answer easier to evaluate.

However, a better prompt cannot guarantee truth.

Human review includes:

• Defining the purpose of the task

• Providing relevant context

• Checking important claims

• Comparing information with reliable sources

• Revising the output

• Disclosing AI use when appropriate

• Taking responsibility for the final result

The model proposes language. The person decides how, whether and where that language should be used.

9. Classroom Activity: Trace Two Answers

Choose one neutral study question. Ask an AI tool the same question twice without changing the prompt.

Save:

• The prompt

• Both answers

• Any sources provided

• Your notes about the differences

Use three colours to examine each answer:

• Green: useful explanations or structure

• Yellow: factual claims requiring verification

• Red: unsupported, contradictory or possibly fabricated information

Then answer:

1. What changed between the two answers?

2. What remained stable?

3. Which claims require outside verification?

4. Did the model express uncertainty?

5. Were any sources provided?

6. Do the sources actually support the claims?

7. Which parts were useful?

8. What did you contribute as the human reviewer?

Don’t use personal, confidential or sensitive information in the activity.

10. Reflection Questions

Write short answers to the following questions:

1. What is a token?

2. How does context influence the meaning of a word?

3. Why can next-token prediction produce fluent language?

4. Why can fluent language still be wrong?

5. When is variation useful?

6. When could variation create a problem?

7. Why doesn’t lower temperature guarantee accuracy?

8. How is tool-grounded information different from ordinary model output?

9. What responsibility remains with the person using the model?

11. Key Vocabulary

Token:
A small language unit processed by a model, such as a word, part of a word or punctuation.

Next-token prediction:
Estimating which tokens could plausibly appear next based on the available context.

Attention:
A mechanism that helps a model weigh relationships among tokens in the context.

Context window:
The amount of text available to a model while it produces an answer.

Sampling:
Selecting among plausible next tokens, which can create variation.

Temperature:
A generation setting that generally influences how predictable or varied an output may be.

Hallucination:
Plausible-sounding information that is false, unsupported or fabricated.

Tool grounding:
Using information from an external tool, such as search, a document, calculator or database, to support an answer.

Human review:
The process of checking, interpreting, revising and taking responsibility for an AI-generated output.

12. Summary

Language models produce answers by processing tokens, using available context and repeatedly predicting plausible next tokens.

This process can create fluent and useful language, but it doesn’t guarantee truth. Sampling can produce variation, and hallucinations can appear convincing. External tools may improve grounding, but their sources and results must still be checked.

The central lesson is:

Language models produce candidate answers, not automatic truth. The model proposes language; people establish truth, meaning and responsibility.

Lesson 3 Interactive Quiz: How Language Models Produce Answers

Choose one answer for each question. Then click Check my answers. This self-checking quiz does not collect names or scores.

1. What is a token?

A. A complete human thought stored by the model B. A small language unit processed by the model C. A verified fact retrieved from an external source D. A complete sentence that cannot be divided further

2. How does context help a language model interpret the word “bank”?

A. It guarantees that the model knows the writer’s intention B. It gives every possible meaning the same importance C. Surrounding words help indicate whether “bank” means a financial institution or a river bank D. It converts the word into a verified external source

3. How does a language model generate an answer?

A. It repeatedly estimates plausible next tokens from the available context B. It retrieves every answer unchanged from one stored document C. It checks every sentence against reality before displaying it D. It waits until it understands the topic as a person would

4. Why isn’t fluent language proof that an answer is correct?

A. Correct information must always be written in simple language B. Language models cannot produce grammatically correct sentences C. Detailed answers are less reliable than short answers in every situation D. Plausible next-token generation can produce fluent but unsupported claims

5. Why might the same prompt produce two different answers?

A. The meaning of every word changes each time the prompt is submitted B. Sampling may select among several plausible next tokens C. The model must use a different training dataset for every answer D. One answer must be factual and the other must be fabricated

6. What is a likely effect of lowering the temperature setting?

A. The model automatically searches for reliable sources B. Every factual claim becomes accurate C. The output generally becomes more predictable D. The model gains access to a larger context window

7. Why can a language model produce a hallucination?

A. It generates plausible continuations without independently verifying every claim B. It deliberately decides to mislead the user C. It can only use information written by the current user D. It rejects every source that contains uncertainty

8. What distinguishes tool-grounded information from ordinary model-generated output?

A. Tool-grounded information never requires human checking B. It is always written in a more confident style C. It comes only from patterns learned during the model’s training D. It is supported by information obtained from an external tool

9. An AI answer includes a source link. What should you check?

A. Whether the link makes the answer look professional B. Whether the source is real, reliable, current and supports the claim C. Whether the source title uses the same words as the AI answer D. Whether the answer provides several links, regardless of their content

10. Which action demonstrates responsible human review?

A. Accepting the answer when it sounds detailed and confident B. Asking the model to confirm that its own answer is correct C. Checking important claims, revising the output and taking responsibility for its use D. Using a lower temperature instead of checking external evidence

Reflection: After checking your answers, identify one factual claim from an AI-generated answer that you would verify. Explain which source or tool you would use and why.