Skip to content

Session 1: Why LLMs Need External Knowledge

Synopsis

Explains the limitations of model memory and static training data, and introduces retrieval-augmented generation as a method for improving factuality, freshness, and domain relevance. Learners understand when RAG is preferable to prompting alone.

Session Content

Session 1: Why LLMs Need External Knowledge

Session Overview

Duration: ~45 minutes
Audience: Python developers with basic programming knowledge
Goal: Understand why large language models (LLMs) are powerful but incomplete on their own, and learn how external knowledge improves accuracy, freshness, grounding, and usefulness.

Learning Objectives

By the end of this session, learners will be able to:

  • Explain what an LLM knows and what it does not know reliably
  • Describe key limitations of standalone LLMs:
  • stale knowledge
  • hallucinations
  • lack of private/business context
  • inability to verify facts by itself
  • Understand the motivation for retrieval and tool use
  • Build a simple Python example using the OpenAI Responses API to compare:
  • answering from model knowledge alone
  • answering with externally supplied context

Agenda

  1. What an LLM actually does
  2. Why model-only answers are often insufficient
  3. The role of external knowledge
  4. Common patterns: retrieval, tools, files, APIs
  5. Hands-on: compare ungrounded vs grounded answers
  6. Wrap-up and takeaways

1. What an LLM Actually Does

A large language model is a system trained to predict and generate text based on patterns learned from huge amounts of data.

Important idea

An LLM is not a database in the traditional sense.

It does not look up facts from a live source unless you explicitly connect it to one. Instead, it generates likely next tokens based on training and input context.

What this means in practice

LLMs are very good at:

  • summarizing text
  • drafting emails and reports
  • transforming data into natural language
  • explaining code
  • generating structured output
  • answering many general questions

But they are weaker when a task requires:

  • up-to-date information
  • organization-specific knowledge
  • exact legal/medical/financial certainty
  • citation-backed answers
  • access to changing systems or data

Mental model

Think of an LLM as:

  • a strong reasoning and language engine
  • with broad but imperfect background knowledge
  • that becomes much more useful when connected to external information

2. Why Model-Only Answers Are Often Insufficient

There are several reasons LLMs need external knowledge.

2.1 Knowledge can be stale

Training happens at a point in time. The world changes after that.

Examples:

  • product pricing changes
  • company policies are updated
  • software libraries release new versions
  • regulations evolve
  • today’s weather is different from yesterday’s

If the model is not connected to current data, it may answer confidently but incorrectly.

2.2 The model does not know your private context

Even a highly capable model does not automatically know:

  • your company handbook
  • internal support documentation
  • customer account data
  • private contracts
  • project-specific architecture docs
  • proprietary datasets

Without access to your actual materials, the model can only guess or give generic advice.

2.3 Hallucinations happen

An LLM may generate plausible-sounding but false content.

Common hallucination patterns:

  • inventing policies
  • making up URLs or citations
  • stating incorrect technical details
  • claiming certainty without evidence
  • blending multiple concepts into a wrong answer

This is especially risky when users ask for exact facts.

2.4 The model cannot verify claims by itself

A standalone prompt like:

“What is our company refund policy?”

has a problem if the model has not been given that policy.

Even if it responds fluently, it is not verifying against a source. It is generating a likely answer.

That is not enough for:

  • compliance-sensitive tasks
  • customer-facing answers
  • internal operations
  • decision support
  • technical troubleshooting

2.5 Many tasks require precise source grounding

Users often need answers that are:

  • faithful to a document
  • limited to a known set of facts
  • auditable
  • reproducible
  • traceable to a source

This is where external knowledge becomes essential.

3. The Role of External Knowledge

External knowledge means providing the model with information outside its built-in training patterns.

This can include:

  • retrieved documents
  • database records
  • API responses
  • uploaded files
  • search results
  • internal wiki pages
  • product catalogs
  • support tickets
  • structured business data

Why this helps

When the model receives relevant context, it can:

  • answer using real source material
  • become more accurate
  • reduce hallucinations
  • tailor answers to your domain
  • work with fresh information
  • cite or summarize specific documents

Simple formula

A very useful mental model is:

LLM + Relevant Context = Better Answers

Not perfect answers, but typically much more trustworthy answers.

4. Common Patterns for Adding External Knowledge

4.1 Retrieval-Augmented Generation (RAG)

RAG means:

  1. find relevant documents or passages
  2. send them to the model as context
  3. ask the model to answer based on those materials

This is one of the most common ways to ground model outputs.

Typical use cases:

  • internal knowledge assistants
  • support bots
  • document Q&A
  • policy lookup
  • enterprise search experiences

4.2 Tool and API access

Sometimes the model should not guess a value. It should fetch it.

Examples:

  • checking a shipping status via API
  • retrieving account balance
  • looking up inventory
  • calculating a quote
  • querying a database
  • calling a weather service

In these cases, the model acts more like an orchestrator.

4.3 File-based grounding

A user can provide:

  • PDFs
  • text files
  • CSVs
  • JSON
  • reports
  • meeting notes

The model can then summarize, analyze, or answer questions based on that file content.

4.4 Structured context injection

For simple workflows, you can manually provide context in the prompt.

Example:

  • policy excerpt
  • product info block
  • troubleshooting checklist

This is a good starting point before building a full retrieval pipeline.

5. Ungrounded vs Grounded Prompting

Let’s compare two patterns.

Ungrounded prompt

What is the refund policy for premium subscriptions?

Problem: - The model may answer generically - It may invent details - It has no guarantee of matching your real policy

Grounded prompt

Use only the policy text below to answer the question.

Policy:
Premium subscriptions can be canceled within 14 days of purchase for a full refund.
After 14 days, refunds are not available.
Annual plans upgraded from monthly plans are prorated based on unused monthly credit.

Question:
What is the refund policy for premium subscriptions?

Benefit: - the model has concrete source material - the answer is constrained - accuracy is much more likely

6. Hands-On Exercise 1: Compare Model-Only vs Context-Grounded Answers

Objective

See how external context changes answer quality.

What you will build

A Python script that:

  1. asks a question with no context
  2. asks the same question with a provided policy snippet
  3. compares the two outputs

Setup

Install the OpenAI Python SDK:

pip install openai

Set your API key:

export OPENAI_API_KEY="your_api_key_here"

On Windows PowerShell:

$env:OPENAI_API_KEY="your_api_key_here"

Example Code

"""
Session 1 - Hands-on Exercise 1
Compare ungrounded vs grounded LLM responses using the OpenAI Responses API.

Requirements:
    pip install openai

Environment:
    OPENAI_API_KEY must be set
"""

from openai import OpenAI

# Create a reusable client instance.
client = OpenAI()

# We will use the requested model for all examples.
MODEL_NAME = "gpt-5.4-mini"

# A simple business policy that the model would not know unless we provide it.
POLICY_TEXT = """
Premium subscriptions can be canceled within 14 days of purchase for a full refund.
After 14 days, refunds are not available.
If a customer upgrades from monthly to annual, unused monthly credit is prorated.
Team plans are non-refundable once the billing cycle begins.
""".strip()


def ask_model_without_context(question: str) -> str:
    """
    Ask the model a question without supplying any domain-specific policy context.
    This simulates a model-only response.
    """
    response = client.responses.create(
        model=MODEL_NAME,
        input=[
            {
                "role": "system",
                "content": [
                    {
                        "type": "input_text",
                        "text": "You are a helpful assistant. Answer clearly and concisely."
                    }
                ],
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": question
                    }
                ],
            },
        ],
    )

    return response.output_text


def ask_model_with_context(question: str, policy_text: str) -> str:
    """
    Ask the model the same question, but ground it with supplied business policy text.
    The instructions explicitly tell the model to use only the provided context.
    """
    grounded_prompt = f"""
Use only the policy text below to answer the user's question.
If the answer is not present in the policy, say: "The policy text does not specify that."

Policy text:
{policy_text}

Question:
{question}
""".strip()

    response = client.responses.create(
        model=MODEL_NAME,
        input=[
            {
                "role": "system",
                "content": [
                    {
                        "type": "input_text",
                        "text": (
                            "You answer questions strictly from provided context. "
                            "Do not invent policy details."
                        )
                    }
                ],
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": grounded_prompt
                    }
                ],
            },
        ],
    )

    return response.output_text


def main() -> None:
    question = "What is the refund policy for premium subscriptions and team plans?"

    print("=" * 80)
    print("QUESTION")
    print(question)

    print("\n" + "=" * 80)
    print("MODEL-ONLY ANSWER")
    model_only_answer = ask_model_without_context(question)
    print(model_only_answer)

    print("\n" + "=" * 80)
    print("CONTEXT-GROUNDED ANSWER")
    grounded_answer = ask_model_with_context(question, POLICY_TEXT)
    print(grounded_answer)


if __name__ == "__main__":
    main()

Example Output

================================================================================
QUESTION
What is the refund policy for premium subscriptions and team plans?

================================================================================
MODEL-ONLY ANSWER
Refund policies for premium subscriptions and team plans vary by company. In many
cases, premium subscriptions may be eligible for a refund within a limited time,
while team plans may follow separate billing terms. You should check the official
policy or support documentation for exact details.

================================================================================
CONTEXT-GROUNDED ANSWER
Premium subscriptions can be canceled within 14 days of purchase for a full refund.
After 14 days, refunds are not available. If a customer upgraded from monthly to
annual, unused monthly credit is prorated. Team plans are non-refundable once the
billing cycle begins.

Discussion

What changed?

  • The first answer is generic and cautious
  • The second answer is specific and tied to actual provided text
  • Grounding gives the model a factual basis

Key lesson

Without context, the model may still sound useful.
With context, the answer becomes much more reliable for domain-specific questions.

7. Hands-On Exercise 2: Force the Model to Admit When Context Is Missing

Objective

Learn an important grounding pattern: if the source doesn’t contain the answer, the model should say so.

This helps reduce hallucinations.

Example Code

"""
Session 1 - Hands-on Exercise 2
Teach the model to answer only from supplied context and explicitly say when
information is missing.
"""

from openai import OpenAI

client = OpenAI()
MODEL_NAME = "gpt-5.4-mini"

POLICY_TEXT = """
Standard subscriptions can be canceled anytime.
Premium subscriptions can be canceled within 14 days of purchase for a full refund.
After 14 days, refunds are not available.
""".strip()


def ask_grounded_question(question: str) -> str:
    """
    Ask a question that must be answered only from the provided policy text.
    """
    prompt = f"""
Answer using only the policy text below.
If the answer is missing, reply exactly:
The policy text does not specify that.

Policy text:
{POLICY_TEXT}

Question:
{question}
""".strip()

    response = client.responses.create(
        model=MODEL_NAME,
        input=prompt,
    )
    return response.output_text


def main() -> None:
    questions = [
        "Can premium subscriptions be refunded?",
        "Do student plans get a discount?",
    ]

    for question in questions:
        print("=" * 80)
        print(f"Question: {question}")
        answer = ask_grounded_question(question)
        print(f"Answer: {answer}\n")


if __name__ == "__main__":
    main()

Example Output

================================================================================
Question: Can premium subscriptions be refunded?
Answer: Premium subscriptions can be canceled within 14 days of purchase for a full refund. After 14 days, refunds are not available.

================================================================================
Question: Do student plans get a discount?
Answer: The policy text does not specify that.

Why this matters

A grounded assistant should not fill gaps with guesses.

This is one of the most practical habits in GenAI system design: prefer explicit uncertainty over confident invention.

8. Theory: Why This Becomes Critical in Real Applications

Customer support assistants

Without external documentation: - answers may be generic - policy details may be wrong - support risk increases

With external knowledge: - answers can reflect real policy articles - agent handoff can improve - consistency improves

Internal enterprise assistants

Without retrieval: - the assistant only knows general public patterns - it cannot answer questions about your internal process

With retrieval: - it can summarize onboarding docs - explain internal runbooks - answer architecture questions from company documents

Developer tools

Without external data: - coding advice may be generic - dependency versions may be wrong

With external knowledge: - the model can use your codebase docs - read API specs - incorporate current project conventions

Analytics and operations

Without live access: - the model cannot know current metrics - it may fabricate values

With tools or APIs: - it can fetch the actual data - explain the numbers - help with decision support

9. Design Principles for External Knowledge

As you build GenAI applications, keep these principles in mind.

9.1 Retrieve the minimum useful context

Too little context: - missing facts

Too much context: - noise - higher cost - weaker focus

The goal is the most relevant context, not all context.

9.2 Instruct the model clearly

Good instructions help a lot.

Examples:

  • “Answer only from the provided text.”
  • “If the answer is not in the source, say so.”
  • “Quote the relevant sentence.”
  • “Do not infer policy beyond what is written.”

9.3 Separate reasoning from facts

The model can reason well, but facts should come from trusted sources whenever possible.

A healthy division of labor is:

  • external systems provide facts
  • the model explains, summarizes, compares, and communicates

9.4 Prefer source-aware outputs for high-trust use cases

In many applications, you want:

  • answer + source excerpt
  • answer + citation
  • answer + confidence constraints
  • answer + link to original record

This increases transparency and user trust.

10. Mini Exercise: Improve a Bad Prompt

Task

Here is a weak prompt:

What is our password reset policy?

Improve it

Rewrite it so the model must answer only from provided policy text and admit when information is missing.

Example improved prompt

Use only the policy text below to answer the question.
If the answer is not stated in the text, reply: "The policy text does not specify that."

Policy text:
[insert password reset policy here]

Question:
What is our password reset policy?

Reflection

This small change is a big mindset shift: you are no longer asking the model to guess; you are asking it to operate over evidence.

11. Common Mistakes Beginners Make

Mistake 1: Treating the LLM like a source of truth

Problem: - fluent output feels authoritative

Fix: - ground important answers in documents, data, or APIs

Mistake 2: Providing context but not constraining behavior

Problem: - the model may use both context and its own prior assumptions

Fix: - explicitly say “use only the provided context”

Mistake 3: Not handling missing information

Problem: - the model fills gaps

Fix: - define a fallback response such as: - “The document does not specify that.”

Mistake 4: Dumping too much irrelevant text

Problem: - signal gets diluted

Fix: - provide concise, relevant context

12. Useful Resources

  • OpenAI Responses API guide: https://developers.openai.com/api/docs/guides/migrate-to-responses
  • OpenAI API docs: https://platform.openai.com/docs
  • OpenAI Python SDK: https://github.com/openai/openai-python
  • Prompt engineering guide: https://platform.openai.com/docs/guides/prompt-engineering

13. Recap

Main ideas from this session

  • LLMs are strong language engines, not guaranteed fact stores
  • They can be stale, generic, or wrong without grounding
  • External knowledge improves:
  • accuracy
  • relevance
  • freshness
  • trustworthiness
  • Grounded prompting is the first step toward retrieval-augmented and agentic systems

One-sentence takeaway

LLMs become far more reliable when they are connected to the right information at the right time.

14. End-of-Session Challenge

Extend Exercise 1 so that it:

  1. asks three different policy questions
  2. prints both model-only and grounded answers for each
  3. flags whether the grounded answer contains the fallback phrase:
  4. "The policy text does not specify that."

Suggested starter code

"""
End-of-session challenge:
Loop through multiple questions and compare ungrounded vs grounded responses.
"""

from openai import OpenAI

client = OpenAI()
MODEL_NAME = "gpt-5.4-mini"

POLICY_TEXT = """
Premium subscriptions can be canceled within 14 days of purchase for a full refund.
After 14 days, refunds are not available.
Team plans are non-refundable once the billing cycle begins.
""".strip()

QUESTIONS = [
    "Can premium subscriptions be refunded?",
    "Are team plans refundable?",
    "Do students get a special discount?",
]


def get_response(prompt: str) -> str:
    """
    Helper function to send a single prompt to the Responses API.
    """
    response = client.responses.create(
        model=MODEL_NAME,
        input=prompt,
    )
    return response.output_text


def main() -> None:
    for question in QUESTIONS:
        print("=" * 80)
        print(f"Question: {question}")

        ungrounded_prompt = question
        grounded_prompt = f"""
Use only the policy text below to answer the question.
If the answer is not present, reply exactly:
The policy text does not specify that.

Policy text:
{POLICY_TEXT}

Question:
{question}
""".strip()

        ungrounded_answer = get_response(ungrounded_prompt)
        grounded_answer = get_response(grounded_prompt)

        print("\nModel-only answer:")
        print(ungrounded_answer)

        print("\nGrounded answer:")
        print(grounded_answer)

        missing = "The policy text does not specify that." in grounded_answer
        print(f"\nMissing-info fallback used: {missing}\n")


if __name__ == "__main__":
    main()

15. Preview of the Next Session

Next, learners typically move from: - manually providing context

to: - systematically retrieving relevant context

That leads directly into retrieval concepts such as: - chunking - embeddings - semantic search - RAG pipelines

Back to Chapter | Back to Master Plan | Next Session