Testing - Evaluating AI Output

10.0 What This Teaches

Testing AI-generated output is harder than testing ordinary functions because the same prompt can produce different text on different runs. You cannot simply compare strings with assertEqual. This tutorial covers:

Why AI output resists traditional unit tests
Structural tests: things you can check deterministically
Semantic tests: running generated code and checking its behavior
Using temperature=0 to improve test stability
Snapshot (golden file) testing for regression detection
Human-in-the-loop evaluation for subjective quality

10.1 Why AI Output Is Hard to Test

AI models are non-deterministic by default. The same prompt sent twice may return two structurally different responses. This breaks the fundamental assumption of automated unit testing - that a function returns the same value for the same input.

The practical consequence: you cannot use assertEqual(response_text, expected) for anything beyond trivial prompts. Instead, test properties of the output: does it contain the right structure? Does the code it generated actually run?

10.2 Structural Tests

Structural tests check properties you can evaluate without running the code. Examples: does the response contain a function definition? Does it parse as valid Python?

# structural_test.py - check that the response contains a function and parses.
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=512,
    temperature=0,
    messages=[{"role": "user", "content": "Write a Python function that adds two numbers."}]
)

text = response.content[0].text

# Structural check 1 - does the response contain a function definition?
assert "def " in text, "Response does not contain a function definition"

# Structural check 2 - does the code block parse as valid Python?
try:
    compile(text, "<string>", "exec")
    print("Syntax OK")
except SyntaxError as e:
    print(f"Syntax error in generated code: {e}")

compile() parses the source without executing it. It catches syntax errors but not logic bugs - that requires running the code.

10.3 Semantic Tests

A semantic test runs the generated code and checks its behavior with known inputs. Python's built-in exec() executes a string as code in a namespace you control:

# semantic_test.py - exec the generated function and test its behavior.
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=256,
    temperature=0,
    messages=[{"role": "user", "content":
        "Write only a Python function named add(a, b) that returns a + b. "
        "No explanation, just the function."}]
)

code = response.content[0].text

# Execute the generated code in an isolated namespace
namespace = {}
exec(code, namespace)

# Call the generated function with known inputs
add = namespace["add"]
assert add(2, 3) == 5,  f"Expected 5, got {add(2, 3)}"
assert add(0, 0) == 0,  f"Expected 0, got {add(0, 0)}"
assert add(-1, 1) == 0, f"Expected 0, got {add(-1, 1)}"

print("All semantic tests passed.")

The prompt asks for only the function to make parsing straightforward. Asking for "no explanation" reduces the chance that the model wraps the function in markdown or prose that would cause exec() to fail.

10.4 Temperature=0 for Reproducibility

With temperature=0, the model picks the highest-probability token at every step. This does not guarantee identical output across model versions, but it is far more stable than the default temperature. For automated tests, always set temperature=0:

# Always use temperature=0 in automated test scripts.
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=512,
    temperature=0,        # makes output as stable as possible
    messages=[{"role": "user", "content": "Write a Python function to reverse a string."}]
)

Without this, a test that passes today may fail tomorrow because the model sampled a different but equally valid function structure.

10.5 Snapshot Testing

A snapshot test (also called a golden file test) generates output once, saves it, and compares future runs against the saved version. This catches regressions when the model or prompt changes:

# snapshot_test.py - generate output, save it, compare on later runs.
import anthropic
from pathlib import Path

client = anthropic.Anthropic()
GOLDEN_FILE = Path("golden_reverse.txt")

PROMPT = ("Write only a Python function named reverse_string(s) that returns s reversed. "
          "No explanation.")

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=256,
    temperature=0,
    messages=[{"role": "user", "content": PROMPT}]
)
current = response.content[0].text

if not GOLDEN_FILE.exists():
    GOLDEN_FILE.write_text(current)
    print("Golden file created.")
else:
    golden = GOLDEN_FILE.read_text()
    if current == golden:
        print("Snapshot matches.")
    else:
        print("SNAPSHOT MISMATCH - output changed since baseline was saved.")
        print("--- golden ---")
        print(golden)
        print("--- current ---")
        print(current)

Delete the golden file to reset the baseline after a deliberate prompt change.

10.6 Human-in-the-Loop Evaluation

For subjective qualities - clarity, style, correctness on complex tasks - automated checks are not enough. A simple scoring script lets you rate multiple responses manually:

# human_eval.py - generate N responses and prompt the user to rate each.
import anthropic

client = anthropic.Anthropic()

PROMPT = "Explain what a Python context manager is. Be clear and concise."
NUM_RESPONSES = 3

scores = []
for i in range(NUM_RESPONSES):
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=256,
        # deliberately not setting temperature=0 to get varied responses
        messages=[{"role": "user", "content": PROMPT}]
    )
    text = response.content[0].text
    print(f"\n--- Response {i + 1} ---")
    print(text)
    score = input("Rate this response 1-5: ").strip()
    scores.append(int(score))

print(f"\nScores: {scores}")
print(f"Average: {sum(scores) / len(scores):.1f}")

Running this over time reveals whether prompt changes consistently improve quality, which no automated metric can judge for complex outputs.

10.7 Exercise

Exercise

Write a test script that:

Asks Claude to write a Python function that returns True if a string is a palindrome (ignore case).
Extracts the code from the response by finding the first line that starts with def and taking everything from there to the end.
Uses exec() to load the function into a local namespace.
Calls the function on ["racecar", "hello", "level"] and asserts the correct results: True, False, True.
Prints "All tests passed." if all assertions succeed.

10.8 Common Mistakes

Testing exact output text with assertEqual

# Bad - breaks whenever the model rephrases the same correct answer.
assert response.content[0].text == "def add(a, b):\n    return a + b\n"

# Good - test behavior, not exact text.
exec(response.content[0].text, ns := {})
assert ns["add"](2, 3) == 5

Exact string comparison is brittle. The model may use different whitespace, variable names, or add a docstring - all valid, but all failing the string check.

Not setting temperature=0 for automated tests

# Bad - default temperature causes flaky tests.
response = client.messages.create(model="claude-sonnet-4-6", max_tokens=256,
    messages=[{"role": "user", "content": "Write a sort function."}])

# Good - pin temperature to reduce variation.
response = client.messages.create(model="claude-sonnet-4-6", max_tokens=256,
    temperature=0,
    messages=[{"role": "user", "content": "Write a sort function."}])

Random variation makes tests fail intermittently. Always use temperature=0 in automated test scripts.

exec()ing model output without any sanity check

# Risky - no check before executing arbitrary model output.
exec(response.content[0].text)

# Safer - compile first to catch syntax errors, then exec in an isolated namespace.
code = response.content[0].text
compile(code, "<string>", "exec")   # raises SyntaxError if malformed
ns = {}
exec(code, ns)

Model output is not guaranteed to be safe or even syntactically valid. At minimum, run compile() before exec() and always use an isolated namespace rather than exec(code) directly into the global scope.

10.9 Key Terms

Term	Meaning
structural test	A test that checks the shape or format of the output without running it
semantic test	A test that runs the generated code and checks its behavior with known inputs
exec()	Python built-in that executes a string as code; used to run model-generated functions
compile()	Python built-in that parses source code and catches syntax errors without executing it
golden file	A saved reference output used to detect changes in future runs
snapshot test	A test that compares current output against a saved golden file
temperature=0	Setting that makes model output as deterministic as possible; required for stable automated tests
human evaluation	Manual scoring of model output for subjective qualities that automated checks cannot measure