App: Test Generator - Generate Tests

A4.0 What This Teaches

This tutorial builds a tool that reads a Python source file and generates a pytest test suite for it. It covers:

Asking Claude to generate runnable Python code rather than prose
Writing generated code to a file with the correct naming convention for pytest
Running pytest from Python using subprocess.run()
Reading returncode to detect test failures
Diagnosing and fixing generated tests when they fail

A4.1 Application Design

The tool follows four steps:

Read the source file
Ask Claude to generate pytest tests for every function in the file
Write the generated tests to test_<filename>.py
Optionally run pytest immediately and report the result

Writing first and running second is important: if the generated code has a syntax error, pytest reports the exact line number - but only if the file exists on disk.

A4.2 The Prompt for Test Generation

The system prompt instructs the model to return only valid Python code with no explanatory prose. Any extra text would break the generated file when pytest tries to import it.

# gen_tests_prompt.py - prompts for test generation.
SYSTEM = (
    "You are a Python test writer. "
    "Write pytest tests for every function in the code. "
    "Cover: normal inputs, edge cases (empty input, zero, None), "
    "and expected failures where appropriate. "
    "Return only valid Python code. No prose."
)

def build_test_prompt(code: str) -> str:
    return f"Write tests for:\n\n```python\n{code}\n```"

"No prose" is doing real work here. Without it, the model often wraps the code in a markdown fence with an explanation, producing a file that starts with a backtick and cannot be parsed as Python.

A4.3 Complete Implementation

# gen_tests.py - generate pytest tests for a Python file.
# Usage: python gen_tests.py <filename.py> [--run]

import sys
import subprocess
from pathlib import Path
import anthropic

SYSTEM = (
    "You are a Python test writer. "
    "Write pytest tests for every function in the code. "
    "Cover: normal inputs, edge cases (empty input, zero, None), "
    "and expected failures where appropriate. "
    "Return only valid Python code. No prose."
)

def main():
    if len(sys.argv) < 2:
        print("Usage: python gen_tests.py <filename.py> [--run]")
        sys.exit(1)

    source_path = sys.argv[1]
    run_after = "--run" in sys.argv

    try:
        code = Path(source_path).read_text(encoding="utf-8")
    except FileNotFoundError:
        print(f"Error: file not found: {source_path}")
        sys.exit(1)

    client = anthropic.Anthropic()
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=2048,
        system=SYSTEM,
        messages=[{
            "role": "user",
            "content": f"Write tests for:\n\n```python\n{code}\n```",
        }],
    )

    test_code = response.content[0].text

    stem = Path(source_path).stem
    test_filename = f"test_{stem}.py"
    Path(test_filename).write_text(test_code, encoding="utf-8")
    print(f"Tests written to {test_filename}")

    if run_after:
        result = subprocess.run(
            ["pytest", test_filename, "-v"],
            capture_output=False,
        )
        if result.returncode != 0:
            print(f"\nSome tests failed. Review {test_filename} and fix any incorrect assertions.")

if __name__ == "__main__":
    main()

A4.4 Running Generated Tests

subprocess.run() launches pytest as a child process and waits for it to finish. capture_output=False lets pytest write directly to the terminal so you see its colored output in real time.

# run_tests.py - run pytest and check the exit code.
import subprocess

def run_pytest(test_filename: str) -> bool:
    result = subprocess.run(["pytest", test_filename, "-v"])
    return result.returncode == 0   # True if all tests passed

passed = run_pytest("test_mylib.py")
if not passed:
    print("Tests failed - open the file and check the assertions.")

pytest exits with code 0 when all tests pass, 1 when any test fails, and 2 or higher for collection errors (such as a syntax error in the test file). Any non-zero code means something needs attention.

A4.5 Sample Generated Tests

Given a source file defining add() and clamp(), Claude typically generates tests like these:

# test_mylib.py - generated by gen_tests.py
import pytest
from mylib import add, clamp

def test_add_positive():
    assert add(2, 3) == 5

def test_add_negative():
    assert add(-1, -4) == -5

def test_add_zero():
    assert add(0, 0) == 0

def test_clamp_below():
    assert clamp(1, 5, 10) == 5

def test_clamp_above():
    assert clamp(15, 5, 10) == 10

def test_clamp_within():
    assert clamp(7, 5, 10) == 7

The import line is the first thing to check manually. If your file is in a subdirectory the import path may need to be adjusted.

A4.6 When Generated Tests Fail

Generated tests fail for two common reasons: the expected value is wrong, or the function is called with the wrong argument order. Both are easy to spot in pytest output.

FAILED test_mylib.py::test_clamp_below - AssertionError: assert 1 == 5

  Where test_clamp_below calls: clamp(1, 5, 10)
  Expected 5, got 1

In this case the model assumed clamp(value, lo, hi) but the actual signature is clamp(lo, hi, value). Open the generated test file, fix the argument order, and re-run pytest. The file is plain Python - edit it exactly as you would any other test file.

A4.7 Exercise

Exercise

Add a --function argument so the tool generates tests for only one named function:

python gen_tests.py mylib.py --function clamp

Use argparse to parse the flag, then adjust the prompt:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("source")
parser.add_argument("--function", default=None)
parser.add_argument("--run", action="store_true")
args = parser.parse_args()

if args.function:
    prompt = (
        f"Write pytest tests only for the function named '{args.function}':\n\n"
        f"```python\n{code}\n```"
    )
else:
    prompt = f"Write tests for:\n\n```python\n{code}\n```"

Test both paths: once without --function (all functions) and once with it (just the named one).

A4.8 Common Mistakes

Running AI-generated tests without reviewing them first

A generated test that asserts divide(10, 0) == 0 will pass if your function silently returns 0 on division by zero - even though that behavior is probably wrong. Always read through the generated file before trusting a green test run. The tests confirm behavior, not correctness.

Wrong import in the generated test file

The model imports from a module name it guesses from your prompt. If your file is src/utils.py but the generated test says from utils import ..., pytest cannot find the module and every test fails with ModuleNotFoundError. Check the import line first whenever tests fail at collection time.

Generating tests for a file with syntax errors

If the source file cannot be parsed by Python, the model still generates tests - but they import a broken module and fail immediately. Run

python -m py_compile
        <filename>

to check for syntax errors before generating tests.

Key Terms

Term	Meaning
pytest	Python's standard test runner; discovers test_ functions automatically
subprocess.run()	Launches a child process, waits for it to finish, and returns a CompletedProcess object
returncode	Integer exit code from a subprocess; 0 means success, non-zero means failure
edge case	An input at the boundary of valid range, such as zero, None, or an empty list
collection error	pytest term for failing to import or parse a test file; returncode is 2
py_compile	Standard library module that checks Python syntax without running the file