How to Guide AI Coding Agents to Write Better pytest Tests

AI agents often test the change instead of the responsibility.

This is the pattern I see in practice: an agent changes a low-level detail, notices that the changed line has no direct coverage, and adds a test for the detail itself.

Suppose an internal symbol changes from "abc" to "abcd":

# before
_INTERNAL_SYMBOL = "abc"

# after
_INTERNAL_SYMBOL = "abcd"

The agent adds this test:

def test_internal_symbol():
    assert _INTERNAL_SYMBOL == "abcd"

The test is green. Coverage increases. The pull request looks more complete.

The test is also useless unless the exact literal is a real public contract.

It does not protect what the unit is responsible for. It only records the latest implementation. If the symbol changes again during a safe refactor, the test fails even though no user-visible behavior, data contract, security boundary, or operational requirement changed.

A useful test starts from a different question:

The answer might be that an unknown symbol is classified correctly, an API response preserves its schema, a payment is never retried after a decline, or an unauthorized user cannot access another tenant’s data. The test should exercise that contract through the most stable practical boundary.

For example, a behavioral test might look like this:

def test_unknown_symbol_is_classified_as_unrecognized():
    result = classify_symbol("not-supported")

    assert result is SymbolStatus.UNRECOGNIZED

This test does not care which private sentinel, lookup table, or helper implements the rule. It fails only when the responsibility changes.

The fix is not a cleverer one-off prompt. Give the agent durable instructions that define good tests, then use a repeatable workflow that forces it to identify the contract before generating assertions.

Put your definition of a useful test in the repository.

If you explain the same testing preferences in every task, they will eventually be shortened, forgotten, or contradicted. Store the stable rules where your coding agent reads project guidance.

Depending on the tool and repository, that may be AGENTS.md, CLAUDE.md, a repository rules file, or a focused testing skill. The filename matters less than three properties:

1. 01
   The agent loads it before editing.
2. 02
   The instructions are versioned with the project.
3. 03
   The rules point to executable commands that verify the result.

Here is the baseline instruction block I use. It combines general code quality with a strict definition of valuable tests:

Follow these preferences in all projects unless a project's own instructions
say otherwise.

## Readability and simplicity come first

Prefer the simplest solution that fully solves the problem. Clarity beats
cleverness, brevity, and premature generalization. When two implementations
are equally correct, pick the one a reader understands faster.

## Do not write redundant comments

Code should explain itself through naming and structure. Write a comment only
when it carries context the code cannot: a non-obvious constraint, a workaround
and the reason for it, a deliberate trade-off, or a link to an external spec
or issue. Never restate what the next line already says.

## Follow SOLID principles

Apply single responsibility, open/closed, Liskov substitution, interface
segregation, and dependency inversion as design pressure, not as ceremony.
They justify splitting a module that has grown two reasons to change, not
wrapping every class in an interface it will only ever have one implementation
of.

## Test behavior, not implementation

Tests must protect stable user-visible behavior, security boundaries, data
contracts, and operational requirements.

Do not write:

- micro-tests that mirror individual lines or private implementation details
- assertions on exact markup or copy that can change without the requirement
  changing
- tests that exercise a framework's or library's own behavior rather than our
  use of it

A test that has to be rewritten every time the code is refactored, without the
requirement having changed, is testing the wrong thing.

## Identify the contract before writing tests

Before changing tests:

1. Identify the unit under test and state its responsibility.
2. List the stable behaviors and boundaries the tests must protect.
3. List private details that tests must not depend on.
4. Inspect nearby tests and reuse the project's established public boundaries,
   fixtures, and commands.
5. Propose the smallest set of tests in which every case protects a distinct
   behavior, boundary, regression, or failure mode.
6. Ask for clarification when the intended behavior cannot be established from
   requirements, accepted behavior, or existing contracts.

## Prove that new tests work

For a bug fix or new behavior, add the focused test before changing production
code. Run it and confirm that it fails for the expected reason. Then implement
the smallest correction and rerun the focused test.

Run the related tests and the complete project quality gate before finishing.
Report the exact commands and results. Never weaken an assertion, delete a
test, or change expected behavior merely to obtain a green result.

The first half guides implementation quality. The testing sections stop the agent from treating coverage as a request to mirror every line.

Keep the rules focused. A repository instruction file should describe stable expectations, not every detail of the current task. Put the specific behavior and acceptance criteria in the task prompt.

Make the agent explain what the unit owns before it writes code.

Do not start with “add tests for this file.” A file can contain several responsibilities, and its private structure is not automatically a contract.

Start with an inspection-only prompt:

Inspect the implementation, its callers, nearby tests, and relevant project
instructions. Do not edit files yet.

Before proposing tests, report:

1. the unit under test;
2. its single responsibility;
3. the public or stable boundary through which that responsibility is observed;
4. the behaviors, boundaries, failures, security rules, or data contracts that
   must remain true;
5. implementation details that tests should not assert directly;
6. the smallest test set in which every case protects a distinct requirement.

Flag any expected result that you cannot trace to a requirement, accepted
behavior, existing public contract, or confirmed bug report.

Review the response before allowing implementation.

Reject plans that propose tests because a line, literal, branch, or private helper exists. Ask what requirement each proposed test protects. If the agent cannot answer, remove the test or clarify the contract.

A compact test plan should use outcomes such as:

• returns a retryable result for a transient failure before the attempt limit;
• denies access when the authenticated user belongs to another tenant;
• preserves the documented response keys when optional data is missing;
• records one operational failure event when an external call times out;
• raises the public validation error at the exact input boundary.

These statements tell you why a test deserves to exist. “Covers line 42” does not.

For existing code, the implementation can reveal current behavior but not always intended behavior. Compare the agent’s explanation with product requirements, API documentation, bug reports, and your own understanding. Do not let the current code silently define the expected result when the code itself may be wrong.

If you are still learning pytest syntax and discovery, first write and run a small pytest test by hand. Reviewing generated tests is much easier once you know how an assertion, failure report, and boundary case work.

Separate the test from the implementation change.

When an agent writes a test and production code in the same step, a green result is weak evidence. The test may have passed before the change, reproduced the implementation, or exercised a different path.

For a bug fix, use two explicit phases.

Phase one: reproduce

Add only the smallest regression test that proves the reported behavior is
wrong. Do not change production code.

Run the focused test. It must fail for the expected reason. Report the failure
and stop.

Read the failure. Confirm that it represents the defect rather than an import error, bad fixture, or invented expectation.

Phase two: correct

Now implement the smallest production change that satisfies the regression
test without changing the established contract.

Run the focused test, the related test module, and the complete project gate.
Do not modify the regression test unless new requirement information proves its
expectation was wrong.

The same pattern works for new behavior. The first test should fail because the capability does not exist yet. The implementation should make it pass without broad unrelated edits.

Characterization tests around legacy code are different. They may be green immediately because their purpose is to record verified existing behavior before a refactor. In that case, ask the agent to explain why the behavior is worth preserving. When practical, make a temporary local mutation to confirm the test is sensitive to the relevant rule, restore the implementation, and rerun the suite.

Red-green is not ceremony. It answers two concrete questions:

1. Could the test detect the missing or broken behavior?
2. Did this implementation change make that behavior pass?

Without the red result, you have only the second half of the evidence.

Use a 100% coverage gate to expose untested paths.

AI makes test generation cheap enough to challenge an old compromise: leaving project-owned Python code partly uncovered because the team does not have time to write the tests.

A strict gate makes missing statements and branches visible on every run. It also stops an agent from declaring success after adding one happy-path test.

For a project whose import package is named shop, install pytest and pytest-cov with uv:

uv add --dev pytest pytest-cov

Then add the gate to pyproject.toml:

[tool.pytest.ini_options]
addopts = "--cov=shop --cov-report=term-missing"
testpaths = ["tests"]

[tool.coverage.run]
branch = true

[tool.coverage.report]
fail_under = 100
show_missing = true

Run the project gate:

uv run pytest

The pytest-cov configuration documentation supports adding coverage options through pytest’s addopts. Coverage.py reads branch, fail_under, and show_missing from pyproject.toml; a result below fail_under exits unsuccessfully.

Replace shop with your project’s actual import package. Keep the coverage scope on application-owned code. Framework internals, generated files, migrations, and other out-of-scope code should be handled deliberately in configuration, not hidden by adding meaningless tests.

A 100% gate creates useful pressure, but it does not define test quality. An agent can reach the number with assertions that never protect a requirement. Coverage answers “which code ran?” It does not answer:

• Was the expected result correct?
• Could the assertion detect the regression?
• Did a mock remove the boundary that can fail?
• Is important behavior missing from the implementation entirely?
• Will the test survive a safe refactor?

Use coverage to find gaps. Use the contract and review process to decide how those gaps should be tested.

Do not approve a micro-test of _INTERNAL_SYMBOL == "abcd" merely because one line remains uncovered. Exercise the public responsibility that reaches the line. If no meaningful behavior can reach it, investigate whether the code is dead, the design hides a responsibility, or the coverage scope needs an explicit and justified exception.

Review generated tests by asking what change should break them.

A generated test suite can be tidy, fast, and fully covered while protecting the wrong things. Review it as production code.

For every test, ask:

1. 01
   Which stable requirement does the test protect?
2. 02
   What realistic defect should make it fail?
3. 03
   Does it use the public or most stable practical boundary?
4. 04
   Does the expected value come from the contract rather than the implementation?
5. 05
   Would a safe refactor leave the test unchanged?
6. 06
   Is a mock replacing an external boundary or hiding the unit’s real work?
7. 07
   Does this case add a distinct behavior beyond the nearby tests?

Remove or rewrite tests with these smells:

The changed-literal test

def test_internal_symbol():
    assert _INTERNAL_SYMBOL == "abcd"

This protects the diff, not the responsibility.

The copied-condition test

expected = status_code in RETRYABLE_CODES and attempt < max_attempts
assert should_retry(status_code, attempt, max_attempts) == expected

The test can repeat the same bug as the implementation. Use explicit cases derived from the retry contract.

The private-call test

repository._load_from_cache.assert_called_once()

This blocks refactoring unless the collaborator interaction is itself required. Prefer the returned result, persisted state, emitted event, or externally visible side effect.

The exact-copy test

assert response.content == b"<h1>Welcome back, Artem</h1>"

Exact markup or copy is useful only when that exact output is a product, accessibility, or SEO contract. Otherwise assert the semantic element, response data, or behavior that must survive copy and layout changes.

The framework test

A test that proves pytest discovers test_ functions, Django saves a normal model, or a standard library function behaves according to its own documentation protects someone else’s code. Test your validation, permissions, defaults, integration, or business rule instead.

The fastest review question is: “If I refactor the implementation without changing requirements, should this test fail?” If the answer is yes, the test probably owns the wrong detail.

Use one task prompt to drive the complete testing loop.

The repository instructions define the standard. The task prompt supplies the behavior for one change.

Here is a complete example you can adapt:

Inspect the relevant production code, callers, tests, and repository
instructions before editing.

Testing goal:
When the payment provider times out, checkout must return the existing
retryable error, must not create a completed order, and must record one failure
event. The public response schema must not change.

First report:
1. the unit and its responsibility;
2. the stable boundary to test;
3. the expected behavior and important edge cases;
4. private details the tests must not assert;
5. the smallest useful test plan.

Do not write tests until the plan is clear. Ask about ambiguous expected
behavior instead of deriving it only from the current implementation.

Then:
1. add the focused behavioral test without changing production code;
2. run it and confirm the expected red result;
3. implement the smallest production change;
4. rerun the focused test;
5. run related tests;
6. run the full test and 100% statement-and-branch coverage gate;
7. review the final diff for micro-tests, duplicated implementation logic,
   unnecessary mocks, exact copy or markup assertions, and unrelated changes.

Report every command and result. Do not weaken tests or change requirements to
make the gate pass.

Replace the example checkout contract with concrete behavior from your task. Do not replace it with a filename or a request to “increase coverage.” The agent can discover files. It cannot safely invent the behavior your software should promise.

This prompt is intentionally procedural. Agents produce better tests when they receive feedback between planning, red, green, and full verification instead of doing everything in one unobserved pass.

Make useful AI-generated tests the normal path.

You do not need a different testing philosophy for AI. You need to make proven engineering practices explicit enough for an agent to follow.

Start with one Python project:

1. 01
   Add the durable behavior-first instructions.
2. 02
   Configure pytest-cov to measure project-owned code and fail below 100% statement and branch coverage.
3. 03
   Choose one real bug or missing behavior.
4. 04
   Make the agent identify the responsibility and contract before editing.
5. 05
   Observe the regression test fail.
6. 06
   Let the agent implement the smallest correction.
7. 07
   Review every generated test for a stable reason to exist.
8. 08
   Run the complete gate and inspect the final diff.

Once the workflow works, package it into repository guidance or a reusable skill. Keep the coverage command and quality gate deterministic. Update the instructions when repeated failures reveal a missing rule, not for every isolated mistake.

For the broader production workflow around planning, reviews, debugging, documentation, deployment checks, and safe experimentation, read AI Coding Agents: Production Reliability Matters.

The next time an agent adds a literal assertion for a literal it just changed, do not patch that one test. Fix the instruction that allowed the agent to confuse code coverage with a contract.