Skip to main content
API Testing · lesson 4 of 11

Beginner

3

Intermediate

5

Advanced

3

Also part of API Testing

2
Open the API Testing page →
API Testing intermediate Lesson 4 of 11

Validating JSON Responses and Contracts

Why comparing a whole response body with == breaks on the first dynamic field, and how jsonschema catches a missing or mistyped field that a value check would miss.

By SkillByExample · Published

api-testing python jsonschema contract-testing pytest

A response body has fields your test cares about and fields it doesn’t. Confusing the two is the most common way an API test suite becomes fragile.

The bug: comparing the whole body

import requests

BASE_URL = "https://httpbin.org"


def test_whole_response_body_matches_exactly():
    response = requests.get(f"{BASE_URL}/get", params={"category": "books"})
    assert response.json() == {
        "args": {"category": "books"},
        "url": "https://httpbin.org/get?category=books",
    }
__________________ test_whole_response_body_matches_exactly ___________________

    def test_whole_response_body_matches_exactly():
        response = requests.get(f"{BASE_URL}/get", params={"category": "books"})
>       assert response.json() == {"args": {"category": "books"}, "url": "https://httpbin.org/get?category=books"}
E       AssertionError: assert {'args': {'ca...tegory=books'} == {'args': {'ca...tegory=books'}
E
E         Omitting 2 identical items, use -vv to show
E         Left contains 2 more items:
E         {'headers': {'Accept': '*/*',
E                      'Accept-Encoding': 'gzip, deflate',
E                      'Host': 'httpbin.org',
E                      'User-Agent': 'python-requests/2.34.2',...
E
E         ...Full output truncated (18 lines hidden), use '-vv' to show

test_whole_body_bug.py:8: AssertionError

The test asserted the two fields it cared about correctly — and failed anyway, because the real response also carries headers and origin, which the test never mentioned. Every API response has fields like this: request IDs, the caller’s IP, server timing. An exact-equality test breaks the instant any of them exist.

The fix: assert a subset

def test_response_contains_the_fields_we_sent():
    response = requests.get(f"{BASE_URL}/get", params={"category": "books"})
    data = response.json()
    expected = {
        "args": {"category": "books"},
        "url": "https://httpbin.org/get?category=books",
    }
    assert expected.items() <= data.items()
test_subset.py::test_response_contains_the_fields_we_sent PASSED         [100%]

============================== 1 passed in 1.32s ==============================

dict.items() returns a set-like view, and <= between two of them is a subset check — true when every pair in expected shows up in data, no matter what else data contains. This test now only fails when a field it actually asked about is wrong.

It has its own problem, though. Change one value so the check fails:

E       AssertionError: assert dict_items([('args', {'category': 'toys'}), ('url', '...')])
                          <= dict_items([('args', {'category': 'books'}),
                          ('headers', {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate',
                          'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.34.2',
                          'X-Amzn-Trace-Id': 'Root=1-6aa67f0e-51f154c6788fcec77043b883'}),
                          ('origin', '203.0.113.42'),
                          ('url', 'https://httpbin.org/get?category=books')])

Both full dicts get dumped as repr() text — no diff, no indication of which field actually mismatched. A subset check is good for “does this contain what I sent,” and bad as a failure message. For checking the shape of a contract rather than one or two values, reach for something built for it.

jsonschema: checking structure, not values

pip install jsonschema
import jsonschema
import requests

BASE_URL = "https://httpbin.org"

RESPONSE_SCHEMA = {
    "type": "object",
    "required": ["args", "headers", "origin", "url"],
    "properties": {
        "args": {"type": "object"},
        "headers": {"type": "object"},
        "origin": {"type": "string"},
        "url": {"type": "string"},
    },
}


def test_get_response_matches_schema():
    response = requests.get(f"{BASE_URL}/get")
    jsonschema.validate(instance=response.json(), schema=RESPONSE_SCHEMA)
test_schema.py::test_get_response_matches_schema PASSED                  [100%]

============================== 1 passed in 1.32s ==============================

No custom assertion needed — validate() raises on any mismatch and does nothing on success. The schema says nothing about values (it doesn’t care if origin is this IP or that one), only about shape: these four keys must exist, and each must be the declared type.

Catching contract drift

Suppose the API team changes origin from a string to a list — say, to support multiple proxy hops. The schema catches it immediately:

E       jsonschema.exceptions.ValidationError: '203.0.113.42' is not of type 'array'
E
E       Failed validating 'type' in schema['properties']['origin']:
E           {'type': 'array'}
E
E       On instance['origin']:
E           '203.0.113.42'

This is the failure a schema test is for: not “this specific value changed” but “this field is no longer the type every caller was promised.” A subset check would have missed this entirely, since it only inspects keys the test happens to name — it never asked about origin at all.

Practice

1. Assert on one deeply nested field directly, without a schema. 
def test_nested_header_field_directly():
    response = requests.get(f"{BASE_URL}/get")
    data = response.json()
    assert data["headers"]["Host"] == "httpbin.org"
test_practice.py::test_nested_header_field_directly PASSED               [ 33%]

For one specific value, direct key access with a clear assertion still beats reaching for a schema — save the schema for checking overall shape.

2. Confirm the schema also catches a required field that's simply missing, not just wrongly typed. 
def test_schema_catches_a_missing_required_field():
    incomplete_payload = {"args": {}, "headers": {}, "url": "https://httpbin.org/get"}
    with pytest.raises(jsonschema.exceptions.ValidationError, match="'origin' is a required property"):
        jsonschema.validate(instance=incomplete_payload, schema=RESPONSE_SCHEMA)
test_practice.py::test_schema_catches_a_missing_required_field PASSED    [ 66%]

required and properties catch two different failure modes — a field that vanished entirely, and a field that’s present but the wrong type. A contract test needs both.

3. Assert a header exists and has a sane value, without pinning down the exact number. 
def test_response_has_a_content_length_header_of_some_value():
    response = requests.get(f"{BASE_URL}/get")
    assert "Content-Length" in response.headers
    assert int(response.headers["Content-Length"]) > 0
test_practice.py::test_response_has_a_content_length_header_of_some_value PASSED [100%]

Content-Length genuinely varies between requests — asserting it’s present and positive tests the contract (“this header exists and is meaningful”) without asserting an exact byte count that would break the next time the response body changes shape even slightly.

Next: building a small local API to test against, so lessons stop depending on a third party.

Frequently Asked Questions

Why not just assert response.json() == expected_dict for the whole body?
Real APIs return fields you didn't ask for and can't predict — request IDs, timestamps, the caller's IP, pagination metadata. Comparing the whole body for exact equality means the test breaks the moment the API adds a field, even one you don't care about, which trains people to stop trusting failures.
What does expected.items() <= data.items() actually check?
dict.items() returns a set-like view of (key, value) pairs, and <= on two such views is a subset test — it's true when every pair in expected also appears in data, regardless of what else data contains. It's a quick way to check 'contains at least these fields' without jsonschema, but its failure message is a wall of repr() text, not a clean diff.
What does jsonschema.validate() check that a dict comparison can't?
Structure and type, independent of exact values: is this field present, is it a string versus a number versus an object, is a nested object shaped the way you expect. It's the right tool for 'is this response still shaped like our contract' rather than 'does this field have this exact value.'
Does jsonschema fail if the response has extra fields I didn't list in the schema?
No, not by default — a schema with `properties` but no `additionalProperties: false` allows any extra keys. That default is usually right for API contract tests, since you want to catch missing or mistyped fields, not fail every time the API team adds something new and unrelated.
Should I validate a whole schema per test, or build a schema once and reuse it?
Reuse it. A schema is a contract, and duplicating it per test is how one test file has three subtly different opinions about what a valid response looks like. Define it once as a module-level constant — or import it from a shared file if several test modules need the same contract — the same way you'd share a fixture.