How much of NGSI-LD does the test suite test?

TS 104-175 (the API) and TS 104-176 (the HTTP binding) are now published, so there is a fixed target to measure the suite against. The full suite was run against a broker, and two things were measured: which parts of the API the suite's requests use, and how much of the broker's code those requests reach.

1. API coverage

Every request a Test Case makes goes through RequestsLibrary, which writes the method, the URL, the headers and the response status into output.xml. A script maps each of these requests to its operation in TS 104-176 and counts, for each operation:

• the requests, and the Test Cases that sent them;
• the URL parameters given, out of those in the operation's table;
• the status codes received, out of those the operation lists.

The operations and the parameters depend only on the Test Cases. The status codes also depend on the broker.

By clause of TS 104-176

Not one Test Case exists for these 40 operations:

• HEAD on every resource that has it (23 operations).
• Snapshots (§13): all 7.
• EntityMaps (§11): all 10, the temporal ones included.
• Retrieve Attribute (§7.5 GET) and the Attribute value sub-resource (§7.6 GET and PUT).
• Context source identity (§12, /info/sourceIdentity).

Among the operations that are tested, the parameters are the largest gap.

• Query Entities is sent 18 of its 28 parameters. It is never sent collation, containedBy, csf, entityMap, entityMapLifetime, geometryProperty, lang, orderFrom, orderGeometry or splitEntities.
• Merge Entity (PATCH) is sent none of its 5.
• type is the one parameter of most write operations, and no write is ever sent it.

Status codes:

• 207 never comes back from the single-entity writes.
• 422 never comes back from the creates.
• 201 and 203 never come back from the queries.

2. Code coverage

The broker was built with coverage instrumentation. For comparison, the same build also ran its own functional tests (952 cases). That build had a few files fewer, so the percentages of the last column have slightly smaller denominators.

The functional areas where the broker's own tests reach far more code than the suite does:

• Query Entities: 37 % against 85 %. These are the untested parameters above, plus pagination and the combinations of q, geo and scope.
• Snapshots: 0 % against about 80 %.
• EntityMaps: 4 % against 81 %.
• Distributed subscriptions: 17 % against 88 %.
• Distributed operations: 54 % against 86 %.
• Discovery forwarding: 23 % against 92 %.
• Tenants: 28 % against 87 %.
• Retrieve and query of the temporal evolution: 34–45 % against 87–89 %.

3. Plan: 100 % of the API

python3 scripts/api_coverage/api_coverage.py results/output.xml                         # the table
python3 scripts/api_coverage/api_coverage.py results/output.xml --json develop.json     # keep it
python3 scripts/api_coverage/api_coverage.py branch/output.xml --baseline develop.json  # what a branch adds

All in all: about 350–400 new Test Cases on top of today's 1 112, with each batch measured as it comes in.

4. The MRs

The schedule and the list of MRs are in the proposal to the TTF.

How this was measured

• Suite: develop plus the open fixes, 1 112 Test Cases, all passing.
• Spec: TS 104-176 from forge.etsi.org/rep/data/specifications/ts-104-176, commit 1446ed11 (2026-06-30).
• Broker: Coraine with MongoDB and TimescaleDB, built with --coverage -O0; the report was produced with gcovr.
• A GET/HEAD section of the spec counts as two operations, as the two methods are tested separately. A HEAD's status codes are 204 and the GET's errors.
• 22 of the 4 533 requests match no operation. They are Test Cases that send a malformed URL on purpose.