ETSI CIM · Testing Task Force

All of NGSI-LD in the test suite by September 2027

SEAMWARE · 31 October 2026 · a proposal to the TTF

The suite tests about half of the published API. Seamware proposes to bring it to every operation, every URL parameter and every status code of TS 104-176. The work goes in as about 50 small coverage MRs and 19 fix MRs, each one small enough to review and merge in a day or two. The last one is merged by the end of February 2027, which leaves March to August for scripts that run the suite against several real brokers, and September for the final measurement and the TTF's documents.

1. Where the suite was on 30 October 2026

The measurement, operation by operation, is on How much of NGSI-LD does the test suite test?

Where it is today, 5 November 2026

Measured on develop at c16cd2ec with scripts/api_coverage, after the MRs merged since 30 October. The MRs still in review (HEAD, the attribute value, the first fix MRs) are not counted yet. The share of a broker's code that the suite reaches is measured again at the milestones.

2. The goal

• 90 of 90 operations, HEAD included.
• Every URL parameter and every status code that the operation tables list and a Test Case can provoke.
• Then what the resource tables do not show: the notification formats, the MQTT binding, the q-language, the attribute types, the geo-relations and timerel.
• As a result, the share of a broker's code that the suite reaches goes from 58 % towards 80 %.

3. How: many small MRs

• One MR covers one operation, or one family of parameters, with at most about 15 Test Cases.
• An MR brings its own keywords and documentation-generator entries, so it stands on its own.
• Each MR states what it adds, measured with scripts/api_coverage (MR 1) against develop.
• Each MR has passed against at least one broker before it is sent.
• Optional features (the 501 answers) are tagged, so that a broker without them can skip those Test Cases.

4. The first three, as examples

MRs 1, 4 and 7 of the list below were sent on 31 October 2026, to show the size and the description every MR will have. All three are based on develop and do not depend on each other.

• MR 1, !326, feat/api-coverage: the measurement script. No Test Case changes.
• MR 4, !327, feat/retrieve-attribute: Retrieve Attribute, 8 Test Cases. Operations 50 → 51, URL parameters 84 → 85, status codes 135 → 138.
• MR 7, !328, feat/context-source-identity: the context source identity, 1 Test Case. Operations 50 → 51, status codes 135 → 136.

5. Seamware's schedule

Seamware's part of the TTF is task T2, Extend coverage up to version 1.9.1 (September 2026 to February 2027), and a part of task T4, Interoperability Test Automation via Scripted Orchestration (November 2026 to August 2027). The schedule follows the two.

When	What	Coverage MRs (§7)	Fix MRs (§8)
Nov 2026	The measurement script, HEAD, the attribute value and the context source identity	1–8	1–7
Dec 2026	EntityMaps and Snapshots: every operation tested	9–16	8–11
Jan 2027	The untested parameters and status codes, Query Entities and the temporal queries first	17–34	12–16
Feb 2027	The rest of the parameters and status codes, then beyond the HTTP resources: notifications, MQTT, q, attribute types, geo, timerel. The last MR merged by 28 February	35–52	17–19
Mar – Aug 2027	Scripts that set up several real brokers, and the interoperability tests with real brokers (IOP_TP)
Sep 2027	The final measurement, the TTF's documents and the report to the group

That is 52 coverage MRs and 19 fix MRs by the end of February, about four a week. How fast the reviews come back sets the real pace. A doubt about the specification that an MR brings up goes to the weekly NGSI-LD API call (Tuesdays) or to Discord, with the aim of an answer within a week.

6. What Seamware asks of the group

• A reviewer for each MR, with an answer within two working days.
• Agreement on the tag for the optional features.
• The measurement script in develop (MR 1), so that every later MR can be measured the same way.

7. The coverage MRs

One line per MR. The number in brackets is its number of Test Cases. Those marked ready are on the forge already.

1. ready scripts/api_coverage: the measurement script and the TS 104-176 inventory (0) feat/api-coverage
2. ready The Request With HEAD keyword and HEAD in the documentation generator, plus 061_01: 204 on 16 resources (16) !337 feat/head-success
3. ready 061_02: HEAD errors, the same 400 or 404 as the GET, no body (16) !338 feat/head-errors
4. ready Retrieve Attribute (§7.5 GET): fragment, datasetId, 400 and 404 (8) feat/retrieve-attribute
5. ready Retrieve Attribute Value (§7.6 GET): value only, 400 and 404 (5); after the one above feat/retrieve-attribute-value
6. ready Set Attribute Value (§7.6 PUT): type kept, observedAt, errors (9); after Retrieve Attribute feat/set-attribute-value
7. ready Context source identity (§12 GET) (1) feat/context-source-identity
8. ready HEAD on the attribute, the attribute value and /info/sourceIdentity: 204, and the 404 of the GET (6) !339 feat/head-attribute-value
9. EntityMaps: create through POST /entityMaps/, retrieve, 400 and 404 (8)
10. EntityMaps: create through GET /entityMaps/, plus HEAD (6)
11. EntityMaps: /temporal/entityMaps, GET, POST and HEAD (8)
12. EntityMaps: update (PATCH) and delete, plus HEAD on /entityMaps/{entityMapId} (10)
13. Snapshots: create, retrieve status, 400 and 409 (8)
14. Snapshots: update status and delete, plus HEAD (8)
15. Snapshots: purge, 207 among them (6)
16. Snapshots: clone, 404 and 409 (5)
17. Query Entities: lang and collation (8)
18. Query Entities: geometryProperty, orderGeometry and orderFrom (10)
19. Query Entities: entityMap and entityMapLifetime, 201 (6)
20. Query Entities: containedBy, csf and splitEntities, 203 (10)
21. Query Entities (POST /entityOperations/query): 201, 203 and 400 (6)
22. Retrieve Entity: type, datasetId and containedBy (8)
23. Retrieve Entity: entityMap and entityMapLifetime, 201 and 203 (6)
24. Merge Entity: format, lang, observedAt and options (10)
25. type on the eight single-entity writes (8)
26. 207 from the single-entity writes, with a registered context source (8)
27. Purge Entities: the geo parameters (8)
28. Purge Entities: idPattern, scopeQ, csf, splitEntities, 207 (8)
29. 422 from Create Entity, the temporal create and Register Context Source (6)
30. 400 from /types/{type}, /attributes/ and /attributes/{attrId} (6)
31. Query Temporal Evolution: lang, collation, expandValues and jsonKeys (10)
32. Query Temporal Evolution: scopeQ, csf, splitEntities, entityMap, 201 and 203 (10)
33. Query Temporal Evolution (POST): 201 and 400 (4)
34. Retrieve Temporal Evolution: datasetId and lang, 203 (6)
35. Query Context Source Registrations: pick, omit, lang and endTimeAt (8)
36. Query Context Source Registrations: geometryProperty, scopeQ and csf (6)
37. Context source registration subscriptions: limit and 409 (4)
38. Serve @context: core, 400 and 422 (5)
39. 501 for the optional features, tagged so a broker without them can skip it (8)
40. Step 5 inventory: TS 104-175 and TS 104-243 in scripts/api_coverage (0)
41. Data representation in scripts/api_coverage: from the request bodies, which attribute types and which sub-attributes, such as unitCode and valueType, the suite sends (0); suggested in the review of !326
42. Notification formats: normalized, concise and keyValues, plus the attribute selection (12)
43. Notifications: GeoJSON, plus showChanges and sysAttrs (10)
44. The MQTT binding: notifications over MQTT and MQTTS, QoS (10)
45. q-language: the comparison operators and the ranges (15)
46. q-language: the pattern match, lists, ;, | and parentheses (15)
47. q-language: the attribute path, the compound values and the metadata (12)
48. Attribute types: LanguageProperty and VocabProperty (12)
49. Attribute types: JsonProperty, ListProperty and ListRelationship (15)
50. The geo-relations: near with minDistance and maxDistance, equals and disjoint (12)
51. The geo-relations: within, contains, intersects and overlaps on every geometry type (15)
52. timerel: before, after and between on observedAt, createdAt and modifiedAt (12)

Steps 4 and 5 also have parameters and status codes beyond the ones named here. Each of those gets an MR of its own, once the ones in this list are in.

8. The fix MRs

Running the suite against a broker, Test Cases were found whose expectation contradicts TS 104-175 or TS 104-176, or whose setup or teardown is incomplete. On today's develop they are 132 Test Cases. Their fixes go in as a list of their own, the same way: one theme per MR, at most about 15 Test Cases, each one with the clause it follows. Those marked rework were sent once and closed in favour of a forge issue; they come back changed as the review asked.

1. Request-level errors: an empty array to Create Entity is BadRequestData; Replace and Merge without an id; an empty attribute name in a temporal path; Delete Entity without an id (5)
2. closed A batch whose @context and Content-Type do not match is a 400 for the whole request (2) !341 - the API call of 10 November kept the 207 of a per-entity error
3. merged The NGSI-LD Null: a deleted multi-attribute notifies one Null per instance; expiresAt removed with urn:ngsi-ld:null (2) !342
4. merged A Subscription's q in a JSON body is not URL-encoded (1) !343
5. sent Temporal expectations: 021_25 with both entities, 021_08 only the matching instances (2) !346
6. merged Temporal pagination (TS 104-176 §6.4.7.3), Retrieve Temporal Evolution: 020_13, 020_14 (12) !347
7. merged Temporal pagination, Query Temporal Evolution: 021_15, 021_16 (9) !348
8. sent CSR discovery 037: entity-id matching, a real polygon for within, three-page pagination (9) !350
9. sent Exclusive registrations name an entity id and do not overlap local data (§9.3.3): 033_01_02 and the exclusive and redirect batch setups (12) !357
10. sent Context source registration subscriptions 047, part 1: the initial notification (§12.4.7), the CSR ids, timesFailed (8) !358
11. sent Context source registration subscriptions 047, part 2: the triggers, the @context on PATCH (10) !359
12. Distributed query D011: the @context in a POST body, the forwarded pick (14)
13. Distributed operations: the assertions of D001 to D005, D010 and D016 (15)
14. sent Loop detection D018_01 on a redirect registration, where 508 is the answer (1) !360
15. merged logUtils: the console argument is documented but ignored, every request goes to the terminal (0) !356
16. merged A registration's attributes as attributeNames or as the deprecated propertyNames and relationshipNames (12) !349
17. sent Housekeeping: the Delete keyword of context source registration subscriptions accepts the 4xx a Test Case expects; 041_03; the @context tests' ${uri} - already fixed on develop (4) !361
18. rework Temporal setup: the current-state twin of a temporal entity, a second temporal POST appends: 020_17 to 020_20, 008_01 (7)
19. rework The notification-result members of a Subscription are checked, not ignored; the status after the initial notification (8)

They go in between the coverage MRs of section 7, a few at a time, so that neither list waits for the other.

The interoperability Test Cases with several real brokers (IOP_TP) have a plan of their own: Interoperability tests with real brokers.