AI JSON passes Schema but is still wrong: verify a ticket against its source

Define the agreement before asking for extraction

Our original source says: “Ticket T-007: User cannot export PDF. Product not identified. Owner unassigned.” Extract the ticket ID, issue category, owner and evidence. Categories are export, login and unknown; an unsupported owner is null. These are editorial task rules, not business requirements invented by a model.

The official JSON Schema object guide explains properties, required and additionalProperties. Use them to control structure while defining field meaning and accepted scope yourself. Permitting a property does not establish that its value was confirmed in the source.

Copy a Schema with explicit boundaries

This original simple Schema fits KitNelo’s Draft-07 validator. It rejects extra properties, requires four fields and checks the identifier, category and types. It does not query a ticket system or determine whether an owner’s name is supported.

To try extraction with AI, use the original prompt below and the same source agreement. Preserve the model’s raw response instead of silently cleaning it into a passing object before review.

{
  "type": "object",
  "properties": {
    "ticketId": {
      "type": "string",
      "pattern": "^T-[0-9]{3}$"
    },
    "category": {
      "enum": [
        "export",
        "login",
        "unknown"
      ]
    },
    "owner": {
      "type": [
        "string",
        "null"
      ]
    },
    "evidence": {
      "type": "string",
      "minLength": 1
    }
  },
  "required": [
    "ticketId",
    "category",
    "owner",
    "evidence"
  ],
  "additionalProperties": false
}

Extract JSON from this source without adding facts.
Keep ticketId, category, owner and evidence.
category must be export, login or unknown; use null for owner without evidence.
Use an actual problem excerpt for evidence. Return JSON only, without extra fields.
Source: Ticket T-007: User cannot export PDF. Product not identified. Owner unassigned.

Three outputs need different acceptance decisions

The first constructed object uses numeric 7 for ticketId, urgent for category and an extra priority field, violating the agreement. The second fixes structure but retains owner: Mina, a deliberately unsupported name. The third changes owner to null, matching the unassigned fact in this example.

The second object is the important counterexample: a string owner is allowed, but the Schema cannot know whether Mina owns this ticket. The validator does not replace source reading, and a pass cannot overrule the original material.

Run the validator, then return to the source

Open the JSON query and Schema tool. Use the separate JSON input and JSON Schema areas; the Schema is not another field inside the ticket. Paste schema.json, then inspect each candidate in the sequence below. Confirm the current candidate and input before every run so a previous response is not mistaken for the next result.

The result can be downloaded as json-result.txt. It is the validator’s text response, not a formal export of the ticket JSON or a human evidence certificate. Rename each downloaded response by candidate and retain the original JSON plus the independent review ledger.

1. 1
   Paste invalid-syntax.json and run “Validate input”. Expect a JSON parsing error. Restore the correct input rather than deleting business fields to obtain a pass.
2. 2
   Paste candidate one with the original Schema. Expect constraint errors concerning the extra priority field, ticketId type or category enum. Error order and wording may differ between validator versions.
3. 3
   Paste candidate two and validate: it should pass. Query the JSON Pointer path /owner: expect the string “Mina”. Compare source.txt and record that this owner lacks evidence.
4. 4
   Paste candidate three and validate against the same Schema: it should also pass. Query /owner for null, then /ticketId for the string “T-007”.
5. 5
   Compare category export and the evidence fragment cannot export PDF with the source, while retaining owner:null. Record structural and evidence decisions separately in review-ledger.csv. Only candidate three is accepted in this exercise.
6. 6
   Download the individual validator responses and reopen them against the candidate filenames. The response does not replace the source JSON; a single line saying pass cannot reconstruct the validated object.

Keep uncertainty instead of repairing it into a pass

For ambiguous source text, unclear versions or undefined fields, retain uncertainty for the responsible person. One validation does not establish complete category coverage or cross-record relationships. Use the missing-versus-zero review to define unknown handling instead of filling fields with arbitrary zeroes, empty strings or names.

The tool supports Draft-07 with a combined limit of two million characters across input, comparison text and Schema. Validation runs locally. Separate syntax, Schema and business-evidence problems. Before table export, use the CSV and JSON conversion checks for identifiers and missing values; conversion cannot repair unsupported facts.

Keep inputs, validator results and review decisions separate

The exercise ZIP contains source.txt, schema.json, output-01-invalid-structure.json, output-02-valid-unverified.json, output-03-source-aligned.json, invalid-syntax.json and review-ledger.csv. The three outputs retain the original article’s cases; invalid-syntax.json adds a syntax fault. The source establishes T-007, inability to export PDF, unidentified product and an unassigned owner.

These are three candidates for one ticket, not three real tickets to append to an operations report. Each ledger row describes review of one candidate. Keep the source, candidates, validator responses and acceptance decision under different filenames so a pass from candidate two cannot be mistaken for an evidence decision on candidate three.

Two validation passes and one accepted candidate

Two candidates pass the structure layer; only one is accepted against the source. These are three editorially constructed candidates, not a measured model accuracy of one third or two completed tickets. No model run was used to estimate how often the error occurs.

An absent owner field, an empty string and null are different states. required makes the field mandatory; the current owner type permits strings and null, so an empty string may still pass this Schema while violating the exercise’s explicit unknown-value convention. State a null-for-unknown rule in the review contract or a stricter Schema instead of inferring it from a green pass.

To recover, correct the unsupported value, rerun the original Schema and repeat the source comparison. Changing only the ledger to accepted while losing the final JSON does not produce a reviewable handoff. For later tabular conversion, use the CSV/JSON checking guide and account for distinctions that may collapse into blank cells.