Build an evaluation dataset

An evaluation dataset is a JSON file containing the questions you want Herman to answer and the reference material used to judge those answers. Each entry is one test case. Use the same reviewed dataset when comparing runs so a score change reflects the system being tested. Start with a small, focused set from one supplier or document collection. Write specific questions, verify the reference answers against the original sources, and record where each expected fact came from.

Download a fill-in template

Copy the case object for each question and replace every REPLACE_WITH value.

Download the worked example

A complete example using fictional training data, with provenance and review metadata.

The template is valid JSON with unfinished content. The worked example teaches the format; it is not an approved benchmark or a real product datasheet. Keep new cases marked draft until their facts and answers have been reviewed.

Start with the file structure

The outer object contains qa_pairs, an array of case objects. One object inside that array becomes one question in an evaluation run.

{
  "qa_pairs": [
    {
      "question": "What hardness does the training datasheet report for Training Grade A, and under which test conditions?",
      "answer": "The training datasheet reports a hardness of 60 Shore A, measured using ASTM D2240 at 23 degrees C.",
      "relevant_facts": [
        "Training Grade A has a reported hardness of 60 Shore A.",
        "The hardness test method is ASTM D2240.",
        "The hardness test temperature is 23 degrees C."
      ],
      "type_of_question": "short_answer",
      "_meta": {
        "internal_ref": "TRAINING-001",
        "category": "Material Properties",
        "difficulty": "easy",
        "source_url": "https://docs.plastics.com/evaluations/dataset-json#training-source",
        "source_ref": "Training source: hardness row",
        "author": "Documentation example",
        "status": "draft"
      }
    }
  ]
}

Use double quotes for keys and strings. Do not add comments or trailing commas. Add another object to qa_pairs for the next question; do not create a new top-level key for each question. Save the file as UTF-8 with a .json extension.

The fields in each case

Field	JSON type	What to put here
`question`	String	The exact prompt Herman should receive. Every case needs a non-empty question.
`answer`	String	A verified reference answer. Required for Technical Correctness and Completeness (TCC), even when `relevant_facts` is provided.
`relevant_facts`	Array of strings	The individual claims a correct response must include. Recommended for TCC; it uses these for the technical-correctness part of its rubric when supplied.
`type_of_question`	String	The case format. Use `short_answer` for ordinary questions, `Numerical` for numerical cases, or `MCQ` for multiple-choice cases, preserving the spelling and case.
`_meta`	Object	Source references, stable case ID, category, authoring notes, and review information. Add descriptive fields here rather than mixing them into the prompt.

These are the recommended fields for a hand-authored dataset. The importer also accepts query in place of question, expected_output in place of answer, and meta in place of _meta. Choose one naming convention throughout a file. If both versions are present, the importer prefers a non-empty query, a non-null expected_output, and an object-valued meta; conflicting copies can therefore hide an edit.

question: the input to Herman

Write the question as a buyer, processor, or engineer would ask it. Name the grade or document when the answer depends on it. Include units and operating conditions when they are part of the request. Keep each case focused on one task. A request for a hardness value and its test conditions is one task; asking for that value and an unrelated commercial recommendation should usually become two cases. Avoid putting the expected answer into the question.

answer: the reference response

Write what a correct response should say, using the supplier page or uploaded document as the source. Preserve units, ranges, test methods, and conditions. Do not fill missing product-specific values with general materials knowledge. The reference answer is authored before the run. It is different from the actual output, which is the answer Herman generates during the run. Do not paste a generated response into answer unless it has been independently verified. For a fuller material-selection answer, use a direct answer followed by these sections where the dataset’s review process requires them: KEY MATERIAL FACTS, MECHANISM REASONING, APPLICATION REQUIREMENTS, EVALUATE FOR, and CERTIFICATIONS TO VERIFY. In a JSON string, encode line breaks as \n. Keep supplier claims distinguishable from your own technical reasoning.

relevant_facts: the claims being checked

Each string should express one independently checkable claim. In the example, hardness, test method, and test temperature are separate claims so a response can be assessed on each one.

Aim for 2 to 8 meaningful facts per case. Do not pad a simple question to reach a count.
Include the qualifiers needed to make a claim correct: the grade, units, thickness, color, or test conditions, as applicable.
Do not combine several unrelated standards or properties into one string.
Keep the list complete: a response satisfying every fact should satisfy the intended technical question.

TCC still needs answer. The fact list guides its technical-correctness subscore; the reference answer also supports the broader completeness and reasoning assessment. If the list is absent or empty, technical correctness falls back to the reference answer. Changing relevant_facts changes what counts as correct. Treat that as a new dataset revision and rerun the baseline before comparing results.

type_of_question: the question format

Use short_answer for the normal reference-answer workflow. A Numerical case should state the expected quantity, units, and relevant conditions clearly. For MCQ, include the choices and a reference identifying the correct option; setting the type alone does not create options or select a scoring metric. The importer supports a choices object alongside question and appends its labeled options to the prompt.

_meta: identity, coverage, and review

Metadata makes a case traceable. These fields are authoring and review conventions; a successful upload does not prove that the sources are correct or that review has happened.

Field inside `_meta`	Type	Purpose
`internal_ref`	String	A unique, stable case ID, such as `SUPPLIER-001`. Keep it when revising the same case.
`category`	String	The primary topic, such as `Material Properties`. Match the agreed category labels exactly so filtering stays consistent.
`supplier`, `product_family`, `application_domain`	Strings	Describe coverage and let reviewers identify gaps or an unintended concentration of cases.
`persona`	String	Intended user: `design_engineer`, `procurement`, `processor`, `sustainability_lead`, or `quality`.
`difficulty`	String	`easy`, `medium`, or `hard`. This describes the question, not confidence in the source.
`status`	String	`draft`, `reviewed`, or `approved`. Start at `draft`; update it after the corresponding review.
`author`, `authoring_note`	Strings	Who wrote the case and why it is framed this way.
`source_url`	String	The primary source’s canonical URL or document identifier.
`source_ref`	String	The exact page, section, table, or row supporting the answer.
`fact_verification`	String	What was checked against the source.
`verified_on`, `recheck_after`	Date strings	The last verification date and next review date, written as `YYYY-MM-DD`.
`verified_by`	String	The person who performed the verification.
`premise_case`	Boolean	`true` when the case intentionally tests whether Herman corrects a false premise. Use JSON `true` or `false`, not quoted text.

Keep status meaningful: the importer does not automatically exclude draft cases. Select or export approved cases before using a dataset as a benchmark.

Confidence and claim provenance

For a dataset that will be maintained or audited, add the following fields. The worked example download shows them together.

Field	Type	How to use it
`confidence`	String	`high`, `medium`, or `low`, describing how well the reference claims are supported.
`confidence_note`	String	Explain uncertainty; include it whenever confidence is not `high`.
`fact_basis`	Array of strings	One entry per `relevant_facts` item: `published`, `absence`, `reasoning`, or `judgment`.
`fact_sources`	Array	One source reference per fact, in the same order. In these examples each entry is a source URL; use `null` for a reasoning claim without a supplier source.
`source_pages`	Array of objects	Every source used by the case. Each object records `url`, `ref`, and `supplies` (what the source supports).

published means the supplier or document states the fact. absence means you checked the source and the relevant information is not published there. reasoning identifies your own materials-science explanation. judgment identifies an interpretive or commercial assessment. The three arrays must stay aligned: relevant_facts[0], fact_basis[0], and fact_sources[0] describe the same claim. Add, remove, or reorder their entries together. As a review convention, use high when every graded claim is directly supported or its absence verified on one source; use medium for cases spanning sources or including technical reasoning; and use low when a graded claim lacks adequate support. Fix or remove low-confidence cases before approval.

Cases for individual uploaded documents

Add document_id to identify the uploaded file and corpus_checked_on to record when you last checked the question against the full document collection. Use 1 to 3 focused questions per document, each anchored to facts specific to it. A single source_pages entry records one source, but does not prove that another document cannot answer the question. Check the rest of the corpus explicitly, and repeat that check when documents are added. Use directly published claims for these retrieval tests.

Optional file-level evaluation settings

The simplest file only needs the qa_pairs wrapper. The current file importer also recognizes optional metrics and judge_config fields next to it. These set defaults for evaluation; they are not case metadata.

{
  "metrics": ["tcc", "answer_relevancy", "faithfulness_context", "ctr"],
  "judge_config": {
    "default": "default",
    "overrides": {
      "tcc": "strong"
    }
  },
  "qa_pairs": [
    {
      "question": "What hardness does the training datasheet report for Training Grade A?",
      "answer": "60 Shore A, measured using ASTM D2240 at 23 degrees C.",
      "relevant_facts": ["Training Grade A has a reported hardness of 60 Shore A."],
      "type_of_question": "short_answer",
      "_meta": {"internal_ref": "TRAINING-002", "status": "draft"}
    }
  ]
}

metrics is an array of recognized metric IDs. judge_config can be a configured preset name, such as "default", or an object with a default preset and metric-specific overrides. It selects the evaluator used to score responses, not the model answering the questions. Run settings can override dataset defaults. Leave these fields out if you will configure scoring when starting the run.

How the fields connect to the scores

Metric	Dataset input	What it evaluates
Technical Correctness and Completeness (`tcc`)	`question`, `answer`, and optional `relevant_facts`	The actual response against the reference answer; supplied facts guide the technical-correctness subscore.
Answer relevancy (`answer_relevancy`)	`question`	Whether the actual response addresses the question. It does not establish that the answer is factually correct.
Faithfulness with runtime context (`faithfulness_context`)	`question`, plus evidence captured during the run	Whether the response’s claims are supported by the context available to Herman.
Contextual Technical Relevance (`ctr`)	`question`, plus retrieval evidence captured during the run	Whether retrieved evidence fits the technical request.

Putting a URL in _meta documents your source; it does not supply retrieved context to the runtime metrics. Those metrics need the relevant evidence from the run. A dataset with questions but no reference answers can therefore produce some scores while being unsuitable for TCC.

Check the file before using it

Confirm it parses as JSON and contains a non-empty qa_pairs array of objects.
Give every case a non-empty question and a stable, unique internal_ref.
Include a verified, non-empty reference answer for every case intended for TCC.
Check every numeric value, unit, standard, and qualification against its source.
Keep facts atomic and keep the provenance arrays the same length and in the same order.
Remove all template placeholders, duplicate questions, and accidental answer hints.
Check category spelling, source locations, confidence notes, and review status.
Have another reviewer inspect the content. JSON parsing and upload acceptance only verify part of the job.

Use a descriptive filename such as herman_eval_supplier_20261022_v1.json. Keep the case IDs stable across revisions. Record substantive changes to questions, reference answers, or facts before using the new file for score comparisons.

Training source

This fictional source is included only to make the worked example self-contained:

Product	Property	Value	Test method	Test temperature
Training Grade A	Hardness	60 Shore A	ASTM D2240	23 degrees C

The numbers describe an invented teaching example. Replace this row and its source references with a verified supplier document when building a real dataset.

Get Started

Merchant Portal

Support

Build an evaluation dataset

Download a fill-in template

Download the worked example

Start with the file structure

The fields in each case

question: the input to Herman

answer: the reference response

relevant_facts: the claims being checked

type_of_question: the question format

_meta: identity, coverage, and review

Confidence and claim provenance

Cases for individual uploaded documents

Optional file-level evaluation settings

How the fields connect to the scores

Check the file before using it

Training source

Download a fill-in template

Download the worked example

​Start with the file structure

​The fields in each case

​question: the input to Herman

​answer: the reference response

​relevant_facts: the claims being checked

​type_of_question: the question format

​_meta: identity, coverage, and review

​Confidence and claim provenance

​Cases for individual uploaded documents

​Optional file-level evaluation settings

​How the fields connect to the scores

​Check the file before using it

​Training source

Start with the file structure

The fields in each case

question: the input to Herman

answer: the reference response

relevant_facts: the claims being checked

type_of_question: the question format

_meta: identity, coverage, and review

Confidence and claim provenance

Cases for individual uploaded documents

Optional file-level evaluation settings

How the fields connect to the scores

Check the file before using it

Training source