> ## Documentation Index > Fetch the complete documentation index at: https://docs.plastics.com/llms.txt > Use this file to discover all available pages before exploring further. # Build an evaluation dataset > Walk through an evaluation JSON file, understand each field, and prepare questions, reference answers, facts, and source metadata. An evaluation dataset is a JSON file containing the questions you want Herman to answer and the reference material used to judge those answers. Each entry is one test case. Use the same reviewed dataset when comparing runs so a score change reflects the system being tested. Start with a small, focused set from one supplier or document collection. Write specific questions, verify the reference answers against the original sources, and record where each expected fact came from. > **[Download a fill-in template](https://docs.plastics.com/examples/evaluation-dataset-template.json)** > > Copy the case object for each question and replace every REPLACE\_WITH value. > **[Download the worked example](https://docs.plastics.com/examples/evaluation-dataset-example.json)** > > A complete example using fictional training data, with provenance and review metadata. > [!NOTE] > > The template is valid JSON with unfinished content. The worked example teaches the format; it is not an approved benchmark or a real product datasheet. Keep new cases marked `draft` until their facts and answers have been reviewed. ## Start with the file structure The outer object contains `qa_pairs`, an array of case objects. One object inside that array becomes one question in an evaluation run. ```json theme={null} { "qa_pairs": [ { "question": "What hardness does the training datasheet report for Training Grade A, and under which test conditions?", "answer": "The training datasheet reports a hardness of 60 Shore A, measured using ASTM D2240 at 23 degrees C.", "relevant_facts": [ "Training Grade A has a reported hardness of 60 Shore A.", "The hardness test method is ASTM D2240.", "The hardness test temperature is 23 degrees C." ], "type_of_question": "short_answer", "_meta": { "internal_ref": "TRAINING-001", "category": "Material Properties", "difficulty": "easy", "source_url": "https://docs.plastics.com/evaluations/dataset-json#training-source", "source_ref": "Training source: hardness row", "author": "Documentation example", "status": "draft" } } ] } ``` Use double quotes for keys and strings. Do not add comments or trailing commas. Add another object to `qa_pairs` for the next question; do not create a new top-level key for each question. Save the file as UTF-8 with a `.json` extension. ## The fields in each case | Field | JSON type | What to put here | | - | - | - | | `question` | String | The exact prompt Herman should receive. Every case needs a non-empty question. | | `answer` | String | A verified reference answer. Required for Technical Correctness and Completeness (TCC), even when `relevant_facts` is provided. | | `relevant_facts` | Array of strings | The individual claims a correct response must include. Recommended for TCC; it uses these for the technical-correctness part of its rubric when supplied. | | `type_of_question` | String | The case format. Use `short_answer` for ordinary questions, `Numerical` for numerical cases, or `MCQ` for multiple-choice cases, preserving the spelling and case. | | `_meta` | Object | Source references, stable case ID, category, authoring notes, and review information. Add descriptive fields here rather than mixing them into the prompt. | These are the recommended fields for a hand-authored dataset. The importer also accepts `query` in place of `question`, `expected_output` in place of `answer`, and `meta` in place of `_meta`. Choose one naming convention throughout a file. If both versions are present, the importer prefers a non-empty `query`, a non-null `expected_output`, and an object-valued `meta`; conflicting copies can therefore hide an edit. ### question: the input to Herman Write the question as a buyer, processor, or engineer would ask it. Name the grade or document when the answer depends on it. Include units and operating conditions when they are part of the request. Keep each case focused on one task. A request for a hardness value and its test conditions is one task; asking for that value and an unrelated commercial recommendation should usually become two cases. Avoid putting the expected answer into the question. ### answer: the reference response Write what a correct response should say, using the supplier page or uploaded document as the source. Preserve units, ranges, test methods, and conditions. Do not fill missing product-specific values with general materials knowledge. The reference answer is authored before the run. It is different from the **actual output**, which is the answer Herman generates during the run. Do not paste a generated response into `answer` unless it has been independently verified. For a fuller material-selection answer, use a direct answer followed by these sections where the dataset's review process requires them: `KEY MATERIAL FACTS`, `MECHANISM REASONING`, `APPLICATION REQUIREMENTS`, `EVALUATE FOR`, and `CERTIFICATIONS TO VERIFY`. In a JSON string, encode line breaks as `\n`. Keep supplier claims distinguishable from your own technical reasoning. ### relevant\_facts: the claims being checked Each string should express one independently checkable claim. In the example, hardness, test method, and test temperature are separate claims so a response can be assessed on each one. * Aim for 2 to 8 meaningful facts per case. Do not pad a simple question to reach a count. * Include the qualifiers needed to make a claim correct: the grade, units, thickness, color, or test conditions, as applicable. * Do not combine several unrelated standards or properties into one string. * Keep the list complete: a response satisfying every fact should satisfy the intended technical question. TCC still needs `answer`. The fact list guides its technical-correctness subscore; the reference answer also supports the broader completeness and reasoning assessment. If the list is absent or empty, technical correctness falls back to the reference answer. Changing `relevant_facts` changes what counts as correct. Treat that as a new dataset revision and rerun the baseline before comparing results. ### type\_of\_question: the question format Use `short_answer` for the normal reference-answer workflow. A `Numerical` case should state the expected quantity, units, and relevant conditions clearly. For `MCQ`, include the choices and a reference identifying the correct option; setting the type alone does not create options or select a scoring metric. The importer supports a `choices` object alongside `question` and appends its labeled options to the prompt. ## \_meta: identity, coverage, and review Metadata makes a case traceable. These fields are authoring and review conventions; a successful upload does not prove that the sources are correct or that review has happened. | Field inside `_meta` | Type | Purpose | | - | - | - | | `internal_ref` | String | A unique, stable case ID, such as `SUPPLIER-001`. Keep it when revising the same case. | | `category` | String | The primary topic, such as `Material Properties`. Match the agreed category labels exactly so filtering stays consistent. | | `supplier`, `product_family`, `application_domain` | Strings | Describe coverage and let reviewers identify gaps or an unintended concentration of cases. | | `persona` | String | Intended user: `design_engineer`, `procurement`, `processor`, `sustainability_lead`, or `quality`. | | `difficulty` | String | `easy`, `medium`, or `hard`. This describes the question, not confidence in the source. | | `status` | String | `draft`, `reviewed`, or `approved`. Start at `draft`; update it after the corresponding review. | | `author`, `authoring_note` | Strings | Who wrote the case and why it is framed this way. | | `source_url` | String | The primary source's canonical URL or document identifier. | | `source_ref` | String | The exact page, section, table, or row supporting the answer. | | `fact_verification` | String | What was checked against the source. | | `verified_on`, `recheck_after` | Date strings | The last verification date and next review date, written as `YYYY-MM-DD`. | | `verified_by` | String | The person who performed the verification. | | `premise_case` | Boolean | `true` when the case intentionally tests whether Herman corrects a false premise. Use JSON `true` or `false`, not quoted text. | Keep `status` meaningful: the importer does not automatically exclude draft cases. Select or export approved cases before using a dataset as a benchmark. ### Confidence and claim provenance For a dataset that will be maintained or audited, add the following fields. The worked example download shows them together. | Field | Type | How to use it | | - | - | - | | `confidence` | String | `high`, `medium`, or `low`, describing how well the reference claims are supported. | | `confidence_note` | String | Explain uncertainty; include it whenever confidence is not `high`. | | `fact_basis` | Array of strings | One entry per `relevant_facts` item: `published`, `absence`, `reasoning`, or `judgment`. | | `fact_sources` | Array | One source reference per fact, in the same order. In these examples each entry is a source URL; use `null` for a reasoning claim without a supplier source. | | `source_pages` | Array of objects | Every source used by the case. Each object records `url`, `ref`, and `supplies` (what the source supports). | `published` means the supplier or document states the fact. `absence` means you checked the source and the relevant information is not published there. `reasoning` identifies your own materials-science explanation. `judgment` identifies an interpretive or commercial assessment. The three arrays must stay aligned: `relevant_facts[0]`, `fact_basis[0]`, and `fact_sources[0]` describe the same claim. Add, remove, or reorder their entries together. As a review convention, use `high` when every graded claim is directly supported or its absence verified on one source; use `medium` for cases spanning sources or including technical reasoning; and use `low` when a graded claim lacks adequate support. Fix or remove low-confidence cases before approval. ### Cases for individual uploaded documents Add `document_id` to identify the uploaded file and `corpus_checked_on` to record when you last checked the question against the full document collection. Use 1 to 3 focused questions per document, each anchored to facts specific to it. A single `source_pages` entry records one source, but does not prove that another document cannot answer the question. Check the rest of the corpus explicitly, and repeat that check when documents are added. Use directly published claims for these retrieval tests. ## Optional file-level evaluation settings The simplest file only needs the `qa_pairs` wrapper. The current file importer also recognizes optional `metrics` and `judge_config` fields next to it. These set defaults for evaluation; they are not case metadata. ```json theme={null} { "metrics": ["tcc", "answer_relevancy", "faithfulness_context", "ctr"], "judge_config": { "default": "default", "overrides": { "tcc": "strong" } }, "qa_pairs": [ { "question": "What hardness does the training datasheet report for Training Grade A?", "answer": "60 Shore A, measured using ASTM D2240 at 23 degrees C.", "relevant_facts": ["Training Grade A has a reported hardness of 60 Shore A."], "type_of_question": "short_answer", "_meta": {"internal_ref": "TRAINING-002", "status": "draft"} } ] } ``` `metrics` is an array of recognized metric IDs. `judge_config` can be a configured preset name, such as `"default"`, or an object with a default preset and metric-specific overrides. It selects the evaluator used to score responses, not the model answering the questions. Run settings can override dataset defaults. Leave these fields out if you will configure scoring when starting the run. ## How the fields connect to the scores | Metric | Dataset input | What it evaluates | | - | - | - | | Technical Correctness and Completeness (`tcc`) | `question`, `answer`, and optional `relevant_facts` | The actual response against the reference answer; supplied facts guide the technical-correctness subscore. | | Answer relevancy (`answer_relevancy`) | `question` | Whether the actual response addresses the question. It does not establish that the answer is factually correct. | | Faithfulness with runtime context (`faithfulness_context`) | `question`, plus evidence captured during the run | Whether the response's claims are supported by the context available to Herman. | | Contextual Technical Relevance (`ctr`) | `question`, plus retrieval evidence captured during the run | Whether retrieved evidence fits the technical request. | Putting a URL in `_meta` documents your source; it does not supply retrieved context to the runtime metrics. Those metrics need the relevant evidence from the run. A dataset with questions but no reference answers can therefore produce some scores while being unsuitable for TCC. ## Check the file before using it 1. Confirm it parses as JSON and contains a non-empty `qa_pairs` array of objects. 2. Give every case a non-empty question and a stable, unique `internal_ref`. 3. Include a verified, non-empty reference answer for every case intended for TCC. 4. Check every numeric value, unit, standard, and qualification against its source. 5. Keep facts atomic and keep the provenance arrays the same length and in the same order. 6. Remove all template placeholders, duplicate questions, and accidental answer hints. 7. Check category spelling, source locations, confidence notes, and review status. 8. Have another reviewer inspect the content. JSON parsing and upload acceptance only verify part of the job. Use a descriptive filename such as `herman_eval_supplier_20261022_v1.json`. Keep the case IDs stable across revisions. Record substantive changes to questions, reference answers, or facts before using the new file for score comparisons. ## Training source This fictional source is included only to make the worked example self-contained: | Product | Property | Value | Test method | Test temperature | | - | - | - | - | - | | Training Grade A | Hardness | 60 Shore A | ASTM D2240 | 23 degrees C | The numbers describe an invented teaching example. Replace this row and its source references with a verified supplier document when building a real dataset. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.