Lessons learned overview

A lesson is a short piece of guidance for one step of Herman’s answer pipeline. It records a correction a reviewer verified in an evaluation, in a form Herman can apply to future answers that match the same situation. Each lesson has two parts:

• Guidance: what Herman should do, written as an instruction. For example: “When a comparison uses estimated values, label them as estimates and do not rank grades on them.”
• When it applies: a rule over the inputs that step can see. For example: apply when estimated values were used, except when the comparison fell back to a simpler method.

Lessons change how Herman reasons and presents an answer. They never hold product data. Grade properties, certifications, prices, inventory, and lead times always come from your catalog and other systems of record.

​

Where lessons come from

Every lesson starts from a verified correction. A reviewer looks at one of Herman’s saved answers, writes what the answer should have said, and cites the source document, revision, and location that supports the correction. A second person independently checks that correction against the source. Only an approved review case can produce lessons. This requirement keeps lessons traceable. Each lesson links back to the evaluation it came from, so anyone reviewing it can see the original question, the correction, and the source.

​

The lifecycle of a lesson

​

Statuses

Lessons are versioned. Editing a lesson creates a new version with its own status and history. Earlier versions keep their decisions and test results.

​

The Lessons Learned page

Open Lessons Learned from the side navigation. The page lists every lesson version in your workspace.

• Tabs filter by status and show how many lessons each status holds.
• Search matches the guidance text.
• Each row shows the guidance, the step, the kind of lesson, its lifecycle, the version, and the date.

Select a lesson to open its record:

​

Testing

​

Rule tests

Rule tests run in seconds and cost nothing. They check that:

• the lesson’s step is still registered and the lesson matches the step’s current input contract
• every example gives the outcome it claims: Applies, Does not apply, or Unknown

Select Run tests in the record. A passing result reads Ready for release, not released. If the step’s contract changes later, the same lesson fails with “Lesson uses a stale step contract”; edit it as a new version.

​

Quality gates

A quality gate measures whether the lesson helps. It answers every case in one or more approved evaluation datasets twice, once with the lesson and once without, and scores both answers with the same judge used in Evaluations. You choose the datasets, the number of trials, and the margin. The setup shows how many answers the run generates and its cost cap before you start. A gate run needs passing rule tests for the same version. Each gate ends with one outcome: Proven lift and non-inferior count as passing. Open a gate record with View in Evaluations to compare both answers case by case. A case that got worse can be added as a counterexample to a new version of the lesson. Quality gates appear only in workspaces with benchmark runs turned on and at least one approved evaluation dataset.

​

What a lesson must not contain

Approvers confirm that a lesson contains none of the following. Herman must get these from systems of record, never from a lesson:

• grade properties or test values
• certification or compliance facts
• prices
• inventory or availability
• lead times

If a correction was about a wrong value, fix the data source instead. Write a lesson only when the problem was in how Herman used or presented the information.