In progress

The whole sign-off, judged print by print.

When Vivo's B2B billing changes, someone has to prove the change worked: today that gets checked by hand, line by line, across three documents. This POC reads the entire sign-off package and judges every step in the test script, with the print as proof.

114steps judged per round
253prints read
4billing scenarios

The test script is the ruler. Every change generates a package: the Story (the request and its acceptance criteria), the CDT (the test script, the ruler) and the ECT (the evidence: the prints of what QA ran). Signing off means proving the evidence meets the script.

Why it hurts. Manual, line-by-line comparison across documents, step by step, print by print; rework every round; slow feedback back to QA.

The POC's real panel: each test project with its conforming steps, blockers and verdict. Internal identifiers blurred in the image.
The executive view: what is signed off, what is blocked, and why.
The executive view: what is signed off, what is blocked, and why.
How it judges

Nothing is approved in bulk. Every step in the script gets its own status, and a failure becomes a named callout back to QA:

OKthe evidence covers the step
OK IN SETthe proof is in the case's full set of prints
INSUFFICIENT EVIDENCEthe print does not prove what the step requires
CONTRADICTORY EVIDENCEthe print proves the opposite
NO EVIDENCEno print covers the step

Step-by-step judgment, with one hard rule: a field without proof does not count as conforming.

An independent audit redoes every number on this screen before publishing.
An independent audit redoes every number on this screen before publishing.
The reliability screen: an independent judge, without seeing the automation's result, judges the same steps, and the panel publishes the agreement. The number is re-measured every round.
Under the hood
The sign-off package comes indeterministic extraction in code (no model)the judgments run on Telaand from here down, it's the judgment flow:
Extraction into fields and joinsstory, script and prints matched without a model,down to print-to-caption by geometryjudge each print (vision)consolidate prints per stepjudge the case as a setscenario opinioncoverage of what the project asked forFeedback by recipientQA, the script and the project each get something differentSign-off panel
474 print-to-step judgments per round761 automated tests in the panelblind judgment: the judge never sees the automation's result
Tela PlatformPythonNuxtLLMs
Reliability

Trust here is not an adjective, it is a measurement: three independent checks audit the pipeline every round.

Blind judgesone per scenario, they judge without seeing the automation's result
Technical metersre-derive every count without reusing the pipeline's own code
Adversarial checkeropens the print down to the pixel, hunting for the error

And one rule that runs through the whole project: agreement never becomes accuracy. Two evaluators can be wrong together, so the number published is always the most conservative one.

← All projectsNext project →