Skip to content

Reading results ​

This page explains the verdict, the exit code, findings, test outcomes and the statuses that appear in Veritas output. The exact definitions are in Semantics.

Verdict and exit code ​

veritas run and veritas prove print both on their first line:

text
run run-20261120T105508829Z-000-017911  verdict: policy_failure (exit 1)
VerdictExit codeMeaning
pass0Every stage succeeded, nothing is inconclusive and nothing needs review
pass_with_warnings0No policy failure, but there are findings to review or inconclusive results
policy_failure1A confirmed specification violation, or a configured threshold is not met
tool_error3A stage failed
inconclusive4Something could not be decided and policy.inconclusiveBehavior is fail

Exit code 2 means a configuration or usage error, printed as error[CODE]: message followed by a hint: line, for example:

text
error[APPROVAL_MISMATCH]: the approval token does not match plan IP-a14164357e16 (the plan changed or the token is wrong)
hint: review the plan again with `veritas improve --dry-run` and use the token it prints

Among findings, only confirmed ones fail the policy; a failed maxNewSurvivors or minPropertyCoverage threshold also does. Inconclusive results never produce pass: under the default warn policy the verdict is pass_with_warnings and each inconclusive item is listed. CI and GitHub Action shows how to configure the policy.

With --json, stdout contains only the JSON result (status, exitCode, runId, dir, reasons, stages, findings) and logs go to stderr.

Stages ​

Each block under the verdict is a stage and its status. Only succeeded is a success. failed and error are tool errors. partial, skipped, unsupported, timeout and cancelled are inconclusive.

Findings ​

text
F-89435b62ae1e  [confirmed high] src/pricing.ts#discountAmount does not satisfy "total >= 1000" for {"total":"1000"} (within the model)

Each finding has an ID, a certainty, a severity and a kind.

CertaintyMeaning
confirmedShown by a real run (a mutation run, or a counterexample replayed in JavaScript); for a specification violation, the property is accepted
suspectedNeeds your review, for example because the property is only proposed
informationalDescribes a limitation of the analysis

Severity is critical, high, medium, low or info.

KindTypical cause
spec_implementation_mismatchThe code disagrees with a documented rule
counterexampleA property-based test or the solver found a violating input
spec_violation_candidateA test built from the specification fails against the code
spec_conflictTwo sources state different rules
survived_mutantNo test notices a change in code that a rule depends on
test_gapA boundary or behaviour is not checked
tautological_testA test does not check project behaviour
unsupported_semanticsCode outside the supported formal subset
stage_failureA stage could not produce its result

A confirmed finding of one of the first three kinds is a specification violation and, with the default policy.confirmedViolation: fail, sets exit code 1.

See the evidence for a finding ​

bash
veritas explain <finding-id>

explain reads the latest run and prints the source and quoted line, the properties, the tests that call the code and their assessments, the mutants, the solver claims with their files, and the suggested next steps. The tutorial shows a full example.

Test outcomes ​

OutcomeMeaning
meaningful_with_evidenceThe test asserts the result of a project function, kills specification-relevant mutants and asserts every decisive boundary input
weak_or_incompleteThe test checks something, but relevant mutants survive it or it misses a boundary
tautological_or_disconnectedThe test does not check project behaviour (constant assertions, the target is never called)
spec_conflictThe test relates to a property whose sources conflict
inconclusiveThere is not enough evidence: the run has no mutation results, the code is reached through a dynamic import or a child process, or the test checks only types (expectTypeOf, assertType)

The outcome summarizes the evidence vector below; it is not a grade.

Evidence vector ​

Each assessment in assessments.json has these fields and no combined score:

FieldContents
specificationRelevanceProperties the test relates to, with their review status
assertionRelevancechecks_result, weak, none or unknown
mutationDetectionMutants the test killed, mutants that survived it, others
boundaryCoverageEach relevant boundary and whether the test uses it (yes, no, unknown)
formalEvidenceIDs of formal claims about the related properties
sourceConfidenceNotes on the quality of the specification sources
unknownsWhat could not be determined

veritas report prints the same information as a rationale line:

text
T-97fef42e4cf7: weak_or_incomplete; rationale boundary not exercised: total >= 1000 (decisive inputs 999, 1000; asserted: none); formal evidence C-8aad536f8e71 counterexample_found (conformance, assurance none); unknowns no mutation results for this run; detection power not measured

Verification statuses ​

StatusMeaning
proven_within_modelNo counterexample exists within the stated model, domain bounds and assumptions
counterexample_foundThe solver found a violating input, and replaying it in JavaScript gave the same result
semantic_mismatchThe solver's counterexample does not reproduce in JavaScript
unknownThe solver gave no answer
timeoutThe solver hit formal.maxSolverMs
unsupportedThe code or property is outside the supported subset
errorThe tool failed

unknown, timeout, unsupported, semantic_mismatch and error mean the question was not answered, so Veritas never counts them as success. They make the run inconclusive, and policy.inconclusiveBehavior decides whether that fails CI. Verification explains each engine and the assurance levels smt_bounded and kernel_checked.

Mutant statuses ​

StatusMeaning
killedAt least one test failed under the mutant
survivedTests ran the mutant and none failed: a test gap candidate, not a bug and not equivalence
timeoutThe tests did not finish under the mutant; counted as detected but credited to no single test
suspected_equivalentNo difference was observed; equivalence is not shown
equivalent_provenNo input in the stated bounded domain distinguishes it from the original
compile_error, runtime_error, ignored, no_coverageReported by the mutation backend
pending, unknownNot run, or not recognized; the mutation stage becomes partial

A surviving mutant becomes equivalent_proven only through a bounded equivalence proof.

bash
veritas report                          # text summary of the latest run
veritas report --format markdown
veritas report --format html > report.html
veritas report --format sarif > veritas.sarif
veritas report --format json
veritas report --run <run-id>           # a specific run

veritas run and veritas prove also write the report files listed in reports.formats (by default report.json, report.md, report.html and report.sarif) to the run directory; for runs of other commands, veritas report renders them on request. Reports and artifacts describes the sections and files.

Released under the Apache-2.0 License.