Skip to content

Semantics ​

Every result Veritas reports has a precise, limited meaning. This page states what each status means, under which model and bounds it holds, and what it must not be read as. The vocabularies are fixed: no other status is ever reported, and none of them is collapsed into a single quality or confidence score.

Status vocabularies ​

SubjectStatuses
Verification (claim)proven_within_model, counterexample_found, unknown, unsupported, timeout, semantic_mismatch, error
Specification (review)proposed, reviewed, accepted, rejected, conflicted, superseded
Test outcomemeaningful_with_evidence, weak_or_incomplete, tautological_or_disconnected, spec_conflict, inconclusive
Mutantkilled, survived, timeout, compile_error, runtime_error, ignored, no_coverage, pending, unknown, suspected_equivalent, equivalent_proven
Finding certaintyconfirmed, suspected, informational

What a status means ​

StatusMeaningNot to be read as
proven_within_modelThe solver found no counterexample for the stated property, within the stated model, domain bounds and assumptionsThe implementation is correct; the specification is correct
counterexample_foundThe solver produced an assignment violating the property, and re-running it in JavaScript gave the same verdictA confirmed bug in the specification's intent
semantic_mismatchThe solver's counterexample does not reproduce in JavaScript; the model and the code disagreeEither a bug or a proof
unknown, timeoutThe solver gave no answer, or the model could not be compared with the JavaScript codeSuccess
unsupportedThe code or property is outside the supported subsetSuccess
errorThe tool failedSuccess

Lean results carry kernel_checked assurance only when the proof uses no sorry, admit, native_decide or axioms beyond Lean's standard ones (ADR 0005); a proof that uses any of them is never treated as verified. SMT results carry smt_bounded assurance. Reports show the difference.

A run fails the policy (exit code 1) only through a confirmed specification violation or a missed policy threshold; suspected findings never fail it (see CLI exit codes). unknown, timeout and unsupported are never counted as success. Under policy.inconclusiveBehavior: warn they leave the exit code at 0 but keep the verdict at pass_with_warnings, never pass.

What is checked ​

Every formal claim is the same bounded query, "for all inputs within the domains, the preconditions imply the body"; its kind says which relation the body encodes. veritas prove checks conformance; the others need mutation results (veritas run --profile deep).

KindQuestioncounterexample_foundproven_within_model
conformanceDoes the implementation satisfy the property?An input where it does not, replayed in JavaScriptNo such input within the bounds
bounded_equivalenceDoes any input tell a surviving mutant from the original?A distinguishing input (a test with it and the specified result would kill the mutant)None within the bounds: the mutant is equivalent_proven
distinguishabilityCan the inputs a test already uses tell a surviving mutant from the original?One of the test's inputs does: the test reaches the difference but its assertion does not check itNone of them does: the test needs a new input
property_detectionDoes the property tell a mutant from the original?An input where the original satisfies the property and the mutant does not: the property detects the changeNo such input within the bounds: the property cannot distinguish this mutant

Distinguishability uses only calls whose arguments are all integer or boolean literals inside the domain; a test with other arguments gets no claim. In the three mutant relations the mutant exists only as a model of the mutated source: the original side is replayed in JavaScript, the mutant side is not. At most 40 distinguishability and property-detection checks run per run; the stage messages say how many were skipped. Undecided results (unknown, timeout) leave the stage partial, as for conformance.

Supported formal subset ​

  • Pure functions with no side effects, I/O, async, this, closures over mutable state, eval or Proxy.
  • Parameters and results that are booleans or integers. Integer parameters must have an explicit bounded domain inside the safe-integer range (default [-10000, 10000]).
  • Parameters that are records of integer and boolean properties, or arrays of integers or booleans with at most formal.maxArrayLength (default 3, at most 8) elements. Results stay integers or booleans.
  • Comparison, integer + - *, logical operators, if/else, ternaries and conditional return.

Records and arrays ​

Records and arrays are flattened into scalar model parameters: the field order.total becomes order__total, and an array xs becomes a length xs__length in [0, maxArrayLength] plus one parameter per element slot (xs__0, xs__1, ...). Slots at or beyond the length are not part of the JavaScript value. Counterexamples are shown, and replayed in JavaScript, in the shape of the original arguments ({"order":{"total":"1000","vip":false}}, {"xs":["7"]}).

Reading xs[k] when k >= xs.length gives undefined in JavaScript, which the model does not describe. Every element read records the condition under which it is out of bounds (taking if, ternary, && and || guards into account), and a property is only proven_within_model when the solver shows that no input in the domain satisfying the preconditions can reach such a read. Otherwise the claim is unsupported, with an input that reads beyond the length; an undecided check gives unknown. Arrays longer than maxArrayLength are not covered.

JavaScript numbers versus the integer model ​

The model uses mathematical integers. JavaScript uses IEEE-754 doubles. They agree only for safe integers when no intermediate result leaves the safe range and no division or fractional value appears. Veritas therefore:

  • restricts domains to safe integers and checks intermediate bounds;
  • treats /, % on possibly negative operands, Math.*, NaN, Infinity, -0, floats, BigInt mixing and implicit coercion as unsupported;
  • re-executes every counterexample in JavaScript and reports semantic_mismatch if the verdict differs.

Mutants ​

  • killed: at least one test failed under the mutant.
  • survived: tests ran the mutant and none failed. This is a test gap candidate, not a bug.
  • suspected_equivalent: no difference observed; equivalence not shown.
  • equivalent_proven: no input in the stated bounded domain distinguishes it (a bounded_equivalence claim); the bounds are part of the result.
  • timeout: the test run did not finish under the mutant (usually a mutated loop). Counted as detected, as in other mutation tools, but never credited to a specific test as a kill. Each test that covers a specification-relevant timeout mutant lists it under unknowns.
  • pending, unknown: the mutant was not run, or the backend reported a status Veritas does not recognize. An unknown mutant makes the mutation stage partial, so the run cannot pass; so does a pending one, except in an incremental run (--since), where mutants outside the change are left out on purpose.

Mutator runs only the test files that could kill the mutants in scope. A test file is left out only when it imports project sources and none of them is in scope. Files are always kept when they, or a local file they import, run code in a child process, load modules at run time (dynamic import(), require, createRequire) or import a relative module that does not resolve. Files in which no test was recognized are also kept. Code that reaches the project only through Vitest configuration (globalSetup, plugins) is not seen; if your tests depend on it, set the scope so that those test files import a source in scope.

Specifications ​

  • Properties start as proposed. Only an explicit approval record makes one accepted.
  • Properties inferred from code, tests or runtime observation (origin inferred or observed) are never independent evidence for themselves. With analysis.inferSpecsFromCode: false, code-inferred candidates are not proposed at all.
  • Conflicting sources produce conflicted; Veritas proposes a preferred reading with both sources but does not pick one, and never merges them into one property.
  • Specifications and expected values are never changed to match the implementation. See Specifications for the review workflow.

Test outcomes ​

Assessments are vectors (specification relevance, assertion relevance, mutation detection, boundary coverage, formal evidence, source confidence, unknowns), summarized as one of meaningful_with_evidence, weak_or_incomplete, tautological_or_disconnected, spec_conflict, inconclusive. There is no single quality score.

A test is meaningful_with_evidence only when an asserted call to a project function kills specification-relevant mutants and asserts every decisive boundary input. Mutation results are recorded per test. When any call in the test goes through a relative import that TypeScript could not resolve, or the test runs code in a child process or through a dynamic import, that code may be what detected the mutants, so the outcome is inconclusive and the reason is listed under unknowns.

Released under the Apache-2.0 License.