1 API

The following is the list of modules provided by this library. The stylesheet and the script that present this chapter in a browser are written beside it, as dokgen.css and dokgen.js; the manual says how a build points at them.


1.1 (veritas all)

1.1.1 Overview

  • Given — An undocumented macro.
  • Then — An undocumented macro.
  • When — An undocumented macro.
  • and-step — An undocumented macro.
  • assert-match — An undocumented macro.
  • but-step — An undocumented macro.
  • convenit — An undocumented macro.
  • dato — An undocumented macro.
  • etiam — An undocumented macro.
  • evaluate-thunks-concurrently — Run thunks concurrently in fibers and collect what they return.
  • fabula — An undocumented macro.
  • feature — An undocumented macro.
  • given — An undocumented macro.
  • lazy — An undocumented macro.
  • lazy-value? — An undocumented macro.
  • quando — An undocumented macro.
  • scaena — An undocumented macro.
  • scenario — An undocumented macro.
  • sed — An undocumented macro.
  • story — An undocumented macro.
  • suite — An undocumented macro.
  • test — An undocumented macro.
  • then — An undocumented macro.
  • tunc — An undocumented macro.
  • when-step — An undocumented macro.
  • veritas-coverage-data-path — A parameter object.
  • veritas-current-file — A parameter object.
  • veritas-junit-data-path — A parameter object.
  • veritas-last-failed-path — A parameter object.
  • veritas-only-failed — A parameter object.
  • veritas-result-counts — A parameter object.
  • veritas-result-counts-data-path — A parameter object.
  • veritas-retries — A parameter object.
  • veritas-trace-frames — A parameter object.
  • veritas-tracing? — A parameter object.
  • %and-step — Define an And (continuation) step named name.
  • %but-step — Define a But (continuation) step named name.
  • %given — Define a Given (setup) step named name.
  • %scenario — Define a Given/When/Then scenario named name; the procedure behind the macro.
  • %step — Define a step named name, labeled with label; the procedure behind the step macros.
  • %suite — Define a suite named name; the procedure behind the suite macro.
  • %test — Define a test named name; the procedure behind the test macro.
  • %then — Define a Then (outcome) step named name.
  • %when-step — Define a When (action) step named name.
  • aequalis — Assert that got is equal? to expect.
  • aequivalens — Assert that got is eqv? to expect.
  • assert — Return an assertion comparing got with expect using compare,
  • assert-approximate — Assert that got is within epsilon of expect.
  • assert-call-count — Assert that spy was called exactly expect times.
  • assert-called — Assert that spy was called at least once.
  • assert-called-with — Assert that spy was called with exactly args.
  • assert-contains — Assert that the container expect contains got.
  • assert-empty — Assert that got is an empty list, vector or string.
  • assert-eq — Assert that got is eq? to expect.
  • assert-equal — Assert that got is equal? to expect.
  • assert-eqv — Assert that got is eqv? to expect.
  • assert-error — Return an assertion that thunk raises.
  • assert-false — Assert that x is #f.
  • assert-in-range — Assert that got is within the inclusive range min–max.
  • assert-length — Assert that got has expect elements (list, vector or string length).
  • assert-matches — Assert that the string got matches pattern.
  • assert-no-error — Return an assertion that thunk does not raise.
  • assert-not-approximate — Assert that got differs from expect by at least epsilon.
  • assert-not-contains — Assert that the container expect does not contain got.
  • assert-not-empty — Assert that got is a non-empty list, vector or string.
  • assert-not-eq — Assert that got is not eq? to expect.
  • assert-not-equal — Assert that got is not equal? to expect.
  • assert-not-eqv — Assert that got is not eqv? to expect.
  • assert-not-matches — Assert that string got does not match regular expression pattern.
  • assert-one-of — Assert that got is equal? to one of the elements of list expect.
  • assert-raises — Assert that calling thunk raises an error.
  • assert-snapshot — Assert that got matches the snapshot recorded under name.
  • assert-true — Assert that x is #t.
  • benchmark — Define a benchmark named name that times thunk over runs runs.
  • call-with-timeout — Run thunk with an optional deadline of seconds seconds.
  • cases->table — Index cases by key, giving an alist of key to case.
  • collect-assertion-successful — Return the outcome of every assertion in xs as a flat list.
  • congruit — Assert that the string got matches pattern.
  • continet — Assert that the container expect contains got.
  • current-seed — Return the seed this run shuffles with, choosing one when none was set.
  • defer-body-forms — Wrap each of forms in a thunk, so a body is evaluated when its task runs.
  • do-if-string-contained — Run result when name matches restriction.
  • errat — Return an assertion that thunk raises.
  • failed-test-names — Return the names that failed in all-results, for a last-failed rerun.
  • failed-test-wanted? — Return #t when name should run in a last-failed rerun.
  • falsus — Assert that x is #f.
  • fisher-yates-shuffle — Return the elements of xs in random order (Fisher-Yates).
  • fresh-seed — Return a new seed for a shuffle, drawn from the platform’s entropy source.
  • generate-beer — Return a generator of beer styles.
  • generate-boolean — Return a generator of #t and #f.
  • generate-car — Return a generator of car makes and models.
  • generate-char — Return a generator of printable ASCII characters, 32 to 126.
  • generate-constant — Return a generator that always answers value.
  • generate-email — Return a generator of first.last@example.com email addresses.
  • generate-first-name — Return a generator of first names.
  • generate-integer — Return a generator of integers between 0 and max-bound (exclusive).
  • generate-last-name — Return a generator of last names.
  • generate-list — Return a generator of lists whose elements come from elem-generator.
  • generate-name — Return a generator of full names, a first name and a last name.
  • generate-natural — Return a generator of integers between 0 and max-bound inclusive.
  • generate-one-of — Return a generator that picks one of choices at random.
  • generate-real — Return a generator of real numbers in the range min to max.
  • generate-string — Return a generator of printable-ASCII strings.
  • generate-symbol — Return a generator of lowercase symbols, min-len to max-len long.
  • generate-tuple — Return a generator of lists holding one value from each of generators.
  • generate-vector — Return a generator of vectors whose elements come from elem-generator.
  • generator-shrink — Return smaller replacements for value taken from generator.
  • idem — Assert that got is eq? to expect.
  • inanis — Assert that got is an empty list, vector or string.
  • intra — Assert that got is within the inclusive range min–max.
  • is-approximate? — Return #t when a and b differ by less than epsilon.
  • longitudo — Assert that got has expect elements (list, vector or string length).
  • make-assertion — Return assertion, a procedure of one validation client, as a case to run.
  • make-spy — Return a procedure that calls proc and records every call.
  • make-stub — Return a stand-in procedure that ignores its arguments and returns returns.
  • merge-failed-names — Combine what several runs recorded as failing into one record.
  • non-aequalis — Assert that got is not equal? to expect.
  • non-aequivalens — Assert that got is not eqv? to expect.
  • non-congruit — Assert that string got does not match regular expression pattern.
  • non-continet — Assert that the container expect does not contain got.
  • non-errat — Return an assertion that thunk does not raise.
  • non-idem — Assert that got is not eq? to expect.
  • non-inanis — Assert that got is a non-empty list, vector or string.
  • non-propinquus — Assert that got differs from expect by at least epsilon.
  • proba — Return an assertion comparing got with expect using compare,
  • property — Return a property named name, a task like a test.
  • propinquus — Assert that got is within epsilon of expect.
  • propinquus? — Return #t when a and b differ by less than epsilon.
  • read-last-failed — Return the test names recorded as failing at path, or #f.
  • read-result-counts — Return the counts recorded at path, or #f.
  • read-trace — Return the frames stored at path, oldest first.
  • render-trace — Print the trace at path as a numbered timeline.
  • seeded-shuffle — Return xs in the order this run’s seed prescribes.
  • show-last-trace — Print the trace of the last run, and return whether there was one.
  • snapshot-path — Return the file holding the snapshot called name.
  • snapshot-update? — Return #t when snapshots are to be rewritten instead of compared.
  • split-keyword-arguments — Split args, the syntax of an argument list, into its parts.
  • spy-call-count — Return how many times spy was called.
  • spy-called-with? — Return #t when spy was called with exactly args.
  • spy-calls — Return the argument lists spy was called with, oldest first.
  • spy-results — Return what spy returned, oldest first, one entry per call.
  • spy? — Return #t when value was made by make-spy or make-stub.
  • suite-each — Like test-each, but case->suite returns a suite.
  • suscitat — Assert that calling thunk raises an error.
  • test-each — Expand cases into one test per case, under a suite named group-name.
  • trace-event! — Record one step of the run when tracing is on.
  • trace-file — Return where the last trace of this run is written.
  • trace-reset! — Forget the frames recorded so far.
  • unus-ex — Assert that got is equal? to one of the elements of list expect.
  • veritas-run — Run tasks, the top-level tests, suites and benchmarks.
  • verus — Assert that x is #t.
  • with-temporary-binding — Call thunk with module’s var set to value, and restore it.
  • write-last-failed! — Record the names that failed, so the next run can rerun just those.
  • write-last-failed-names! — Record names at path as the tests a rerun should select.
  • write-result-counts — Write counts to path so another process can add them up.
  • write-trace! — Write the frames recorded so far to path and return it.
  • veritas-rerun-all-marker — A variable.

1.1.2 Macros

Macro: Given stx

Undocumented macro.

Macro: Then stx

Undocumented macro.

Macro: When stx

Undocumented macro.

Macro: and-step stx

Undocumented macro.

Macro: assert-match x

Undocumented macro.

Macro: but-step stx

Undocumented macro.

Macro: convenit x

Undocumented macro.

Macro: dato stx

Undocumented macro.

Macro: etiam stx

Undocumented macro.

Macro: evaluate-thunks-concurrently x

Run thunks concurrently in fibers and collect what they return.

Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.

(evaluate-thunks-concurrently
 (list (lambda () (sleep 1) 'delayed)
       (lambda () 'instant)))
⇒ (delayed instant)
Macro: fabula stx

Undocumented macro.

Macro: feature stx

Undocumented macro.

Macro: given stx

Undocumented macro.

Macro: lazy x

Undocumented macro.

Macro: lazy-value? x

Undocumented macro.

Macro: quando stx

Undocumented macro.

Macro: scaena stx

Undocumented macro.

Macro: scenario stx

Undocumented macro.

Macro: sed stx

Undocumented macro.

Macro: story stx

Undocumented macro.

Macro: suite stx

Undocumented macro.

Macro: test stx

Undocumented macro.

Macro: then stx

Undocumented macro.

Macro: tunc stx

Undocumented macro.

Macro: when-step stx

Undocumented macro.

1.1.3 Parameters

Parameter: veritas-coverage-data-path

Default value:

#f
Parameter: veritas-current-file

Default value:

#f
Parameter: veritas-junit-data-path

Default value:

#f
Parameter: veritas-last-failed-path

Default value:

"reports/last-failed.scm"
Parameter: veritas-only-failed

Default value:

#f
Parameter: veritas-result-counts

Default value:

((files . 0) (suites . 0) (tests . 0) (assertions . 0) (flaky . 0))
Parameter: veritas-result-counts-data-path

Default value:

#f
Parameter: veritas-retries

Default value:

0
Parameter: veritas-trace-frames

Default value:

()
Parameter: veritas-tracing?

Default value:

#f

1.1.4 Procedures

Procedure: %and-step name . cases

Define an And (continuation) step named name.

Procedure: %but-step name . cases

Define a But (continuation) step named name.

Procedure: %given name . cases

Define a Given (setup) step named name.

Procedure: %scenario name KEY: #:skip? #:debug? #:shuffle? #:concurrent? #:timeout #:tags #:before-all #:after-all . steps

Define a Given/When/Then scenario named name; the procedure behind the macro.

A scenario is a suite, so its steps run in declaration order. Keywords are forwarded to suite, and steps come from given, when-step, then, and-step and but-step.

timeout, when given, caps the whole scenario in seconds, exactly like suite; a step can carry its own #:timeout as well, since steps are ordinary tests.

(scenario "emptying the cart"
  (given "a cart with one item"
    (assert-equal #:expect 1 #:got (cart-size cart)))
  (when-step "the cart is emptied"
    (empty-cart! cart))
  (then "the cart is empty"
    (assert-true (cart-empty? cart))))
Procedure: %step label name . cases

Define a step named name, labeled with label; the procedure behind the step macros.

A step is an ordinary test, so it takes the same keywords and cases.

Procedure: %suite name KEY: #:debug? #:shuffle? #:skip? #:timeout #:tags #:concurrent? #:before-all #:after-all #:source . tests

Define a suite named name; the procedure behind the suite macro.

The arguments after name are the children. Each form is evaluated when the suite runs, and every test, suite or benchmark built while it is evaluated is a child: one already built, a list of them, or a thunk of them are all understood. Children run concurrently unless concurrent? is #f, and in shuffled order unless shuffle? is #f. before-all and after-all run once around them, timeout caps the whole suite in seconds including those hooks, skip? drops it from the run and tags decide which filter may select it.

Procedure: %test name KEY: #:debug? #:shuffle? #:skip? #:timeout #:tags #:before #:after #:source #:retries . cases

Define a test named name; the procedure behind the test macro.

The arguments after name are the body. Each form is evaluated when the test runs, and every assertion built while it is evaluated is a case: an assertion, a list of them, a thunk of them, or a helper procedure that builds them are all understood. before and after run around the body, timeout caps the whole test in seconds, hooks included, skip? drops it from the run and tags decide which filter may select it.

A body is read in order, because a body is a sequence: one form prepares what the next one checks. shuffle? asks for the order of the body forms to be randomized anyway, which is a way to find checks that depend on each other by accident.

retries reruns a failing test, hooks and body together, up to that many extra attempts. A test that fails and then passes is reported as flaky with its attempt count, so an intermittent failure is visible instead of hidden.

(test "Addition Validation"
      #:before (lambda (vc) (display "Initializing..."))
      (assert-equal #:expect 4 #:got (+ 2 2)))
Procedure: %then name . cases

Define a Then (outcome) step named name.

Procedure: %when-step name . cases

Define a When (action) step named name.

Called when-step rather than when so it does not shadow Guile’s when syntax.

Procedure: aequalis KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to expect.

The usual choice for lists, strings, vectors and other compound values.

Procedure: aequivalens KEY: #:expect #:got #:name #:timeout

Assert that got is eqv? to expect.

Good for numbers, characters and symbols, where value equivalence is what matters.

Procedure: assert KEY: #:expect #:got #:compare #:name #:timeout

Return an assertion comparing got with expect using compare, a two-argument predicate such as equal?.

The assert-* wrappers below are built on this one. got and expect are usually plain values, but either may be wrapped in lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1))) evaluates the expression when the assertion runs, inside its timeout, which is what makes an expensive comparison something a deadline can catch. A lazy value keeps the lexical scope of the code that wrote it and is evaluated once per assertion that receives it.

name labels the assertion in the report. timeout caps it in seconds; #f, the default, means no deadline.

(assert #:expect 3
        #:got (+ 1 1 1)
        #:compare equal?
        #:name "Ensuring addition works")
Procedure: assert-approximate KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got is within epsilon of expect.

The default epsilon is 1.0e-8.

Procedure: assert-call-count spy KEY: #:expect #:name

Assert that spy was called exactly expect times.

Checked when the assertion runs, so it is written as a case lambda: (lambda (vc) ((assert-call-count spy #:expect 2) vc)).

Procedure: assert-called spy KEY: #:name

Assert that spy was called at least once.

Checked when the assertion runs, so it is written as a case lambda: (lambda (vc) ((assert-called spy) vc)).

Procedure: assert-called-with spy args KEY: #:name

Assert that spy was called with exactly args.

Checked when the assertion runs, so it is written as a case lambda: (lambda (vc) ((assert-called-with spy '(1 2)) vc)).

Procedure: assert-contains KEY: #:expect #:got #:name #:timeout

Assert that the container expect contains got.

Works for strings (substring), lists and vectors (membership) and hash tables (key membership).

(assert-contains #:expect '(1 2 3) #:got 2)
(assert-contains #:expect "hello world" #:got "world")
Procedure: assert-empty KEY: #:got #:name #:timeout

Assert that got is an empty list, vector or string.

Procedure: assert-eq KEY: #:expect #:got #:name #:timeout

Assert that got is eq? to expect.

The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.

Procedure: assert-equal KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to expect.

The usual choice for lists, strings, vectors and other compound values.

Procedure: assert-eqv KEY: #:expect #:got #:name #:timeout

Assert that got is eqv? to expect.

Good for numbers, characters and symbols, where value equivalence is what matters.

Procedure: assert-error thunk KEY: #:name #:timeout

Return an assertion that thunk raises.

It fails if thunk returns normally. name labels it in the report and timeout caps it in seconds, as in assert.

(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
              #:name "Should fail when list contains non-numbers")
Procedure: assert-false x KEY: #:name #:timeout

Assert that x is #f.

Procedure: assert-in-range KEY: #:min #:max #:got #:name #:timeout

Assert that got is within the inclusive range min–max.

Procedure: assert-length KEY: #:expect #:got #:name #:timeout

Assert that got has expect elements (list, vector or string length).

Procedure: assert-matches KEY: #:pattern #:got #:name #:timeout

Assert that the string got matches pattern.

pattern is a string or a compiled regexp.

(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Procedure: assert-no-error thunk KEY: #:name #:timeout

Return an assertion that thunk does not raise.

It fails if thunk raises. name labels it in the report and timeout caps it in seconds, as in assert.

Procedure: assert-not-approximate KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got differs from expect by at least epsilon.

Inverse of assert-approximate.

Procedure: assert-not-contains KEY: #:expect #:got #:name #:timeout

Assert that the container expect does not contain got.

Inverse of assert-contains.

Procedure: assert-not-empty KEY: #:got #:name #:timeout

Assert that got is a non-empty list, vector or string.

Procedure: assert-not-eq KEY: #:expect #:got #:name #:timeout

Assert that got is not eq? to expect.

Inverse of assert-eq.

Procedure: assert-not-equal KEY: #:expect #:got #:name #:timeout

Assert that got is not equal? to expect.

Inverse of assert-equal.

Procedure: assert-not-eqv KEY: #:expect #:got #:name #:timeout

Assert that got is not eqv? to expect.

Inverse of assert-eqv.

Procedure: assert-not-matches KEY: #:pattern #:got #:name #:timeout

Assert that string got does not match regular expression pattern.

Inverse of assert-matches.

Procedure: assert-one-of KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to one of the elements of list expect.

Procedure: assert-raises thunk KEY: #:on #:predicate #:name #:timeout

Assert that calling thunk raises an error.

By default any error satisfies the assertion. Pass on to require a specific error key (for example 'wrong-type-arg), or predicate, a procedure of (key args) returning #t for the expected error.

(assert-raises (lambda () (/ 1 0)))
(assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Procedure: assert-snapshot KEY: #:got #:name #:dir

Assert that got matches the snapshot recorded under name.

name labels the assertion and names the file, so it is required; the file lives under VERITAS_SNAPSHOT_DIR (default test/snapshots) or under dir. The first run writes the snapshot and passes. Later runs compare with equal?, and a mismatch fails with both values. A snapshot that cannot be read is a failure too, never a silent rewrite.

A got that is a procedure is called with no arguments when the assertion runs, so state that only exists at test time can be snapshotted.

(assert-snapshot #:name "the rendered cart"
                 #:got (lambda () (render (make-cart))))
Procedure: assert-true x KEY: #:name #:timeout

Assert that x is #t.

Procedure: benchmark KEY: #:skip? #:name #:runs #:thunk #:timeout

Define a benchmark named name that times thunk over runs runs.

Statistics come from Guile’s statprof, so benchmarks are reported separately from tests. timeout caps all the runs together in seconds and skip? drops the benchmark from the run.

Procedure: call-with-timeout seconds thunk KEY: #:on-timeout

Run thunk with an optional deadline of seconds seconds.

When seconds is a positive real number the work runs in its own fiber and, if it has not finished by then, the fiber is aborted and the result of calling on-timeout is returned instead. When seconds is #f (the default, meaning no deadline) thunk is simply called and its result is returned.

thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.

(call-with-timeout 5
                   (lambda () (expensive-computation))
                   #:on-timeout (lambda () 'gave-up))
Procedure: cases->table cases key

Index cases by key, giving an alist of key to case.

Handy for tables with named columns.

Procedure: collect-assertion-successful xs

Return the outcome of every assertion in xs as a flat list.

Booleans only: #t for a pass, #f for a failure.

Procedure: congruit KEY: #:pattern #:got #:name #:timeout

Assert that the string got matches pattern.

pattern is a string or a compiled regexp.

(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Procedure: continet KEY: #:expect #:got #:name #:timeout

Assert that the container expect contains got.

Works for strings (substring), lists and vectors (membership) and hash tables (key membership).

(assert-contains #:expect '(1 2 3) #:got 2)
(assert-contains #:expect "hello world" #:got "world")
Procedure: current-seed KEY: #:seed

Return the seed this run shuffles with, choosing one when none was set.

seed (a number or a numeric string) wins when given, otherwise the value of VERITAS_SEED is used, and when neither is set a fresh seed is drawn from the platform. The chosen seed is remembered, so the order the runner gives the test files and every order chosen here come from one seed, and the run can be replayed by setting VERITAS_SEED to the seed that was reported.

(current-seed #:seed 7)
⇒ 7
Procedure: defer-body-forms forms

Wrap each of forms in a thunk, so a body is evaluated when its task runs.

Deferring is what makes a body form contribute both its own value and the cases it builds: the form is evaluated inside the body’s collector, at run time, instead of while the test file is being loaded.

Procedure: do-if-string-contained KEY: #:name #:restriction #:result

Run result when name matches restriction.

The check is case-insensitive, and a restriction counts as a match when it appears anywhere in name. A #f restriction accepts everything; one that does not appear gives #f.

Procedure: errat thunk KEY: #:name #:timeout

Return an assertion that thunk raises.

It fails if thunk returns normally. name labels it in the report and timeout caps it in seconds, as in assert.

(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
              #:name "Should fail when list contains non-numbers")
Procedure: failed-test-names all-results

Return the names that failed in all-results, for a last-failed rerun.

Every test whose assertions failed is named. A suite that failed as a whole, through its own deadline for instance, contributes the names of the tests it holds, because a suite name is not something a rerun can select. A failure with no test name attached at all contributes #all, which makes the next run run everything rather than silently run nothing. Names come back in the order recorded, without duplicates.

Procedure: failed-test-wanted? name KEY: #:only

Return #t when name should run in a last-failed rerun.

only is the record of failing names to honour; it defaults to the one read from reports/last-failed.scm when VERITAS_ONLY_FAILED asked for a rerun. #f means there is no record, so everything runs; an empty list means the last run passed, so nothing does.

Procedure: falsus x KEY: #:name #:timeout

Assert that x is #f.

Procedure: fisher-yates-shuffle xs KEY: #:seed

Return the elements of xs in random order (Fisher-Yates).

Shuffling happens on a vector copy, so xs is left alone. seed makes the order reproducible, which is how a failing order is replayed; without it a fresh seed is pulled from random-state-from-platform on every call, so two calls on the same list differ. Lists of one element or fewer come back as they are.

(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
        (fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Procedure: fresh-seed

Return a new seed for a shuffle, drawn from the platform’s entropy source.

Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.

Procedure: generate-beer

Return a generator of beer styles.

Procedure: generate-boolean

Return a generator of #t and #f.

Procedure: generate-car

Return a generator of car makes and models.

Procedure: generate-char

Return a generator of printable ASCII characters, 32 to 126.

Procedure: generate-constant value

Return a generator that always answers value.

Procedure: generate-email

Return a generator of first.last@example.com email addresses.

Procedure: generate-first-name

Return a generator of first names.

Procedure: generate-integer KEY: #:max-bound

Return a generator of integers between 0 and max-bound (exclusive).

Call the generator with a random state:

(define gen (generate-integer #:max-bound 100))
(gen (seed->random-state 42))
Procedure: generate-last-name

Return a generator of last names.

Procedure: generate-list elem-generator KEY: #:min-len #:max-len

Return a generator of lists whose elements come from elem-generator.

Lists are min-len to max-len long.

Procedure: generate-name

Return a generator of full names, a first name and a last name.

Procedure: generate-natural KEY: #:max-bound

Return a generator of integers between 0 and max-bound inclusive.

Procedure: generate-one-of choices

Return a generator that picks one of choices at random.

A choice is usually a constant, so there is nothing to shrink.

Procedure: generate-real KEY: #:min #:max

Return a generator of real numbers in the range min to max.

Procedure: generate-string KEY: #:min-len #:max-len

Return a generator of printable-ASCII strings.

Strings are min-len to max-len characters long.

Procedure: generate-symbol KEY: #:min-len #:max-len

Return a generator of lowercase symbols, min-len to max-len long.

Symbols stay unescaped because the characters are always a-z.

Procedure: generate-tuple generators

Return a generator of lists holding one value from each of generators.

Procedure: generate-vector elem-generator KEY: #:min-len #:max-len

Return a generator of vectors whose elements come from elem-generator.

Procedure: generator-shrink generator value

Return smaller replacements for value taken from generator.

The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.

Procedure: idem KEY: #:expect #:got #:name #:timeout

Assert that got is eq? to expect.

The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.

Procedure: inanis KEY: #:got #:name #:timeout

Assert that got is an empty list, vector or string.

Procedure: intra KEY: #:min #:max #:got #:name #:timeout

Assert that got is within the inclusive range min–max.

Procedure: is-approximate? a b KEY: #:epsilon

Return #t when a and b differ by less than epsilon.

Useful for floating point, where exact equality rarely holds.

Procedure: longitudo KEY: #:expect #:got #:name #:timeout

Assert that got has expect elements (list, vector or string length).

Procedure: make-assertion assertion

Return assertion, a procedure of one validation client, as a case to run.

The returned procedure is tagged so the runner can tell an assertion from a task, and it records having run, which is how an assertion the body itself consumes - with value or ok?, say - stays an input to another case instead of becoming a case of its own.

It is also registered with the collector of the body being evaluated, if there is one. That registration is what keeps a body form that builds several assertions, such as the body of a let, from contributing only the value of its last expression. Building an assertion outside any body - at load time, or inside an assertion that runs later - simply registers nothing.

Procedure: make-spy proc KEY: #:name

Return a procedure that calls proc and records every call.

The recording is read back with spy-calls, spy-results and spy-call-count. name is only used in the messages of the assert-* checks below.

(define double (make-spy (lambda (x) (* x 2))))
(double 3)
(spy-calls double)
⇒ ((3))
Procedure: make-stub KEY: #:returns #:name

Return a stand-in procedure that ignores its arguments and returns returns.

A stub is a spy whose procedure ignores its arguments, so spy-calls, spy-call-count and the assert-* checks work on it unchanged.

Procedure: merge-failed-names records

Combine what several runs recorded as failing into one record.

records holds one record per run: a list of names, or #f when a run left no record at all. When every run recorded an empty list the result is an empty list, so a parallel run that passed does not look like a run to rerun. When no run left a record the result is #f, which stands for no information rather than for nothing having failed. Names are deduplicated and keep the order they were found in.

Procedure: non-aequalis KEY: #:expect #:got #:name #:timeout

Assert that got is not equal? to expect.

Inverse of assert-equal.

Procedure: non-aequivalens KEY: #:expect #:got #:name #:timeout

Assert that got is not eqv? to expect.

Inverse of assert-eqv.

Procedure: non-congruit KEY: #:pattern #:got #:name #:timeout

Assert that string got does not match regular expression pattern.

Inverse of assert-matches.

Procedure: non-continet KEY: #:expect #:got #:name #:timeout

Assert that the container expect does not contain got.

Inverse of assert-contains.

Procedure: non-errat thunk KEY: #:name #:timeout

Return an assertion that thunk does not raise.

It fails if thunk raises. name labels it in the report and timeout caps it in seconds, as in assert.

Procedure: non-idem KEY: #:expect #:got #:name #:timeout

Assert that got is not eq? to expect.

Inverse of assert-eq.

Procedure: non-inanis KEY: #:got #:name #:timeout

Assert that got is a non-empty list, vector or string.

Procedure: non-propinquus KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got differs from expect by at least epsilon.

Inverse of assert-approximate.

Procedure: proba KEY: #:expect #:got #:compare #:name #:timeout

Return an assertion comparing got with expect using compare, a two-argument predicate such as equal?.

The assert-* wrappers below are built on this one. got and expect are usually plain values, but either may be wrapped in lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1))) evaluates the expression when the assertion runs, inside its timeout, which is what makes an expensive comparison something a deadline can catch. A lazy value keeps the lexical scope of the code that wrote it and is evaluated once per assertion that receives it.

name labels the assertion in the report. timeout caps it in seconds; #f, the default, means no deadline.

(assert #:expect 3
        #:got (+ 1 1 1)
        #:compare equal?
        #:name "Ensuring addition works")
Procedure: property name KEY: #:generators #:thunk #:runs #:seed #:shrink?

Return a property named name, a task like a test.

generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.

(property \"reverse of reverse is the original list\"
          #:generators (list (generate-list (generate-integer)))
          #:thunk (lambda (xs)
                    (assert-equal #:expect xs
                                  #:got (reverse (reverse xs)))))

A failing input is shrunk to the smallest value from the same generators that still fails, and both values are reported. seed makes the draw reproducible; without one, VERITAS_SEED is used, and failing that a fresh seed is drawn and reported with the failure.

Procedure: propinquus KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got is within epsilon of expect.

The default epsilon is 1.0e-8.

Procedure: propinquus? a b KEY: #:epsilon

Return #t when a and b differ by less than epsilon.

Useful for floating point, where exact equality rarely holds.

Procedure: read-last-failed KEY: #:path

Return the test names recorded as failing at path, or #f.

path defaults to the record of the whole run. #f means there is no record of a previous run: the file is missing or unreadable, and nothing is filtered.

Procedure: read-result-counts path

Return the counts recorded at path, or #f.

#f means there is nothing to add up: the file is missing, unreadable or not the alist write-result-counts writes.

Procedure: read-trace KEY: #:path

Return the frames stored at path, oldest first.

Procedure: render-trace KEY: #:path #:port

Print the trace at path as a numbered timeline.

Procedure: seeded-shuffle xs

Return xs in the order this run’s seed prescribes.

Both the runner, which shuffles test files, and veritas-run, which shuffles tasks, tests and cases, shuffle through this, so a single seed reproduces a whole run.

Procedure: show-last-trace

Print the trace of the last run, and return whether there was one.

Procedure: snapshot-path name KEY: #:dir

Return the file holding the snapshot called name.

dir overrides VERITAS_SNAPSHOT_DIR, which itself defaults to test/snapshots.

Procedure: snapshot-update?

Return #t when snapshots are to be rewritten instead of compared.

Set VERITAS_UPDATE_SNAPSHOTS to true to accept the current value of every snapshot a run reaches.

Procedure: split-keyword-arguments args

Split args, the syntax of an argument list, into its parts.

Two values come back: the #:keyword value pairs in the order they were written, and the forms that are not part of a pair, also in order. The test and suite macros use it to tell a body from the flags around it, wherever those flags were written.

Procedure: spy-call-count spy

Return how many times spy was called.

Procedure: spy-called-with? spy args

Return #t when spy was called with exactly args.

Procedure: spy-calls spy

Return the argument lists spy was called with, oldest first.

Procedure: spy-results spy

Return what spy returned, oldest first, one entry per call.

Procedure: spy? value

Return #t when value was made by make-spy or make-stub.

Procedure: suite-each group-name cases case->suite KEY: #:shuffle? #:concurrent?

Like test-each, but case->suite returns a suite.

Use it when one case needs several tests rather than one.

Procedure: suscitat thunk KEY: #:on #:predicate #:name #:timeout

Assert that calling thunk raises an error.

By default any error satisfies the assertion. Pass on to require a specific error key (for example 'wrong-type-arg), or predicate, a procedure of (key args) returning #t for the expected error.

(assert-raises (lambda () (/ 1 0)))
(assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Procedure: test-each group-name cases case->test KEY: #:shuffle? #:concurrent?

Expand cases into one test per case, under a suite named group-name.

case->test takes one case and returns a test:

(test-each "integer addition"
           '((0 0 0) (1 2 3) (10 20 30))
           (lambda (case)
             (let ((a (list-ref case 0))
                   (b (list-ref case 1))
                   (expected (list-ref case 2)))
               (test (format #f "~a + ~a = ~a" a b expected)
                 (assert-equal #:expect expected #:got (+ a b))))))
Procedure: trace-event! kind name OPT: detail

Record one step of the run when tracing is on.

kind is a symbol such as test-start, assertion or test-end, name is what the step is about, and detail is a short string a reader can use to tell one frame from another.

Procedure: trace-file

Return where the last trace of this run is written.

Procedure: trace-reset!

Forget the frames recorded so far.

Procedure: unus-ex KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to one of the elements of list expect.

Procedure: veritas-run KEY: #:shuffle? #:concurrent? #:debug? #:reporter #:with-coverage? #:coverage-dir #:coverage-html-dir #:coverage-include #:coverage-exclude #:seed . tasks

Run tasks, the top-level tests, suites and benchmarks.

shuffle? randomizes the order of the tasks and concurrent? runs them in parallel fibers. reporter names a reporter, otherwise VERITAS_REPORTER decides. seed makes the shuffle reproducible, and the seed actually used is reported, so a failing order can be replayed by setting VERITAS_SEED to it. with-coverage? measures coverage - VERITAS_COVERAGE=true asks for it too - and debug? prints the raw results.

Coverage is written where coverage-dir (default VERITAS_COVERAGE_DIR, "reports/lcov") says, with the HTML report in coverage-html-dir (default VERITAS_COVERAGE_HTML_DIR, "reports/html"). Only the files under coverage-include (default VERITAS_COVERAGE_INCLUDE, the current directory) are reported, minus the ones under coverage-exclude (default VERITAS_COVERAGE_EXCLUDE).

Procedure: verus x KEY: #:name #:timeout

Assert that x is #t.

Procedure: with-temporary-binding module var value thunk

Call thunk with module’s var set to value, and restore it.

The previous value comes back even when thunk raises, so substituting a dependency is safe to try.

Guile resolves a call to a known procedure while it compiles, so a compiled call site does *not* see the change: a call written in a module the compiler can see through keeps calling the original procedure. Measured on Guile 3.0, swapping a binding is observed by module-ref, by code passed to eval, and by call sites in a module declared #:declarative? #f; it is not observed by a call compiled in a declarative module, which is the default.

That leaves the ordinary Scheme answer, which needs no substitution at all: pass the dependency in, or hand the caller a make-spy recording. Use this when the code under test reaches the binding at run time.

(with-temporary-binding (resolve-module '(my app)) 'now
                        (lambda () 0)
                        (lambda ()
                          (assert-equal #:expect 0
                                        #:got (eval '((@ (my app) now))
                                                    (resolve-module '(my app))))))
Procedure: write-last-failed! all-results

Record the names that failed, so the next run can rerun just those.

Each worker process writes its own record - the path is a parameter - and the --jobs parent merges them into the run’s, which is why the writing and the merging are separate.

Procedure: write-last-failed-names! names KEY: #:path

Record names at path as the tests a rerun should select.

An empty list is written when everything passed, which is what makes the next VERITAS_ONLY_FAILED run do nothing instead of rerunning the world. A trouble with the file is swallowed: the run has already happened.

Procedure: write-result-counts counts KEY: #:path

Write counts to path so another process can add them up.

An alist is plain Scheme data, so read restores it exactly: this is how a worker process hands its totals to the process that runs the other files. Returns path.

Procedure: write-trace! KEY: #:path

Write the frames recorded so far to path and return it.

The newest frame is last in the file, so the timeline reads top to bottom.

1.1.5 Variables

Variable: veritas-rerun-all-marker
"#all"

1.2 (veritas assert)

Assertions for Veritas. The core procedure is ‘assert’, which compares a value against an expected one using whatever predicate you give it. The assert-* procedures below are thin wrappers around it, together with a few helpers for checking error handling.

Every assertion also has a Latin alias, following the Latin naming already used by the BDD helpers (dato, quando, tunc, ...): (proba ...) is (assert ...), aequalis is assert-equal, verus is assert-true, continet is assert-contains, and so on. An alias is the exact same procedure object as its English name, so keyword arguments, semantics and reporting are identical. The complete mapping is documented in the README and repeated in the alias block at the end of this file.

1.2.1 Overview

  • assert-match — An undocumented macro.
  • convenit — An undocumented macro.
  • lazy — An undocumented macro.
  • lazy-value? — An undocumented macro.
  • aequalis — Assert that got is equal? to expect.
  • aequivalens — Assert that got is eqv? to expect.
  • assert — Return an assertion comparing got with expect using compare,
  • assert-approximate — Assert that got is within epsilon of expect.
  • assert-contains — Assert that the container expect contains got.
  • assert-empty — Assert that got is an empty list, vector or string.
  • assert-eq — Assert that got is eq? to expect.
  • assert-equal — Assert that got is equal? to expect.
  • assert-eqv — Assert that got is eqv? to expect.
  • assert-error — Return an assertion that thunk raises.
  • assert-false — Assert that x is #f.
  • assert-in-range — Assert that got is within the inclusive range min–max.
  • assert-length — Assert that got has expect elements (list, vector or string length).
  • assert-matches — Assert that the string got matches pattern.
  • assert-no-error — Return an assertion that thunk does not raise.
  • assert-not-approximate — Assert that got differs from expect by at least epsilon.
  • assert-not-contains — Assert that the container expect does not contain got.
  • assert-not-empty — Assert that got is a non-empty list, vector or string.
  • assert-not-eq — Assert that got is not eq? to expect.
  • assert-not-equal — Assert that got is not equal? to expect.
  • assert-not-eqv — Assert that got is not eqv? to expect.
  • assert-not-matches — Assert that string got does not match regular expression pattern.
  • assert-one-of — Assert that got is equal? to one of the elements of list expect.
  • assert-raises — Assert that calling thunk raises an error.
  • assert-true — Assert that x is #t.
  • congruit — Assert that the string got matches pattern.
  • continet — Assert that the container expect contains got.
  • errat — Return an assertion that thunk raises.
  • falsus — Assert that x is #f.
  • idem — Assert that got is eq? to expect.
  • inanis — Assert that got is an empty list, vector or string.
  • intra — Assert that got is within the inclusive range min–max.
  • is-approximate? — Return #t when a and b differ by less than epsilon.
  • longitudo — Assert that got has expect elements (list, vector or string length).
  • make-assertion — Return assertion, a procedure of one validation client, as a case to run.
  • non-aequalis — Assert that got is not equal? to expect.
  • non-aequivalens — Assert that got is not eqv? to expect.
  • non-congruit — Assert that string got does not match regular expression pattern.
  • non-continet — Assert that the container expect does not contain got.
  • non-errat — Return an assertion that thunk does not raise.
  • non-idem — Assert that got is not eq? to expect.
  • non-inanis — Assert that got is a non-empty list, vector or string.
  • non-propinquus — Assert that got differs from expect by at least epsilon.
  • proba — Return an assertion comparing got with expect using compare,
  • propinquus — Assert that got is within epsilon of expect.
  • propinquus? — Return #t when a and b differ by less than epsilon.
  • suscitat — Assert that calling thunk raises an error.
  • unus-ex — Assert that got is equal? to one of the elements of list expect.
  • verus — Assert that x is #t.

1.2.2 Macros

Macro: assert-match x

Undocumented macro.

Macro: convenit x

Undocumented macro.

Macro: lazy x

Undocumented macro.

Macro: lazy-value? x

Undocumented macro.

1.2.3 Procedures

Procedure: aequalis KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to expect.

The usual choice for lists, strings, vectors and other compound values.

Procedure: aequivalens KEY: #:expect #:got #:name #:timeout

Assert that got is eqv? to expect.

Good for numbers, characters and symbols, where value equivalence is what matters.

Procedure: assert KEY: #:expect #:got #:compare #:name #:timeout

Return an assertion comparing got with expect using compare, a two-argument predicate such as equal?.

The assert-* wrappers below are built on this one. got and expect are usually plain values, but either may be wrapped in lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1))) evaluates the expression when the assertion runs, inside its timeout, which is what makes an expensive comparison something a deadline can catch. A lazy value keeps the lexical scope of the code that wrote it and is evaluated once per assertion that receives it.

name labels the assertion in the report. timeout caps it in seconds; #f, the default, means no deadline.

(assert #:expect 3
        #:got (+ 1 1 1)
        #:compare equal?
        #:name "Ensuring addition works")
Procedure: assert-approximate KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got is within epsilon of expect.

The default epsilon is 1.0e-8.

Procedure: assert-contains KEY: #:expect #:got #:name #:timeout

Assert that the container expect contains got.

Works for strings (substring), lists and vectors (membership) and hash tables (key membership).

(assert-contains #:expect '(1 2 3) #:got 2)
(assert-contains #:expect "hello world" #:got "world")
Procedure: assert-empty KEY: #:got #:name #:timeout

Assert that got is an empty list, vector or string.

Procedure: assert-eq KEY: #:expect #:got #:name #:timeout

Assert that got is eq? to expect.

The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.

Procedure: assert-equal KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to expect.

The usual choice for lists, strings, vectors and other compound values.

Procedure: assert-eqv KEY: #:expect #:got #:name #:timeout

Assert that got is eqv? to expect.

Good for numbers, characters and symbols, where value equivalence is what matters.

Procedure: assert-error thunk KEY: #:name #:timeout

Return an assertion that thunk raises.

It fails if thunk returns normally. name labels it in the report and timeout caps it in seconds, as in assert.

(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
              #:name "Should fail when list contains non-numbers")
Procedure: assert-false x KEY: #:name #:timeout

Assert that x is #f.

Procedure: assert-in-range KEY: #:min #:max #:got #:name #:timeout

Assert that got is within the inclusive range min–max.

Procedure: assert-length KEY: #:expect #:got #:name #:timeout

Assert that got has expect elements (list, vector or string length).

Procedure: assert-matches KEY: #:pattern #:got #:name #:timeout

Assert that the string got matches pattern.

pattern is a string or a compiled regexp.

(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Procedure: assert-no-error thunk KEY: #:name #:timeout

Return an assertion that thunk does not raise.

It fails if thunk raises. name labels it in the report and timeout caps it in seconds, as in assert.

Procedure: assert-not-approximate KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got differs from expect by at least epsilon.

Inverse of assert-approximate.

Procedure: assert-not-contains KEY: #:expect #:got #:name #:timeout

Assert that the container expect does not contain got.

Inverse of assert-contains.

Procedure: assert-not-empty KEY: #:got #:name #:timeout

Assert that got is a non-empty list, vector or string.

Procedure: assert-not-eq KEY: #:expect #:got #:name #:timeout

Assert that got is not eq? to expect.

Inverse of assert-eq.

Procedure: assert-not-equal KEY: #:expect #:got #:name #:timeout

Assert that got is not equal? to expect.

Inverse of assert-equal.

Procedure: assert-not-eqv KEY: #:expect #:got #:name #:timeout

Assert that got is not eqv? to expect.

Inverse of assert-eqv.

Procedure: assert-not-matches KEY: #:pattern #:got #:name #:timeout

Assert that string got does not match regular expression pattern.

Inverse of assert-matches.

Procedure: assert-one-of KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to one of the elements of list expect.

Procedure: assert-raises thunk KEY: #:on #:predicate #:name #:timeout

Assert that calling thunk raises an error.

By default any error satisfies the assertion. Pass on to require a specific error key (for example 'wrong-type-arg), or predicate, a procedure of (key args) returning #t for the expected error.

(assert-raises (lambda () (/ 1 0)))
(assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Procedure: assert-true x KEY: #:name #:timeout

Assert that x is #t.

Procedure: congruit KEY: #:pattern #:got #:name #:timeout

Assert that the string got matches pattern.

pattern is a string or a compiled regexp.

(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Procedure: continet KEY: #:expect #:got #:name #:timeout

Assert that the container expect contains got.

Works for strings (substring), lists and vectors (membership) and hash tables (key membership).

(assert-contains #:expect '(1 2 3) #:got 2)
(assert-contains #:expect "hello world" #:got "world")
Procedure: errat thunk KEY: #:name #:timeout

Return an assertion that thunk raises.

It fails if thunk returns normally. name labels it in the report and timeout caps it in seconds, as in assert.

(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
              #:name "Should fail when list contains non-numbers")
Procedure: falsus x KEY: #:name #:timeout

Assert that x is #f.

Procedure: idem KEY: #:expect #:got #:name #:timeout

Assert that got is eq? to expect.

The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.

Procedure: inanis KEY: #:got #:name #:timeout

Assert that got is an empty list, vector or string.

Procedure: intra KEY: #:min #:max #:got #:name #:timeout

Assert that got is within the inclusive range min–max.

Procedure: is-approximate? a b KEY: #:epsilon

Return #t when a and b differ by less than epsilon.

Useful for floating point, where exact equality rarely holds.

Procedure: longitudo KEY: #:expect #:got #:name #:timeout

Assert that got has expect elements (list, vector or string length).

Procedure: make-assertion assertion

Return assertion, a procedure of one validation client, as a case to run.

The returned procedure is tagged so the runner can tell an assertion from a task, and it records having run, which is how an assertion the body itself consumes - with value or ok?, say - stays an input to another case instead of becoming a case of its own.

It is also registered with the collector of the body being evaluated, if there is one. That registration is what keeps a body form that builds several assertions, such as the body of a let, from contributing only the value of its last expression. Building an assertion outside any body - at load time, or inside an assertion that runs later - simply registers nothing.

Procedure: non-aequalis KEY: #:expect #:got #:name #:timeout

Assert that got is not equal? to expect.

Inverse of assert-equal.

Procedure: non-aequivalens KEY: #:expect #:got #:name #:timeout

Assert that got is not eqv? to expect.

Inverse of assert-eqv.

Procedure: non-congruit KEY: #:pattern #:got #:name #:timeout

Assert that string got does not match regular expression pattern.

Inverse of assert-matches.

Procedure: non-continet KEY: #:expect #:got #:name #:timeout

Assert that the container expect does not contain got.

Inverse of assert-contains.

Procedure: non-errat thunk KEY: #:name #:timeout

Return an assertion that thunk does not raise.

It fails if thunk raises. name labels it in the report and timeout caps it in seconds, as in assert.

Procedure: non-idem KEY: #:expect #:got #:name #:timeout

Assert that got is not eq? to expect.

Inverse of assert-eq.

Procedure: non-inanis KEY: #:got #:name #:timeout

Assert that got is a non-empty list, vector or string.

Procedure: non-propinquus KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got differs from expect by at least epsilon.

Inverse of assert-approximate.

Procedure: proba KEY: #:expect #:got #:compare #:name #:timeout

Return an assertion comparing got with expect using compare, a two-argument predicate such as equal?.

The assert-* wrappers below are built on this one. got and expect are usually plain values, but either may be wrapped in lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1))) evaluates the expression when the assertion runs, inside its timeout, which is what makes an expensive comparison something a deadline can catch. A lazy value keeps the lexical scope of the code that wrote it and is evaluated once per assertion that receives it.

name labels the assertion in the report. timeout caps it in seconds; #f, the default, means no deadline.

(assert #:expect 3
        #:got (+ 1 1 1)
        #:compare equal?
        #:name "Ensuring addition works")
Procedure: propinquus KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got is within epsilon of expect.

The default epsilon is 1.0e-8.

Procedure: propinquus? a b KEY: #:epsilon

Return #t when a and b differ by less than epsilon.

Useful for floating point, where exact equality rarely holds.

Procedure: suscitat thunk KEY: #:on #:predicate #:name #:timeout

Assert that calling thunk raises an error.

By default any error satisfies the assertion. Pass on to require a specific error key (for example 'wrong-type-arg), or predicate, a procedure of (key args) returning #t for the expected error.

(assert-raises (lambda () (/ 1 0)))
(assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Procedure: unus-ex KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to one of the elements of list expect.

Procedure: verus x KEY: #:name #:timeout

Assert that x is #t.


1.3 (veritas bdd)

Behaviour-driven (Given/When/Then) helpers for Veritas, in the spirit of Cucumber/Gherkin scenarios. A scenario is a suite whose steps are ordinary tests, so every existing reporter, filter and runner feature keeps working. Step names are prefixed with their keyword so the output reads naturally.

Steps run in order by default: scenarios disable shuffling and concurrency so that Given -> When -> Then always executes sequentially.

State is shared between steps simply by closing over a ‘let’ binding in the scenario body, for example:

(scenario "withdrawing too much" (let ((account (make-account 100))) (given "an account with 100" (assert-equal #:expect 100 #:got (account-balance account))) (when-step "withdrawing 150" (assert-false (withdraw account 150))) (then "the balance is unchanged" (assert-equal #:expect 100 #:got (account-balance account)))))

Every step of that scenario runs: a scenario body is read for the steps it builds, not for the value of its last expression, so a ‘let’ around the steps is the natural way to share state and it does not hide the steps from the runner. The same holds inside a step: a ‘let’ body contributes every assertion it builds.

Latin aliases are provided for those who prefer them: fabula/scaena for scenario, dato for given, quando for when, tunc for then, etiam for and, and sed for but.

The three steps also have PascalCase aliases – Given, When and Then – for readers who expect the keywords to look like the Cucumber ones.

The When step is called ‘when-step’ rather than ‘when’ because Guile’s core ‘when’ is a syntax keyword and importing a conflicting binding would break ‘(when ...)’ for every module that imports Veritas BDD. The Latin ‘quando’ is unshadowed and reads naturally.

1.3.1 Overview

  • Given — An undocumented macro.
  • Then — An undocumented macro.
  • When — An undocumented macro.
  • and-step — An undocumented macro.
  • but-step — An undocumented macro.
  • dato — An undocumented macro.
  • etiam — An undocumented macro.
  • fabula — An undocumented macro.
  • feature — An undocumented macro.
  • given — An undocumented macro.
  • quando — An undocumented macro.
  • scaena — An undocumented macro.
  • scenario — An undocumented macro.
  • sed — An undocumented macro.
  • story — An undocumented macro.
  • then — An undocumented macro.
  • tunc — An undocumented macro.
  • when-step — An undocumented macro.
  • %and-step — Define an And (continuation) step named name.
  • %but-step — Define a But (continuation) step named name.
  • %given — Define a Given (setup) step named name.
  • %scenario — Define a Given/When/Then scenario named name; the procedure behind the macro.
  • %step — Define a step named name, labeled with label; the procedure behind the step macros.
  • %then — Define a Then (outcome) step named name.
  • %when-step — Define a When (action) step named name.

1.3.2 Macros

Macro: Given stx

Undocumented macro.

Macro: Then stx

Undocumented macro.

Macro: When stx

Undocumented macro.

Macro: and-step stx

Undocumented macro.

Macro: but-step stx

Undocumented macro.

Macro: dato stx

Undocumented macro.

Macro: etiam stx

Undocumented macro.

Macro: fabula stx

Undocumented macro.

Macro: feature stx

Undocumented macro.

Macro: given stx

Undocumented macro.

Macro: quando stx

Undocumented macro.

Macro: scaena stx

Undocumented macro.

Macro: scenario stx

Undocumented macro.

Macro: sed stx

Undocumented macro.

Macro: story stx

Undocumented macro.

Macro: then stx

Undocumented macro.

Macro: tunc stx

Undocumented macro.

Macro: when-step stx

Undocumented macro.

1.3.3 Procedures

Procedure: %and-step name . cases

Define an And (continuation) step named name.

Procedure: %but-step name . cases

Define a But (continuation) step named name.

Procedure: %given name . cases

Define a Given (setup) step named name.

Procedure: %scenario name KEY: #:skip? #:debug? #:shuffle? #:concurrent? #:timeout #:tags #:before-all #:after-all . steps

Define a Given/When/Then scenario named name; the procedure behind the macro.

A scenario is a suite, so its steps run in declaration order. Keywords are forwarded to suite, and steps come from given, when-step, then, and-step and but-step.

timeout, when given, caps the whole scenario in seconds, exactly like suite; a step can carry its own #:timeout as well, since steps are ordinary tests.

(scenario "emptying the cart"
  (given "a cart with one item"
    (assert-equal #:expect 1 #:got (cart-size cart)))
  (when-step "the cart is emptied"
    (empty-cart! cart))
  (then "the cart is empty"
    (assert-true (cart-empty? cart))))
Procedure: %step label name . cases

Define a step named name, labeled with label; the procedure behind the step macros.

A step is an ordinary test, so it takes the same keywords and cases.

Procedure: %then name . cases

Define a Then (outcome) step named name.

Procedure: %when-step name . cases

Define a When (action) step named name.

Called when-step rather than when so it does not shadow Guile’s when syntax.


1.4 (veritas cli)

Command line interface for the Veritas test runner.

The CLI is intentionally a thin layer over the runner: every flag is translated into the matching VERITAS_* environment variable that the runner and the rest of Veritas already understand. This keeps the environment variables as the single, reproducible source of truth while offering ergonomic syntactic sugar for interactive use and CI scripts.

The runner module is resolved lazily (only after the environment has been updated), so that parameters which are initialised from the environment when their module loads - such as VERITAS_RUN_BENCHMARKS and VERITAS_ONLY_TEST - observe the values supplied on the command line.

The run itself is a straight line, and every stage of it is a module:

Mermaid diagram, as source:

graph LR
  cli["(veritas cli)"] --> env["VERITAS_* settings"]
  env --> runner["(veritas runner)"]
  runner --> files["test files"]
  files --> suites["suites and tests"]
  suites --> results["(veritas model) results"]
  results --> reports["(veritas reporter)"]

Only the first stage is this module’s: a run started by hand and a run started by an editor meet at the environment variables and go on from there.

1.4.1 Overview

1.4.2 Procedures

Procedure: main args

Entry point suitable for a launcher or a Guile -c invocation.

args is the list of user arguments (typically (cdr (command-line))). Exits the process with the status returned by run-cli.

Procedure: run-cli args

Parse args and run the Veritas runner.

args is a list of command-line strings without the program name. Returns an exit status: 0 on success, 1 on failure.

(run-cli '("--test-type" "unit" "--no-benchmarks"))
Procedure: show-help

Print the full Veritas CLI help page to standard output.

1.4.3 Variables

Variable: veritas-help-text
"veritas 1.5.0 - test runner for the Veritas framework\n\nusage: veritas [OPTIONS]\n\nDiscover and run the test suites, tests and benchmarks of a project.\nEvery option is mapped to the corresponding VERITAS_* environment\nvariable, so the command line is only syntactic sugar over the exact\nsame configuration used programmatically and in CI.\n\noptions:\n  -d, --scan-dir DIR       root directory to scan for tests (VERITAS_SCAN_DIR)\n  -T, --test-top-dir DIR   directory holding the test hierarchy (VERITAS_TEST_TOP_DIR)\n  -t, --test-type TYPE     test category / sub-directory to load (VERITAS_TEST_TYPE)\n  -e, --entrypoint NAME    no-argument function to call per file (VERITAS_ENTRYPOINT)\n  -m, --test-module NAME   top level module prefix of the tests (VERITAS_TEST_MODULE)\n  -r, --reporter NAME      reporter to use, e.g. emoji-reporter,\n                           monotone-reporter, json-reporter,\n                           junit-reporter or github-reporter (VERITAS_REPORTER)\n      --only-test SUBSTR   only run tests whose name contains SUBSTR (VERITAS_ONLY_TEST)\n      --only-suite SUBSTR  only run suites whose name contains SUBSTR (VERITAS_ONLY_SUITE)\n      --only-tag TAG       only run tests/suites carrying TAG (VERITAS_ONLY_TAG)\n      --only-failed        rerun only what failed in the previous run\n                           (VERITAS_ONLY_FAILED)\n      --no-only-failed     ignore the recorded failures (the default)\n      --seed N             seed every shuffle of this run (VERITAS_SEED)\n      --partition I/T      run slice I of the T slices of the test files\n                           (VERITAS_PARTITION)\n  -j, --jobs N             run the test files in N worker processes, one\n                           file per process (VERITAS_JOBS)\n      --changed            run only the test files the working tree change\n                           reaches (VERITAS_CHANGED)\n      --no-changed         run every test file (the default)\n      --since REF          run only the test files the change since REF\n                           reaches (VERITAS_SINCE)\n      --src-dir DIR        source root the change graph is read from\n                           (VERITAS_SRC_DIR, default src)\n      --retries N          rerun a failing test up to N extra times\n                           (VERITAS_RETRIES)\n      --trace              record every step of the run, to\n                           reports/traces/last.scm (VERITAS_TRACE)\n      --no-trace           do not record a trace (the default)\n      --show-trace         print the timeline of the last run and exit\n      --list               print the test files this run would run, one\n                           per line, and run none of them (VERITAS_LIST)\n      --coverage           measure code coverage and write the reports\n                           (VERITAS_COVERAGE)\n      --no-coverage        do not measure coverage (the default)\n      --coverage-dir DIR   where the LCOV trace and the JSON summary go\n                           (VERITAS_COVERAGE_DIR, default reports/lcov)\n      --coverage-html-dir DIR\n                           where the HTML report goes, or \"false\" to skip\n                           it (VERITAS_COVERAGE_HTML_DIR, default reports/html)\n      --coverage-include DIRS\n                           colon-separated source roots the report covers\n                           (VERITAS_COVERAGE_INCLUDE, default the current\n                           directory)\n      --coverage-exclude DIRS\n                           colon-separated source roots left out\n                           (VERITAS_COVERAGE_EXCLUDE)\n  -s, --shuffle            shuffle the order of discovered files (default)\n      --no-shuffle         keep the discovered file order (VERITAS_SHUFFLE_FILES=false)\n  -b, --benchmarks         run benchmarks (default) (VERITAS_RUN_BENCHMARKS=true)\n      --no-benchmarks      skip benchmarks (VERITAS_RUN_BENCHMARKS=false)\n  -h, --help               show this help and exit\n  -V, --version            show version information and exit\n\nenvironment variables:\n  VERITAS_ENTRYPOINT        entry-point function name, defaults to \"spec\"\n  VERITAS_SCAN_DIR          directory to scan, defaults to \"./\"\n  VERITAS_TEST_TYPE         test type directory, defaults to \"unit\"\n  VERITAS_TEST_TOP_DIR      test hierarchy root, defaults to \"test\"\n  VERITAS_TEST_MODULE       module prefix, defaults to \"veritas\"\n  VERITAS_SHUFFLE_FILES     \"true\" or \"false\", defaults to \"true\"\n  VERITAS_RUN_BENCHMARKS    \"true\" or \"false\", defaults to \"true\"\n  VERITAS_ONLY_TEST         optional test-name substring filter\n  VERITAS_ONLY_SUITE        optional suite-name substring filter\n  VERITAS_ONLY_TAG          optional tag filter\n  VERITAS_ONLY_FAILED       \"true\" to rerun only the tests that failed last time\n  VERITAS_REPORTER          reporter symbol, defaults to emoji-reporter\n  VERITAS_SEED              seed for every shuffle, printed by a shuffled run\n  VERITAS_PARTITION         \"INDEX/TOTAL\" slice of the discovered test files\n  VERITAS_JOBS              whole number of worker processes for the files,\n                            defaults to 1 and then they run in this process\n  VERITAS_CHANGED           \"true\" to run only the tests the change reaches\n  VERITAS_SINCE             revision the change is read against, for example\n                            origin/trunk\n  VERITAS_SRC_DIR           source root of the change graph, defaults to src\n  VERITAS_RETRIES           extra attempts for a failing test, defaults to 0\n  VERITAS_TRACE             \"true\" to record a trace of the run\n  VERITAS_LIST              \"true\" to print the selected files and run none\n  VERITAS_COVERAGE          \"true\" to measure code coverage\n  VERITAS_COVERAGE_DIR      where the LCOV trace is written, defaults to\n                            reports/lcov\n  VERITAS_COVERAGE_HTML_DIR where the HTML report is written, defaults to\n                            reports/html, and \"false\" turns it off\n  VERITAS_COVERAGE_INCLUDE  colon-separated source roots to report on,\n                            defaults to the current directory\n  VERITAS_COVERAGE_EXCLUDE  colon-separated source roots to leave out\n  VERITAS_COBERTURA_PATH    where the Cobertura XML is written, defaults to\n                            cobertura.xml in the coverage directory\n  VERITAS_JACOCO_PATH       where the JaCoCo XML is written, defaults to\n                            jacoco.xml in the coverage directory\n  VERITAS_LAST_FAILED_PATH  where the failing names are recorded, defaults\n                            to reports/last-failed.scm\n  VERITAS_JUNIT_PATH        where junit-reporter writes, defaults to\n                            reports/junit/veritas-junit.xml\n\nexamples:\n  veritas                          run unit tests with the project defaults\n  veritas --test-type integration  run the integration test suite\n  veritas --no-shuffle --reporter monotone-reporter\n  veritas --only-suite auth        run only the suites mentioning \"auth\"\n  veritas --only-tag slow          run only tests/suites tagged \"slow\"\n  veritas --no-benchmarks          fast feedback loop, skip benchmarks\n  veritas --only-failed            rerun what failed in the previous run\n  veritas --seed 42                repeat a shuffled order exactly\n  veritas --partition 1/3          run one third of the test files\n  veritas --jobs 4                 run the files in four worker processes\n  veritas --changed                run only the tests this change can reach\n  veritas --since origin/trunk     run only the tests a branch touches\n  veritas --retries 2              ride out an intermittent failure\n  veritas --trace                  record the run, then\n  veritas --show-trace             print the recorded timeline\n  veritas --changed --list         show what --changed would run\n  veritas --coverage               measure coverage and write LCOV and HTML\n  veritas --coverage --coverage-exclude test\n                                   measure the sources, not the tests\n  veritas --reporter junit-reporter  write a JUnit XML report\n\ninteractive use:\n  The same runner is available from a Guile REPL.  Load the CLI module and\n  call (run-cli '(\"--help\")) or (show-help) to inspect the options, and\n  ((@@ (veritas runner) run-tests)) to run with the current environment.\n"
Variable: veritas-man-page-name
"veritas(1)"
Variable: veritas-option-grammar
((scan-dir (single-char #\d) (value #t)) (test-top-dir (single-char #\T) (value #t)) (test-type (single-char #\t) (value #t)) (entrypoint (single-char #\e) (value #t)) (test-module (single-char #\m) (value #t)) (reporter (single-char #\r) (value #t)) (only-test (value #t)) (only-suite (value #t)) (only-tag (value #t)) (only-failed (value #f)) (no-only-failed (value #f)) (seed (value #t)) (partition (value #t)) (jobs (single-char #\j) (value #t)) (changed (value #f)) (no-changed (value #f)) (since (value #t)) (src-dir (value #t)) (list (value #f)) (trace (value #f)) (no-trace (value #f)) (show-trace (value #f)) (retries (value #t)) (coverage (value #f)) (no-coverage (value #f)) (coverage-dir (value #t)) (coverage-html-dir (value #t)) (coverage-include (value #t)) (coverage-exclude (value #t)) (shuffle (single-char #\s) (value #f)) (no-shuffle (value #f)) (benchmarks (single-char #\b) (value #f)) (no-benchmarks (value #f)) (help (single-char #\h) (value #f)) (version (single-char #\V) (value #f)))
Variable: veritas-version
"1.5.0"

1.5 (veritas container)

Docker/Podman helpers for integration and black box testing, following the well known container-test pattern: start a container, wait until it is ready, exercise it (exec, logs, host ports), then clean it up.

The runtime is auto-detected (Podman first, then Docker) and can be forced with the VERITAS_CONTAINER_RUNTIME environment variable, mirroring the way testcontainers libraries target a Docker-compatible endpoint. Everything shells out to the CLI, so no daemon bindings are required.

Typical use inside a suite:

(with-container (lambda (c) (assert-true (container-running? c)) (assert-matches #:pattern "ready" #:got (container-exec c ’("app" "–version")))) #:image "alpine" #:command ’("sh" "-c" "echo ready && sleep 600") #:wait (lambda (c) (container-wait-for-log c "ready" #:timeout 10)))

1.5.1 Overview

1.5.2 Procedures

Procedure: assert-container-running KEY: #:container #:name

Return an assertion that container is running.

Unlike assert-true, the check is performed when the assertion runs, not when it is constructed.

Procedure: container-build KEY: #:tag #:dockerfile #:context #:extra-args

Build an image with tag from dockerfile in context.

Procedure: container-exec container command

Run command (a list of strings) inside running container.

Returns the captured combined output and raises on failure.

Procedure: container-exec-status container command

Run command inside container, returning two values: status and output.

Procedure: container-logs container

Return the logs of container as a string.

Procedure: container-remove container

Force-stop and remove container, returning its exit status.

Procedure: container-run KEY: #:image #:name #:env #:ports #:volumes #:command #:detach? #:extra-args

Start image and return the new container id (or name when given).

env entries are passed as -e variables, ports as -p bindings and volumes as -v mounts.

Procedure: container-running? container

Return #t when container exists and is running.

Procedure: container-runtime

Return the container runtime executable to use.

Honours VERITAS_CONTAINER_RUNTIME, otherwise prefers podman and falls back to docker.

Procedure: container-runtime-available?

Return #t when a container runtime is available.

Procedure: container-stop container

Stop container, returning its exit status.

Procedure: container-wait-for-command container command KEY: #:timeout #:interval

Wait until command succeeds inside container.

Procedure: container-wait-for-log container pattern KEY: #:timeout #:interval

Wait until container logs match pattern (regexp or string).

Procedure: container-wait-for-port host port KEY: #:timeout #:interval

Wait until a TCP connection to host:port succeeds.

Procedure: run-command command KEY: #:input

Run command, a list of strings, without a shell.

Returns two values: the exit status (an integer, -1 when the process could not be started) and the captured standard output as a string.

(call-with-values (lambda () (run-command (list "echo" "hi")))
  (lambda (status output) (list status output)))
⇒ (0 "hi\n")
Procedure: run-command! command KEY: #:input

Like run-command, but raise an error on a non-zero exit status.

Returns the captured standard output.

Procedure: wait-until predicate KEY: #:timeout #:interval #:message

Poll predicate every interval seconds until it is true.

Returns #t, or raises an error after timeout seconds. Exceptions raised by predicate are treated as a false result.

Procedure: with-container thunk KEY: #:image #:name #:env #:ports #:volumes #:command #:detach? #:extra-args #:wait #:keep?

Start a container, call thunk with its id, and always clean it up.

wait, when supplied, is a procedure called with the container id after start (for example to wait for a log line or a port). keep? leaves the container running for debugging.


1.6 (veritas coverage)

Code coverage for Veritas.

Guile collects execution counts by firing a VM hook on every instruction it retires. That hook only exists in the ‘debug’ VM engine: in the ‘regular’ engine the same code runs silently, which is why a run that merely calls ‘(system vm coverage)’ records no hits at all. Every collection here therefore happens inside ‘with-debug-vm’.

What comes out of the VM is an IP-to-count table and a census of the source locations of every loaded program. It is turned into a report - one record per source file, holding the execution count of each instrumented line - and that report can be written as LCOV, as a self-contained HTML report, or as JSON, and merged with other reports (the files of a ‘–jobs’ run, say).

1.6.1 Overview

1.6.2 Macros

Macro: coverage-file-lines x

Undocumented macro.

Macro: coverage-file-path x

Undocumented macro.

Macro: coverage-file? x

Undocumented macro.

Macro: coverage-report-files x

Undocumented macro.

Macro: coverage-report? x

Undocumented macro.

1.6.3 Procedures

Procedure: collect-coverage thunk KEY: #:include #:exclude #:base

Run thunk and return its values preceded by its coverage report.

The report is a coverage-report record; the values of thunk follow it unchanged. The run happens on the debug engine, and the engine in use before the call is restored afterwards.

(call-with-values
 (lambda () (collect-coverage (lambda () (run-the-tests))))
 (lambda (report result) ...))
Procedure: coverage-cobertura-path KEY: #:dir

Return the path of the Cobertura XML report of a run.

The VERITAS_COBERTURA_PATH environment variable wins when it is set, which is how a CI job puts the document where its dashboard looks for it.

Procedure: coverage-data->report data KEY: #:include #:exclude #:base

Turn Guile’s coverage data into a report.

Every instrumented source file is kept, with the count of each of its lines - including the lines that never ran, which is the whole point of a coverage report. include and exclude filter the files by absolute path.

Procedure: coverage-directory KEY: #:dir

Return the directory the LCOV report is written to.

dir wins; otherwise VERITAS_COVERAGE_DIR decides, and "reports/lcov" is the default. The directory holds the LCOV trace, the JSON summary and the Cobertura and JaCoCo documents.

Procedure: coverage-disabled-by-environment? KEY: #:value

Return #t when value (the environment by default) refuses coverage.

An explicit refusal beats a test file that asks for measurement with #:with-coverage? #t: a task that says ‘maak test-fast’ or passes --no-coverage means it, and it is the only way to run a file that always measures without paying for it.

Procedure: coverage-enabled-by-environment? KEY: #:value

Return #t when value (the environment by default) asks for coverage.

Procedure: coverage-exclude-paths KEY: #:exclude #:base

Return the prefixes left out of a report.

An explicit exclude wins, otherwise VERITAS_COVERAGE_EXCLUDE is read. Nothing is excluded by default.

Procedure: coverage-file-covered file

Return how many instrumented lines of file ran at least once.

Procedure: coverage-file-instrumented file

Return how many lines of file hold code that could be counted.

Procedure: coverage-file-line-count file line

Return how often line of file ran, or #f when it holds no code.

A line that was instrumented but never reached counts 0; a line with no code at all - a comment, a blank line - has no entry and gives #f.

Procedure: coverage-file-percentage file

Return the percentage of file’s instrumented lines that ran.

Procedure: coverage-html-directory KEY: #:html-dir

Return the directory the HTML report is written to, or #f to skip it.

html-dir wins; otherwise VERITAS_COVERAGE_HTML_DIR decides. An empty value, "false" or "0" turns the HTML report off.

Procedure: coverage-html-path KEY: #:html-dir

Return the path of the index page of the HTML report, or #f when off.

Procedure: coverage-include-paths KEY: #:include #:base

Return the prefixes a report is limited to.

An explicit include wins; otherwise VERITAS_COVERAGE_INCLUDE is read, and when neither is set the current directory is the boundary, which is what keeps Guile, Fibers and the other dependencies out of the report.

Procedure: coverage-jacoco-path KEY: #:dir

Return the path of the JaCoCo XML report of a run.

The VERITAS_JACOCO_PATH environment variable wins when it is set.

Procedure: coverage-json-path KEY: #:dir

Return the path of the JSON summary of a coverage run.

Procedure: coverage-lcov-path KEY: #:dir

Return the path of the merged LCOV file of a coverage run.

Procedure: coverage-path-included? path KEY: #:include #:exclude

Return #t when path belongs in a coverage report.

include is a list of directory prefixes; an empty list means every file. exclude is a list of prefixes that win over it, so the tests of a project can be left out of the coverage of the code they exercise.

Procedure: coverage-report-covered report

Return how many instrumented lines of report ran at least once.

Procedure: coverage-report-instrumented report

Return how many lines of report hold code that could be counted.

Procedure: coverage-report-line-count report path line

Return how often line of path ran in report, or #f.

path has to be the absolute path the report holds.

Procedure: coverage-report-percentage report

Return the percentage of report’s instrumented lines that ran.

Procedure: coverage-report-summary-string report KEY: #:title

Return report as the one line a run prints.

"Code coverage: 82.4% of 340 lines (280 covered) in 12 files"
Procedure: empty-coverage-report

Return a report that holds no file at all.

Procedure: lcov-string report

Return report as an LCOV trace file held in a string.

Procedure: make-coverage-file path lines

Return the coverage of path, given its lines.

lines is an alist of (line-number . execution-count). It is sorted by line number here, so every report has one shape however it was built.

Procedure: make-coverage-report files

Return a report holding files, sorted by path.

files may hold a number of coverage-file records for the same path; they are merged, keeping the highest count seen for each line, so a report read from several LCOV files can be handed to this directly.

Procedure: merge-coverage-reports . reports

Return one report holding every file of reports.

The line counts of a file present in several reports are merged by taking the highest count of each line: a line that ran in any of the runs was covered.

Procedure: merge-lcov-files paths

Return the reports of every file of paths merged into one.

A path that does not exist is skipped, so the caller can pass the files a run was expected to write without checking each one.

Procedure: normalize-coverage-path name KEY: #:base

Return the source file name points at.

The VM records source files the way the compiler saw them: an absolute path for a system library, a path relative to the current directory for a project file, or a path relative to one of Guile’s load path entries. Each is resolved here, in that order, so a report always holds absolute paths and two spellings of one file cannot end up as two rows.

Procedure: publish-coverage! report KEY: #:dir #:html-dir #:json-file #:cobertura-file #:jacoco-file #:sources #:log

Merge report into this run’s coverage and write the reports.

The report of the run so far is kept per destination directory and merged with report, so a process that runs several test files ends with all of them in one report. The merged report is written as LCOV, as HTML (unless html-dir or the environment turns it off), as JSON, and as the two XML formats the Java world reads: Cobertura and JaCoCo. sources names the roots the XML documents report the files relative to. log, when given, is called with a one-line summary and with the paths written.

Nothing is written when no file has been measured yet: an empty report is the absence of coverage, and a report of zero lines is not something to publish.

Returns the merged report.

Procedure: read-lcov port

Return the LCOV trace data read from port as a report.

Only SF and DA records carry information here; the other lines are accepted and ignored, so a file written by genhtml or by another tool can be read as well.

Procedure: read-lcov-file path

Return the LCOV trace file at path as a report.

A missing file is an empty report, which is what a run that produced no coverage looks like.

Procedure: regenerate-coverage-reports KEY: #:lcov-file #:html-dir #:json-file #:cobertura-file #:jacoco-file #:sources

Rebuild the HTML, JSON, Cobertura and JaCoCo reports from the last LCOV run.

lcov-file defaults to the report of the last run, reports/lcov/veritas.info, and sources names the roots the two XML documents report the files relative to (the include paths of the environment by default). Returns the report read back, or #f when no LCOV file has been written yet, in which case nothing is written.

Procedure: with-debug-vm thunk KEY: #:engine

Call thunk on a VM whose engine is engine, then put it back.

Guile fires the VM hooks that count instructions only in the debug engine, so this is what makes coverage - and any other hook - record anything at all. The engine that was current before is restored even when thunk raises, which keeps the switch invisible to the rest of the run.

The switch is bracketed with call-with-fiber-safe-wind, not with a plain dynamic-wind: a fiber that yields would otherwise put the old engine back in the middle of the measured run, and everything after the first suspension would be counted as not covered.

Procedure: write-cobertura-xml report KEY: #:path #:sources

Write report as a Cobertura XML document to path.

Cobertura is the format Jenkins, GitLab, Azure DevOps and SonarQube accept most readily. Every line carries its execution count, each file is a class, and a directory is a package. sources defaults to the source roots of the run, and the filenames are written relative to them.

path defaults to cobertura.xml next to the LCOV file; the VERITAS_COBERTURA_PATH environment variable overrides it. Returns the path written.

Procedure: write-coverage-json report KEY: #:path

Write a JSON summary of report to path.

The document holds the totals, and for each file its counts and the numbers of the lines that never ran, which is what a CI job needs to annotate a pull request without parsing HTML. path defaults to coverage-summary.json next to the LCOV file.

Procedure: write-html-report report KEY: #:directory #:title

Write report as one self-contained HTML page into directory.

The page holds the summary, the table of files and the annotated source of every file, so the whole project’s coverage is one document: nothing is fetched, nothing is split across pages, and it opens from a file:// URL. Source lines that never ran are marked, and the page can be filtered by path.

directory defaults to reports/html, or to whatever VERITAS_COVERAGE_HTML_DIR says. Returns the path of the page.

Procedure: write-jacoco-xml report KEY: #:path #:sources

Write report as a JaCoCo XML document to path.

JaCoCo’s format is what the coverage gutters of IDEs, SonarQube and Jenkins’ JaCoCo plugin read. Each file becomes a class and a source file, every line carries its missed and covered instruction counts, and the counters at class, package and report level let a tool compute the totals without reading a line.

path defaults to jacoco.xml next to the LCOV file; the VERITAS_JACOCO_PATH environment variable overrides it. Returns the path written.

Procedure: write-lcov report port

Write report to port in the LCOV trace file format.

Procedure: write-lcov-file report path

Write report to path as LCOV, creating the directory as needed.

Returns path, so a caller can report where the file went.


1.7 (veritas diff)

A short explanation of where two values differ, for an assertion that failed. The expect/got pair already says what the values were; this says which element or character went wrong, which is what a reader misses when the values are long.

1.7.1 Overview

  • render-diff — Return a short explanation of where expect and got differ.

1.7.2 Procedures

Procedure: render-diff expect got

Return a short explanation of where expect and got differ.

Only strings, lists and vectors are described; #f means there is nothing useful to add - the values are equal, or they are of a kind a positional diff says nothing about.

(render-diff '(1 2 3) '(1 2 9))
⇒ "  lists differ at index 2 (expected 3, got 9); lengths 3 and 3"

1.8 (veritas generator)

Generator combinators for the property-based testing subsystem. Every generator is a closure taking a random state, so the calling convention is always: (lambda (rand) ...). Sizes are picked inside the generator, which saves threading a size argument through the pipeline by hand.

A generator may also carry a shrinker: a procedure of one value that returns a list of smaller replacements, smallest first. A property uses it to walk a failing input down to the smallest value that still fails. Generators made by hand need no shrinker; they simply do not shrink.

1.8.1 Overview

  • generate-beer — Return a generator of beer styles.
  • generate-boolean — Return a generator of #t and #f.
  • generate-car — Return a generator of car makes and models.
  • generate-char — Return a generator of printable ASCII characters, 32 to 126.
  • generate-constant — Return a generator that always answers value.
  • generate-email — Return a generator of first.last@example.com email addresses.
  • generate-first-name — Return a generator of first names.
  • generate-integer — Return a generator of integers between 0 and max-bound (exclusive).
  • generate-last-name — Return a generator of last names.
  • generate-list — Return a generator of lists whose elements come from elem-generator.
  • generate-name — Return a generator of full names, a first name and a last name.
  • generate-natural — Return a generator of integers between 0 and max-bound inclusive.
  • generate-one-of — Return a generator that picks one of choices at random.
  • generate-real — Return a generator of real numbers in the range min to max.
  • generate-string — Return a generator of printable-ASCII strings.
  • generate-symbol — Return a generator of lowercase symbols, min-len to max-len long.
  • generate-tuple — Return a generator of lists holding one value from each of generators.
  • generate-vector — Return a generator of vectors whose elements come from elem-generator.
  • generator-shrink — Return smaller replacements for value taken from generator.

1.8.2 Procedures

Procedure: generate-beer

Return a generator of beer styles.

Procedure: generate-boolean

Return a generator of #t and #f.

Procedure: generate-car

Return a generator of car makes and models.

Procedure: generate-char

Return a generator of printable ASCII characters, 32 to 126.

Procedure: generate-constant value

Return a generator that always answers value.

Procedure: generate-email

Return a generator of first.last@example.com email addresses.

Procedure: generate-first-name

Return a generator of first names.

Procedure: generate-integer KEY: #:max-bound

Return a generator of integers between 0 and max-bound (exclusive).

Call the generator with a random state:

(define gen (generate-integer #:max-bound 100))
(gen (seed->random-state 42))
Procedure: generate-last-name

Return a generator of last names.

Procedure: generate-list elem-generator KEY: #:min-len #:max-len

Return a generator of lists whose elements come from elem-generator.

Lists are min-len to max-len long.

Procedure: generate-name

Return a generator of full names, a first name and a last name.

Procedure: generate-natural KEY: #:max-bound

Return a generator of integers between 0 and max-bound inclusive.

Procedure: generate-one-of choices

Return a generator that picks one of choices at random.

A choice is usually a constant, so there is nothing to shrink.

Procedure: generate-real KEY: #:min #:max

Return a generator of real numbers in the range min to max.

Procedure: generate-string KEY: #:min-len #:max-len

Return a generator of printable-ASCII strings.

Strings are min-len to max-len characters long.

Procedure: generate-symbol KEY: #:min-len #:max-len

Return a generator of lowercase symbols, min-len to max-len long.

Symbols stay unescaped because the characters are always a-z.

Procedure: generate-tuple generators

Return a generator of lists holding one value from each of generators.

Procedure: generate-vector elem-generator KEY: #:min-len #:max-len

Return a generator of vectors whose elements come from elem-generator.

Procedure: generator-shrink generator value

Return smaller replacements for value taken from generator.

The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.


1.9 (veritas impact)

1.9.1 Overview

1.9.2 Procedures

Procedure: affected-test-files files KEY: #:src-dir #:changed

Return the files from files that the changes in changed can affect.

A test is affected when it imports a module the change reaches, when the change touched the test file itself, when a changed file’s name matches the test’s name, or when the test names the changed file as a string: the last two cover a project whose tests name their subject instead of importing it, and the data a test cannot run without.

Procedure: changed-selection files KEY: #:since #:src-dir

Return the changed files and the subset of files they affect.

Procedure: git-changed-files KEY: #:since

Return the files a change touched, relative to the repository root.

Without since, the working tree and the index are compared with HEAD, and untracked files are included. With since, the working tree is compared with that revision, which is what a CI job wants: the diff of a branch against its base.

Procedure: module-imports text

Return the modules text imports with #:use-module, as lists.

Only the plain form is read - (veritas reporter) - because a #:select list belongs to the importer, not to the dependency.

Procedure: module-name-of-file path KEY: #:root

Return the module path defines, as a list of strings.

"src/veritas/reporter.scm" is ("veritas" "reporter"). root is the directory the path is relative to, so a project with a different layout can say so.

Procedure: referenced-paths text

Return the path-like string literals text mentions.

A test that reads a data file names it as a string, so those strings are the other dependency edge a project has: not a module, but a file the test cannot run without. Only strings that look like a path are kept - one with a slash, a dot or both - because every other literal in a test is prose.


1.10 (veritas mock)

Test doubles: spies, stubs and temporary bindings, in the spirit of a mocking library. Scheme needs less of one than most languages do - a module’s procedures can be passed in, so a fake is often just a lambda - but recording what was called, and checking it after the fact, is worth having.

(define send (make-spy real-send)) ; records every call (define clock (make-stub #:returns 0)) ; ignores its arguments

(test "the receipt is sent once" (lambda (vc) (send "ada" "thanks") ; the call under test ((assert-called-with send ’("ada" "thanks")) vc) ((assert-call-count send #:expect 1) vc)))

The assertions read the spy when they run, not when they are built, which is why they are written as a case lambda: a spy’s calls are test-time state.

A stub can also stand in for a module binding without editing the module, as long as the call site reads that binding when it runs - see ‘with-temporary-binding’ for exactly when that is.

1.10.1 Overview

  • assert-call-count — Assert that spy was called exactly expect times.
  • assert-called — Assert that spy was called at least once.
  • assert-called-with — Assert that spy was called with exactly args.
  • make-spy — Return a procedure that calls proc and records every call.
  • make-stub — Return a stand-in procedure that ignores its arguments and returns returns.
  • spy-call-count — Return how many times spy was called.
  • spy-called-with? — Return #t when spy was called with exactly args.
  • spy-calls — Return the argument lists spy was called with, oldest first.
  • spy-results — Return what spy returned, oldest first, one entry per call.
  • spy? — Return #t when value was made by make-spy or make-stub.
  • with-temporary-binding — Call thunk with module’s var set to value, and restore it.

1.10.2 Procedures

Procedure: assert-call-count spy KEY: #:expect #:name

Assert that spy was called exactly expect times.

Checked when the assertion runs, so it is written as a case lambda: (lambda (vc) ((assert-call-count spy #:expect 2) vc)).

Procedure: assert-called spy KEY: #:name

Assert that spy was called at least once.

Checked when the assertion runs, so it is written as a case lambda: (lambda (vc) ((assert-called spy) vc)).

Procedure: assert-called-with spy args KEY: #:name

Assert that spy was called with exactly args.

Checked when the assertion runs, so it is written as a case lambda: (lambda (vc) ((assert-called-with spy '(1 2)) vc)).

Procedure: make-spy proc KEY: #:name

Return a procedure that calls proc and records every call.

The recording is read back with spy-calls, spy-results and spy-call-count. name is only used in the messages of the assert-* checks below.

(define double (make-spy (lambda (x) (* x 2))))
(double 3)
(spy-calls double)
⇒ ((3))
Procedure: make-stub KEY: #:returns #:name

Return a stand-in procedure that ignores its arguments and returns returns.

A stub is a spy whose procedure ignores its arguments, so spy-calls, spy-call-count and the assert-* checks work on it unchanged.

Procedure: spy-call-count spy

Return how many times spy was called.

Procedure: spy-called-with? spy args

Return #t when spy was called with exactly args.

Procedure: spy-calls spy

Return the argument lists spy was called with, oldest first.

Procedure: spy-results spy

Return what spy returned, oldest first, one entry per call.

Procedure: spy? value

Return #t when value was made by make-spy or make-stub.

Procedure: with-temporary-binding module var value thunk

Call thunk with module’s var set to value, and restore it.

The previous value comes back even when thunk raises, so substituting a dependency is safe to try.

Guile resolves a call to a known procedure while it compiles, so a compiled call site does *not* see the change: a call written in a module the compiler can see through keeps calling the original procedure. Measured on Guile 3.0, swapping a binding is observed by module-ref, by code passed to eval, and by call sites in a module declared #:declarative? #f; it is not observed by a call compiled in a declarative module, which is the default.

That leaves the ordinary Scheme answer, which needs no substitution at all: pass the dependency in, or hand the caller a make-spy recording. Use this when the code under test reaches the binding at run time.

(with-temporary-binding (resolve-module '(my app)) 'now
                        (lambda () 0)
                        (lambda ()
                          (assert-equal #:expect 0
                                        #:got (eval '((@ (my app) now))
                                                    (resolve-module '(my app))))))

1.11 (veritas model)

1.11.1 Overview

1.11.2 Macros

Macro: assertion-result-index x

Undocumented macro.

Macro: assertion-result-name x

Undocumented macro.

Macro: assertion-result-output x

Undocumented macro.

Macro: assertion-result-successful x

Undocumented macro.

Macro: assertion-result? x

Undocumented macro.

Macro: benchmark-result-name x

Undocumented macro.

Macro: benchmark-result-results x

Undocumented macro.

Macro: benchmark-result? x

Undocumented macro.

Macro: make-assertion-result x

Undocumented macro.

Macro: make-benchmark-result x

Undocumented macro.

Macro: make-raw-pre-test-result x

Undocumented macro.

Macro: make-result-and-output x

Undocumented macro.

Macro: make-suite-result x

Undocumented macro.

Macro: make-test-result x

Undocumented macro.

Macro: pre-test-result-attempts x

Undocumented macro.

Macro: pre-test-result-duration x

Undocumented macro.

Macro: pre-test-result-file x

Undocumented macro.

Macro: pre-test-result-line x

Undocumented macro.

Macro: pre-test-result-name x

Undocumented macro.

Macro: pre-test-result-results x

Undocumented macro.

Macro: pre-test-result? x

Undocumented macro.

Macro: result-and-output-output x

Undocumented macro.

Macro: result-and-output-result x

Undocumented macro.

Macro: result-and-output? x

Undocumented macro.

Macro: set-suite-result-output! x

Undocumented macro.

Macro: suite-result-name x

Undocumented macro.

Macro: suite-result-output x

Undocumented macro.

Macro: suite-result-test-results x

Undocumented macro.

Macro: suite-result? x

Undocumented macro.

Macro: test-result-index x

Undocumented macro.

Macro: test-result-output x

Undocumented macro.

Macro: test-result-results x

Undocumented macro.

Macro: test-result? x

Undocumented macro.

1.11.3 Procedures

Procedure: make-pre-test-result name results KEY: #:file #:line #:duration #:attempts

Return the result of a test named name holding results.

file and line are where the test was declared, when that is known, duration is how many seconds it took, and attempts is how many times the test ran - more than one when a retry was needed. They are what a report needs to place a test and time it: the JUnit document writes them out, and a CI annotation uses the file and line to attach a failure to the right place in the source.

1.11.4 Record Types

Record type: assertion-result

This record type has the following fields:

  • index
  • output
  • name
  • successful
Record type: benchmark-result

This record type has the following fields:

  • name
  • results
Record type: pre-test-result

This record type has the following fields:

  • name
  • results
  • file
  • line
  • duration
  • attempts
Record type: result-and-output

This record type has the following fields:

  • result
  • output
Record type: suite-result

This record type has the following fields:

  • name
  • test-results
  • output
Record type: test-result

This record type has the following fields:

  • index
  • output
  • results

1.12 (veritas prelude)

1.12.1 Overview

1.12.2 Macros

Macro: evaluate-thunks x

Evaluate thunks one after another, keeping their order.

The result list lines up with the input list.

Macro: evaluate-thunks-concurrently x

Run thunks concurrently in fibers and collect what they return.

Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.

(evaluate-thunks-concurrently
 (list (lambda () (sleep 1) 'delayed)
       (lambda () 'instant)))
⇒ (delayed instant)

1.12.3 Parameters

Parameter: current-collector

Default value:

#f

1.12.4 Procedures

Procedure: add-result-counts a b

Return the sum of the count alists a and b.

Procedure: call-with-fiber-safe-wind before thunk after

Call thunk between before and after, safely under fibers.

Fibers suspends a task by unwinding to the scheduler and resumes it by winding back in, so a plain dynamic-wind runs its guards again every time the task yields - and a port operation is enough to yield. A guard that has a side effect - a module binding swapped, a directory removed, a VM engine changed - would then fire in the middle of the thunk. dynamic-wind* from (fibers scheduler) skips the guards during those round trips and runs them when the extent is really entered and really left, which is what every bracketing helper here wants.

(call-with-fiber-safe-wind
 (lambda () (module-set! module 'now 0))
 (lambda () (read-the-clock))
 (lambda () (module-set! module 'now original)))
Procedure: call-with-timeout seconds thunk KEY: #:on-timeout

Run thunk with an optional deadline of seconds seconds.

When seconds is a positive real number the work runs in its own fiber and, if it has not finished by then, the fiber is aborted and the result of calling on-timeout is returned instead. When seconds is #f (the default, meaning no deadline) thunk is simply called and its result is returned.

thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.

(call-with-timeout 5
                   (lambda () (expensive-computation))
                   #:on-timeout (lambda () 'gave-up))
Procedure: capture-result-and-output thunk

Call thunk and return a result-and-output record.

The record holds what thunk returned together with everything it printed, so a test’s output can be reported next to its result.

Procedure: collect-assertion-successful xs

Return the outcome of every assertion in xs as a flat list.

Booleans only: #t for a pass, #f for a failure.

Procedure: collect-assertion-successful* xs

Return the assertion outcomes found inside xs.

Knows assertion, pre-test, test and suite results, and recurses into lists. Unspecified values, benchmarks and #t contribute nothing; an unrecognized type raises.

Procedure: collect-in! item

Record item in the active collector, and return item.

Does nothing else when no collector is bound, which is what building a case outside a body - at load time, or inside an assertion that runs later - looks like. #f is never recorded: a case a filter dropped is not a case.

Procedure: collector-items collector

Return what collector recorded, in the order the items were built.

Procedure: collector? value

Return #t when value is a collector made by make-collector.

Procedure: deduplicate-values values

Return values without the entries that are the same object as an earlier one.

Identity, not equality, decides: the same case built twice is two cases, while the same case both recorded and returned by one form is one.

Procedure: display-all-results vc all-results

Print and log every test outcome inside all-results using reporter client vc.

Iterates over the compilation of all-results. Evaluates types dynamically:

  • suite-result: Displays suite output, followed by inner test results.
  • pre-test-result: Logs the test identity via vc and displays inner assertions.
  • benchmark-result: Dispatches to log-benchmark.

Throws an explicit error if an unrecognized result type is processed.

Procedure: display-assertion-results assertion-results

Print the captured output of each record in assertion-results.

A record that is not an assertion gets printed as an error instead of bringing down the reporter.

Procedure: flatten-cases value case? expand OPT: depth

Return the cases value stands for, in order, leaving out what is not one.

case? recognises a case, expand evaluates a thunk-shaped body form, lists (of any depth) are walked, and anything else - an unspecified value, a number, a nested task in a place that wants assertions - contributes nothing. depth bounds the recursion, so a thunk that keeps returning thunks is a value that is ignored rather than a run that never ends.

Procedure: flatten-list xs

Flatten xs into a single list, however deeply it nests.

An atom that is not a list comes back wrapped in a one-element list.

Procedure: getenv-or var KEY: #:default

Return environment variable var, or default when it is unset.

var is a symbol. Anything that goes wrong while reading it yields default as well.

Procedure: make-collector

Return a fresh collector: a mutable list head of recorded items.

Procedure: make-timeout-assertion-result KEY: #:vc #:name #:timeout

Build the failed assertion-result reported for a timeout.

name is the timed task, timeout the seconds it was given and vc the client used to report it. A timeout fails the run exactly like any other failed assertion.

Procedure: nullary-procedure? proc

Return #t when proc can be called with no arguments at all.

A body form is wrapped in such a thunk by the DSL, and a thunk of this shape is therefore a body still to be evaluated rather than a case to run.

Procedure: procedure-required-arity proc

Return how many arguments proc needs, or #f when that is unknown.

Guile reports a procedure’s arity as the list (required optional rest?), or as several values on older versions; both are read here.

Procedure: result-counts result KEY: #:files

Return the files, suites, tests and assertions result records.

result is a result record or a list of them, at any depth: an assertion counts as one assertion, a test as one test, a suite as one suite, and a suite or a test that holds further results counts what it holds as well. files is how many test files the results stand for - one for the run of a single file - and is added to the files count.

A benchmark measures instead of asserting, and a task that raised arrives as the error string, so neither contributes a test or an assertion of its own.

(result-counts (list a-suite a-test) #:files 1)
⇒ ((files . 1) (suites . 1) (tests . 3) (assertions . 7))
Procedure: result-counts-summary counts

Return counts as one readable line.

(result-counts-summary '((files . 2) (suites . 3) (tests . 12) (assertions . 48)))
⇒ "2 files, 3 suites, 12 tests and 48 assertions"

1.12.5 Variables

Variable: empty-result-counts
((files . 0) (suites . 0) (tests . 0) (assertions . 0) (flaky . 0))

1.13 (veritas property)

Properties: one assertion, many generated inputs. A property draws a value from each of its generators, calls its thunk, and repeats; a thunk that fails for one input makes the property fail, and the failing input is shrunk to the smallest value that still fails before it is reported. The seed is reported too, so a failure can be replayed with VERITAS_SEED.

1.13.1 Overview

  • property — Return a property named name, a task like a test.

1.13.2 Procedures

Procedure: property name KEY: #:generators #:thunk #:runs #:seed #:shrink?

Return a property named name, a task like a test.

generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.

(property \"reverse of reverse is the original list\"
          #:generators (list (generate-list (generate-integer)))
          #:thunk (lambda (xs)
                    (assert-equal #:expect xs
                                  #:got (reverse (reverse xs)))))

A failing input is shrunk to the smallest value from the same generators that still fails, and both values are reported. seed makes the draw reproducible; without one, VERITAS_SEED is used, and failing that a fresh seed is drawn and reported with the failure.


1.14 (veritas reporter)

1.14.1 Overview

1.14.2 Macros

Macro: log x

Print msg on a line of its own.

(log "Hello World")
;; Outputs: \nHello World
Macro: log-assert x

Announce an assertion that is about to be checked, by name.

Macro: log-benchmark x

Report benchmark results, given a benchmark-result record.

Macro: log-debug x

Log a message for debugging.

Macro: log-error x

Report a runtime error.

Macro: log-fail x

Report a failing check.

Macro: log-flaky x

Report a test that passed only after a retry.

Macro: log-info x

Log an informational message.

Macro: log-pass x

Report a passing check.

Macro: log-suite x

Announce a suite by name.

Macro: log-test x

Announce a test by name.

1.14.3 Parameters

Parameter: veritas-clock

Default value:

#<procedure 7f5add991190 at veritas/reporter.scm:331:18 ()>
Parameter: veritas-current-location

Default value:

#f

1.14.4 Procedures

Procedure: append-github-summary-row counts KEY: #:name #:summary-path

Append one row of counts to the GitHub Actions step summary.

The table header is written the first time, and every test file of the run adds a row, so the summary shows a file beside its numbers and still says something when the run is failing. Returns summary-path, or #f when there is no summary to append to.

Procedure: dispatch-log vc log-type msg

Send msg to the reporter chosen in vc.

The client names one under the key 'reporter; anything else falls back to emoji-reporter. log-type picks the entry to call ('log-pass, 'log-fail, and so on).

Procedure: ensure-parent-directories path

Create every directory in front of path, ignoring ones that exist.

Procedure: github-actions?

Return #t when the run is inside a GitHub Actions workflow.

Procedure: junit-report-data all-results KEY: #:seconds

Return the JUnit report of all-results as data rather than a file.

The result is an alist holding the numbers a testsuites element needs - tests, failures, errors and skipped - and xml, the testsuite elements themselves as a string. Writing the file is write-junit-document’s job; a separate process can be handed this alist and its run joined to others with merge-junit-data, which is how one document is made out of the parallel runs of --jobs.

Procedure: merge-junit-data datas

Combine the JUnit data of several runs into one.

The counts are summed and the testsuite elements are concatenated in the order given, so the merged document keeps the order the runs were scheduled in.

Procedure: now

Undocumented procedure.

Procedure: read-junit-data path

Return the JUnit data recorded at path, or #f.

#f means there is nothing to merge: the file is missing, unreadable or not the alist write-junit-data writes.

Procedure: write-github-annotations all-results KEY: #:limit

Print workflow-command annotations for the failures in all-results.

GitHub keeps ten error annotations per step, so only the first limit are written and a notice says how many were left out; the JUnit document carries every one of them. Returns how many annotations were written.

Procedure: write-junit-data data KEY: #:path

Write JUnit data to path so another process can read it back.

An alist is plain Scheme data, so read restores it exactly: this is how a worker process hands its report to the process that merges them. Returns path.

Procedure: write-junit-document data KEY: #:seconds #:path

Write JUnit data, as made by junit-report-data, to path.

seconds is the wall clock of the run and becomes the document’s total time; for a run split over worker processes it is the wall clock of the whole parallel run, not the sum of the workers. Returns path.

Every testcase carries the file it came from, the suite path it sat in and its own duration; each testsuite sums the durations of its cases.

Procedure: write-junit-report all-results KEY: #:seconds #:path

Write all-results to path as one JUnit XML document.

seconds is the wall clock of the run and becomes the document’s total time. Every test becomes a testcase inside a testsuite named after the suites it sits in; loose tests land in a suite called veritas. A benchmark is reported as skipped, because it measures instead of asserting, and a task that raised is reported as an error rather than a failure. Returns path.

Every testcase carries the file it came from, the suite path it sat in and its own duration; each testsuite sums the durations of its cases.

1.14.5 Variables

Variable: ansi-color-escapes
((yellow . "\x1b[0;33m") (magenta . "\x1b[0;35m") (green . "\x1b[0;32m") (default . "\x1b[0;39m") (bold . "\x1b[1m") (red . "\x1b[0;31m") (blue . "\x1b[0;34m") (cyan . "\x1b[0;36m") (reset . "\x1b[0m"))
Variable: emoji-reporter
((reporter-name . emoji-reporter) (log-flaky . #<procedure 7f5add991000 at veritas/reporter.scm:145:13 (msg)>) (log-pass . #<procedure 7f5add991010 at veritas/reporter.scm:149:47 (msg)>) (log-fail . #<procedure 7f5add991020 at veritas/reporter.scm:159:14 (msg)>) (log-debug . #<procedure 7f5add991030 at veritas/reporter.scm:166:15 (msg)>) (log-info . #<procedure 7f5add991040 at veritas/reporter.scm:173:14 (msg)>) (log-error . #<procedure 7f5add991050 at veritas/reporter.scm:180:15 (msg)>) (log-test . #<procedure 7f5add991060 at veritas/reporter.scm:187:14 (msg)>) (log-benchmark . #<procedure 7f5add991070 at veritas/reporter.scm:200:19 (msg)>) (log-suite . #<procedure 7f5add991080 at veritas/reporter.scm:234:15 (msg)>) (log-assert . #<procedure 7f5add991090 at veritas/reporter.scm:248:16 (msg)>))
Variable: github-reporter
((reporter-name . github-reporter) (log-flaky . #<procedure junit-quiet (msg)>) (log-pass . #<procedure junit-quiet (msg)>) (log-assert . #<procedure junit-quiet (msg)>) (log-test . #<procedure junit-quiet (msg)>) (log-suite . #<procedure junit-quiet (msg)>) (log-debug . #<procedure junit-quiet (msg)>) (log-benchmark . #<procedure junit-quiet (msg)>) (log-fail . #<procedure 7f5add991620 at veritas/reporter.scm:955:14 (msg)>) (log-error . #<procedure 7f5add991630 at veritas/reporter.scm:958:15 (msg)>) (log-info . #<procedure 7f5add991640 at veritas/reporter.scm:961:14 (msg)>))
Variable: json-reporter
((reporter-name . json-reporter) (log-flaky . #<procedure 7f5add6f7a00 at veritas/reporter.scm:343:14 (msg)>) (log-pass . #<procedure 7f5add6f79e0 at veritas/reporter.scm:347:46 (msg)>) (log-fail . #<procedure 7f5add6f79c0 at veritas/reporter.scm:356:14 (msg)>) (log-error . #<procedure 7f5add6f79a0 at veritas/reporter.scm:362:15 (msg)>) (log-test . #<procedure 7f5add6f7980 at veritas/reporter.scm:368:14 (msg)>) (log-benchmark . #<procedure 7f5add6f7960 at veritas/reporter.scm:374:19 (msg)>) (log-debug . #<procedure 7f5add6f7940 at veritas/reporter.scm:401:15 (msg)>) (log-info . #<procedure 7f5add6f7920 at veritas/reporter.scm:407:14 (msg)>) (log-suite . #<procedure 7f5add6f7900 at veritas/reporter.scm:413:15 (msg)>) (log-assert . #<procedure 7f5add6f78e0 at veritas/reporter.scm:419:16 (msg)>))
Variable: junit-reporter
((reporter-name . junit-reporter) (log-flaky . #<procedure junit-quiet (msg)>) (log-pass . #<procedure junit-quiet (msg)>) (log-assert . #<procedure junit-quiet (msg)>) (log-test . #<procedure junit-quiet (msg)>) (log-suite . #<procedure junit-quiet (msg)>) (log-debug . #<procedure junit-quiet (msg)>) (log-benchmark . #<procedure junit-quiet (msg)>) (log-fail . #<procedure 7f5add991518 at veritas/reporter.scm:828:14 (msg)>) (log-error . #<procedure 7f5add991528 at veritas/reporter.scm:831:15 (msg)>) (log-info . #<procedure 7f5add991538 at veritas/reporter.scm:834:14 (msg)>))
Variable: monotone-reporter
((reporter-name . monotone-reporter) (log-flaky . #<procedure 7f5add9910d8 at veritas/reporter.scm:263:14 (msg)>) (log-pass . #<procedure 7f5add9910e8 at veritas/reporter.scm:265:50 (msg)>) (log-fail . #<procedure 7f5add9910f8 at veritas/reporter.scm:269:14 (msg)>) (log-error . #<procedure 7f5add991108 at veritas/reporter.scm:274:15 (msg)>) (log-test . #<procedure 7f5add991118 at veritas/reporter.scm:279:14 (msg)>) (log-benchmark . #<procedure 7f5add991128 at veritas/reporter.scm:284:19 (msg)>) (log-debug . #<procedure 7f5add991138 at veritas/reporter.scm:304:15 (msg)>) (log-info . #<procedure 7f5add991148 at veritas/reporter.scm:309:14 (msg)>) (log-suite . #<procedure 7f5add991158 at veritas/reporter.scm:314:15 (msg)>) (log-assert . #<procedure 7f5add991168 at veritas/reporter.scm:320:16 (msg)>))

1.15 (veritas runner)

1.15.1 Overview

1.15.2 Procedures

Procedure: discover-tests KEY: #:test-type #:scan-dir #:test-top-dir #:test-module #:dir

Walk the cleaned file tree dir and collect candidate test files.

Descends through scan-dir, test-top-dir and test-module, then hands the contents of a folder named test-type to flatten-test-files.

Procedure: display-result-counts counts

Print the global totals of a run: its files, suites, tests and assertions.

Procedure: merge-worker-counts indexes

Return the totals of the workers at indexes added up.

A worker that left no counts - it failed before its file ran - contributes nothing, because there is nothing to count.

Procedure: merge-worker-coverage indexes

Return the coverage of the workers at indexes merged into one report.

A worker that was not asked for coverage left no fragment, and a run without coverage merges to an empty report, which is then left alone.

Procedure: parse-jobs value

Return the number of worker processes value asks for, or #f.

#f or an absent value means the run stays in one process. A value that is not a whole number of processes is an error rather than a silent fallback, because a CI job that asked for parallelism and got none should be told.

(parse-jobs "4")
⇒ 4
Procedure: parse-partition value

Return the (index . total) a partition string describes, or #f.

A partition is written "2/3": this machine runs the second of three slices. A malformed or impossible partition raises instead of quietly running everything, because that would hide a misconfigured CI job.

Procedure: partition-files files index total

Return the files of partition index of total from files.

The split is round-robin over files, so the same project always splits the same way and the partitions together are exactly files.

(partition-files '("a" "b" "c" "d" "e") 2 3)
⇒ ("b" "e")
Procedure: run-test-file KEY: #:test-file #:test-type #:entrypoint-function

Load one test file and call its entrypoint.

test-file is a path relative to the test root and test-type is the module prefix, so "math/algebra.scm" with unit loads (veritas unit math algebra). entrypoint-function names the zero-argument procedure to call in that module.

Procedure: run-test-file-guarded KEY: #:test-file #:test-type #:entrypoint-function

Run one test file, returning #t when it passed and #f when not.

veritas-run reports a failing file by exiting the process, and a file that cannot even be loaded raises; either would end the whole single-process run and take the files after it, and the totals of the run, with it. Both are caught here and turned into a returned answer, with what went wrong printed against the file.

Procedure: run-test-files files KEY: #:test-type #:entrypoint-function #:junit-data-paths

Run files one after another in this process, returning the failed ones.

A file that fails no longer ends the run: the files after it still run, so the run reports every failure and its global totals before it exits, which is what --jobs already did for the files it spread over workers.

When junit-data-paths is given, the ith file leaves its JUnit report at that path as data instead of writing it as the run’s document, so the caller can merge one document out of all of them.

Procedure: run-tests

Find every test file under the scan root and run each one.

Behaviour comes from the environment:

  • VERITAS_SCAN_DIR: Root folder to scan. Defaults to "./".
  • VERITAS_TEST_TOP_DIR: The subfolder boundary identifier containing test suites. Defaults to "test".
  • VERITAS_ENTRYPOINT: The signature method executed inside test targets. Defaults to "spec".
  • VERITAS_TEST_MODULE: Top level test parent folder filter. Defaults to "veritas".
  • VERITAS_TEST_TYPE: Category discriminator keyword used during routing. Defaults to "unit".
  • VERITAS_SHUFFLE_FILES: Set to string "true" to shuffle execution sequences using a Fisher-Yates algorithm. Defaults to "true".
  • VERITAS_SEED: the seed every shuffle of this run uses, reported when a run is shuffled so a failing order can be replayed.
  • VERITAS_PARTITION: INDEX/TOTAL to run only one slice of the discovered files, which is how a run is split across CI machines.
  • VERITAS_JOBS: how many worker processes to run the discovered files in. Absent or 1 means one process, as before; 2 and up run one file per process and merge their output, their record of failures and their JUnit reports.
  • VERITAS_COVERAGE: "true" measures code coverage even for a test file that does not ask for it, and the worker fragments are merged into one report.
Procedure: select-changed-files files KEY: #:test-type #:changed? #:since #:src-dir

Return the files of files a change can reach.

Without changed? or since every file comes back, and the run is the whole suite it always was. With them, the changed files are read from git, the modules they reach are followed through imports, and only the test files that import one of those modules - or that the change touched - come back.

A change that touches nothing returns every file rather than none: a run is never silently empty, because an empty run looks exactly like a passing one.


1.16 (veritas shuffle)

1.16.1 Overview

  • fisher-yates-shuffle — Return the elements of xs in random order (Fisher-Yates).
  • fresh-seed — Return a new seed for a shuffle, drawn from the platform’s entropy source.

1.16.2 Procedures

Procedure: fisher-yates-shuffle xs KEY: #:seed

Return the elements of xs in random order (Fisher-Yates).

Shuffling happens on a vector copy, so xs is left alone. seed makes the order reproducible, which is how a failing order is replayed; without it a fresh seed is pulled from random-state-from-platform on every call, so two calls on the same list differ. Lists of one element or fewer come back as they are.

(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
        (fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Procedure: fresh-seed

Return a new seed for a shuffle, drawn from the platform’s entropy source.

Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.


1.17 (veritas snapshot)

Snapshot testing: compare a value with the one recorded the first time the check ran. A missing snapshot is written and the check passes, so a new test does not fail merely for being new. A mismatch fails and shows both sides, and VERITAS_UPDATE_SNAPSHOTS=true rewrites every snapshot the run reaches, which is how an intended change is accepted.

A snapshot is Scheme data, read back with ‘read’: lists, vectors, strings, numbers, symbols and alists travel exactly. Procedures, records and ports do not, so snapshot the string they print instead.

1.17.1 Overview

  • assert-snapshot — Assert that got matches the snapshot recorded under name.
  • snapshot-path — Return the file holding the snapshot called name.
  • snapshot-update? — Return #t when snapshots are to be rewritten instead of compared.

1.17.2 Procedures

Procedure: assert-snapshot KEY: #:got #:name #:dir

Assert that got matches the snapshot recorded under name.

name labels the assertion and names the file, so it is required; the file lives under VERITAS_SNAPSHOT_DIR (default test/snapshots) or under dir. The first run writes the snapshot and passes. Later runs compare with equal?, and a mismatch fails with both values. A snapshot that cannot be read is a failure too, never a silent rewrite.

A got that is a procedure is called with no arguments when the assertion runs, so state that only exists at test time can be snapshotted.

(assert-snapshot #:name "the rendered cart"
                 #:got (lambda () (render (make-cart))))
Procedure: snapshot-path name KEY: #:dir

Return the file holding the snapshot called name.

dir overrides VERITAS_SNAPSHOT_DIR, which itself defaults to test/snapshots.

Procedure: snapshot-update?

Return #t when snapshots are to be rewritten instead of compared.

Set VERITAS_UPDATE_SNAPSHOTS to true to accept the current value of every snapshot a run reaches.


1.18 (veritas table)

Table-driven tests, in the spirit of pytest’s parametrize, JUnit’s parameterized tests and Go’s table tests. One test definition is expanded over a list of cases and each case is reported as its own test, so a failure points at the exact input.

1.18.1 Overview

  • cases->table — Index cases by key, giving an alist of key to case.
  • suite-each — Like test-each, but case->suite returns a suite.
  • test-each — Expand cases into one test per case, under a suite named group-name.

1.18.2 Procedures

Procedure: cases->table cases key

Index cases by key, giving an alist of key to case.

Handy for tables with named columns.

Procedure: suite-each group-name cases case->suite KEY: #:shuffle? #:concurrent?

Like test-each, but case->suite returns a suite.

Use it when one case needs several tests rather than one.

Procedure: test-each group-name cases case->test KEY: #:shuffle? #:concurrent?

Expand cases into one test per case, under a suite named group-name.

case->test takes one case and returns a test:

(test-each "integer addition"
           '((0 0 0) (1 2 3) (10 20 30))
           (lambda (case)
             (let ((a (list-ref case 0))
                   (b (list-ref case 1))
                   (expected (list-ref case 2)))
               (test (format #f "~a + ~a = ~a" a b expected)
                 (assert-equal #:expect expected #:got (+ a b))))))

1.19 (veritas trace)

1.19.1 Overview

1.19.2 Parameters

Parameter: veritas-trace-frames

Default value:

()
Parameter: veritas-tracing?

Default value:

#f

1.19.3 Procedures

Procedure: read-trace KEY: #:path

Return the frames stored at path, oldest first.

Procedure: render-trace KEY: #:path #:port

Print the trace at path as a numbered timeline.

Procedure: show-last-trace

Print the trace of the last run, and return whether there was one.

Procedure: trace-event! kind name OPT: detail

Record one step of the run when tracing is on.

kind is a symbol such as test-start, assertion or test-end, name is what the step is about, and detail is a short string a reader can use to tell one frame from another.

Procedure: trace-file

Return where the last trace of this run is written.

Procedure: trace-reset!

Forget the frames recorded so far.

Procedure: write-trace! KEY: #:path

Write the frames recorded so far to path and return it.

The newest frame is last in the file, so the timeline reads top to bottom.


1.20 (veritas veritas)

1.20.1 Overview

1.20.2 Macros

Macro: assert-match x

Undocumented macro.

Macro: convenit x

Undocumented macro.

Macro: evaluate-thunks-concurrently x

Run thunks concurrently in fibers and collect what they return.

Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.

(evaluate-thunks-concurrently
 (list (lambda () (sleep 1) 'delayed)
       (lambda () 'instant)))
⇒ (delayed instant)
Macro: lazy x

Undocumented macro.

Macro: lazy-value? x

Undocumented macro.

Macro: suite stx

Undocumented macro.

Macro: test stx

Undocumented macro.

1.20.3 Parameters

Parameter: veritas-coverage-data-path

Default value:

#f
Parameter: veritas-current-file

Default value:

#f
Parameter: veritas-junit-data-path

Default value:

#f
Parameter: veritas-last-failed-path

Default value:

"reports/last-failed.scm"
Parameter: veritas-only-failed

Default value:

#f
Parameter: veritas-result-counts

Default value:

((files . 0) (suites . 0) (tests . 0) (assertions . 0) (flaky . 0))
Parameter: veritas-result-counts-data-path

Default value:

#f
Parameter: veritas-retries

Default value:

0

1.20.4 Procedures

Procedure: %suite name KEY: #:debug? #:shuffle? #:skip? #:timeout #:tags #:concurrent? #:before-all #:after-all #:source . tests

Define a suite named name; the procedure behind the suite macro.

The arguments after name are the children. Each form is evaluated when the suite runs, and every test, suite or benchmark built while it is evaluated is a child: one already built, a list of them, or a thunk of them are all understood. Children run concurrently unless concurrent? is #f, and in shuffled order unless shuffle? is #f. before-all and after-all run once around them, timeout caps the whole suite in seconds including those hooks, skip? drops it from the run and tags decide which filter may select it.

Procedure: %test name KEY: #:debug? #:shuffle? #:skip? #:timeout #:tags #:before #:after #:source #:retries . cases

Define a test named name; the procedure behind the test macro.

The arguments after name are the body. Each form is evaluated when the test runs, and every assertion built while it is evaluated is a case: an assertion, a list of them, a thunk of them, or a helper procedure that builds them are all understood. before and after run around the body, timeout caps the whole test in seconds, hooks included, skip? drops it from the run and tags decide which filter may select it.

A body is read in order, because a body is a sequence: one form prepares what the next one checks. shuffle? asks for the order of the body forms to be randomized anyway, which is a way to find checks that depend on each other by accident.

retries reruns a failing test, hooks and body together, up to that many extra attempts. A test that fails and then passes is reported as flaky with its attempt count, so an intermittent failure is visible instead of hidden.

(test "Addition Validation"
      #:before (lambda (vc) (display "Initializing..."))
      (assert-equal #:expect 4 #:got (+ 2 2)))
Procedure: aequalis KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to expect.

The usual choice for lists, strings, vectors and other compound values.

Procedure: aequivalens KEY: #:expect #:got #:name #:timeout

Assert that got is eqv? to expect.

Good for numbers, characters and symbols, where value equivalence is what matters.

Procedure: assert-approximate KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got is within epsilon of expect.

The default epsilon is 1.0e-8.

Procedure: assert-contains KEY: #:expect #:got #:name #:timeout

Assert that the container expect contains got.

Works for strings (substring), lists and vectors (membership) and hash tables (key membership).

(assert-contains #:expect '(1 2 3) #:got 2)
(assert-contains #:expect "hello world" #:got "world")
Procedure: assert-empty KEY: #:got #:name #:timeout

Assert that got is an empty list, vector or string.

Procedure: assert-eq KEY: #:expect #:got #:name #:timeout

Assert that got is eq? to expect.

The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.

Procedure: assert-equal KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to expect.

The usual choice for lists, strings, vectors and other compound values.

Procedure: assert-eqv KEY: #:expect #:got #:name #:timeout

Assert that got is eqv? to expect.

Good for numbers, characters and symbols, where value equivalence is what matters.

Procedure: assert-error thunk KEY: #:name #:timeout

Return an assertion that thunk raises.

It fails if thunk returns normally. name labels it in the report and timeout caps it in seconds, as in assert.

(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
              #:name "Should fail when list contains non-numbers")
Procedure: assert-false x KEY: #:name #:timeout

Assert that x is #f.

Procedure: assert-in-range KEY: #:min #:max #:got #:name #:timeout

Assert that got is within the inclusive range min–max.

Procedure: assert-length KEY: #:expect #:got #:name #:timeout

Assert that got has expect elements (list, vector or string length).

Procedure: assert-matches KEY: #:pattern #:got #:name #:timeout

Assert that the string got matches pattern.

pattern is a string or a compiled regexp.

(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Procedure: assert-no-error thunk KEY: #:name #:timeout

Return an assertion that thunk does not raise.

It fails if thunk raises. name labels it in the report and timeout caps it in seconds, as in assert.

Procedure: assert-not-approximate KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got differs from expect by at least epsilon.

Inverse of assert-approximate.

Procedure: assert-not-contains KEY: #:expect #:got #:name #:timeout

Assert that the container expect does not contain got.

Inverse of assert-contains.

Procedure: assert-not-empty KEY: #:got #:name #:timeout

Assert that got is a non-empty list, vector or string.

Procedure: assert-not-eq KEY: #:expect #:got #:name #:timeout

Assert that got is not eq? to expect.

Inverse of assert-eq.

Procedure: assert-not-equal KEY: #:expect #:got #:name #:timeout

Assert that got is not equal? to expect.

Inverse of assert-equal.

Procedure: assert-not-eqv KEY: #:expect #:got #:name #:timeout

Assert that got is not eqv? to expect.

Inverse of assert-eqv.

Procedure: assert-not-matches KEY: #:pattern #:got #:name #:timeout

Assert that string got does not match regular expression pattern.

Inverse of assert-matches.

Procedure: assert-one-of KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to one of the elements of list expect.

Procedure: assert-raises thunk KEY: #:on #:predicate #:name #:timeout

Assert that calling thunk raises an error.

By default any error satisfies the assertion. Pass on to require a specific error key (for example 'wrong-type-arg), or predicate, a procedure of (key args) returning #t for the expected error.

(assert-raises (lambda () (/ 1 0)))
(assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Procedure: assert-true x KEY: #:name #:timeout

Assert that x is #t.

Procedure: benchmark KEY: #:skip? #:name #:runs #:thunk #:timeout

Define a benchmark named name that times thunk over runs runs.

Statistics come from Guile’s statprof, so benchmarks are reported separately from tests. timeout caps all the runs together in seconds and skip? drops the benchmark from the run.

Procedure: call-with-timeout seconds thunk KEY: #:on-timeout

Run thunk with an optional deadline of seconds seconds.

When seconds is a positive real number the work runs in its own fiber and, if it has not finished by then, the fiber is aborted and the result of calling on-timeout is returned instead. When seconds is #f (the default, meaning no deadline) thunk is simply called and its result is returned.

thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.

(call-with-timeout 5
                   (lambda () (expensive-computation))
                   #:on-timeout (lambda () 'gave-up))
Procedure: collect-assertion-successful xs

Return the outcome of every assertion in xs as a flat list.

Booleans only: #t for a pass, #f for a failure.

Procedure: congruit KEY: #:pattern #:got #:name #:timeout

Assert that the string got matches pattern.

pattern is a string or a compiled regexp.

(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Procedure: continet KEY: #:expect #:got #:name #:timeout

Assert that the container expect contains got.

Works for strings (substring), lists and vectors (membership) and hash tables (key membership).

(assert-contains #:expect '(1 2 3) #:got 2)
(assert-contains #:expect "hello world" #:got "world")
Procedure: current-seed KEY: #:seed

Return the seed this run shuffles with, choosing one when none was set.

seed (a number or a numeric string) wins when given, otherwise the value of VERITAS_SEED is used, and when neither is set a fresh seed is drawn from the platform. The chosen seed is remembered, so the order the runner gives the test files and every order chosen here come from one seed, and the run can be replayed by setting VERITAS_SEED to the seed that was reported.

(current-seed #:seed 7)
⇒ 7
Procedure: defer-body-forms forms

Wrap each of forms in a thunk, so a body is evaluated when its task runs.

Deferring is what makes a body form contribute both its own value and the cases it builds: the form is evaluated inside the body’s collector, at run time, instead of while the test file is being loaded.

Procedure: do-if-string-contained KEY: #:name #:restriction #:result

Run result when name matches restriction.

The check is case-insensitive, and a restriction counts as a match when it appears anywhere in name. A #f restriction accepts everything; one that does not appear gives #f.

Procedure: errat thunk KEY: #:name #:timeout

Return an assertion that thunk raises.

It fails if thunk returns normally. name labels it in the report and timeout caps it in seconds, as in assert.

(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
              #:name "Should fail when list contains non-numbers")
Procedure: failed-test-names all-results

Return the names that failed in all-results, for a last-failed rerun.

Every test whose assertions failed is named. A suite that failed as a whole, through its own deadline for instance, contributes the names of the tests it holds, because a suite name is not something a rerun can select. A failure with no test name attached at all contributes #all, which makes the next run run everything rather than silently run nothing. Names come back in the order recorded, without duplicates.

Procedure: failed-test-wanted? name KEY: #:only

Return #t when name should run in a last-failed rerun.

only is the record of failing names to honour; it defaults to the one read from reports/last-failed.scm when VERITAS_ONLY_FAILED asked for a rerun. #f means there is no record, so everything runs; an empty list means the last run passed, so nothing does.

Procedure: falsus x KEY: #:name #:timeout

Assert that x is #f.

Procedure: fisher-yates-shuffle xs KEY: #:seed

Return the elements of xs in random order (Fisher-Yates).

Shuffling happens on a vector copy, so xs is left alone. seed makes the order reproducible, which is how a failing order is replayed; without it a fresh seed is pulled from random-state-from-platform on every call, so two calls on the same list differ. Lists of one element or fewer come back as they are.

(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
        (fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Procedure: fresh-seed

Return a new seed for a shuffle, drawn from the platform’s entropy source.

Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.

Procedure: generate-beer

Return a generator of beer styles.

Procedure: generate-boolean

Return a generator of #t and #f.

Procedure: generate-car

Return a generator of car makes and models.

Procedure: generate-char

Return a generator of printable ASCII characters, 32 to 126.

Procedure: generate-constant value

Return a generator that always answers value.

Procedure: generate-email

Return a generator of first.last@example.com email addresses.

Procedure: generate-first-name

Return a generator of first names.

Procedure: generate-integer KEY: #:max-bound

Return a generator of integers between 0 and max-bound (exclusive).

Call the generator with a random state:

(define gen (generate-integer #:max-bound 100))
(gen (seed->random-state 42))
Procedure: generate-last-name

Return a generator of last names.

Procedure: generate-list elem-generator KEY: #:min-len #:max-len

Return a generator of lists whose elements come from elem-generator.

Lists are min-len to max-len long.

Procedure: generate-name

Return a generator of full names, a first name and a last name.

Procedure: generate-natural KEY: #:max-bound

Return a generator of integers between 0 and max-bound inclusive.

Procedure: generate-one-of choices

Return a generator that picks one of choices at random.

A choice is usually a constant, so there is nothing to shrink.

Procedure: generate-real KEY: #:min #:max

Return a generator of real numbers in the range min to max.

Procedure: generate-string KEY: #:min-len #:max-len

Return a generator of printable-ASCII strings.

Strings are min-len to max-len characters long.

Procedure: generate-symbol KEY: #:min-len #:max-len

Return a generator of lowercase symbols, min-len to max-len long.

Symbols stay unescaped because the characters are always a-z.

Procedure: generate-tuple generators

Return a generator of lists holding one value from each of generators.

Procedure: generate-vector elem-generator KEY: #:min-len #:max-len

Return a generator of vectors whose elements come from elem-generator.

Procedure: generator-shrink generator value

Return smaller replacements for value taken from generator.

The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.

Procedure: idem KEY: #:expect #:got #:name #:timeout

Assert that got is eq? to expect.

The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.

Procedure: inanis KEY: #:got #:name #:timeout

Assert that got is an empty list, vector or string.

Procedure: intra KEY: #:min #:max #:got #:name #:timeout

Assert that got is within the inclusive range min–max.

Procedure: is-approximate? a b KEY: #:epsilon

Return #t when a and b differ by less than epsilon.

Useful for floating point, where exact equality rarely holds.

Procedure: longitudo KEY: #:expect #:got #:name #:timeout

Assert that got has expect elements (list, vector or string length).

Procedure: make-assertion assertion

Return assertion, a procedure of one validation client, as a case to run.

The returned procedure is tagged so the runner can tell an assertion from a task, and it records having run, which is how an assertion the body itself consumes - with value or ok?, say - stays an input to another case instead of becoming a case of its own.

It is also registered with the collector of the body being evaluated, if there is one. That registration is what keeps a body form that builds several assertions, such as the body of a let, from contributing only the value of its last expression. Building an assertion outside any body - at load time, or inside an assertion that runs later - simply registers nothing.

Procedure: merge-failed-names records

Combine what several runs recorded as failing into one record.

records holds one record per run: a list of names, or #f when a run left no record at all. When every run recorded an empty list the result is an empty list, so a parallel run that passed does not look like a run to rerun. When no run left a record the result is #f, which stands for no information rather than for nothing having failed. Names are deduplicated and keep the order they were found in.

Procedure: non-aequalis KEY: #:expect #:got #:name #:timeout

Assert that got is not equal? to expect.

Inverse of assert-equal.

Procedure: non-aequivalens KEY: #:expect #:got #:name #:timeout

Assert that got is not eqv? to expect.

Inverse of assert-eqv.

Procedure: non-congruit KEY: #:pattern #:got #:name #:timeout

Assert that string got does not match regular expression pattern.

Inverse of assert-matches.

Procedure: non-continet KEY: #:expect #:got #:name #:timeout

Assert that the container expect does not contain got.

Inverse of assert-contains.

Procedure: non-errat thunk KEY: #:name #:timeout

Return an assertion that thunk does not raise.

It fails if thunk raises. name labels it in the report and timeout caps it in seconds, as in assert.

Procedure: non-idem KEY: #:expect #:got #:name #:timeout

Assert that got is not eq? to expect.

Inverse of assert-eq.

Procedure: non-inanis KEY: #:got #:name #:timeout

Assert that got is a non-empty list, vector or string.

Procedure: non-propinquus KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got differs from expect by at least epsilon.

Inverse of assert-approximate.

Procedure: proba KEY: #:expect #:got #:compare #:name #:timeout

Return an assertion comparing got with expect using compare, a two-argument predicate such as equal?.

The assert-* wrappers below are built on this one. got and expect are usually plain values, but either may be wrapped in lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1))) evaluates the expression when the assertion runs, inside its timeout, which is what makes an expensive comparison something a deadline can catch. A lazy value keeps the lexical scope of the code that wrote it and is evaluated once per assertion that receives it.

name labels the assertion in the report. timeout caps it in seconds; #f, the default, means no deadline.

(assert #:expect 3
        #:got (+ 1 1 1)
        #:compare equal?
        #:name "Ensuring addition works")
Procedure: property name KEY: #:generators #:thunk #:runs #:seed #:shrink?

Return a property named name, a task like a test.

generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.

(property \"reverse of reverse is the original list\"
          #:generators (list (generate-list (generate-integer)))
          #:thunk (lambda (xs)
                    (assert-equal #:expect xs
                                  #:got (reverse (reverse xs)))))

A failing input is shrunk to the smallest value from the same generators that still fails, and both values are reported. seed makes the draw reproducible; without one, VERITAS_SEED is used, and failing that a fresh seed is drawn and reported with the failure.

Procedure: propinquus KEY: #:expect #:got #:name #:timeout #:epsilon

Assert that got is within epsilon of expect.

The default epsilon is 1.0e-8.

Procedure: propinquus? a b KEY: #:epsilon

Return #t when a and b differ by less than epsilon.

Useful for floating point, where exact equality rarely holds.

Procedure: read-last-failed KEY: #:path

Return the test names recorded as failing at path, or #f.

path defaults to the record of the whole run. #f means there is no record of a previous run: the file is missing or unreadable, and nothing is filtered.

Procedure: read-result-counts path

Return the counts recorded at path, or #f.

#f means there is nothing to add up: the file is missing, unreadable or not the alist write-result-counts writes.

Procedure: seeded-shuffle xs

Return xs in the order this run’s seed prescribes.

Both the runner, which shuffles test files, and veritas-run, which shuffles tasks, tests and cases, shuffle through this, so a single seed reproduces a whole run.

Procedure: split-keyword-arguments args

Split args, the syntax of an argument list, into its parts.

Two values come back: the #:keyword value pairs in the order they were written, and the forms that are not part of a pair, also in order. The test and suite macros use it to tell a body from the flags around it, wherever those flags were written.

Procedure: suscitat thunk KEY: #:on #:predicate #:name #:timeout

Assert that calling thunk raises an error.

By default any error satisfies the assertion. Pass on to require a specific error key (for example 'wrong-type-arg), or predicate, a procedure of (key args) returning #t for the expected error.

(assert-raises (lambda () (/ 1 0)))
(assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Procedure: unus-ex KEY: #:expect #:got #:name #:timeout

Assert that got is equal? to one of the elements of list expect.

Procedure: veritas-run KEY: #:shuffle? #:concurrent? #:debug? #:reporter #:with-coverage? #:coverage-dir #:coverage-html-dir #:coverage-include #:coverage-exclude #:seed . tasks

Run tasks, the top-level tests, suites and benchmarks.

shuffle? randomizes the order of the tasks and concurrent? runs them in parallel fibers. reporter names a reporter, otherwise VERITAS_REPORTER decides. seed makes the shuffle reproducible, and the seed actually used is reported, so a failing order can be replayed by setting VERITAS_SEED to it. with-coverage? measures coverage - VERITAS_COVERAGE=true asks for it too - and debug? prints the raw results.

Coverage is written where coverage-dir (default VERITAS_COVERAGE_DIR, "reports/lcov") says, with the HTML report in coverage-html-dir (default VERITAS_COVERAGE_HTML_DIR, "reports/html"). Only the files under coverage-include (default VERITAS_COVERAGE_INCLUDE, the current directory) are reported, minus the ones under coverage-exclude (default VERITAS_COVERAGE_EXCLUDE).

Procedure: verus x KEY: #:name #:timeout

Assert that x is #t.

Procedure: write-last-failed! all-results

Record the names that failed, so the next run can rerun just those.

Each worker process writes its own record - the path is a parameter - and the --jobs parent merges them into the run’s, which is why the writing and the merging are separate.

Procedure: write-last-failed-names! names KEY: #:path

Record names at path as the tests a rerun should select.

An empty list is written when everything passed, which is what makes the next VERITAS_ONLY_FAILED run do nothing instead of rerunning the world. A trouble with the file is swallowed: the run has already happened.

Procedure: write-result-counts counts KEY: #:path

Write counts to path so another process can add them up.

An alist is plain Scheme data, so read restores it exactly: this is how a worker process hands its totals to the process that runs the other files. Returns path.

1.20.5 Variables

Variable: veritas-rerun-all-marker
"#all"