The following is the list of modules provided by this library. The stylesheet and the script that present this chapter in a browser are written beside it, as dokgen.css and dokgen.js; the manual says how a build points at them.
suite
macro.
test
macro.
equal? to expect.
eqv? to expect.
eq? to expect.
equal? to expect.
eqv? to expect.
#f.
eq? to expect.
equal? to expect.
eqv? to expect.
equal? to one of the elements of list
expect.
#t.
#t when name should run in a last-failed rerun.
#f.
#t and #f.
first.last@example.com email addresses.
eq? to expect.
#t when a and b differ by less than
epsilon.
equal? to expect.
eqv? to expect.
eq? to expect.
#t when a and b differ by less than
epsilon.
#f.
#f.
#t when snapshots are to be rewritten instead of compared.
#t when spy was called with exactly args.
#t when value was made by make-spy or
make-stub.
test-each, but case->suite returns a suite.
equal? to one of the elements of list
expect.
#t.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Run thunks concurrently in fibers and collect what they return.
Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.
(evaluate-thunks-concurrently
(list (lambda () (sleep 1) 'delayed)
(lambda () 'instant)))
⇒ (delayed instant)
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Default value:
#f
Default value:
#f
Default value:
#f
Default value:
"reports/last-failed.scm"
Default value:
#f
Default value:
((files . 0) (suites . 0) (tests . 0) (assertions . 0) (flaky . 0))
Default value:
#f
Default value:
0
Default value:
()
Default value:
#f
Define an And (continuation) step named name.
Define a But (continuation) step named name.
Define a Given (setup) step named name.
Define a Given/When/Then scenario named name; the procedure behind the macro.
A scenario is a suite, so its steps run in declaration order. Keywords
are forwarded to suite, and steps come from given,
when-step, then, and-step and but-step.
timeout, when given, caps the whole scenario in seconds, exactly
like suite; a step can carry its own #:timeout as well,
since steps are ordinary tests.
(scenario "emptying the cart"
(given "a cart with one item"
(assert-equal #:expect 1 #:got (cart-size cart)))
(when-step "the cart is emptied"
(empty-cart! cart))
(then "the cart is empty"
(assert-true (cart-empty? cart))))
Define a step named name, labeled with label; the procedure behind the step macros.
A step is an ordinary test, so it takes the same keywords and cases.
Define a suite named name; the procedure behind the suite
macro.
The arguments after name are the children. Each form is evaluated
when the suite runs, and every test, suite or benchmark built while it
is evaluated is a child: one already built, a list of them, or a thunk
of them are all understood. Children run concurrently unless
concurrent? is #f, and in shuffled order unless
shuffle? is #f. before-all and after-all run
once around them, timeout caps the whole suite in seconds
including those hooks, skip? drops it from the run and tags
decide which filter may select it.
Define a test named name; the procedure behind the test
macro.
The arguments after name are the body. Each form is evaluated when the test runs, and every assertion built while it is evaluated is a case: an assertion, a list of them, a thunk of them, or a helper procedure that builds them are all understood. before and after run around the body, timeout caps the whole test in seconds, hooks included, skip? drops it from the run and tags decide which filter may select it.
A body is read in order, because a body is a sequence: one form prepares what the next one checks. shuffle? asks for the order of the body forms to be randomized anyway, which is a way to find checks that depend on each other by accident.
retries reruns a failing test, hooks and body together, up to that many extra attempts. A test that fails and then passes is reported as flaky with its attempt count, so an intermittent failure is visible instead of hidden.
(test "Addition Validation"
#:before (lambda (vc) (display "Initializing..."))
(assert-equal #:expect 4 #:got (+ 2 2)))
Define a Then (outcome) step named name.
Define a When (action) step named name.
Called when-step rather than when so it does not shadow
Guile’s when syntax.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Assert that spy was called exactly expect times.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-call-count spy #:expect 2) vc)).
Assert that spy was called at least once.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-called spy) vc)).
Assert that spy was called with exactly args.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-called-with spy '(1 2)) vc)).
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Assert that got is an empty list, vector or string.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Assert that x is #f.
Assert that got is within the inclusive range min–max.
Assert that got has expect elements (list, vector or string length).
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Assert that got is a non-empty list, vector or string.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that got is equal? to one of the elements of list
expect.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Assert that got matches the snapshot recorded under name.
name labels the assertion and names the file, so it is required;
the file lives under VERITAS_SNAPSHOT_DIR (default
test/snapshots) or under dir. The first run writes the
snapshot and passes. Later runs compare with equal?, and a
mismatch fails with both values. A snapshot that cannot be read is a
failure too, never a silent rewrite.
A got that is a procedure is called with no arguments when the assertion runs, so state that only exists at test time can be snapshotted.
(assert-snapshot #:name "the rendered cart"
#:got (lambda () (render (make-cart))))
Assert that x is #t.
Define a benchmark named name that times thunk over runs runs.
Statistics come from Guile’s statprof, so benchmarks are reported
separately from tests. timeout caps all the runs together in
seconds and skip? drops the benchmark from the run.
Run thunk with an optional deadline of seconds seconds.
When seconds is a positive real number the work runs in its own
fiber and, if it has not finished by then, the fiber is aborted and the
result of calling on-timeout is returned instead. When
seconds is #f (the default, meaning no deadline)
thunk is simply called and its result is returned.
thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.
(call-with-timeout 5
(lambda () (expensive-computation))
#:on-timeout (lambda () 'gave-up))
Index cases by key, giving an alist of key to case.
Handy for tables with named columns.
Return the outcome of every assertion in xs as a flat list.
Booleans only: #t for a pass, #f for a failure.
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Return the seed this run shuffles with, choosing one when none was set.
seed (a number or a numeric string) wins when given, otherwise the
value of VERITAS_SEED is used, and when neither is set a fresh
seed is drawn from the platform. The chosen seed is remembered, so the
order the runner gives the test files and every order chosen here come
from one seed, and the run can be replayed by setting
VERITAS_SEED to the seed that was reported.
(current-seed #:seed 7) ⇒ 7
Wrap each of forms in a thunk, so a body is evaluated when its task runs.
Deferring is what makes a body form contribute both its own value and the cases it builds: the form is evaluated inside the body’s collector, at run time, instead of while the test file is being loaded.
Run result when name matches restriction.
The check is case-insensitive, and a restriction counts as a match when
it appears anywhere in name. A #f restriction accepts
everything; one that does not appear gives #f.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Return the names that failed in all-results, for a last-failed rerun.
Every test whose assertions failed is named. A suite that failed as a
whole, through its own deadline for instance, contributes the names of
the tests it holds, because a suite name is not something a rerun can
select. A failure with no test name attached at all contributes
#all, which makes the next run run everything rather than
silently run nothing. Names come back in the order recorded, without
duplicates.
Return #t when name should run in a last-failed rerun.
only is the record of failing names to honour; it defaults to the
one read from reports/last-failed.scm when
VERITAS_ONLY_FAILED asked for a rerun. #f means there is
no record, so everything runs; an empty list means the last run passed,
so nothing does.
Assert that x is #f.
Return the elements of xs in random order (Fisher-Yates).
Shuffling happens on a vector copy, so xs is left alone.
seed makes the order reproducible, which is how a failing order is
replayed; without it a fresh seed is pulled from
random-state-from-platform on every call, so two calls on the
same list differ. Lists of one element or fewer come back as they are.
(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
(fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Return a new seed for a shuffle, drawn from the platform’s entropy source.
Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.
Return a generator of beer styles.
Return a generator of #t and #f.
Return a generator of car makes and models.
Return a generator of printable ASCII characters, 32 to 126.
Return a generator that always answers value.
Return a generator of first.last@example.com email addresses.
Return a generator of first names.
Return a generator of integers between 0 and max-bound (exclusive).
Call the generator with a random state:
(define gen (generate-integer #:max-bound 100)) (gen (seed->random-state 42))
Return a generator of last names.
Return a generator of lists whose elements come from elem-generator.
Lists are min-len to max-len long.
Return a generator of full names, a first name and a last name.
Return a generator of integers between 0 and max-bound inclusive.
Return a generator that picks one of choices at random.
A choice is usually a constant, so there is nothing to shrink.
Return a generator of real numbers in the range min to max.
Return a generator of printable-ASCII strings.
Strings are min-len to max-len characters long.
Return a generator of lowercase symbols, min-len to max-len long.
Symbols stay unescaped because the characters are always a-z.
Return a generator of lists holding one value from each of generators.
Return a generator of vectors whose elements come from elem-generator.
Return smaller replacements for value taken from generator.
The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is an empty list, vector or string.
Assert that got is within the inclusive range min–max.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Assert that got has expect elements (list, vector or string length).
Return assertion, a procedure of one validation client, as a case to run.
The returned procedure is tagged so the runner can tell an assertion
from a task, and it records having run, which is how an assertion the
body itself consumes - with value or ok?, say - stays an
input to another case instead of becoming a case of its own.
It is also registered with the collector of the body being evaluated, if
there is one. That registration is what keeps a body form that builds
several assertions, such as the body of a let, from contributing
only the value of its last expression. Building an assertion outside
any body - at load time, or inside an assertion that runs later - simply
registers nothing.
Return a procedure that calls proc and records every call.
The recording is read back with spy-calls, spy-results and
spy-call-count. name is only used in the messages of the
assert-* checks below.
(define double (make-spy (lambda (x) (* x 2)))) (double 3) (spy-calls double) ⇒ ((3))
Return a stand-in procedure that ignores its arguments and returns returns.
A stub is a spy whose procedure ignores its arguments, so
spy-calls, spy-call-count and the assert-* checks
work on it unchanged.
Combine what several runs recorded as failing into one record.
records holds one record per run: a list of names, or #f
when a run left no record at all. When every run recorded an empty list
the result is an empty list, so a parallel run that passed does not look
like a run to rerun. When no run left a record the result is #f,
which stands for no information rather than for nothing having failed.
Names are deduplicated and keep the order they were found in.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is a non-empty list, vector or string.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Return a property named name, a task like a test.
generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.
(property \"reverse of reverse is the original list\"
#:generators (list (generate-list (generate-integer)))
#:thunk (lambda (xs)
(assert-equal #:expect xs
#:got (reverse (reverse xs)))))
A failing input is shrunk to the smallest value from the same generators
that still fails, and both values are reported. seed makes the
draw reproducible; without one, VERITAS_SEED is used, and failing
that a fresh seed is drawn and reported with the failure.
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Return the test names recorded as failing at path, or #f.
path defaults to the record of the whole run. #f means
there is no record of a previous run: the file is missing or unreadable,
and nothing is filtered.
Return the counts recorded at path, or #f.
#f means there is nothing to add up: the file is missing,
unreadable or not the alist write-result-counts writes.
Return the frames stored at path, oldest first.
Print the trace at path as a numbered timeline.
Return xs in the order this run’s seed prescribes.
Both the runner, which shuffles test files, and veritas-run,
which shuffles tasks, tests and cases, shuffle through this, so a single
seed reproduces a whole run.
Print the trace of the last run, and return whether there was one.
Return the file holding the snapshot called name.
dir overrides VERITAS_SNAPSHOT_DIR, which itself defaults
to test/snapshots.
Return #t when snapshots are to be rewritten instead of compared.
Set VERITAS_UPDATE_SNAPSHOTS to true to accept the current
value of every snapshot a run reaches.
Split args, the syntax of an argument list, into its parts.
Two values come back: the #:keyword value pairs in the order they
were written, and the forms that are not part of a pair, also in order.
The test and suite macros use it to tell a body from the
flags around it, wherever those flags were written.
Return how many times spy was called.
Return #t when spy was called with exactly args.
Return the argument lists spy was called with, oldest first.
Return what spy returned, oldest first, one entry per call.
Return #t when value was made by make-spy or
make-stub.
Like test-each, but case->suite returns a suite.
Use it when one case needs several tests rather than one.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Expand cases into one test per case, under a suite named group-name.
case->test takes one case and returns a test:
(test-each "integer addition"
'((0 0 0) (1 2 3) (10 20 30))
(lambda (case)
(let ((a (list-ref case 0))
(b (list-ref case 1))
(expected (list-ref case 2)))
(test (format #f "~a + ~a = ~a" a b expected)
(assert-equal #:expect expected #:got (+ a b))))))
Record one step of the run when tracing is on.
kind is a symbol such as test-start, assertion or
test-end, name is what the step is about, and detail
is a short string a reader can use to tell one frame from another.
Return where the last trace of this run is written.
Forget the frames recorded so far.
Assert that got is equal? to one of the elements of list
expect.
Run tasks, the top-level tests, suites and benchmarks.
shuffle? randomizes the order of the tasks and concurrent?
runs them in parallel fibers. reporter names a reporter,
otherwise VERITAS_REPORTER decides. seed makes the shuffle
reproducible, and the seed actually used is reported, so a failing order
can be replayed by setting VERITAS_SEED to it.
with-coverage? measures coverage - VERITAS_COVERAGE=true
asks for it too - and debug? prints the raw results.
Coverage is written where coverage-dir (default
VERITAS_COVERAGE_DIR, "reports/lcov") says, with the HTML
report in coverage-html-dir (default
VERITAS_COVERAGE_HTML_DIR, "reports/html"). Only the
files under coverage-include (default
VERITAS_COVERAGE_INCLUDE, the current directory) are reported,
minus the ones under coverage-exclude (default
VERITAS_COVERAGE_EXCLUDE).
Assert that x is #t.
Call thunk with module’s var set to value, and restore it.
The previous value comes back even when thunk raises, so substituting a dependency is safe to try.
Guile resolves a call to a known procedure while it compiles, so a
compiled call site does *not* see the change: a call written in a module
the compiler can see through keeps calling the original procedure.
Measured on Guile 3.0, swapping a binding is observed by
module-ref, by code passed to eval, and by call sites in a
module declared #:declarative? #f; it is not observed by a call
compiled in a declarative module, which is the default.
That leaves the ordinary Scheme answer, which needs no substitution at
all: pass the dependency in, or hand the caller a make-spy
recording. Use this when the code under test reaches the binding at run
time.
(with-temporary-binding (resolve-module '(my app)) 'now
(lambda () 0)
(lambda ()
(assert-equal #:expect 0
#:got (eval '((@ (my app) now))
(resolve-module '(my app))))))
Record the names that failed, so the next run can rerun just those.
Each worker process writes its own record - the path is a parameter -
and the --jobs parent merges them into the run’s, which is why
the writing and the merging are separate.
Record names at path as the tests a rerun should select.
An empty list is written when everything passed, which is what makes the
next VERITAS_ONLY_FAILED run do nothing instead of rerunning the
world. A trouble with the file is swallowed: the run has already
happened.
Write counts to path so another process can add them up.
An alist is plain Scheme data, so read restores it exactly: this
is how a worker process hands its totals to the process that runs the
other files. Returns path.
Write the frames recorded so far to path and return it.
The newest frame is last in the file, so the timeline reads top to bottom.
Assertions for Veritas. The core procedure is ‘assert’, which compares a value against an expected one using whatever predicate you give it. The assert-* procedures below are thin wrappers around it, together with a few helpers for checking error handling.
Every assertion also has a Latin alias, following the Latin naming already used by the BDD helpers (dato, quando, tunc, ...): (proba ...) is (assert ...), aequalis is assert-equal, verus is assert-true, continet is assert-contains, and so on. An alias is the exact same procedure object as its English name, so keyword arguments, semantics and reporting are identical. The complete mapping is documented in the README and repeated in the alias block at the end of this file.
equal? to expect.
eqv? to expect.
eq? to expect.
equal? to expect.
eqv? to expect.
#f.
eq? to expect.
equal? to expect.
eqv? to expect.
equal? to one of the elements of list
expect.
#t.
#f.
eq? to expect.
#t when a and b differ by less than
epsilon.
equal? to expect.
eqv? to expect.
eq? to expect.
#t when a and b differ by less than
epsilon.
equal? to one of the elements of list
expect.
#t.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Assert that got is an empty list, vector or string.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Assert that x is #f.
Assert that got is within the inclusive range min–max.
Assert that got has expect elements (list, vector or string length).
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Assert that got is a non-empty list, vector or string.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that got is equal? to one of the elements of list
expect.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Assert that x is #t.
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Assert that x is #f.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is an empty list, vector or string.
Assert that got is within the inclusive range min–max.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Assert that got has expect elements (list, vector or string length).
Return assertion, a procedure of one validation client, as a case to run.
The returned procedure is tagged so the runner can tell an assertion
from a task, and it records having run, which is how an assertion the
body itself consumes - with value or ok?, say - stays an
input to another case instead of becoming a case of its own.
It is also registered with the collector of the body being evaluated, if
there is one. That registration is what keeps a body form that builds
several assertions, such as the body of a let, from contributing
only the value of its last expression. Building an assertion outside
any body - at load time, or inside an assertion that runs later - simply
registers nothing.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is a non-empty list, vector or string.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Assert that got is equal? to one of the elements of list
expect.
Assert that x is #t.
Behaviour-driven (Given/When/Then) helpers for Veritas, in the spirit of Cucumber/Gherkin scenarios. A scenario is a suite whose steps are ordinary tests, so every existing reporter, filter and runner feature keeps working. Step names are prefixed with their keyword so the output reads naturally.
Steps run in order by default: scenarios disable shuffling and concurrency so that Given -> When -> Then always executes sequentially.
State is shared between steps simply by closing over a ‘let’ binding in the scenario body, for example:
(scenario "withdrawing too much" (let ((account (make-account 100))) (given "an account with 100" (assert-equal #:expect 100 #:got (account-balance account))) (when-step "withdrawing 150" (assert-false (withdraw account 150))) (then "the balance is unchanged" (assert-equal #:expect 100 #:got (account-balance account)))))
Every step of that scenario runs: a scenario body is read for the steps it builds, not for the value of its last expression, so a ‘let’ around the steps is the natural way to share state and it does not hide the steps from the runner. The same holds inside a step: a ‘let’ body contributes every assertion it builds.
Latin aliases are provided for those who prefer them: fabula/scaena for scenario, dato for given, quando for when, tunc for then, etiam for and, and sed for but.
The three steps also have PascalCase aliases – Given, When and Then – for readers who expect the keywords to look like the Cucumber ones.
The When step is called ‘when-step’ rather than ‘when’ because Guile’s core ‘when’ is a syntax keyword and importing a conflicting binding would break ‘(when ...)’ for every module that imports Veritas BDD. The Latin ‘quando’ is unshadowed and reads naturally.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Define an And (continuation) step named name.
Define a But (continuation) step named name.
Define a Given (setup) step named name.
Define a Given/When/Then scenario named name; the procedure behind the macro.
A scenario is a suite, so its steps run in declaration order. Keywords
are forwarded to suite, and steps come from given,
when-step, then, and-step and but-step.
timeout, when given, caps the whole scenario in seconds, exactly
like suite; a step can carry its own #:timeout as well,
since steps are ordinary tests.
(scenario "emptying the cart"
(given "a cart with one item"
(assert-equal #:expect 1 #:got (cart-size cart)))
(when-step "the cart is emptied"
(empty-cart! cart))
(then "the cart is empty"
(assert-true (cart-empty? cart))))
Define a step named name, labeled with label; the procedure behind the step macros.
A step is an ordinary test, so it takes the same keywords and cases.
Define a Then (outcome) step named name.
Define a When (action) step named name.
Called when-step rather than when so it does not shadow
Guile’s when syntax.
Command line interface for the Veritas test runner.
The CLI is intentionally a thin layer over the runner: every flag is
translated into the matching VERITAS_* environment variable that
the runner and the rest of Veritas already understand. This keeps the
environment variables as the single, reproducible source of truth while
offering ergonomic syntactic sugar for interactive use and CI scripts.
The runner module is resolved lazily (only after the environment has
been updated), so that parameters which are initialised from the
environment when their module loads - such as
VERITAS_RUN_BENCHMARKS and VERITAS_ONLY_TEST - observe the
values supplied on the command line.
The run itself is a straight line, and every stage of it is a module:
Mermaid diagram, as source:
graph LR cli["(veritas cli)"] --> env["VERITAS_* settings"] env --> runner["(veritas runner)"] runner --> files["test files"] files --> suites["suites and tests"] suites --> results["(veritas model) results"] results --> reports["(veritas reporter)"]
Only the first stage is this module’s: a run started by hand and a run started by an editor meet at the environment variables and go on from there.
-c invocation.
Entry point suitable for a launcher or a Guile -c invocation.
args is the list of user arguments (typically (cdr
(command-line))). Exits the process with the status returned by
run-cli.
Parse args and run the Veritas runner.
args is a list of command-line strings without the program name.
Returns an exit status: 0 on success, 1 on failure.
(run-cli '("--test-type" "unit" "--no-benchmarks"))
Print the full Veritas CLI help page to standard output.
"veritas 1.5.0 - test runner for the Veritas framework\n\nusage: veritas [OPTIONS]\n\nDiscover and run the test suites, tests and benchmarks of a project.\nEvery option is mapped to the corresponding VERITAS_* environment\nvariable, so the command line is only syntactic sugar over the exact\nsame configuration used programmatically and in CI.\n\noptions:\n -d, --scan-dir DIR root directory to scan for tests (VERITAS_SCAN_DIR)\n -T, --test-top-dir DIR directory holding the test hierarchy (VERITAS_TEST_TOP_DIR)\n -t, --test-type TYPE test category / sub-directory to load (VERITAS_TEST_TYPE)\n -e, --entrypoint NAME no-argument function to call per file (VERITAS_ENTRYPOINT)\n -m, --test-module NAME top level module prefix of the tests (VERITAS_TEST_MODULE)\n -r, --reporter NAME reporter to use, e.g. emoji-reporter,\n monotone-reporter, json-reporter,\n junit-reporter or github-reporter (VERITAS_REPORTER)\n --only-test SUBSTR only run tests whose name contains SUBSTR (VERITAS_ONLY_TEST)\n --only-suite SUBSTR only run suites whose name contains SUBSTR (VERITAS_ONLY_SUITE)\n --only-tag TAG only run tests/suites carrying TAG (VERITAS_ONLY_TAG)\n --only-failed rerun only what failed in the previous run\n (VERITAS_ONLY_FAILED)\n --no-only-failed ignore the recorded failures (the default)\n --seed N seed every shuffle of this run (VERITAS_SEED)\n --partition I/T run slice I of the T slices of the test files\n (VERITAS_PARTITION)\n -j, --jobs N run the test files in N worker processes, one\n file per process (VERITAS_JOBS)\n --changed run only the test files the working tree change\n reaches (VERITAS_CHANGED)\n --no-changed run every test file (the default)\n --since REF run only the test files the change since REF\n reaches (VERITAS_SINCE)\n --src-dir DIR source root the change graph is read from\n (VERITAS_SRC_DIR, default src)\n --retries N rerun a failing test up to N extra times\n (VERITAS_RETRIES)\n --trace record every step of the run, to\n reports/traces/last.scm (VERITAS_TRACE)\n --no-trace do not record a trace (the default)\n --show-trace print the timeline of the last run and exit\n --list print the test files this run would run, one\n per line, and run none of them (VERITAS_LIST)\n --coverage measure code coverage and write the reports\n (VERITAS_COVERAGE)\n --no-coverage do not measure coverage (the default)\n --coverage-dir DIR where the LCOV trace and the JSON summary go\n (VERITAS_COVERAGE_DIR, default reports/lcov)\n --coverage-html-dir DIR\n where the HTML report goes, or \"false\" to skip\n it (VERITAS_COVERAGE_HTML_DIR, default reports/html)\n --coverage-include DIRS\n colon-separated source roots the report covers\n (VERITAS_COVERAGE_INCLUDE, default the current\n directory)\n --coverage-exclude DIRS\n colon-separated source roots left out\n (VERITAS_COVERAGE_EXCLUDE)\n -s, --shuffle shuffle the order of discovered files (default)\n --no-shuffle keep the discovered file order (VERITAS_SHUFFLE_FILES=false)\n -b, --benchmarks run benchmarks (default) (VERITAS_RUN_BENCHMARKS=true)\n --no-benchmarks skip benchmarks (VERITAS_RUN_BENCHMARKS=false)\n -h, --help show this help and exit\n -V, --version show version information and exit\n\nenvironment variables:\n VERITAS_ENTRYPOINT entry-point function name, defaults to \"spec\"\n VERITAS_SCAN_DIR directory to scan, defaults to \"./\"\n VERITAS_TEST_TYPE test type directory, defaults to \"unit\"\n VERITAS_TEST_TOP_DIR test hierarchy root, defaults to \"test\"\n VERITAS_TEST_MODULE module prefix, defaults to \"veritas\"\n VERITAS_SHUFFLE_FILES \"true\" or \"false\", defaults to \"true\"\n VERITAS_RUN_BENCHMARKS \"true\" or \"false\", defaults to \"true\"\n VERITAS_ONLY_TEST optional test-name substring filter\n VERITAS_ONLY_SUITE optional suite-name substring filter\n VERITAS_ONLY_TAG optional tag filter\n VERITAS_ONLY_FAILED \"true\" to rerun only the tests that failed last time\n VERITAS_REPORTER reporter symbol, defaults to emoji-reporter\n VERITAS_SEED seed for every shuffle, printed by a shuffled run\n VERITAS_PARTITION \"INDEX/TOTAL\" slice of the discovered test files\n VERITAS_JOBS whole number of worker processes for the files,\n defaults to 1 and then they run in this process\n VERITAS_CHANGED \"true\" to run only the tests the change reaches\n VERITAS_SINCE revision the change is read against, for example\n origin/trunk\n VERITAS_SRC_DIR source root of the change graph, defaults to src\n VERITAS_RETRIES extra attempts for a failing test, defaults to 0\n VERITAS_TRACE \"true\" to record a trace of the run\n VERITAS_LIST \"true\" to print the selected files and run none\n VERITAS_COVERAGE \"true\" to measure code coverage\n VERITAS_COVERAGE_DIR where the LCOV trace is written, defaults to\n reports/lcov\n VERITAS_COVERAGE_HTML_DIR where the HTML report is written, defaults to\n reports/html, and \"false\" turns it off\n VERITAS_COVERAGE_INCLUDE colon-separated source roots to report on,\n defaults to the current directory\n VERITAS_COVERAGE_EXCLUDE colon-separated source roots to leave out\n VERITAS_COBERTURA_PATH where the Cobertura XML is written, defaults to\n cobertura.xml in the coverage directory\n VERITAS_JACOCO_PATH where the JaCoCo XML is written, defaults to\n jacoco.xml in the coverage directory\n VERITAS_LAST_FAILED_PATH where the failing names are recorded, defaults\n to reports/last-failed.scm\n VERITAS_JUNIT_PATH where junit-reporter writes, defaults to\n reports/junit/veritas-junit.xml\n\nexamples:\n veritas run unit tests with the project defaults\n veritas --test-type integration run the integration test suite\n veritas --no-shuffle --reporter monotone-reporter\n veritas --only-suite auth run only the suites mentioning \"auth\"\n veritas --only-tag slow run only tests/suites tagged \"slow\"\n veritas --no-benchmarks fast feedback loop, skip benchmarks\n veritas --only-failed rerun what failed in the previous run\n veritas --seed 42 repeat a shuffled order exactly\n veritas --partition 1/3 run one third of the test files\n veritas --jobs 4 run the files in four worker processes\n veritas --changed run only the tests this change can reach\n veritas --since origin/trunk run only the tests a branch touches\n veritas --retries 2 ride out an intermittent failure\n veritas --trace record the run, then\n veritas --show-trace print the recorded timeline\n veritas --changed --list show what --changed would run\n veritas --coverage measure coverage and write LCOV and HTML\n veritas --coverage --coverage-exclude test\n measure the sources, not the tests\n veritas --reporter junit-reporter write a JUnit XML report\n\ninteractive use:\n The same runner is available from a Guile REPL. Load the CLI module and\n call (run-cli '(\"--help\")) or (show-help) to inspect the options, and\n ((@@ (veritas runner) run-tests)) to run with the current environment.\n"
"veritas(1)"
((scan-dir (single-char #\d) (value #t)) (test-top-dir (single-char #\T) (value #t)) (test-type (single-char #\t) (value #t)) (entrypoint (single-char #\e) (value #t)) (test-module (single-char #\m) (value #t)) (reporter (single-char #\r) (value #t)) (only-test (value #t)) (only-suite (value #t)) (only-tag (value #t)) (only-failed (value #f)) (no-only-failed (value #f)) (seed (value #t)) (partition (value #t)) (jobs (single-char #\j) (value #t)) (changed (value #f)) (no-changed (value #f)) (since (value #t)) (src-dir (value #t)) (list (value #f)) (trace (value #f)) (no-trace (value #f)) (show-trace (value #f)) (retries (value #t)) (coverage (value #f)) (no-coverage (value #f)) (coverage-dir (value #t)) (coverage-html-dir (value #t)) (coverage-include (value #t)) (coverage-exclude (value #t)) (shuffle (single-char #\s) (value #f)) (no-shuffle (value #f)) (benchmarks (single-char #\b) (value #f)) (no-benchmarks (value #f)) (help (single-char #\h) (value #f)) (version (single-char #\V) (value #f)))
"1.5.0"
Docker/Podman helpers for integration and black box testing, following the well known container-test pattern: start a container, wait until it is ready, exercise it (exec, logs, host ports), then clean it up.
The runtime is auto-detected (Podman first, then Docker) and can be forced with the VERITAS_CONTAINER_RUNTIME environment variable, mirroring the way testcontainers libraries target a Docker-compatible endpoint. Everything shells out to the CLI, so no daemon bindings are required.
Typical use inside a suite:
(with-container (lambda (c) (assert-true (container-running? c)) (assert-matches #:pattern "ready" #:got (container-exec c ’("app" "–version")))) #:image "alpine" #:command ’("sh" "-c" "echo ready && sleep 600") #:wait (lambda (c) (container-wait-for-log c "ready" #:timeout 10)))
#t when container exists and is running.
#t when a container runtime is available.
run-command, but raise an error on a non-zero exit status.
Return an assertion that container is running.
Unlike assert-true, the check is performed when the assertion
runs, not when it is constructed.
Build an image with tag from dockerfile in context.
Run command (a list of strings) inside running container.
Returns the captured combined output and raises on failure.
Run command inside container, returning two values: status and output.
Return the logs of container as a string.
Force-stop and remove container, returning its exit status.
Start image and return the new container id (or name when given).
env entries are passed as -e variables, ports as
-p bindings and volumes as -v mounts.
Return #t when container exists and is running.
Return the container runtime executable to use.
Honours VERITAS_CONTAINER_RUNTIME, otherwise prefers
podman and falls back to docker.
Return #t when a container runtime is available.
Stop container, returning its exit status.
Wait until command succeeds inside container.
Wait until container logs match pattern (regexp or string).
Wait until a TCP connection to host:port succeeds.
Run command, a list of strings, without a shell.
Returns two values: the exit status (an integer, -1 when the
process could not be started) and the captured standard output as a
string.
(call-with-values (lambda () (run-command (list "echo" "hi"))) (lambda (status output) (list status output))) ⇒ (0 "hi\n")
Like run-command, but raise an error on a non-zero exit status.
Returns the captured standard output.
Poll predicate every interval seconds until it is true.
Returns #t, or raises an error after timeout seconds.
Exceptions raised by predicate are treated as a false result.
Start a container, call thunk with its id, and always clean it up.
wait, when supplied, is a procedure called with the container id after start (for example to wait for a log line or a port). keep? leaves the container running for debugging.
Code coverage for Veritas.
Guile collects execution counts by firing a VM hook on every instruction it retires. That hook only exists in the ‘debug’ VM engine: in the ‘regular’ engine the same code runs silently, which is why a run that merely calls ‘(system vm coverage)’ records no hits at all. Every collection here therefore happens inside ‘with-debug-vm’.
What comes out of the VM is an IP-to-count table and a census of the source locations of every loaded program. It is turned into a report - one record per source file, holding the execution count of each instrumented line - and that report can be written as LCOV, as a self-contained HTML report, or as JSON, and merged with other reports (the files of a ‘–jobs’ run, say).
#t when value (the environment by default) refuses
coverage.
#t when value (the environment by default) asks for
coverage.
#f when it
holds no code.
#f to skip
it.
#f when
off.
#t when path belongs in a coverage report.
#f.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Run thunk and return its values preceded by its coverage report.
The report is a coverage-report record; the values of thunk
follow it unchanged. The run happens on the debug engine, and the
engine in use before the call is restored afterwards.
(call-with-values (lambda () (collect-coverage (lambda () (run-the-tests)))) (lambda (report result) ...))
Return the path of the Cobertura XML report of a run.
The VERITAS_COBERTURA_PATH environment variable wins when it is
set, which is how a CI job puts the document where its dashboard looks
for it.
Turn Guile’s coverage data into a report.
Every instrumented source file is kept, with the count of each of its lines - including the lines that never ran, which is the whole point of a coverage report. include and exclude filter the files by absolute path.
Return the directory the LCOV report is written to.
dir wins; otherwise VERITAS_COVERAGE_DIR decides, and
"reports/lcov" is the default. The directory holds the LCOV
trace, the JSON summary and the Cobertura and JaCoCo documents.
Return #t when value (the environment by default) refuses
coverage.
An explicit refusal beats a test file that asks for measurement with
#:with-coverage? #t: a task that says ‘maak test-fast’ or passes
--no-coverage means it, and it is the only way to run a file that
always measures without paying for it.
Return #t when value (the environment by default) asks for
coverage.
Return the prefixes left out of a report.
An explicit exclude wins, otherwise
VERITAS_COVERAGE_EXCLUDE is read. Nothing is excluded by
default.
Return how many instrumented lines of file ran at least once.
Return how many lines of file hold code that could be counted.
Return how often line of file ran, or #f when it
holds no code.
A line that was instrumented but never reached counts 0; a line with no
code at all - a comment, a blank line - has no entry and gives
#f.
Return the percentage of file’s instrumented lines that ran.
Return the directory the HTML report is written to, or #f to skip
it.
html-dir wins; otherwise VERITAS_COVERAGE_HTML_DIR decides.
An empty value, "false" or "0" turns the HTML report off.
Return the path of the index page of the HTML report, or #f when
off.
Return the prefixes a report is limited to.
An explicit include wins; otherwise
VERITAS_COVERAGE_INCLUDE is read, and when neither is set the
current directory is the boundary, which is what keeps Guile, Fibers and
the other dependencies out of the report.
Return the path of the JaCoCo XML report of a run.
The VERITAS_JACOCO_PATH environment variable wins when it is set.
Return the path of the JSON summary of a coverage run.
Return the path of the merged LCOV file of a coverage run.
Return #t when path belongs in a coverage report.
include is a list of directory prefixes; an empty list means every file. exclude is a list of prefixes that win over it, so the tests of a project can be left out of the coverage of the code they exercise.
Return how many instrumented lines of report ran at least once.
Return how many lines of report hold code that could be counted.
Return how often line of path ran in report, or
#f.
path has to be the absolute path the report holds.
Return the percentage of report’s instrumented lines that ran.
Return report as the one line a run prints.
"Code coverage: 82.4% of 340 lines (280 covered) in 12 files"
Return a report that holds no file at all.
Return report as an LCOV trace file held in a string.
Return the coverage of path, given its lines.
lines is an alist of (line-number . execution-count). It
is sorted by line number here, so every report has one shape however it
was built.
Return a report holding files, sorted by path.
files may hold a number of coverage-file records for the
same path; they are merged, keeping the highest count seen for each
line, so a report read from several LCOV files can be handed to this
directly.
Return one report holding every file of reports.
The line counts of a file present in several reports are merged by taking the highest count of each line: a line that ran in any of the runs was covered.
Return the reports of every file of paths merged into one.
A path that does not exist is skipped, so the caller can pass the files a run was expected to write without checking each one.
Return the source file name points at.
The VM records source files the way the compiler saw them: an absolute path for a system library, a path relative to the current directory for a project file, or a path relative to one of Guile’s load path entries. Each is resolved here, in that order, so a report always holds absolute paths and two spellings of one file cannot end up as two rows.
Merge report into this run’s coverage and write the reports.
The report of the run so far is kept per destination directory and
merged with report, so a process that runs several test files ends
with all of them in one report. The merged report is written as LCOV,
as HTML (unless html-dir or the environment turns it off), as
JSON, and as the two XML formats the Java world reads: Cobertura and
JaCoCo. sources names the roots the XML documents report the
files relative to. log, when given, is called with a one-line
summary and with the paths written.
Nothing is written when no file has been measured yet: an empty report is the absence of coverage, and a report of zero lines is not something to publish.
Returns the merged report.
Return the LCOV trace data read from port as a report.
Only SF and DA records carry information here; the other
lines are accepted and ignored, so a file written by genhtml or
by another tool can be read as well.
Return the LCOV trace file at path as a report.
A missing file is an empty report, which is what a run that produced no coverage looks like.
Rebuild the HTML, JSON, Cobertura and JaCoCo reports from the last LCOV run.
lcov-file defaults to the report of the last run,
reports/lcov/veritas.info, and sources names the roots the
two XML documents report the files relative to (the include paths of the
environment by default). Returns the report read back, or #f
when no LCOV file has been written yet, in which case nothing is
written.
Call thunk on a VM whose engine is engine, then put it back.
Guile fires the VM hooks that count instructions only in the
debug engine, so this is what makes coverage - and any other hook
- record anything at all. The engine that was current before is
restored even when thunk raises, which keeps the switch invisible
to the rest of the run.
The switch is bracketed with call-with-fiber-safe-wind, not with
a plain dynamic-wind: a fiber that yields would otherwise put the
old engine back in the middle of the measured run, and everything after
the first suspension would be counted as not covered.
Write report as a Cobertura XML document to path.
Cobertura is the format Jenkins, GitLab, Azure DevOps and SonarQube accept most readily. Every line carries its execution count, each file is a class, and a directory is a package. sources defaults to the source roots of the run, and the filenames are written relative to them.
path defaults to cobertura.xml next to the LCOV file; the
VERITAS_COBERTURA_PATH environment variable overrides it. Returns
the path written.
Write a JSON summary of report to path.
The document holds the totals, and for each file its counts and the
numbers of the lines that never ran, which is what a CI job needs to
annotate a pull request without parsing HTML. path defaults to
coverage-summary.json next to the LCOV file.
Write report as one self-contained HTML page into directory.
The page holds the summary, the table of files and the annotated source of every file, so the whole project’s coverage is one document: nothing is fetched, nothing is split across pages, and it opens from a file:// URL. Source lines that never ran are marked, and the page can be filtered by path.
directory defaults to reports/html, or to whatever
VERITAS_COVERAGE_HTML_DIR says. Returns the path of the page.
Write report as a JaCoCo XML document to path.
JaCoCo’s format is what the coverage gutters of IDEs, SonarQube and Jenkins’ JaCoCo plugin read. Each file becomes a class and a source file, every line carries its missed and covered instruction counts, and the counters at class, package and report level let a tool compute the totals without reading a line.
path defaults to jacoco.xml next to the LCOV file; the
VERITAS_JACOCO_PATH environment variable overrides it. Returns
the path written.
Write report to port in the LCOV trace file format.
Write report to path as LCOV, creating the directory as needed.
Returns path, so a caller can report where the file went.
A short explanation of where two values differ, for an assertion that failed. The expect/got pair already says what the values were; this says which element or character went wrong, which is what a reader misses when the values are long.
Return a short explanation of where expect and got differ.
Only strings, lists and vectors are described; #f means there is
nothing useful to add - the values are equal, or they are of a kind a
positional diff says nothing about.
(render-diff '(1 2 3) '(1 2 9)) ⇒ " lists differ at index 2 (expected 3, got 9); lengths 3 and 3"
Generator combinators for the property-based testing subsystem. Every generator is a closure taking a random state, so the calling convention is always: (lambda (rand) ...). Sizes are picked inside the generator, which saves threading a size argument through the pipeline by hand.
A generator may also carry a shrinker: a procedure of one value that returns a list of smaller replacements, smallest first. A property uses it to walk a failing input down to the smallest value that still fails. Generators made by hand need no shrinker; they simply do not shrink.
#t and #f.
first.last@example.com email addresses.
Return a generator of beer styles.
Return a generator of #t and #f.
Return a generator of car makes and models.
Return a generator of printable ASCII characters, 32 to 126.
Return a generator that always answers value.
Return a generator of first.last@example.com email addresses.
Return a generator of first names.
Return a generator of integers between 0 and max-bound (exclusive).
Call the generator with a random state:
(define gen (generate-integer #:max-bound 100)) (gen (seed->random-state 42))
Return a generator of last names.
Return a generator of lists whose elements come from elem-generator.
Lists are min-len to max-len long.
Return a generator of full names, a first name and a last name.
Return a generator of integers between 0 and max-bound inclusive.
Return a generator that picks one of choices at random.
A choice is usually a constant, so there is nothing to shrink.
Return a generator of real numbers in the range min to max.
Return a generator of printable-ASCII strings.
Strings are min-len to max-len characters long.
Return a generator of lowercase symbols, min-len to max-len long.
Symbols stay unescaped because the characters are always a-z.
Return a generator of lists holding one value from each of generators.
Return a generator of vectors whose elements come from elem-generator.
Return smaller replacements for value taken from generator.
The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.
#:use-module, as
lists.
Return the files from files that the changes in changed can affect.
A test is affected when it imports a module the change reaches, when the change touched the test file itself, when a changed file’s name matches the test’s name, or when the test names the changed file as a string: the last two cover a project whose tests name their subject instead of importing it, and the data a test cannot run without.
Return the changed files and the subset of files they affect.
Return the files a change touched, relative to the repository root.
Without since, the working tree and the index are compared with
HEAD, and untracked files are included. With since, the
working tree is compared with that revision, which is what a CI job
wants: the diff of a branch against its base.
Return the modules text imports with #:use-module, as
lists.
Only the plain form is read - (veritas reporter) - because a
#:select list belongs to the importer, not to the dependency.
Return the module path defines, as a list of strings.
"src/veritas/reporter.scm" is ("veritas" "reporter").
root is the directory the path is relative to, so a project with a
different layout can say so.
Return the path-like string literals text mentions.
A test that reads a data file names it as a string, so those strings are the other dependency edge a project has: not a module, but a file the test cannot run without. Only strings that look like a path are kept - one with a slash, a dot or both - because every other literal in a test is prose.
Test doubles: spies, stubs and temporary bindings, in the spirit of a mocking library. Scheme needs less of one than most languages do - a module’s procedures can be passed in, so a fake is often just a lambda - but recording what was called, and checking it after the fact, is worth having.
(define send (make-spy real-send)) ; records every call (define clock (make-stub #:returns 0)) ; ignores its arguments
(test "the receipt is sent once" (lambda (vc) (send "ada" "thanks") ; the call under test ((assert-called-with send ’("ada" "thanks")) vc) ((assert-call-count send #:expect 1) vc)))
The assertions read the spy when they run, not when they are built, which is why they are written as a case lambda: a spy’s calls are test-time state.
A stub can also stand in for a module binding without editing the module, as long as the call site reads that binding when it runs - see ‘with-temporary-binding’ for exactly when that is.
#t when spy was called with exactly args.
#t when value was made by make-spy or
make-stub.
Assert that spy was called exactly expect times.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-call-count spy #:expect 2) vc)).
Assert that spy was called at least once.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-called spy) vc)).
Assert that spy was called with exactly args.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-called-with spy '(1 2)) vc)).
Return a procedure that calls proc and records every call.
The recording is read back with spy-calls, spy-results and
spy-call-count. name is only used in the messages of the
assert-* checks below.
(define double (make-spy (lambda (x) (* x 2)))) (double 3) (spy-calls double) ⇒ ((3))
Return a stand-in procedure that ignores its arguments and returns returns.
A stub is a spy whose procedure ignores its arguments, so
spy-calls, spy-call-count and the assert-* checks
work on it unchanged.
Return how many times spy was called.
Return #t when spy was called with exactly args.
Return the argument lists spy was called with, oldest first.
Return what spy returned, oldest first, one entry per call.
Return #t when value was made by make-spy or
make-stub.
Call thunk with module’s var set to value, and restore it.
The previous value comes back even when thunk raises, so substituting a dependency is safe to try.
Guile resolves a call to a known procedure while it compiles, so a
compiled call site does *not* see the change: a call written in a module
the compiler can see through keeps calling the original procedure.
Measured on Guile 3.0, swapping a binding is observed by
module-ref, by code passed to eval, and by call sites in a
module declared #:declarative? #f; it is not observed by a call
compiled in a declarative module, which is the default.
That leaves the ordinary Scheme answer, which needs no substitution at
all: pass the dependency in, or hand the caller a make-spy
recording. Use this when the code under test reaches the binding at run
time.
(with-temporary-binding (resolve-module '(my app)) 'now
(lambda () 0)
(lambda ()
(assert-equal #:expect 0
#:got (eval '((@ (my app) now))
(resolve-module '(my app))))))
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Return the result of a test named name holding results.
file and line are where the test was declared, when that is known, duration is how many seconds it took, and attempts is how many times the test ran - more than one when a retry was needed. They are what a report needs to place a test and time it: the JUnit document writes them out, and a CI annotation uses the file and line to attach a failure to the right place in the source.
This record type has the following fields:
index
output
name
successful
This record type has the following fields:
name
results
This record type has the following fields:
name
results
file
line
duration
attempts
This record type has the following fields:
result
output
This record type has the following fields:
name
test-results
output
This record type has the following fields:
index
output
results
result-and-output record.
#t when value is a collector made by
make-collector.
assertion-result reported for a timeout.
#t when proc can be called with no arguments at all.
#f when that is
unknown.
Evaluate thunks one after another, keeping their order.
The result list lines up with the input list.
Run thunks concurrently in fibers and collect what they return.
Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.
(evaluate-thunks-concurrently
(list (lambda () (sleep 1) 'delayed)
(lambda () 'instant)))
⇒ (delayed instant)
Return the sum of the count alists a and b.
Call thunk between before and after, safely under fibers.
Fibers suspends a task by unwinding to the scheduler and resumes it by
winding back in, so a plain dynamic-wind runs its guards again
every time the task yields - and a port operation is enough to yield. A
guard that has a side effect - a module binding swapped, a directory
removed, a VM engine changed - would then fire in the middle of the
thunk. dynamic-wind* from (fibers scheduler) skips the
guards during those round trips and runs them when the extent is really
entered and really left, which is what every bracketing helper here
wants.
(call-with-fiber-safe-wind (lambda () (module-set! module 'now 0)) (lambda () (read-the-clock)) (lambda () (module-set! module 'now original)))
Run thunk with an optional deadline of seconds seconds.
When seconds is a positive real number the work runs in its own
fiber and, if it has not finished by then, the fiber is aborted and the
result of calling on-timeout is returned instead. When
seconds is #f (the default, meaning no deadline)
thunk is simply called and its result is returned.
thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.
(call-with-timeout 5
(lambda () (expensive-computation))
#:on-timeout (lambda () 'gave-up))
Call thunk and return a result-and-output record.
The record holds what thunk returned together with everything it printed, so a test’s output can be reported next to its result.
Return the outcome of every assertion in xs as a flat list.
Booleans only: #t for a pass, #f for a failure.
Return the assertion outcomes found inside xs.
Knows assertion, pre-test, test and suite results, and recurses into
lists. Unspecified values, benchmarks and #t contribute nothing;
an unrecognized type raises.
Record item in the active collector, and return item.
Does nothing else when no collector is bound, which is what building a
case outside a body - at load time, or inside an assertion that runs
later - looks like. #f is never recorded: a case a filter
dropped is not a case.
Return what collector recorded, in the order the items were built.
Return #t when value is a collector made by
make-collector.
Return values without the entries that are the same object as an earlier one.
Identity, not equality, decides: the same case built twice is two cases, while the same case both recorded and returned by one form is one.
Print and log every test outcome inside all-results using reporter client vc.
Iterates over the compilation of all-results. Evaluates types dynamically:
suite-result: Displays suite output, followed by inner test
results.
pre-test-result: Logs the test identity via vc and displays
inner assertions.
benchmark-result: Dispatches to log-benchmark.
Throws an explicit error if an unrecognized result type is processed.
Print the captured output of each record in assertion-results.
A record that is not an assertion gets printed as an error instead of bringing down the reporter.
Return the cases value stands for, in order, leaving out what is not one.
case? recognises a case, expand evaluates a thunk-shaped body form, lists (of any depth) are walked, and anything else - an unspecified value, a number, a nested task in a place that wants assertions - contributes nothing. depth bounds the recursion, so a thunk that keeps returning thunks is a value that is ignored rather than a run that never ends.
Flatten xs into a single list, however deeply it nests.
An atom that is not a list comes back wrapped in a one-element list.
Return environment variable var, or default when it is unset.
var is a symbol. Anything that goes wrong while reading it yields default as well.
Return a fresh collector: a mutable list head of recorded items.
Build the failed assertion-result reported for a timeout.
name is the timed task, timeout the seconds it was given and vc the client used to report it. A timeout fails the run exactly like any other failed assertion.
Return #t when proc can be called with no arguments at all.
A body form is wrapped in such a thunk by the DSL, and a thunk of this shape is therefore a body still to be evaluated rather than a case to run.
Return how many arguments proc needs, or #f when that is
unknown.
Guile reports a procedure’s arity as the list (required optional
rest?), or as several values on older versions; both are read here.
Return the files, suites, tests and assertions result records.
result is a result record or a list of them, at any depth: an
assertion counts as one assertion, a test as one test, a suite as one
suite, and a suite or a test that holds further results counts what it
holds as well. files is how many test files the results stand for
- one for the run of a single file - and is added to the files
count.
A benchmark measures instead of asserting, and a task that raised arrives as the error string, so neither contributes a test or an assertion of its own.
(result-counts (list a-suite a-test) #:files 1) ⇒ ((files . 1) (suites . 1) (tests . 3) (assertions . 7))
Return counts as one readable line.
(result-counts-summary '((files . 2) (suites . 3) (tests . 12) (assertions . 48))) ⇒ "2 files, 3 suites, 12 tests and 48 assertions"
Properties: one assertion, many generated inputs. A property draws a value from each of its generators, calls its thunk, and repeats; a thunk that fails for one input makes the property fail, and the failing input is shrunk to the smallest value that still fails before it is reported. The seed is reported too, so a failure can be replayed with VERITAS_SEED.
Return a property named name, a task like a test.
generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.
(property \"reverse of reverse is the original list\"
#:generators (list (generate-list (generate-integer)))
#:thunk (lambda (xs)
(assert-equal #:expect xs
#:got (reverse (reverse xs)))))
A failing input is shrunk to the smallest value from the same generators
that still fails, and both values are reported. seed makes the
draw reproducible; without one, VERITAS_SEED is used, and failing
that a fresh seed is drawn and reported with the failure.
benchmark-result record.
#t when the run is inside a GitHub Actions workflow.
#f.
junit-report-data, to
path.
Print msg on a line of its own.
(log "Hello World") ;; Outputs: \nHello World
Announce an assertion that is about to be checked, by name.
Report benchmark results, given a benchmark-result record.
Log a message for debugging.
Report a runtime error.
Report a failing check.
Report a test that passed only after a retry.
Log an informational message.
Report a passing check.
Announce a suite by name.
Announce a test by name.
Default value:
#<procedure 7f5add991190 at veritas/reporter.scm:331:18 ()>
Default value:
#f
Append one row of counts to the GitHub Actions step summary.
The table header is written the first time, and every test file of the
run adds a row, so the summary shows a file beside its numbers and still
says something when the run is failing. Returns summary-path, or
#f when there is no summary to append to.
Send msg to the reporter chosen in vc.
The client names one under the key 'reporter; anything else falls
back to emoji-reporter. log-type picks the entry to call
('log-pass, 'log-fail, and so on).
Create every directory in front of path, ignoring ones that exist.
Return #t when the run is inside a GitHub Actions workflow.
Return the JUnit report of all-results as data rather than a file.
The result is an alist holding the numbers a testsuites element
needs - tests, failures, errors and skipped
- and xml, the testsuite elements themselves as a string.
Writing the file is write-junit-document’s job; a separate
process can be handed this alist and its run joined to others with
merge-junit-data, which is how one document is made out of the
parallel runs of --jobs.
Combine the JUnit data of several runs into one.
The counts are summed and the testsuite elements are concatenated
in the order given, so the merged document keeps the order the runs were
scheduled in.
Undocumented procedure.
Return the JUnit data recorded at path, or #f.
#f means there is nothing to merge: the file is missing,
unreadable or not the alist write-junit-data writes.
Print workflow-command annotations for the failures in all-results.
GitHub keeps ten error annotations per step, so only the first limit are written and a notice says how many were left out; the JUnit document carries every one of them. Returns how many annotations were written.
Write JUnit data to path so another process can read it back.
An alist is plain Scheme data, so read restores it exactly: this
is how a worker process hands its report to the process that merges
them. Returns path.
Write JUnit data, as made by junit-report-data, to
path.
seconds is the wall clock of the run and becomes the document’s total time; for a run split over worker processes it is the wall clock of the whole parallel run, not the sum of the workers. Returns path.
Every testcase carries the file it came from, the suite path it
sat in and its own duration; each testsuite sums the durations of
its cases.
Write all-results to path as one JUnit XML document.
seconds is the wall clock of the run and becomes the document’s
total time. Every test becomes a testcase inside a
testsuite named after the suites it sits in; loose tests land in
a suite called veritas. A benchmark is reported as skipped,
because it measures instead of asserting, and a task that raised is
reported as an error rather than a failure. Returns path.
Every testcase carries the file it came from, the suite path it
sat in and its own duration; each testsuite sums the durations of
its cases.
((yellow . "\x1b[0;33m") (magenta . "\x1b[0;35m") (green . "\x1b[0;32m") (default . "\x1b[0;39m") (bold . "\x1b[1m") (red . "\x1b[0;31m") (blue . "\x1b[0;34m") (cyan . "\x1b[0;36m") (reset . "\x1b[0m"))
((reporter-name . emoji-reporter) (log-flaky . #<procedure 7f5add991000 at veritas/reporter.scm:145:13 (msg)>) (log-pass . #<procedure 7f5add991010 at veritas/reporter.scm:149:47 (msg)>) (log-fail . #<procedure 7f5add991020 at veritas/reporter.scm:159:14 (msg)>) (log-debug . #<procedure 7f5add991030 at veritas/reporter.scm:166:15 (msg)>) (log-info . #<procedure 7f5add991040 at veritas/reporter.scm:173:14 (msg)>) (log-error . #<procedure 7f5add991050 at veritas/reporter.scm:180:15 (msg)>) (log-test . #<procedure 7f5add991060 at veritas/reporter.scm:187:14 (msg)>) (log-benchmark . #<procedure 7f5add991070 at veritas/reporter.scm:200:19 (msg)>) (log-suite . #<procedure 7f5add991080 at veritas/reporter.scm:234:15 (msg)>) (log-assert . #<procedure 7f5add991090 at veritas/reporter.scm:248:16 (msg)>))
((reporter-name . github-reporter) (log-flaky . #<procedure junit-quiet (msg)>) (log-pass . #<procedure junit-quiet (msg)>) (log-assert . #<procedure junit-quiet (msg)>) (log-test . #<procedure junit-quiet (msg)>) (log-suite . #<procedure junit-quiet (msg)>) (log-debug . #<procedure junit-quiet (msg)>) (log-benchmark . #<procedure junit-quiet (msg)>) (log-fail . #<procedure 7f5add991620 at veritas/reporter.scm:955:14 (msg)>) (log-error . #<procedure 7f5add991630 at veritas/reporter.scm:958:15 (msg)>) (log-info . #<procedure 7f5add991640 at veritas/reporter.scm:961:14 (msg)>))
((reporter-name . json-reporter) (log-flaky . #<procedure 7f5add6f7a00 at veritas/reporter.scm:343:14 (msg)>) (log-pass . #<procedure 7f5add6f79e0 at veritas/reporter.scm:347:46 (msg)>) (log-fail . #<procedure 7f5add6f79c0 at veritas/reporter.scm:356:14 (msg)>) (log-error . #<procedure 7f5add6f79a0 at veritas/reporter.scm:362:15 (msg)>) (log-test . #<procedure 7f5add6f7980 at veritas/reporter.scm:368:14 (msg)>) (log-benchmark . #<procedure 7f5add6f7960 at veritas/reporter.scm:374:19 (msg)>) (log-debug . #<procedure 7f5add6f7940 at veritas/reporter.scm:401:15 (msg)>) (log-info . #<procedure 7f5add6f7920 at veritas/reporter.scm:407:14 (msg)>) (log-suite . #<procedure 7f5add6f7900 at veritas/reporter.scm:413:15 (msg)>) (log-assert . #<procedure 7f5add6f78e0 at veritas/reporter.scm:419:16 (msg)>))
((reporter-name . junit-reporter) (log-flaky . #<procedure junit-quiet (msg)>) (log-pass . #<procedure junit-quiet (msg)>) (log-assert . #<procedure junit-quiet (msg)>) (log-test . #<procedure junit-quiet (msg)>) (log-suite . #<procedure junit-quiet (msg)>) (log-debug . #<procedure junit-quiet (msg)>) (log-benchmark . #<procedure junit-quiet (msg)>) (log-fail . #<procedure 7f5add991518 at veritas/reporter.scm:828:14 (msg)>) (log-error . #<procedure 7f5add991528 at veritas/reporter.scm:831:15 (msg)>) (log-info . #<procedure 7f5add991538 at veritas/reporter.scm:834:14 (msg)>))
((reporter-name . monotone-reporter) (log-flaky . #<procedure 7f5add9910d8 at veritas/reporter.scm:263:14 (msg)>) (log-pass . #<procedure 7f5add9910e8 at veritas/reporter.scm:265:50 (msg)>) (log-fail . #<procedure 7f5add9910f8 at veritas/reporter.scm:269:14 (msg)>) (log-error . #<procedure 7f5add991108 at veritas/reporter.scm:274:15 (msg)>) (log-test . #<procedure 7f5add991118 at veritas/reporter.scm:279:14 (msg)>) (log-benchmark . #<procedure 7f5add991128 at veritas/reporter.scm:284:19 (msg)>) (log-debug . #<procedure 7f5add991138 at veritas/reporter.scm:304:15 (msg)>) (log-info . #<procedure 7f5add991148 at veritas/reporter.scm:309:14 (msg)>) (log-suite . #<procedure 7f5add991158 at veritas/reporter.scm:314:15 (msg)>) (log-assert . #<procedure 7f5add991168 at veritas/reporter.scm:320:16 (msg)>))
#f.
#t when it passed and #f when
not.
Walk the cleaned file tree dir and collect candidate test files.
Descends through scan-dir, test-top-dir and
test-module, then hands the contents of a folder named
test-type to flatten-test-files.
Print the global totals of a run: its files, suites, tests and assertions.
Return the totals of the workers at indexes added up.
A worker that left no counts - it failed before its file ran - contributes nothing, because there is nothing to count.
Return the coverage of the workers at indexes merged into one report.
A worker that was not asked for coverage left no fragment, and a run without coverage merges to an empty report, which is then left alone.
Return the number of worker processes value asks for, or
#f.
#f or an absent value means the run stays in one process. A
value that is not a whole number of processes is an error rather than a
silent fallback, because a CI job that asked for parallelism and got
none should be told.
(parse-jobs "4") ⇒ 4
Return the (index . total) a partition string describes, or
#f.
A partition is written "2/3": this machine runs the second of
three slices. A malformed or impossible partition raises instead of
quietly running everything, because that would hide a misconfigured CI
job.
Return the files of partition index of total from files.
The split is round-robin over files, so the same project always splits the same way and the partitions together are exactly files.
(partition-files '("a" "b" "c" "d" "e") 2 3)
⇒ ("b" "e")
Load one test file and call its entrypoint.
test-file is a path relative to the test root and test-type
is the module prefix, so "math/algebra.scm" with unit
loads (veritas unit math algebra). entrypoint-function
names the zero-argument procedure to call in that module.
Run one test file, returning #t when it passed and #f when
not.
veritas-run reports a failing file by exiting the process, and a
file that cannot even be loaded raises; either would end the whole
single-process run and take the files after it, and the totals of the
run, with it. Both are caught here and turned into a returned answer,
with what went wrong printed against the file.
Run files one after another in this process, returning the failed ones.
A file that fails no longer ends the run: the files after it still run,
so the run reports every failure and its global totals before it exits,
which is what --jobs already did for the files it spread over
workers.
When junit-data-paths is given, the ith file leaves its JUnit report at that path as data instead of writing it as the run’s document, so the caller can merge one document out of all of them.
Find every test file under the scan root and run each one.
Behaviour comes from the environment:
VERITAS_SCAN_DIR: Root folder to scan. Defaults to "./".
VERITAS_TEST_TOP_DIR: The subfolder boundary identifier
containing test suites. Defaults to "test".
VERITAS_ENTRYPOINT: The signature method executed inside test
targets. Defaults to "spec".
VERITAS_TEST_MODULE: Top level test parent folder filter.
Defaults to "veritas".
VERITAS_TEST_TYPE: Category discriminator keyword used during
routing. Defaults to "unit".
VERITAS_SHUFFLE_FILES: Set to string "true" to shuffle
execution sequences using a Fisher-Yates algorithm. Defaults to
"true".
VERITAS_SEED: the seed every shuffle of this run uses, reported
when a run is shuffled so a failing order can be replayed.
VERITAS_PARTITION: INDEX/TOTAL to run only one slice of
the discovered files, which is how a run is split across CI machines.
VERITAS_JOBS: how many worker processes to run the discovered
files in. Absent or 1 means one process, as before; 2 and
up run one file per process and merge their output, their record of
failures and their JUnit reports.
VERITAS_COVERAGE: "true" measures code coverage even for a
test file that does not ask for it, and the worker fragments are merged
into one report.
Return the files of files a change can reach.
Without changed? or since every file comes back, and the run is the whole suite it always was. With them, the changed files are read from git, the modules they reach are followed through imports, and only the test files that import one of those modules - or that the change touched - come back.
A change that touches nothing returns every file rather than none: a run is never silently empty, because an empty run looks exactly like a passing one.
Return the elements of xs in random order (Fisher-Yates).
Shuffling happens on a vector copy, so xs is left alone.
seed makes the order reproducible, which is how a failing order is
replayed; without it a fresh seed is pulled from
random-state-from-platform on every call, so two calls on the
same list differ. Lists of one element or fewer come back as they are.
(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
(fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Return a new seed for a shuffle, drawn from the platform’s entropy source.
Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.
Snapshot testing: compare a value with the one recorded the first time the check ran. A missing snapshot is written and the check passes, so a new test does not fail merely for being new. A mismatch fails and shows both sides, and VERITAS_UPDATE_SNAPSHOTS=true rewrites every snapshot the run reaches, which is how an intended change is accepted.
A snapshot is Scheme data, read back with ‘read’: lists, vectors, strings, numbers, symbols and alists travel exactly. Procedures, records and ports do not, so snapshot the string they print instead.
#t when snapshots are to be rewritten instead of compared.
Assert that got matches the snapshot recorded under name.
name labels the assertion and names the file, so it is required;
the file lives under VERITAS_SNAPSHOT_DIR (default
test/snapshots) or under dir. The first run writes the
snapshot and passes. Later runs compare with equal?, and a
mismatch fails with both values. A snapshot that cannot be read is a
failure too, never a silent rewrite.
A got that is a procedure is called with no arguments when the assertion runs, so state that only exists at test time can be snapshotted.
(assert-snapshot #:name "the rendered cart"
#:got (lambda () (render (make-cart))))
Return the file holding the snapshot called name.
dir overrides VERITAS_SNAPSHOT_DIR, which itself defaults
to test/snapshots.
Return #t when snapshots are to be rewritten instead of compared.
Set VERITAS_UPDATE_SNAPSHOTS to true to accept the current
value of every snapshot a run reaches.
Table-driven tests, in the spirit of pytest’s parametrize, JUnit’s parameterized tests and Go’s table tests. One test definition is expanded over a list of cases and each case is reported as its own test, so a failure points at the exact input.
test-each, but case->suite returns a suite.
Index cases by key, giving an alist of key to case.
Handy for tables with named columns.
Like test-each, but case->suite returns a suite.
Use it when one case needs several tests rather than one.
Expand cases into one test per case, under a suite named group-name.
case->test takes one case and returns a test:
(test-each "integer addition"
'((0 0 0) (1 2 3) (10 20 30))
(lambda (case)
(let ((a (list-ref case 0))
(b (list-ref case 1))
(expected (list-ref case 2)))
(test (format #f "~a + ~a = ~a" a b expected)
(assert-equal #:expect expected #:got (+ a b))))))
Default value:
()
Default value:
#f
Return the frames stored at path, oldest first.
Print the trace at path as a numbered timeline.
Print the trace of the last run, and return whether there was one.
Record one step of the run when tracing is on.
kind is a symbol such as test-start, assertion or
test-end, name is what the step is about, and detail
is a short string a reader can use to tell one frame from another.
Return where the last trace of this run is written.
Forget the frames recorded so far.
Write the frames recorded so far to path and return it.
The newest frame is last in the file, so the timeline reads top to bottom.
suite
macro.
test
macro.
equal? to expect.
eqv? to expect.
eq? to expect.
equal? to expect.
eqv? to expect.
#f.
eq? to expect.
equal? to expect.
eqv? to expect.
equal? to one of the elements of list
expect.
#t.
#t when name should run in a last-failed rerun.
#f.
#t and #f.
first.last@example.com email addresses.
eq? to expect.
#t when a and b differ by less than
epsilon.
equal? to expect.
eqv? to expect.
eq? to expect.
#t when a and b differ by less than
epsilon.
#f.
#f.
equal? to one of the elements of list
expect.
#t.
Undocumented macro.
Undocumented macro.
Run thunks concurrently in fibers and collect what they return.
Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.
(evaluate-thunks-concurrently
(list (lambda () (sleep 1) 'delayed)
(lambda () 'instant)))
⇒ (delayed instant)
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Default value:
#f
Default value:
#f
Default value:
#f
Default value:
"reports/last-failed.scm"
Default value:
#f
Default value:
((files . 0) (suites . 0) (tests . 0) (assertions . 0) (flaky . 0))
Default value:
#f
Default value:
0
Define a suite named name; the procedure behind the suite
macro.
The arguments after name are the children. Each form is evaluated
when the suite runs, and every test, suite or benchmark built while it
is evaluated is a child: one already built, a list of them, or a thunk
of them are all understood. Children run concurrently unless
concurrent? is #f, and in shuffled order unless
shuffle? is #f. before-all and after-all run
once around them, timeout caps the whole suite in seconds
including those hooks, skip? drops it from the run and tags
decide which filter may select it.
Define a test named name; the procedure behind the test
macro.
The arguments after name are the body. Each form is evaluated when the test runs, and every assertion built while it is evaluated is a case: an assertion, a list of them, a thunk of them, or a helper procedure that builds them are all understood. before and after run around the body, timeout caps the whole test in seconds, hooks included, skip? drops it from the run and tags decide which filter may select it.
A body is read in order, because a body is a sequence: one form prepares what the next one checks. shuffle? asks for the order of the body forms to be randomized anyway, which is a way to find checks that depend on each other by accident.
retries reruns a failing test, hooks and body together, up to that many extra attempts. A test that fails and then passes is reported as flaky with its attempt count, so an intermittent failure is visible instead of hidden.
(test "Addition Validation"
#:before (lambda (vc) (display "Initializing..."))
(assert-equal #:expect 4 #:got (+ 2 2)))
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Assert that got is an empty list, vector or string.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Assert that x is #f.
Assert that got is within the inclusive range min–max.
Assert that got has expect elements (list, vector or string length).
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Assert that got is a non-empty list, vector or string.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that got is equal? to one of the elements of list
expect.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Assert that x is #t.
Define a benchmark named name that times thunk over runs runs.
Statistics come from Guile’s statprof, so benchmarks are reported
separately from tests. timeout caps all the runs together in
seconds and skip? drops the benchmark from the run.
Run thunk with an optional deadline of seconds seconds.
When seconds is a positive real number the work runs in its own
fiber and, if it has not finished by then, the fiber is aborted and the
result of calling on-timeout is returned instead. When
seconds is #f (the default, meaning no deadline)
thunk is simply called and its result is returned.
thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.
(call-with-timeout 5
(lambda () (expensive-computation))
#:on-timeout (lambda () 'gave-up))
Return the outcome of every assertion in xs as a flat list.
Booleans only: #t for a pass, #f for a failure.
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Return the seed this run shuffles with, choosing one when none was set.
seed (a number or a numeric string) wins when given, otherwise the
value of VERITAS_SEED is used, and when neither is set a fresh
seed is drawn from the platform. The chosen seed is remembered, so the
order the runner gives the test files and every order chosen here come
from one seed, and the run can be replayed by setting
VERITAS_SEED to the seed that was reported.
(current-seed #:seed 7) ⇒ 7
Wrap each of forms in a thunk, so a body is evaluated when its task runs.
Deferring is what makes a body form contribute both its own value and the cases it builds: the form is evaluated inside the body’s collector, at run time, instead of while the test file is being loaded.
Run result when name matches restriction.
The check is case-insensitive, and a restriction counts as a match when
it appears anywhere in name. A #f restriction accepts
everything; one that does not appear gives #f.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Return the names that failed in all-results, for a last-failed rerun.
Every test whose assertions failed is named. A suite that failed as a
whole, through its own deadline for instance, contributes the names of
the tests it holds, because a suite name is not something a rerun can
select. A failure with no test name attached at all contributes
#all, which makes the next run run everything rather than
silently run nothing. Names come back in the order recorded, without
duplicates.
Return #t when name should run in a last-failed rerun.
only is the record of failing names to honour; it defaults to the
one read from reports/last-failed.scm when
VERITAS_ONLY_FAILED asked for a rerun. #f means there is
no record, so everything runs; an empty list means the last run passed,
so nothing does.
Assert that x is #f.
Return the elements of xs in random order (Fisher-Yates).
Shuffling happens on a vector copy, so xs is left alone.
seed makes the order reproducible, which is how a failing order is
replayed; without it a fresh seed is pulled from
random-state-from-platform on every call, so two calls on the
same list differ. Lists of one element or fewer come back as they are.
(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
(fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Return a new seed for a shuffle, drawn from the platform’s entropy source.
Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.
Return a generator of beer styles.
Return a generator of #t and #f.
Return a generator of car makes and models.
Return a generator of printable ASCII characters, 32 to 126.
Return a generator that always answers value.
Return a generator of first.last@example.com email addresses.
Return a generator of first names.
Return a generator of integers between 0 and max-bound (exclusive).
Call the generator with a random state:
(define gen (generate-integer #:max-bound 100)) (gen (seed->random-state 42))
Return a generator of last names.
Return a generator of lists whose elements come from elem-generator.
Lists are min-len to max-len long.
Return a generator of full names, a first name and a last name.
Return a generator of integers between 0 and max-bound inclusive.
Return a generator that picks one of choices at random.
A choice is usually a constant, so there is nothing to shrink.
Return a generator of real numbers in the range min to max.
Return a generator of printable-ASCII strings.
Strings are min-len to max-len characters long.
Return a generator of lowercase symbols, min-len to max-len long.
Symbols stay unescaped because the characters are always a-z.
Return a generator of lists holding one value from each of generators.
Return a generator of vectors whose elements come from elem-generator.
Return smaller replacements for value taken from generator.
The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is an empty list, vector or string.
Assert that got is within the inclusive range min–max.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Assert that got has expect elements (list, vector or string length).
Return assertion, a procedure of one validation client, as a case to run.
The returned procedure is tagged so the runner can tell an assertion
from a task, and it records having run, which is how an assertion the
body itself consumes - with value or ok?, say - stays an
input to another case instead of becoming a case of its own.
It is also registered with the collector of the body being evaluated, if
there is one. That registration is what keeps a body form that builds
several assertions, such as the body of a let, from contributing
only the value of its last expression. Building an assertion outside
any body - at load time, or inside an assertion that runs later - simply
registers nothing.
Combine what several runs recorded as failing into one record.
records holds one record per run: a list of names, or #f
when a run left no record at all. When every run recorded an empty list
the result is an empty list, so a parallel run that passed does not look
like a run to rerun. When no run left a record the result is #f,
which stands for no information rather than for nothing having failed.
Names are deduplicated and keep the order they were found in.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is a non-empty list, vector or string.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Return a property named name, a task like a test.
generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.
(property \"reverse of reverse is the original list\"
#:generators (list (generate-list (generate-integer)))
#:thunk (lambda (xs)
(assert-equal #:expect xs
#:got (reverse (reverse xs)))))
A failing input is shrunk to the smallest value from the same generators
that still fails, and both values are reported. seed makes the
draw reproducible; without one, VERITAS_SEED is used, and failing
that a fresh seed is drawn and reported with the failure.
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Return the test names recorded as failing at path, or #f.
path defaults to the record of the whole run. #f means
there is no record of a previous run: the file is missing or unreadable,
and nothing is filtered.
Return the counts recorded at path, or #f.
#f means there is nothing to add up: the file is missing,
unreadable or not the alist write-result-counts writes.
Return xs in the order this run’s seed prescribes.
Both the runner, which shuffles test files, and veritas-run,
which shuffles tasks, tests and cases, shuffle through this, so a single
seed reproduces a whole run.
Split args, the syntax of an argument list, into its parts.
Two values come back: the #:keyword value pairs in the order they
were written, and the forms that are not part of a pair, also in order.
The test and suite macros use it to tell a body from the
flags around it, wherever those flags were written.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Assert that got is equal? to one of the elements of list
expect.
Run tasks, the top-level tests, suites and benchmarks.
shuffle? randomizes the order of the tasks and concurrent?
runs them in parallel fibers. reporter names a reporter,
otherwise VERITAS_REPORTER decides. seed makes the shuffle
reproducible, and the seed actually used is reported, so a failing order
can be replayed by setting VERITAS_SEED to it.
with-coverage? measures coverage - VERITAS_COVERAGE=true
asks for it too - and debug? prints the raw results.
Coverage is written where coverage-dir (default
VERITAS_COVERAGE_DIR, "reports/lcov") says, with the HTML
report in coverage-html-dir (default
VERITAS_COVERAGE_HTML_DIR, "reports/html"). Only the
files under coverage-include (default
VERITAS_COVERAGE_INCLUDE, the current directory) are reported,
minus the ones under coverage-exclude (default
VERITAS_COVERAGE_EXCLUDE).
Assert that x is #t.
Record the names that failed, so the next run can rerun just those.
Each worker process writes its own record - the path is a parameter -
and the --jobs parent merges them into the run’s, which is why
the writing and the merging are separate.
Record names at path as the tests a rerun should select.
An empty list is written when everything passed, which is what makes the
next VERITAS_ONLY_FAILED run do nothing instead of rerunning the
world. A trouble with the file is swallowed: the run has already
happened.
Write counts to path so another process can add them up.
An alist is plain Scheme data, so read restores it exactly: this
is how a worker process hands its totals to the process that runs the
other files. Returns path.