Next: (veritas assert), Up: API
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Run thunks concurrently in fibers and collect what they return.
Blocks until every thunk has reported back through a shared channel. A thunk that raises sends a formatted error string instead of a value, so one failure cannot hang the run. Results come back in the order of thunks, not in completion order: two runs of one suite have to report the same lines, and which fiber finished first is not part of a result.
(evaluate-thunks-concurrently
(list (lambda () (sleep 1) 'delayed)
(lambda () 'instant)))
⇒ (delayed instant)
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Undocumented macro.
Default value:
#f
Default value:
#f
Default value:
#f
Default value:
"reports/last-failed.scm"
Default value:
#f
Default value:
((files . 0) (suites . 0) (tests . 0) (assertions . 0) (flaky . 0))
Default value:
#f
Default value:
0
Default value:
()
Default value:
#f
Define an And (continuation) step named name.
Define a But (continuation) step named name.
Define a Given (setup) step named name.
Define a Given/When/Then scenario named name; the procedure behind the macro.
A scenario is a suite, so its steps run in declaration order. Keywords
are forwarded to suite, and steps come from given,
when-step, then, and-step and but-step.
timeout, when given, caps the whole scenario in seconds, exactly
like suite; a step can carry its own #:timeout as well,
since steps are ordinary tests.
(scenario "emptying the cart"
(given "a cart with one item"
(assert-equal #:expect 1 #:got (cart-size cart)))
(when-step "the cart is emptied"
(empty-cart! cart))
(then "the cart is empty"
(assert-true (cart-empty? cart))))
Define a step named name, labeled with label; the procedure behind the step macros.
A step is an ordinary test, so it takes the same keywords and cases.
Define a suite named name; the procedure behind the suite
macro.
The arguments after name are the children. Each form is evaluated
when the suite runs, and every test, suite or benchmark built while it
is evaluated is a child: one already built, a list of them, or a thunk
of them are all understood. Children run concurrently unless
concurrent? is #f, and in shuffled order unless
shuffle? is #f. before-all and after-all run
once around them, timeout caps the whole suite in seconds
including those hooks, skip? drops it from the run and tags
decide which filter may select it.
Define a test named name; the procedure behind the test
macro.
The arguments after name are the body. Each form is evaluated when the test runs, and every assertion built while it is evaluated is a case: an assertion, a list of them, a thunk of them, or a helper procedure that builds them are all understood. before and after run around the body, timeout caps the whole test in seconds, hooks included, skip? drops it from the run and tags decide which filter may select it.
A body is read in order, because a body is a sequence: one form prepares what the next one checks. shuffle? asks for the order of the body forms to be randomized anyway, which is a way to find checks that depend on each other by accident.
retries reruns a failing test, hooks and body together, up to that many extra attempts. A test that fails and then passes is reported as flaky with its attempt count, so an intermittent failure is visible instead of hidden.
(test "Addition Validation"
#:before (lambda (vc) (display "Initializing..."))
(assert-equal #:expect 4 #:got (+ 2 2)))
Define a Then (outcome) step named name.
Define a When (action) step named name.
Called when-step rather than when so it does not shadow
Guile’s when syntax.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Assert that spy was called exactly expect times.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-call-count spy #:expect 2) vc)).
Assert that spy was called at least once.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-called spy) vc)).
Assert that spy was called with exactly args.
Checked when the assertion runs, so it is written as a case lambda:
(lambda (vc) ((assert-called-with spy '(1 2)) vc)).
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Assert that got is an empty list, vector or string.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is equal? to expect.
The usual choice for lists, strings, vectors and other compound values.
Assert that got is eqv? to expect.
Good for numbers, characters and symbols, where value equivalence is what matters.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Assert that x is #f.
Assert that got is within the inclusive range min–max.
Assert that got has expect elements (list, vector or string length).
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Assert that got is a non-empty list, vector or string.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that got is equal? to one of the elements of list
expect.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Assert that got matches the snapshot recorded under name.
name labels the assertion and names the file, so it is required;
the file lives under VERITAS_SNAPSHOT_DIR (default
test/snapshots) or under dir. The first run writes the
snapshot and passes. Later runs compare with equal?, and a
mismatch fails with both values. A snapshot that cannot be read is a
failure too, never a silent rewrite.
A got that is a procedure is called with no arguments when the assertion runs, so state that only exists at test time can be snapshotted.
(assert-snapshot #:name "the rendered cart"
#:got (lambda () (render (make-cart))))
Assert that x is #t.
Define a benchmark named name that times thunk over runs runs.
Statistics come from Guile’s statprof, so benchmarks are reported
separately from tests. timeout caps all the runs together in
seconds and skip? drops the benchmark from the run.
Run thunk with an optional deadline of seconds seconds.
When seconds is a positive real number the work runs in its own
fiber and, if it has not finished by then, the fiber is aborted and the
result of calling on-timeout is returned instead. When
seconds is #f (the default, meaning no deadline)
thunk is simply called and its result is returned.
thunk must return a single value. The caller never waits for more than seconds, and the runaway fiber is terminated rather than left running.
(call-with-timeout 5
(lambda () (expensive-computation))
#:on-timeout (lambda () 'gave-up))
Index cases by key, giving an alist of key to case.
Handy for tables with named columns.
Return the outcome of every assertion in xs as a flat list.
Booleans only: #t for a pass, #f for a failure.
Assert that the string got matches pattern.
pattern is a string or a compiled regexp.
(assert-matches #:pattern "^veri.*s$" #:got "veritas")
Assert that the container expect contains got.
Works for strings (substring), lists and vectors (membership) and hash tables (key membership).
(assert-contains #:expect '(1 2 3) #:got 2) (assert-contains #:expect "hello world" #:got "world")
Return the seed this run shuffles with, choosing one when none was set.
seed (a number or a numeric string) wins when given, otherwise the
value of VERITAS_SEED is used, and when neither is set a fresh
seed is drawn from the platform. The chosen seed is remembered, so the
order the runner gives the test files and every order chosen here come
from one seed, and the run can be replayed by setting
VERITAS_SEED to the seed that was reported.
(current-seed #:seed 7) ⇒ 7
Wrap each of forms in a thunk, so a body is evaluated when its task runs.
Deferring is what makes a body form contribute both its own value and the cases it builds: the form is evaluated inside the body’s collector, at run time, instead of while the test file is being loaded.
Run result when name matches restriction.
The check is case-insensitive, and a restriction counts as a match when
it appears anywhere in name. A #f restriction accepts
everything; one that does not appear gives #f.
Return an assertion that thunk raises.
It fails if thunk returns normally. name labels it in the
report and timeout caps it in seconds, as in assert.
(assert-error (lambda () (sum-list-elements '(1 2 "a" 4)))
#:name "Should fail when list contains non-numbers")
Return the names that failed in all-results, for a last-failed rerun.
Every test whose assertions failed is named. A suite that failed as a
whole, through its own deadline for instance, contributes the names of
the tests it holds, because a suite name is not something a rerun can
select. A failure with no test name attached at all contributes
#all, which makes the next run run everything rather than
silently run nothing. Names come back in the order recorded, without
duplicates.
Return #t when name should run in a last-failed rerun.
only is the record of failing names to honour; it defaults to the
one read from reports/last-failed.scm when
VERITAS_ONLY_FAILED asked for a rerun. #f means there is
no record, so everything runs; an empty list means the last run passed,
so nothing does.
Assert that x is #f.
Return the elements of xs in random order (Fisher-Yates).
Shuffling happens on a vector copy, so xs is left alone.
seed makes the order reproducible, which is how a failing order is
replayed; without it a fresh seed is pulled from
random-state-from-platform on every call, so two calls on the
same list differ. Lists of one element or fewer come back as they are.
(equal? (fisher-yates-shuffle '(1 2 3) #:seed 7)
(fisher-yates-shuffle '(1 2 3) #:seed 7))
⇒ #t
Return a new seed for a shuffle, drawn from the platform’s entropy source.
Report the seed of a shuffled run and the run can be replayed: the same seed and the same list always produce the same order.
Return a generator of beer styles.
Return a generator of #t and #f.
Return a generator of car makes and models.
Return a generator of printable ASCII characters, 32 to 126.
Return a generator that always answers value.
Return a generator of first.last@example.com email addresses.
Return a generator of first names.
Return a generator of integers between 0 and max-bound (exclusive).
Call the generator with a random state:
(define gen (generate-integer #:max-bound 100)) (gen (seed->random-state 42))
Return a generator of last names.
Return a generator of lists whose elements come from elem-generator.
Lists are min-len to max-len long.
Return a generator of full names, a first name and a last name.
Return a generator of integers between 0 and max-bound inclusive.
Return a generator that picks one of choices at random.
A choice is usually a constant, so there is nothing to shrink.
Return a generator of real numbers in the range min to max.
Return a generator of printable-ASCII strings.
Strings are min-len to max-len characters long.
Return a generator of lowercase symbols, min-len to max-len long.
Symbols stay unescaped because the characters are always a-z.
Return a generator of lists holding one value from each of generators.
Return a generator of vectors whose elements come from elem-generator.
Return smaller replacements for value taken from generator.
The list is smallest first, and is empty when the generator cannot shrink a value - a constant generator, or one a user wrote by hand.
Assert that got is eq? to expect.
The strictest of the equality assertions; use it for symbols, booleans and anything compared by identity.
Assert that got is an empty list, vector or string.
Assert that got is within the inclusive range min–max.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Assert that got has expect elements (list, vector or string length).
Return assertion, a procedure of one validation client, as a case to run.
The returned procedure is tagged so the runner can tell an assertion
from a task, and it records having run, which is how an assertion the
body itself consumes - with value or ok?, say - stays an
input to another case instead of becoming a case of its own.
It is also registered with the collector of the body being evaluated, if
there is one. That registration is what keeps a body form that builds
several assertions, such as the body of a let, from contributing
only the value of its last expression. Building an assertion outside
any body - at load time, or inside an assertion that runs later - simply
registers nothing.
Return a procedure that calls proc and records every call.
The recording is read back with spy-calls, spy-results and
spy-call-count. name is only used in the messages of the
assert-* checks below.
(define double (make-spy (lambda (x) (* x 2)))) (double 3) (spy-calls double) ⇒ ((3))
Return a stand-in procedure that ignores its arguments and returns returns.
A stub is a spy whose procedure ignores its arguments, so
spy-calls, spy-call-count and the assert-* checks
work on it unchanged.
Combine what several runs recorded as failing into one record.
records holds one record per run: a list of names, or #f
when a run left no record at all. When every run recorded an empty list
the result is an empty list, so a parallel run that passed does not look
like a run to rerun. When no run left a record the result is #f,
which stands for no information rather than for nothing having failed.
Names are deduplicated and keep the order they were found in.
Assert that got is not equal? to expect.
Inverse of assert-equal.
Assert that got is not eqv? to expect.
Inverse of assert-eqv.
Assert that string got does not match regular expression pattern.
Inverse of assert-matches.
Assert that the container expect does not contain got.
Inverse of assert-contains.
Return an assertion that thunk does not raise.
It fails if thunk raises. name labels it in the report and
timeout caps it in seconds, as in assert.
Assert that got is not eq? to expect.
Inverse of assert-eq.
Assert that got is a non-empty list, vector or string.
Assert that got differs from expect by at least epsilon.
Inverse of assert-approximate.
Return an assertion comparing got with expect using
compare, a two-argument predicate such as equal?.
The assert-* wrappers below are built on this one. got and
expect are usually plain values, but either may be wrapped in
lazy: (assert-equal #:expect 2 #:got (lazy (+ 1 1)))
evaluates the expression when the assertion runs, inside its
timeout, which is what makes an expensive comparison something a
deadline can catch. A lazy value keeps the lexical scope of the
code that wrote it and is evaluated once per assertion that receives it.
name labels the assertion in the report. timeout caps it in
seconds; #f, the default, means no deadline.
(assert #:expect 3
#:got (+ 1 1 1)
#:compare equal?
#:name "Ensuring addition works")
Return a property named name, a task like a test.
generators is a non-empty list of generators, one per argument of thunk. Each of the runs runs draws one value from every generator and calls the thunk with them; the thunk returns the assertions to check, as a test body does.
(property \"reverse of reverse is the original list\"
#:generators (list (generate-list (generate-integer)))
#:thunk (lambda (xs)
(assert-equal #:expect xs
#:got (reverse (reverse xs)))))
A failing input is shrunk to the smallest value from the same generators
that still fails, and both values are reported. seed makes the
draw reproducible; without one, VERITAS_SEED is used, and failing
that a fresh seed is drawn and reported with the failure.
Assert that got is within epsilon of expect.
The default epsilon is 1.0e-8.
Return #t when a and b differ by less than
epsilon.
Useful for floating point, where exact equality rarely holds.
Return the test names recorded as failing at path, or #f.
path defaults to the record of the whole run. #f means
there is no record of a previous run: the file is missing or unreadable,
and nothing is filtered.
Return the counts recorded at path, or #f.
#f means there is nothing to add up: the file is missing,
unreadable or not the alist write-result-counts writes.
Return the frames stored at path, oldest first.
Print the trace at path as a numbered timeline.
Return xs in the order this run’s seed prescribes.
Both the runner, which shuffles test files, and veritas-run,
which shuffles tasks, tests and cases, shuffle through this, so a single
seed reproduces a whole run.
Print the trace of the last run, and return whether there was one.
Return the file holding the snapshot called name.
dir overrides VERITAS_SNAPSHOT_DIR, which itself defaults
to test/snapshots.
Return #t when snapshots are to be rewritten instead of compared.
Set VERITAS_UPDATE_SNAPSHOTS to true to accept the current
value of every snapshot a run reaches.
Split args, the syntax of an argument list, into its parts.
Two values come back: the #:keyword value pairs in the order they
were written, and the forms that are not part of a pair, also in order.
The test and suite macros use it to tell a body from the
flags around it, wherever those flags were written.
Return how many times spy was called.
Return #t when spy was called with exactly args.
Return the argument lists spy was called with, oldest first.
Return what spy returned, oldest first, one entry per call.
Return #t when value was made by make-spy or
make-stub.
Like test-each, but case->suite returns a suite.
Use it when one case needs several tests rather than one.
Assert that calling thunk raises an error.
By default any error satisfies the assertion. Pass on to require
a specific error key (for example 'wrong-type-arg), or
predicate, a procedure of (key args) returning #t
for the expected error.
(assert-raises (lambda () (/ 1 0))) (assert-raises (lambda () (+ 1 'a)) #:on 'wrong-type-arg)
Expand cases into one test per case, under a suite named group-name.
case->test takes one case and returns a test:
(test-each "integer addition"
'((0 0 0) (1 2 3) (10 20 30))
(lambda (case)
(let ((a (list-ref case 0))
(b (list-ref case 1))
(expected (list-ref case 2)))
(test (format #f "~a + ~a = ~a" a b expected)
(assert-equal #:expect expected #:got (+ a b))))))
Record one step of the run when tracing is on.
kind is a symbol such as test-start, assertion or
test-end, name is what the step is about, and detail
is a short string a reader can use to tell one frame from another.
Return where the last trace of this run is written.
Forget the frames recorded so far.
Assert that got is equal? to one of the elements of list
expect.
Run tasks, the top-level tests, suites and benchmarks.
shuffle? randomizes the order of the tasks and concurrent?
runs them in parallel fibers. reporter names a reporter,
otherwise VERITAS_REPORTER decides. seed makes the shuffle
reproducible, and the seed actually used is reported, so a failing order
can be replayed by setting VERITAS_SEED to it.
with-coverage? measures coverage - VERITAS_COVERAGE=true
asks for it too - and debug? prints the raw results.
Coverage is written where coverage-dir (default
VERITAS_COVERAGE_DIR, "reports/lcov") says, with the HTML
report in coverage-html-dir (default
VERITAS_COVERAGE_HTML_DIR, "reports/html"). Only the
files under coverage-include (default
VERITAS_COVERAGE_INCLUDE, the current directory) are reported,
minus the ones under coverage-exclude (default
VERITAS_COVERAGE_EXCLUDE).
Assert that x is #t.
Call thunk with module’s var set to value, and restore it.
The previous value comes back even when thunk raises, so substituting a dependency is safe to try.
Guile resolves a call to a known procedure while it compiles, so a
compiled call site does *not* see the change: a call written in a module
the compiler can see through keeps calling the original procedure.
Measured on Guile 3.0, swapping a binding is observed by
module-ref, by code passed to eval, and by call sites in a
module declared #:declarative? #f; it is not observed by a call
compiled in a declarative module, which is the default.
That leaves the ordinary Scheme answer, which needs no substitution at
all: pass the dependency in, or hand the caller a make-spy
recording. Use this when the code under test reaches the binding at run
time.
(with-temporary-binding (resolve-module '(my app)) 'now
(lambda () 0)
(lambda ()
(assert-equal #:expect 0
#:got (eval '((@ (my app) now))
(resolve-module '(my app))))))
Record the names that failed, so the next run can rerun just those.
Each worker process writes its own record - the path is a parameter -
and the --jobs parent merges them into the run’s, which is why
the writing and the merging are separate.
Record names at path as the tests a rerun should select.
An empty list is written when everything passed, which is what makes the
next VERITAS_ONLY_FAILED run do nothing instead of rerunning the
world. A trouble with the file is swallowed: the run has already
happened.
Write counts to path so another process can add them up.
An alist is plain Scheme data, so read restores it exactly: this
is how a worker process hands its totals to the process that runs the
other files. Returns path.
Write the frames recorded so far to path and return it.
The newest frame is last in the file, so the timeline reads top to bottom.
"#all"
Next: (veritas assert), Up: API