Documentation for Veritas: PACTA SUNT SERVANDA
Table of Contents
Unit, Integration and Black Box testing framework powered by Lisp (Guile Scheme)
Veritas helps you verify the correctness of software and explore its functioning and robustness. It achieves this by providing a clear structure for writing tests and producing easy-to-read feedback in various formats.
Normal arguments are denoted by their name, keyword arguments are denoted by their name prefixed with #: as in Scheme. When rest arguments are mentioned, it's another terminology for variadic.
Core functions: MUNERA PRIMA
veritas.scm : Core functions
test
test defines a collection of assertions that make up a single test case.
- Parameters:
name(string): A descriptive name for the test.#:debug?(boolean, optional): If#t, prints a debug representation of the test results. Defaults to#f.#:shuffle?(boolean, optional): If#t, shuffles the order of the test cases. Defaults to#t.#:skip?(boolean, optional): If#t, the test is skipped. Defaults to#f.rest(list): A list of assertion procedures to be executed as part of this test.
suite
suite groups a series of tests into a logical suite.
- Parameters:
name(string): A descriptive name for the test suite.#:debug?(boolean, optional): If#t, prints a debug representation of the suite's results. Defaults to#f.#:shuffle?(boolean, optional): If#t, shuffles the order of the tests within the suite. Defaults to#t.#:skip?(boolean, optional): If#t, the entire suite is skipped. Defaults to#f.#:concurrent?(boolean, optional): If#t, runs the tests concurrently using fibers. Defaults to#t.rest(list): A list of test procedures to be executed within this suite.
benchmark
benchmark runs a procedure multiple times and reports performance statistics using statprof.
- Parameters:
#:name(string): A descriptive name for the benchmark.#:runs(number): The number of times to run the provided procedure.#:thunk(function): The procedure to be measured and sampled.#:skip?(boolean, optional): If#t, the benchmark is skipped. Defaults to#f.
veritas-run
veritas-run is the main entry point for executing test suites and benchmarks. It orchestrates the entire test run, collects results, and provides a final report.
- Parameters:
#:shuffle?(boolean, optional): If#t, shuffles the order of the tasks (suites and benchmarks) to be run. Defaults to#t.#:concurrent?(boolean, optional): If#t, runs the tasks concurrently using fibers. Defaults to#t.#:debug?(boolean, optional): If#t, prints debug information about the test results. Defaults to#f.#:reporter(alist of symbol to procedures, optional): The reporter to use for outputting results. Defaults tocolorful-reporter. You can easily write your own too.rest(list): A list of test or benchmark procedures to be executed.
Assertions: ASSERTIONES
assert.scm: Assertion functions
Note: If one desires to delay execution of the #:got argument of the assertions, one can simply quote it and wrap it inside a block with compute as car. You can use compute, computare, or comp as special keywords for lazy evaluation. This works for all assertions, except assert-error and related. See an example here:
;; got will evaluate eagerly
(assert-equal #:expect 1
#:got (+ 10 (- 0 9)))
;; got will evaluate lazily, inside test context
(assert-equal #:expect 1
#:got `(comp (begin
(sleep 4)
(+ 10 (- 0 9)))))
assert-equal
Asserts that two values are equal based on the equal? comparison function.
- Parameters:
#:expect(any): The expected value.#:got(any): The value to test.#:name(string, optional): A descriptive name for the assertion.
assert-eqv
Asserts that two values are equivalent based on the eqv? comparison function. This is typically used for comparing objects that are "the same" in a more fundamental way than equal? would suggest (e.g., numbers and characters).
- Parameters:
#:expect(any): The expected value.#:got(any): The value to test.#:name(string, optional): A descriptive name for the assertion.
assert-eq
Asserts that two values are the exact same object based on the eq? comparison function. This is the strictest comparison.
- Parameters:
#:expect(any): The expected value.#:got(any): The value to test.#:name(string, optional): A descriptive name for the assertion.
assert-true
Asserts that a given value is #t.
- Parameters:
x(any): The value to test.#:name(string, optional): A descriptive name for the assertion.
assert-false
Asserts that a given value is #f.
- Parameters:
x(any): The value to test.#:name(string, optional): A descriptive name for the assertion.
assert-error
Asserts that a function call will result in an error.
- Parameters:
thunk(function): The function to be executed.#:name(string, optional): A descriptive name for the assertion.
assert-no-error
Asserts that a function call will not result in an error.
- Parameters:
thunk(function): The function to be executed.#:name(string, optional): A descriptive name for the assertion.
Prelude: UTILITATES COMMUNIA
prelude.scm: Shared grammar, language constructs and utilities
evaluate-thunks-concurrently
evaluate-thunks-concurrently is a macro that takes a list of functions that take no arguments (known as "thunks"), runs each one concurrently and returns a list of their results once all have completed. This is useful for speeding up tasks that can be performed in parallel.
- Parameters:
thunks(list of functions): A list of procedures to execute.
- Returns:
- A list containing the return value of each thunk.
evaluate-thunks
evaluate-thunks is a macro takes a list of functions (thunks) that don't take any arguments, runs each one sequentially, and returns a list of their results.
- Parameters:
thunks(list of functions): A list of procedures to execute.
- Returns:
- A list containing the return value of each thunk.
Environment Variables: VARIABILES AMBITUS
Some functionalities, including the runner, are configured via environment variables for ease of use and integration in CI environments. In the Guile Scheme implementation this is done leveraging parameters and. All the variables have sane defaults, but are your entry-point for lots of customisation of the test process.
VERITAS_ENTRYPOINTdefaults tospecand determines which function should be called as entry-point to your tests.VERITAS_SCAN_DIRdefaults to./and determines which directory should be used to look for test directories and files.VERITAS_TEST_TYPEdefaults tounitand determines which type of test to use, and thus which directory should be used to load files from.VERITAS_TEST_TOP_DIRdefaults totestand determines which directory to use to look for test hierarchies, typically this means atestdirectory beside yoursrcdirectory, at top repository level.VERITAS_TEST_MODULEdefaults toveritasand determines the test module to load, and also should be equal to the Guile Scheme modules name you use. For example if you have a unit test file a./test/my-program/unit/my-test.scmthen you should change this variable to the valuemy-programand your module declaration should be(define-module (my-program unit my-test)). If you don't change the defaults, place your tests at for example./test/veritas/unit/my-test.scmand name them(define-module (veritas unit my-test)).VERITAS_SHUFFLE_FILESdefaults totrueand determines whether to shuffle test file order or not.VERITAS_RUN_BENCHMARKSdefaults totrueand determines whether benchmarks should be run.VERITAS_ONLY_TESTis an optional variable you can pass to limit the execution to test cases whose name partially matches this variable.VERITAS_ONLY_SUITEis an optional variable you can pass to limit the execution to suites whose name partially matches this variable.