Byggsteg - Manual & Documentation

byggsteg and all of its source code are free software, licensed under the GNU Affero General Public License v3 (or newer at your convenience).

https://www.gnu.org/licenses/agpl-3.0.nl.html

The documentation and examples, including this document, which are provided with byggsteg, are all licensed under the GNU Free Documentation License v1.3 (or newer at your convenience).

https://www.gnu.org/licenses/fdl-1.3.html

This project adheres to the jointhefreeworld code of conduct. Find it here:

https://jointhefreeworld.org/blog/articles/personal/jointhefreeworld-code-of-conduct/index.html

In summary, we foster an inclusive, respectful, and cooperative environment for all contributors and users of this free software project. Inspired by the ideals of the GNU Project, we strive to uphold freedom, equality, and community as guiding principles. We believe that collaboration in a community of mutual respect is essential to creating excellent free software.

The project's main branch is trunk. Find a list of roles and people involved in the project.

Only BDFL (Benevolent Dictator For Life) can merge PRs. Only BDFL is allowed to push directly to trunk in case a quick fix is needed.

MFL (Maintainer For Life) can ascend to BDFL after extensive productive collaboration. MFL has write permissions to all branches (except trunk) and can open PRs.

BDFL

@jjba23 - Josep Bigorra <jjbigorra@gmail.com>

MFL

@NalaGinrut - Mu Lei <NalaGinrut@hardenedlinux.org>

Other contributors must fork byggsteg and create a pull request to contribute back. After productive collaboration other kinds of roles can be added and assigned.

byggsteg is a community driven effort and as such, can use your help in translating to several different languages.

English language (en-US)

100% translated

Dutch language (nl-NL)

100% translated

Spanish language (es-ES)

100% translated

Contributing to free software is a uniquely beautiful act because it embodies principles of generosity, collaboration, and empowerment.

We welcome everyone to feel invited to the byggsteg Project, and encourage active contribution in all forms, to improve it and/or suggest improvements, brainstorm with me, make it more modular/flexible, etc, feel free to contact me <jjbigorra@gmail.com> to chat, discuss or report feedback.

Find here the Backlog and Kanban boards for byggsteg: https://lucidplan.jointhefreeworld.org/tickets/byggsteg

Jobs are simple Lisp (Guile Scheme) code. Profiles are just saved jobs.

byggsteg allows you to send code over the wire (HTTP) and save it too, in a thunk form, for example :

(lambda()
  `((project . "free-alacarte")
    (branch-name . "trunk")
    (task . "stack-test")
    (clone-url . "https://codeberg.org/jjba23/free-alacarte")))

Sending things in thunk form allows one to execute any code arbitrarily, inside the lambda expression. Be responsible and wary of who you give access to your byggsteg instance.

(lambda()
  (define my-project-name "wikimusic-api")
  (define my-branch-name
    (format #f "branch-num-~a" (* 2 (+ 3 (+ 8 10)))))

  ;; remember that a ". ," and a "unquote" are equivalent
  `((project . ,my-project-name)
    (branch-name unquote my-branch-name)
    (task . ("stack-build" "stack-test"))
    (clone-url unquote (format #f "https://codeberg.org/jjba23/~a" my-project-name))))

When defining a pipeline alist, the task field can be either a single step, or multiple. You should either pass a string or a list of strings of job names.

make-test - Run test suite with make test.

stack-test - Haskell project build and test with Stack + Hackage bundle.

stack-build - Haskell project build with Stack + Hackage bundle.

nix-build - Nix build (flake).

byggsteg-version - Byggsteg pull and restart shepherd daemon itself.

pull-and-restart - Git pull and restart wanted shepherd service.

pull-and-deploy - Git pull and run a sudo make deploy.

sbt-test - Scala project test with SBT.

sbt-testcontainers-test - Scala project test with SBT using testcontainers in Podman.

trivial - Allows passing any arbitrary commands to build your own pipeline completely. This means also that you should pass an additional field to the alist you send to byggsteg, namely, trivial-cmds which should be a list of strings, representing commands to run, in order.

As a fallback, if no other task matches are found, byggsteg will default to doing a make build.

project is a string indicating the project's Git repository name

branch-name is a string indicating the name of the branch to checkout and run the job on

clone-url is a string which contains the HTTPS or SSH URL to clone the Git repository

task is either a string or a list of strings that indicate what tasks (pipeline steps) should be run

daemon-name is a string that indicates the name of the daemon, it's used in certain tasks like pull-and-restart, if not specified, this will default to project

trivial-cmds is a list of strings which represent commands to be run

Bewaking is a nice library to help you secure your web applications with Guile Scheme: https://codeberg.org/jjba23/bewaking

With bewaking we use secure tokens encrypted and stored in secure cookies (for browsers) and headers (for REST API)

If you want to be a super user in byggsteg your token will need to have the byggsteg:modify-permissions permission, effectively meaning you can give yourself and others all wanted permissions and do everything.

See more of the permissions here:

• byggsteg:job-create
• byggsteg:job-delete
• byggsteg:profile-create
• byggsteg:profile-delete
• byggsteg:user-create
• byggsteg:user-delete
• byggsteg:modify-permissions

Byggsteg requires an authenticated session for performing administrative tasks such as creating/deleting jobs and profiles. For web browsers you can simply use the login page, for scripts and headless usage, use the /api/v1/login endpoint and take the x-bewaking-byggsteg response header.

Upon starting a blank new byggsteg deployment you will have no users in the database. Check in db/migration and edit the password field and salt, or add more users. An important note is that currently the passwords should NOT contain any special characters (letters and numbers are OK).

Then do a guix shell -m manifest.scm -- art migrate up user to add your users.

If running locally, it's recommended to use localhost instead of 127.0.0.1 as this will work better with the authentication/authorization and other cookies.

Byggsteg provides HTTP endpoints. After logging in, you can use the UI and visit /jobs/request or you can also invoke this via the REST API by first calling /api/v1/login with the right credentials and then calling /api/v1/jobs/manage/submit.

You could also do this for example from your favourite programming language, or system, or via cURL, etc:

curl 'https://byggsteg.jointhefreeworld.org/api/v1/jobs/manage/submit' \
    -X POST -H 'content-type: application/x-www-form-urlencoded' \
    -H 'x-bewaking-byggsteg: <your token here>' \
    --data-urlencode 'job-code=(lambda() `((project . "free-alacarte")(branch-name . "trunk")(task . "stack-test")(clone-url . "https://codeberg.org/jjba23/free-alacarte")))'

#  {
#    "log-id": "ce24da08-1833-4821-b7e3-e2bacaa79326",
#    "human-id": "ZnJlZS1hbGFjYXJ0ZV9fMTQ6NTM6NTdfXzMwLTEyLTIwMjQuYnlnZ3N0ZWcubG9n"
#  }

You can query for the job status via the UI or also via the REST API. You could implement a check to keep querying for job status until success or failure is true.

curl 'http://localhost:50002/api/v1/logs/ce24da08-1833-4821-b7e3-e2bacaa79326'
#
# {
#   "success": 1,
#   "failure": 0,
#   "in-progress": 0,
#   "log-id": "ce24da08-1833-4821-b7e3-e2bacaa79326",  
#   "human-id": "ZnJlZS1hbGFjYXJ0ZV9fMTQ6NTM6NTdfXzMwLTEyLTIwMjQuYnlnZ3N0ZWcubG9n",
#   "log-data": "ZnJlZS1hbGFjYXJ0ZV9fMTQ6NTM6NTdfXzMwLTEyLTIwMjQuYnlnZ3N0ZWcubG9n.........."
# }

Find a nice integration with Woodpecker here, complete with retries on submit and configurable polling for status:

https://codeberg.org/jjba23/byggsteg-woodpecker

These pipeline steps act as a bridge, allowing the Woodpecker CI system to leverage Byggsteg for its powerful Lisp-based CI/CD orchestration, especially when working with source code hosted on platforms like Codeberg.

The pipeline steps are written in shell script, and use Alpine Linux images achieving a great performance, simplicity and minimalism.

byggsteg is quite a performant implementation of a CI/CD system and the numbers might surprise you for a Scheme only full stack implementation (SXML + SQLite + more).

Using the drill HTTP load testing tool written in Rust.

---

concurrency: 8
base: 'https://byggsteg.jointhefreeworld.org'
iterations: 100
rampup: 4

plan:
  - name: Fetch logs
    request:
      url: https://byggsteg.jointhefreeworld.org/logs
  - name: Fetch log by UUID
    request:
      url: https://byggsteg.jointhefreeworld.org/logs/4868ce1e-d029-4fdb-bbbe-d54d1616a631

Fetch logs                Total requests            100
Fetch logs                Successful requests       100
Fetch logs                Failed requests           0
Fetch logs                Median time per request   164ms
Fetch logs                Average time per request  173ms
Fetch logs                Sample standard deviation 47ms
Fetch logs                99.0'th percentile        315ms
Fetch logs                99.5'th percentile        338ms
Fetch logs                99.9'th percentile        338ms

Fetch log by UUID         Total requests            100
Fetch log by UUID         Successful requests       100
Fetch log by UUID         Failed requests           0
Fetch log by UUID         Median time per request   214ms
Fetch log by UUID         Average time per request  196ms
Fetch log by UUID         Sample standard deviation 71ms
Fetch log by UUID         99.0'th percentile        295ms
Fetch log by UUID         99.5'th percentile        297ms
Fetch log by UUID         99.9'th percentile        297ms

Time taken for tests      4.7 seconds
Total requests            200
Successful requests       200
Failed requests           0
Requests per second       42.37 [#/sec]
Median time per request   185ms
Average time per request  184ms
Sample standard deviation 62ms
99.0'th percentile        297ms
99.5'th percentile        315ms
99.9'th percentile        338ms

If you are thinking of running Byggsteg yourself and hosting it on your own here follow some tips.

Firstly, make sure you use the Guix package manager, whether as a standalone distro (Guix system) or on top of any other GNU/Linux distribution.

I include a manifest.scm in the repo so that it's easy to run the project if you use the Guix package manager.

If you want to run without Guix you should check that manifest.scm anyway since it will give you exactly the packages that you need.

For running byggsteg yourself, you'll need to define and load a module called (byggsteg env) where you add some required environment variables. I recommend in the root of the project create a env.scm and exclude it from Git.

(define-module (byggsteg env))

(define-public encryption-key "a2ebe4c5-4a78-4490-af83-cfb05a285bf9")

(define-public signing-key "8d3a5bcc-973e-4f80-aa3d-33be33d6631c")


(define-public pingwing-url
  "my-pingwing-host")

(define-public pingwing-basic-auth-user
  "my-user")

(define-public pingwing-basic-auth-password
  "my-password")

Pingwing is a mail and notification solution written in Guile Scheme and also part of the jointhefreeworld project. It's used in Byggsteg for password resets, among other notifications. Find more about it here: https://codeberg.org/jjba23/pingwing

Then, you will need a database to be present in the root folder of the project (alongside app, lib, etc.).

Byggsteg uses SQLite by default but is also compatible with PostgreSQL and MySQL.

You can automate the creation of the database and the enabling of WAL mode for concurrent writes with make init-db.

You should then look at db/migration folder and edit the migration for the user table to your liking.

Then do a guix shell -m manifest.scm -- art migrate up user to add your users.

If you are hacking away at the project, I would suggest using make dev since this will effectively run the project with hot-reloading capabilities and it will use the default artanis.conf where we configure hostname, db path, etc.

If you are hosting this for production, I would suggest running Byggsteg as a Shepherd service/daemon. You can alternatively just feel free to do this any other way, I recommend using the manifest.scm . Check the Makefile for more.

Here follows an example of a Guix (Shepherd service):

(define-module (wolk-jjba byggsteg)
  #:use-module (gnu)
  #:use-module (guix))

(use-service-modules shepherd)

(define-public (wolk-jjba-byggsteg-service config)
  (list (shepherd-service (documentation "Run byggsteg as a daemon")
                          (provision '(byggsteg))
                          (requirement '())
                          (start #~(make-forkexec-constructor '("make"
                                                                "production-server")
                                    #:directory
                                    "/etc/byggsteg/job-clone/byggsteg/trunk"
                                    #:environment-variables '("GUILE_LOAD_PATH=/run/current-system/profile/share/guile/site/3.0"
                                                              "GUILE_DBD_PATH=/run/current-system/profile/lib"
                                                              "C_INCLUDE_PATH=/run/current-system/profile/include"
                                                              "GUILE_LOAD_COMPILED_PATH=/run/current-system/profile/lib/guile/3.0/site-ccache:/run/current-system/profile/share/guile/site/3.0"
                                                              "LIBRARY_PATH=/run/current-system/profile/lib"
                                                              "GIT_SSL_NO_VERIFY=1"
                                                              "PATH=/run/privileged/bin:/run/current-system/profile/bin:/run/current-system/profile/sbin")))
                          (stop #~(make-kill-destructor))
                          (auto-start? #t)
                          (respawn? #t))))

(define-public wolk-jjba-byggsteg-service-type
  (service-type (name 'byggsteg)
                (description "Run byggsteg as a daemon")
                (extensions (list (service-extension
                                   shepherd-root-service-type
                                   wolk-jjba-byggsteg-service)))
                (default-value '())))

We believe creating accessible websites for everyone, including people with disabilities, is crucial for equality, helps provide a good user experience.

Byggsteg chooses lots of its UI to appeal to hackers, and computer aficionados by default, but is very configurable to allow extensibility.

Find the documentation for Artanis here: https://www.gnu.org/software/artanis/manual/

GNU Artanis is the first production-level modern Web framework of Scheme programming language.

It is designed and maintained to be robust, fast, and easy to use for professional web development.

Here lies not just a framework, but a canvas where your ideas take flight with remarkable speed and grace. Artanis isn't about wrestling with complexity; it's about the joy of creation.

It's about harnessing the power of a robust and mature language to build something truly your own, something beautiful, nimble, flexible, powerful, efficient, understandable and maintainable. GNU Artanis was Certificated as Awesome Project at 2013 Lisp in summer projects.

One day, the folks at GNU were discussing what language they would write the GNU website in - and many chose Python. But I found that strange, because the official extension language of GNU is GNU Guile. And I wondered aloud - why not start a brand new project to provide a web framework written with GNU Guile? To which RMS said, "It's cool, I like this idea."

But at that time, it was just an idea without a plan.

Fortunately, a few months later, the Guile community held a hack-potluck to celebrate Guile2 turning two - which is a contest to write a cool program in a few weeks. And so, Artanis was born. History

February 2013 - Artanis born at the GNU Guile hack-potluck. 2013 - Artanis submitted to "Lisp In Summer Projects" contest. Received "Certificated awesome project award" in 2014. 1st January, 2015 - the first stable version Artanis-0.0.1 was released. 19th January, 2015 - offers Artanis to FSF/GNU, and RMS inducts it as an official GNU project. Artanis becomes GNU Artanis. 29 December 2021 - GNU Artanis donates to HardenedLinux community to connect with product environment more tightly.