This repository is the configuration of the MariaDB Foundation's CI. It runs on a fork of Buildbot (vladbogo/buildbot, branch grid) and builds and tests:
- MariaDB Server, including the packages that go into a release;
- Galera packages;
- the MariaDB Connectors: C, C++ and ODBC;
- the plugins in Foundry.
Two instances run from this repository:
| Environment | Web UI | Deployed from | Saved files | Cross-Reference |
|---|---|---|---|---|
| Production | https://buildbot.mariadb.org | main |
https://ci.mariadb.org | https://buildbot.mariadb.org/cr/ |
| Development | https://buildbot.dev.mariadb.org | dev |
https://ci.dev.mariadb.org | https://buildbot.dev.mariadb.org/cr/ |
Cross-Reference records the MTR test failures of the builds, so that a failure can be looked up in earlier runs. Its code is in MariaDB/cross-reference.
Pull requests target dev, and changes reach main once they have run on dev. The settings that differ between the two are in docker-compose/.env and docker-compose/.env.dev.
Buildbot runs as several masters that share one database. Each project has an entry point builder that every build of the project starts from (How changes reach Buildbot). New builders go on master-migration, which uses the builder framework in configuration/.
- uv, which installs Python 3.9 for the virtual environment;
libvirt-devandlibmariadb-dev, to build the Python bindings;- Docker or Podman, to check the masters' configuration in the masters' images, and for the hadolint hook.
make install
source .venv/bin/activate
make install-pre-commitmake install creates .venv, installs requirements.txt, and builds and installs the same Buildbot fork as the masters run, cloned into .vendor/. make clean removes both, and make help lists the other targets. To run the linters on every commit, run pre-commit install.
The private settings, master-private.cfg and master-config.yaml, are links to their -sample files, whose placeholder values are enough to load every master. The real ones exist only on the master hosts; see Secrets.
| Command | Checks |
|---|---|
make pre-commit-run |
The linters (Python, YAML, shell, Markdown, Dockerfiles, spelling), on the staged files. make pre-commit-run-all checks the whole repository. |
./validate_master_cfg.sh -e DEV |
buildbot checkconfig of every master, in the dev master image with .env.dev. It generates the autogen masters first, and prints how many there are per architecture. |
./validate_master_cfg.sh -e PROD |
The same, with the production image and .env |
make checkconfig |
Both of the above |
make test |
The unit tests of configuration/ |
The masters expect the repository at /srv/buildbot/master and read their settings from environment variables, which is why validate_master_cfg.sh runs them in containers. It also writes checkconfig_summary.md, with the jobs master-migration's builders ask of each worker.
Merging into dev deploys to buildbot.dev.mariadb.org, which builds the maintainers' forks. Read Testing on dev before starting builds there.
Open it against dev, and link the Jira issue (MDBF project), as commit messages do with an MDBF-<number>: prefix. The pull request template links to a checklist for each project: server builders and their install and upgrade VMs, Galera, the Connectors, Foundry, build images and workers.
On a pull request, GitHub runs:
- pre-commit, the same linters;
- bbm-deploy, the
checkconfigof every master with the dev and the production images. Its summary shows, for each master-migration worker, the jobs its builders ask for; - unittests, when
configuration/changes; - the image builds that use a changed Dockerfile. The image workflows are grouped by family, so only the affected family is rebuilt.
The masters share one database and coordinate through a Crossbar message router. The builds, and their load, are spread over the masters.
| Master | What it runs |
|---|---|
| master-web | The web UI and the GitHub change hooks. No workers. |
| master-protected-branches | tarball-docker, the entry point of every server build, and fast server builders that report to GitHub. |
autogen/<arch>-master-<n> |
Generated from os_info.yaml: one test builder and one package (-autobake) builder per server platform. See Server builders. |
| master-libvirt | Install and upgrade tests of the server packages, in VMs. |
| master-galera | Galera packages. |
| master-nonlatent | Windows, macOS, FreeBSD and AIX builders, and the Docker Library and WordPress tests. |
| master-docker-nonstandard, master-docker-nonstandard-2 | Server builders with special configurations: Valgrind, other compilers, full test suites. |
| master-migration | The builder framework in configuration/: the Connectors, Foundry, and the server builders moving to it. |
Add new builders to master-migration. The aim is to move the builders of the other masters there over time.
Builds run on three kinds of workers:
- Docker latent (autogen masters, master-protected-branches, master-galera, master-docker-nonstandard*): for each build, the master starts a container from a build image on a worker host's Docker daemon. The image contains
buildbot-worker, which connects back to the master. Each container is a worker of its own, so the master doesn't know which host a build runs on; worker_locks.yaml caps how many builds start on a host. - Non-latent (master-nonlatent, master-migration): a
buildbot-workerprocess that runs on the host all the time. On master-migration, each worker has a pool of jobs (its vCPUs) and each build takes as many as its builder asks for, so the host's load is controlled per builder. See master-migration. - libvirt (master-libvirt): a VM that starts for a build.
GitHub sends push and pull request events to the change hook on master-web. Production receives them from the official repositories; dev receives them from the maintainers' forks, for every project. Each project has its own entry point builder, which every build of that project starts from:
| Project | Entry point | Master | Documentation |
|---|---|---|---|
| Server | tarball-docker |
master-protected-branches | Server builders |
| Galera | trigger-galera-builds |
master-galera | master-galera |
| Connectors | cc-tarball-docker, ccpp-tarball-docker, codbc-tarball-docker |
master-migration | Connectors |
| Foundry | foundry-trigger-builders |
master-migration | Foundry |
| Path | Holds |
|---|---|
master-*/ |
One directory per master, with its master.cfg |
| master.cfg, os_info.yaml, define_masters.py | The template, platform list and generator of the autogen masters |
| constants.py | Server versions, platforms per version, builders that report to GitHub, extra MTR suites |
| master_common.py | The settings all masters share: database, message queue, GitHub status reporting |
| common_factories.py, utils.py, schedulers_definition.py, locks.py | The server builders' factories, schedulers and helpers, used by every master except master-migration |
| configuration/ | The builder framework used by master-migration |
| ci_build_images/ | Dockerfiles of the build images |
| .github/workflows/ | Image builds, checks and deployment |
| docker-compose/ | The containers that run the masters, and deployment |
| dashboards/release/ | The Server Release Status page |
scripts/ |
Scripts that builders download from GitHub at build time |
| minio/ | The MinIO service used by the S3 tests |
| Dockerfile | The image of the masters |
A push to dev deploys dev. An operator deploys production, from main. See Deploying.
Some changes reach production as soon as they are merged to main, without a deployment:
- scripts that builders download from GitHub when they run: the install and upgrade tests on master-libvirt, the Docker Library tests;
- build images: merging a Dockerfile change to
mainmoves the production tags to the images tested on dev. See Build images.
A Buildbot upgrade is planned, which will rewrite the masters' image, built from Dockerfile. Until then, avoid changes to the current Buildbot version and its fork: they would have to be ported to the new version.
This project would not have gotten off the ground without the help and support of Rasmus Johansson. We thank him for his many contributions to the community, and remember him for his kindness, level headedness, and as an example for us all.