GSoC ’23 OpenWrt PPA Part 2: GitLab packaging

GSoC ’23 OpenWrt PPA Part 2: GitLab packaging

Previous post: https://blog.freifunk.net/2023/05/13/gsoc-23-documenting-the-openwrt-compilation-process-to-set-up-a-ppa

When working on this project we’ve encountered the networking troubles: there is no networking on OpenBuildService builds. This would require packaging everything into tarballs and copying these over to the source files before building, which would not be user-friendly at all. (The purpose of this project is to make packaging easier, not harder!) This seemed impractical.

GitLab’s packaging system solves this issue cleanly. It provides a full CI interface, and allows for networking as well. It works cross-architecture too! The key thing to resolve is to get OpenWrt tools in, as these do not get official status like Debian or Ubuntu, so they need to be implemented in a way that follows GitLab conventions.

Zoobab helped a lot by producing an automated build system: https://gitlab.com/zoobab/openwrtsdkbuild. This takes in a git repository and exports binaries as artifacts. This is really useful as this is essentially the input and output of this process: some code to run on OpenWrt becomes an architecture-specific binary.

I have experimented with different ways to store these binaries (OpenWrt’s opkg expects a very specific file structure, not unlike most other package managers). Over time, I was able to port this into a repository structure: https://gitlab.com/ndren/openwrtsdkbuild/-/blob/7d92349d53befe8e2cc5ce1a89919b68050b9ce2/.gitlab-ci.yml#L50. This uses GitLab’s feature where it integrates an internal package repository with HTTP requests to read and write data. In this case it was very useful so that there is long-term storage for the package binaries. (This uses GitLab’s generic package repositories.)

It would be very useful if the CI system could verify that the binaries actually work! I’ve looked into OpenWrt’s docker repository and I found they provide a root filesystem (rootfs) exactly made for CI testing for different architectures: https://gitlab.com/ndren/openwrtsdkbuild/-/blob/7d92349d53befe8e2cc5ce1a89919b68050b9ce2/.gitlab-ci.yml#L72. With a little bit of tweaking to prepare the SDK, it worked great and meant that we could keep up with any upstream updates to the SDK or demo rootfs.

We have now a dropdown menu where we can choose the SDK and the git repo. We have to check whether a Gitlab Page could provide a better UI (with or without login page in front).

Future plans: Because the main CI system is setup this way, it easily allows for improvements. For example, exploring things like adding support for new architectures takes editing a line of YML and testing. Therefore I can start looking at self-hosted GitLab, and testing more architectures.

TODO list:

1. Try github Actions (see if it is portable)

2. Problems with gitlab package repo URLs (requires authenticated downloads which is not ideal)

3. Problem with the SDK that outputs lots of warnings about missing packages (can this be fixed without breaking the build?)

4. Check whether other OpenWRT SDK images are usable, so the build can be done in different environments.

Finally, a message from our hello world package, running in CI in Docker in Docker in aarch64 in amd64:

Hello world!
MyClass::MyClass()
MyClass::printMessage()
This is my message to print