Author: Mr Andrei

GSoC ’23: OpenWrt PPA Porting to GitLab and Rebuilding

Previous post: https://blog.freifunk.net/2023/07/02/gsoc-23-openwrt-ppa-part-2-gitlab-packaging/

GitHub Port and Feature Parity

Continuing from the successful CI build on GitLab, I started working on a GitHub port. Fortunately, only the specific syntax had to be dealt with, as the actual build code needed the same parameters and configuration scripts. This meant a full port could be done without the need to create a build recipe from scratch: https://github.com/ndren/openwrtsdkbuild/blob/master/.github/workflows/docker-build.yml. Notice how similar the original script is: https://github.com/ndren/openwrtsdkbuild/blob/master/.github/workflows/docker-build.yml.

However one major difference is where the CI artifacts are uploaded. As an actual OpenWrt router needs to consume the packages, this needs to be done in a specific folder-level setup. Fortunately, GitHub releases cleanly matches up with this:

- uses: "marvinpinto/action-automatic-releases@latest"
  with:
    repo_token: "${{ secrets.GITHUB_TOKEN }}"
    automatic_release_tag: "latest"
    title: "Package release"
    files: |
        /home/runner/work/openwrtsdkbuild/openwrtsdkbuild/artifacts/packages/*/*/*.ipk
        /home/runner/work/Packages.gz

Notice the use of a GitHub token, specific to this repository. This is setup by GitHub automatically every CI run, there is no need to manually create one.

This can, of course, be reproduced on a router running OpenWrt by following the shell script used by CI: https://github.com/ndren/openwrtsdkbuild/blob/master/test-gh-release.sh

UPLOAD_REPO="https://github.com/ndren/openwrtsdkbuild/releases/download/latest"
echo "src/gz myrepo ${UPLOAD_REPO}" >> /etc/opkg/customfeeds.conf
# Install package
opkg install "${PACK_NAME}"

To get feature parity with GitLab, we found a method to add a dropdown so that app users can use this to build packages without the need to manually edit the source code in their fork. This also makes it clear which parameters are provided to the build environment without reading the configuration files:

Neater unauthenticated package downloads

With this in mind, I was really happy when I learned that all the builds can be done without authentication when downloading. I discovered that GitLab started allowing guests to download from generic package repositories: https://gitlab.com/gitlab-org/gitlab/-/issues/299384. This makes it a lot easier to download on an actual router. Compare the following example configuration files:

Not only is it easier to type, it also does not require a PERSONAL_ACCESS_TOKEN to copy to each router, so this means that the router cannot leak the GitLab tokens (since it doesn’t need any), which is a real threat since the router may not support HTTPS cleanly in its installation of opkg. Not only easier to type by hand, it also makes more sense in a shared environment. That is, I don’t need to tell you my personal access token, you can add https://gitlab.com/api/v4/projects/ndren%2Fopenwrtsdkbuild/packages/generic/armvirt_64/0.0.3/ to your list of repos and it will work.

What is nice is that the same main code can be used to install from a GitLab or GitHub repository:

# Add repository (any repository: GitLab, GitHub, etc.)
UPLOAD_REPO="https://gitlab.com/api/v4/projects/ndren%2Fopenwrtsdkbuild/packages/generic/armvirt_64/0.0.3/"
echo "src/gz myrepo ${UPLOAD_REPO}" >> /etc/opkg/customfeeds.conf
opkg install "${PACK_NAME}"

Updated SDK

I also took the time to get set up with a newer SDK docker images on my fork of openwrtsdk: https://gitlab.com/ndren/openwrtsdk. I found a surprising amount of packages needed to be included in the newer SDK version, the latest release candidate 23.05.0-rc2. This also required an upgrade to alpine 3.15, as argp-standalone did not exist before. This lost compatibility with python2 but fortunately the OpenWrt developers ported their SDK to python3: https://github.com/openwrt/packages/issues/8892

FROM alpine:3.14
RUN apk add asciidoc bash bc binutils bzip2 cdrkit coreutils diffutils findutils flex g++ gawk gcc gettext git grep intltool libxslt linux-headers make ncurses-dev openssl-dev patch perl python2-dev python3-dev rsync tar unzip util-linux wget zlib-dev sudo xz lighttpd curl

FROM alpine:3.15
RUN apk add asciidoc bash bc binutils bzip2 cdrkit coreutils diffutils findutils flex g++ gawk gcc gettext git grep intltool libxslt linux-headers make ncurses-dev openssl-dev patch perl python3-dev rsync tar unzip util-linux wget zlib-dev sudo xz lighttpd curl alpine-sdk gzip build-base musl-dev musl-libintl musl-utils fts fts-dev musl-obstack musl-obstack-dev musl-nscd-dev musl-nscd argp-standalone

Once this was resolved for a single build building everything was only a matter of CPU time. The builds are here: https://hub.docker.com/r/andreien/openwrtsdk/tags. (Yes, that is all the architectures included!)

Closing words

I invite you to try this out! All it takes is GitHub or GitLab account and a bit of text editing and you should be able to build any OpenWrt package in a full CI environment. Feel free to file an issue if there’s anything to improve, I’m happy to help.

Thanks to everyone at Freifunk for the encouragement and in particular to my mentor Zoobab that carried me through this journey with help with code examples and design decisions. I am really happy to have learned to use Docker and write GitHub and GitLab CI scripts effectively. Thank you for having me on this adventure, I’ll see you later.

GSoC ’23 OpenWrt PPA Part 2: GitLab packaging

Previous post: https://blog.freifunk.net/2023/05/13/gsoc-23-documenting-the-openwrt-compilation-process-to-set-up-a-ppa

When working on this project we’ve encountered the networking troubles: there is no networking on OpenBuildService builds. This would require packaging everything into tarballs and copying these over to the source files before building, which would not be user-friendly at all. (The purpose of this project is to make packaging easier, not harder!) This seemed impractical.

GitLab’s packaging system solves this issue cleanly. It provides a full CI interface, and allows for networking as well. It works cross-architecture too! The key thing to resolve is to get OpenWrt tools in, as these do not get official status like Debian or Ubuntu, so they need to be implemented in a way that follows GitLab conventions.

Zoobab helped a lot by producing an automated build system: https://gitlab.com/zoobab/openwrtsdkbuild. This takes in a git repository and exports binaries as artifacts. This is really useful as this is essentially the input and output of this process: some code to run on OpenWrt becomes an architecture-specific binary.

I have experimented with different ways to store these binaries (OpenWrt’s opkg expects a very specific file structure, not unlike most other package managers). Over time, I was able to port this into a repository structure: https://gitlab.com/ndren/openwrtsdkbuild/-/blob/7d92349d53befe8e2cc5ce1a89919b68050b9ce2/.gitlab-ci.yml#L50. This uses GitLab’s feature where it integrates an internal package repository with HTTP requests to read and write data. In this case it was very useful so that there is long-term storage for the package binaries. (This uses GitLab’s generic package repositories.)

It would be very useful if the CI system could verify that the binaries actually work! I’ve looked into OpenWrt’s docker repository and I found they provide a root filesystem (rootfs) exactly made for CI testing for different architectures: https://gitlab.com/ndren/openwrtsdkbuild/-/blob/7d92349d53befe8e2cc5ce1a89919b68050b9ce2/.gitlab-ci.yml#L72. With a little bit of tweaking to prepare the SDK, it worked great and meant that we could keep up with any upstream updates to the SDK or demo rootfs.

We have now a dropdown menu where we can choose the SDK and the git repo. We have to check whether a Gitlab Page could provide a better UI (with or without login page in front).

Future plans: Because the main CI system is setup this way, it easily allows for improvements. For example, exploring things like adding support for new architectures takes editing a line of YML and testing. Therefore I can start looking at self-hosted GitLab, and testing more architectures.

TODO list:

1. Try github Actions (see if it is portable)

2. Problems with gitlab package repo URLs (requires authenticated downloads which is not ideal)

3. Problem with the SDK that outputs lots of warnings about missing packages (can this be fixed without breaking the build?)

4. Check whether other OpenWRT SDK images are usable, so the build can be done in different environments.

Finally, a message from our hello world package, running in CI in Docker in Docker in aarch64 in amd64:

Hello world!
MyClass::MyClass()
MyClass::printMessage()
This is my message to print

GSoC ’23: Documenting the OpenWrt compilation process to set up a PPA

Getting the OpenWrt PPA set up will require understanding and documenting the current approaches for portable compilation of OpenWrt packages. This is my first impression of this task.

When I verified the “hello world” program provided and verified by Zoobab correctly compiled, I have looked to see how exactly this process works. The Docker build script depends on the toolchain compilation script. They both require an internet connection, and I will have to keep this in mind when porting this to make an OpenBuildService (OBS) package: “Mentioning repositories directly is not allowed (using obsrepositories:/ is ok)“.

Interesting technical details I found while researching how to transfer the current Docker approach to OBS:
– At first I did not know what the flag V was in `make V=s`. It turns out it turns on verbose compilation output on console on the OpenWrt build system. (I was surprised when I saw this was undocumented in the new version of the guide.)
– The package manager for OpenWrt uses ipk files. The helloworld package (that has been compiled under the OpenWrt SDK) has been successfully installed inside an openwrt rootfs and runs great!
– The current system uses cascading Docker images (multi-stage builds). This is done by producing the OpenWrt SDK container image first and then a new Docker image compiles the actual system. This is useful because it means the actual package being compiled is kept separate from the source itself.
– It turns out that obs-build from OpenBuildService assumes a working chroot so one must be created manually. The actual container must be merged in separately into the upstream project, outside of obs-build.

A message from our working hello world package:

root@localhost:/# helloworld
Hello world!
MyClass::MyClass()
MyClass::printMessage()
This is my message to print

I can’t wait to learn more and get up to speed on how to approach this project. I’ll see you later!