GSoC ’23: OpenWrt PPA Porting to GitHub and Rebuilding

GSoC ’23: OpenWrt PPA Porting to GitLab and Rebuilding

Previous post: https://blog.freifunk.net/2023/07/02/gsoc-23-openwrt-ppa-part-2-gitlab-packaging/

GitHub Port and Feature Parity

Continuing from the successful CI build on GitLab, I started working on a GitHub port. Fortunately, only the specific syntax had to be dealt with, as the actual build code needed the same parameters and configuration scripts. This meant a full port could be done without the need to create a build recipe from scratch: https://github.com/ndren/openwrtsdkbuild/blob/master/.github/workflows/docker-build.yml. Notice how similar the original script is: https://github.com/ndren/openwrtsdkbuild/blob/master/.github/workflows/docker-build.yml.

However one major difference is where the CI artifacts are uploaded. As an actual OpenWrt router needs to consume the packages, this needs to be done in a specific folder-level setup. Fortunately, GitHub releases cleanly matches up with this:

- uses: "marvinpinto/action-automatic-releases@latest"
  with:
    repo_token: "${{ secrets.GITHUB_TOKEN }}"
    automatic_release_tag: "latest"
    title: "Package release"
    files: |
        /home/runner/work/openwrtsdkbuild/openwrtsdkbuild/artifacts/packages/*/*/*.ipk
        /home/runner/work/Packages.gz

Notice the use of a GitHub token, specific to this repository. This is setup by GitHub automatically every CI run, there is no need to manually create one.

This can, of course, be reproduced on a router running OpenWrt by following the shell script used by CI: https://github.com/ndren/openwrtsdkbuild/blob/master/test-gh-release.sh

UPLOAD_REPO="https://github.com/ndren/openwrtsdkbuild/releases/download/latest"
echo "src/gz myrepo ${UPLOAD_REPO}" >> /etc/opkg/customfeeds.conf
# Install package
opkg install "${PACK_NAME}"

To get feature parity with GitLab, we found a method to add a dropdown so that app users can use this to build packages without the need to manually edit the source code in their fork. This also makes it clear which parameters are provided to the build environment without reading the configuration files:

Neater unauthenticated package downloads

With this in mind, I was really happy when I learned that all the builds can be done without authentication when downloading. I discovered that GitLab started allowing guests to download from generic package repositories: https://gitlab.com/gitlab-org/gitlab/-/issues/299384. This makes it a lot easier to download on an actual router. Compare the following example configuration files:

Not only is it easier to type, it also does not require a PERSONAL_ACCESS_TOKEN to copy to each router, so this means that the router cannot leak the GitLab tokens (since it doesn’t need any), which is a real threat since the router may not support HTTPS cleanly in its installation of opkg. Not only easier to type by hand, it also makes more sense in a shared environment. That is, I don’t need to tell you my personal access token, you can add https://gitlab.com/api/v4/projects/ndren%2Fopenwrtsdkbuild/packages/generic/armvirt_64/0.0.3/ to your list of repos and it will work.

What is nice is that the same main code can be used to install from a GitLab or GitHub repository:

# Add repository (any repository: GitLab, GitHub, etc.)
UPLOAD_REPO="https://gitlab.com/api/v4/projects/ndren%2Fopenwrtsdkbuild/packages/generic/armvirt_64/0.0.3/"
echo "src/gz myrepo ${UPLOAD_REPO}" >> /etc/opkg/customfeeds.conf
opkg install "${PACK_NAME}"

Updated SDK

I also took the time to get set up with a newer SDK docker images on my fork of openwrtsdk: https://gitlab.com/ndren/openwrtsdk. I found a surprising amount of packages needed to be included in the newer SDK version, the latest release candidate 23.05.0-rc2. This also required an upgrade to alpine 3.15, as argp-standalone did not exist before. This lost compatibility with python2 but fortunately the OpenWrt developers ported their SDK to python3: https://github.com/openwrt/packages/issues/8892

FROM alpine:3.14
RUN apk add asciidoc bash bc binutils bzip2 cdrkit coreutils diffutils findutils flex g++ gawk gcc gettext git grep intltool libxslt linux-headers make ncurses-dev openssl-dev patch perl python2-dev python3-dev rsync tar unzip util-linux wget zlib-dev sudo xz lighttpd curl

FROM alpine:3.15
RUN apk add asciidoc bash bc binutils bzip2 cdrkit coreutils diffutils findutils flex g++ gawk gcc gettext git grep intltool libxslt linux-headers make ncurses-dev openssl-dev patch perl python3-dev rsync tar unzip util-linux wget zlib-dev sudo xz lighttpd curl alpine-sdk gzip build-base musl-dev musl-libintl musl-utils fts fts-dev musl-obstack musl-obstack-dev musl-nscd-dev musl-nscd argp-standalone

Once this was resolved for a single build building everything was only a matter of CPU time. The builds are here: https://hub.docker.com/r/andreien/openwrtsdk/tags. (Yes, that is all the architectures included!)

Closing words

I invite you to try this out! All it takes is GitHub or GitLab account and a bit of text editing and you should be able to build any OpenWrt package in a full CI environment. Feel free to file an issue if there’s anything to improve, I’m happy to help.

Thanks to everyone at Freifunk for the encouragement and in particular to my mentor Zoobab that carried me through this journey with help with code examples and design decisions. I am really happy to have learned to use Docker and write GitHub and GitLab CI scripts effectively. Thank you for having me on this adventure, I’ll see you later.