This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.
Direct links to the work I’ve done this GSoC:
First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.
At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.
The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.
Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.
All three packages created within this GSoC went upstream. The programs are shortly described below:
rpcd-mod-attendedsysupgradeIt’s only purpose is to offer a simple way via
ubusto call the internal
sysupgradecommand. This small package may merge into a different package in future.
rpcd-mod-packagelistOn call returns all user installed packages without the need of
opkg. This can be useful for 4/32 routers not having enough space to install
luci-app-attendedsysupgradeNext to the update server the main work was creating a simple update view in Luci handling all communication between router and server. The app gathers all needed data by requesting the router via
XMLHttpRequests. The main idea behind this was to create code which is Independent of the framework around. Luci1 uses Lua, Luci-ng uses AngularJS. The app should be easily portable to a new Interface.
Image shows the notification for a new snapshot build. This was meant for testing, normally the app would show an upgrade from
17.01.3 or similar.
The project was made possible by the LibreMesh community so it was on crucial to support their distributions. Luckily it only required two small pull requests auto setting the version information received by
ubus call system board and to show the the installed app in their simplified admin view.
As a newcomer to the community I had to do a lot of reading to understand the used technology. I learned a lot about processes and structures keeping such a big project like LEDE/OpenWrt running. Also I had the chance to meet many interesting people in the LibreMesh community.
The server should later handle requests peaks on release days so a lot of work should be done in SQL instead of Python. Studying computer science I worked with SQL but only covering the very basics, now learned some of it’s awesome features and construct a complex layout.
Server / Worker
Rapidly generating requested images would require a powerful server with might be in idle most of the time. To avoid costs the building structure was prepared to offer a server/client setup. The server handles all requests and serves created images. Workers will spawn if needed within seconds in VMs building for a single purpose (target/subtarget) and delete themselves if idle for to long.
The basic structure is ready, still auto spawning needs a bit more of work. See Future.
The final challenge was to create a way to cover packages changes between major releases. Say a packages is merged into another or is spitted due to it’s increasing size. The server has to know how to handle these special cases and deliver a package that still works with all expected features. I came up with a simple
yaml configuration allowing package drops, package replaces and package replaces with a dependence.
The following example was created based on transformations from
transformations: libmicrohttpd: libmicrohttpd-no-ssl: true libmicrohttpd: false openvpn: openvpn-openssl: true openvpn-mbedtls: - polarssl - mbedtls openvpn-nossl: false polarssl: mbedtls kmod-ipv6: False
libmicrohttpdwill be replaced by
libmicrohttpd-no-ssl, in future a menu in Luci could offer listed alternatives.
openvpnis replaced by
mbedtlsinstalled the server chooses to install
kmod-ipv6will simply be dropped as no longer available.
Within the last weeks I found some missing features in the update server which I’ll shortly describe below. All features are planed for the time after this years GSoC. I’ll try to keep on developing the project in the future.
- Ability to sign created images with
- Complete the Server/Worker separation, even splitting the project in three different packages.
- Enable package editing (adding/removing) from within the Luci app.
- Make all builds reproducible
- Improve the Luci web interface
md5delivered by the server
- check for free space before download
- Implement an opt-in report system telling the server on successful upgrades. Enabling release channels in the future.
- Create a simple command line interface working without the web interface
- Rely on signed images instead of encrypted HTTP
All further improvements will happen at my Github repository, it contains all code related to the update server.
At the end I’d especially like to thank my mentor and the LibreMesh community in general! I received a lot of help and am happy to entered the community. Thanks for the great opportunity of participating in this years GSoC!