Hi Freifunk,

This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.

tl;dr:
Direct links to the work I’ve done this GSoC:

• luci-app-attendedsysupgrade
• rpcd-mod-attendedsysupgrade
• rpcd-mod-packagelist
• update-server

First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.

Progress

Server

At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.

The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.

Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.

To try the the server at https://betaupdate.libremesh.org have a look at the demos I prepared.

Router

All three packages created within this GSoC went upstream. The programs are shortly described below:

• rpcd-mod-attendedsysupgrade It’s only purpose is to offer a simple way via ubus to call the internal sysupgrade command. This small package may merge into a different package in future.
• rpcd-mod-packagelist On call returns all user installed packages without the need of opkg. This can be useful for 4/32 routers not having enough space to install opkg.
• luci-app-attendedsysupgrade Next to the update server the main work was creating a simple update view in Luci handling all communication between router and server. The app gathers all needed data by requesting the router via ubus and sends requests to the server. All logic implemented in JavaScript, requests are in JSON using standard XMLHttpRequests. The main idea behind this was to create code which is Independent of the framework around. Luci1 uses Lua, Luci-ng uses AngularJS. The app should be easily portable to a new Interface.

Image shows the notification for a new snapshot build. This was meant for testing, normally the app would show an upgrade from 17.01.2 to 17.01.3 or similar.

LibreMesh

The project was made possible by the LibreMesh community so it was on crucial to support their distributions. Luckily it only required two small pull requests auto setting the version information received by ubus call system board and to show the the installed app in their simplified admin view.

Challenges

As a newcomer to the community I had to do a lot of reading to understand the used technology. I learned a lot about processes and structures keeping such a big project like LEDE/OpenWrt running. Also I had the chance to meet many interesting people in the LibreMesh community.

I’ve never touched Lua code before and barely worked with JavaScript. Creating the web interface was even it’s simple quite a challenge for me. As written above the code should be easily portable to a new Luci version so Lua was no option. As the package should be as small as possible (currently under 10KB), including a framework just for updating was no option, so vanilla JavaScript it is.

Database

The server should later handle requests peaks on release days so a lot of work should be done in SQL instead of Python. Studying computer science I worked with SQL but only covering the very basics, now learned some of it’s awesome features and construct a complex layout.

Server / Worker

Rapidly generating requested images would require a powerful server with might be in idle most of the time. To avoid costs the building structure was prepared to offer a server/client setup. The server handles all requests and serves created images. Workers will spawn if needed within seconds in VMs building for a single purpose (target/subtarget) and delete themselves if idle for to long.

The basic structure is ready, still auto spawning needs a bit more of work. See Future.

Transformations

The final challenge was to create a way to cover packages changes between major releases. Say a packages is merged into another or is spitted due to it’s increasing size. The server has to know how to handle these special cases and deliver a package that still works with all expected features. I came up with a simple yaml configuration allowing package drops, package replaces and package replaces with a dependence.

The following example was created based on transformations from 15.05 to 17.01

transformations:
    libmicrohttpd:
        libmicrohttpd-no-ssl: true
        libmicrohttpd: false
    openvpn:
        openvpn-openssl: true
        openvpn-mbedtls:
            - polarssl
            - mbedtls
        openvpn-nossl: false
    polarssl: mbedtls
    kmod-ipv6: False

• libmicrohttpd will be replaced by libmicrohttpd-no-ssl, in future a menu in Luci could offer listed alternatives.
• openvpn is replaced by openvpn-openssl, if polarssl or mbedtls installed the server chooses to install openvpn-mbedtls.
• polarssl will become mbedtlks.
• kmod-ipv6 will simply be dropped as no longer available.

Future

Within the last weeks I found some missing features in the update server which I’ll shortly describe below. All features are planed for the time after this years GSoC. I’ll try to keep on developing the project in the future.

• Ability to sign created images with usign
• Complete the Server/Worker separation, even splitting the project in three different packages.
  • update-server, update-server-worker & update-server-common
• Enable package editing (adding/removing) from within the Luci app.
• Make all builds reproducible
• Improve the Luci web interface
  • check md5 delivered by the server
  • check for free space before download
• Implement an opt-in report system telling the server on successful  upgrades. Enabling release channels in the future.
• Create a simple command line interface working without the web interface
  • Rely on signed images instead of encrypted HTTP

---

All further improvements will happen at my Github repository, it contains all code related to the update server.

At the end I’d especially like to thank my mentor and the LibreMesh community in general! I received a lot of help and am happy to entered the community. Thanks for the great opportunity of participating in this years GSoC!