Hi, everybody.
This is my last post regarding my GSoC project for 2018.
My work can be found here:
A quick summary of what this project was about: Move on from building Nodewatcher supported imagebuilders from source but instead use upstream provided OpenWrt imagebuiders. Also, build our custom not yet upstreamed packages using OpenWrt SDK.
Current status
Most of the code was merged to their relevant wlanslovenija repositories but some of it is still waiting to be merged.
Nodewatcher
Various fixes to make Nodewatcher run on newer kernels and distributions like Ubuntu 18.04 were merged to the main branch of wlanslovenija/nodewatcher repository.
This includes fixes for known issues with the never pip tool as well as multiple packages with new names.
Still to be submitted is updating the various Python packages which are currently outdated, this is waiting for thorough testing.
firmware-packages-opkg
The name of wlanslovenija repository with our custom packages that are used by Nodewatcher like Tunneldigger is firmware-packages-opkg.
A big part of the changes is already merged, in wlanslovenija/firmware-packages-opkg.
This was a first big cleanup after a long time, a lot of packages that were not used and a lot of those that used custom patches were dropped.
I manually verified that those with custom patches had those patches upstreamed before they were dropped, this now enables us to use new iwinfo versions that include many fixes.
This also enables compiling packages such as curl with GCC7.3.
There are currently fixes for packages refusing to compile or dead sources waiting in my tree on Github.
firmware-core and the build process
The name of wlanslovenija repository where all files pertaining to the building of Nodewatcher compatible Docker-based imagebuilders are located is firmware-core.
This repository was the target of the bulk of my effort.
The current code was almost completely dropped or significantly reworked, which in the end resulted in removing 3,964 lines of code while adding only 255.
This significantly reduces the maintenance burden as almost no maintenance is needed except for adding or removing required Ubuntu packages in our Docker images.
Big changes that were made are:
- LEDE and OpenWrt are remerged in our build process
- Building of old OpenWrt versions prior to 17.01 was completely removed (CC 15.05 etc.)
This was completely unnecessary and only caused the legacy code to stick around.
There is no explanation for use of OpenWrt Chaos Calmer and even older versions now that OpenWrt and LEDE have merged.
Those versions have numerous known exploits that have been fixed in 17.01 and now in 18.06. - Both our build and runtime Docker base images now use Ubuntu 18.04 instead of old 14.04.
This enables us to fully utilize the fact that OpenWrt uses GCC7.3 as default compiler since Ubuntu 18.04 finally ships with it as default too.
Size of the base image has reduced due to less unnecessary packages being shipped with it. - We now use imagebuilders provided by upstream OpenWrt project
This significantly reduces the build time as most of the packages and the whole toolchain are not built anymore.
The fact that we can’t patch the sources with custom patches anymore does not matter as we were not using any important patches.
Unfortunately, due to the fact that most of the packages needed for Nodewatcher to function are custom written and were never upstreamed we still need to custom build them.
Thankfully upstream OpenWrt provides an SDK next to imagebuilders, those are meant for just what we need, for building packages only.
They provide already built toolchain and all of the tools needed so that saves a lot of time, but since our packages have a lot of dependencies it still takes some time to build them.
Then they are simply copied to the imagebuilder, we manually trigger the package index to be regenerated as we use that package index to generate metadata so Nodewatcher knows what and which version of packages are inside each imagebuilder. This enables configuring packages on per version basis.
Since we can now easily download all of the community packages we dont have to compile them in like we did so far.
This completely removes the need for us to have package mirrors.In the end, this has reduced the time needed for each target around 3-4 times. - Configuration of the build process was greatly reduced as well as its complexity.
No more need for a lot of Dockerfiles and configuration for each of the targets.
Currently, all of these changes been merged into the main repository wlanslovenija/firmware-core
Future
I did not have time to do all of the things I wanted.
This is mainly upstreaming as much of our packages as possible, as they are the biggest time consumer during building.
This will be dealt with after GSoC.
Nodewatcher needs to be updated to merge LEDE and OpenWrt as we have some checks to ensure that more advanced features are only enabled on LEDE as OpenWrt did not have them at that time.
This will be dealt with after GSoC too.
I also wanted to add some new features to our imagebuilders, but since hitting a lot of bugs and unexpected stuff during development I did not have for these, so like previous two points, this will be dealt after GSoC.
So to sum this up, this was a really good experience.
I got to focus on two things I enjoy working on: FOSS software and OpenWrt.
This enabled me to learn a lot on the functioning of our Nodewatcher, OpenWrt imagebuilder and especially OpenWrt SDK.
Thanks to Google for organizing GSoC, Freifunk for enabling me to give back to the community in the usefull way.
And special thanks to my mentor Valent Turković.
Best regards
Robert Marko