Hi, everybody.

This is my last post regarding my GSoC project for 2018.
My work can be found here:

• firmware-core
• firmware-packages-opkg
• nodewatcher

A quick summary of what this project was about: Move on from building Nodewatcher supported imagebuilders from source but instead use upstream provided OpenWrt imagebuiders. Also, build our custom not yet upstreamed packages using OpenWrt SDK.

Current status

Most of the code was merged to their relevant wlanslovenija repositories but some of it is still waiting to be merged.

Nodewatcher

Various fixes to make Nodewatcher run on newer kernels and distributions like Ubuntu 18.04 were merged to the main branch of wlanslovenija/nodewatcher repository.
This includes fixes for known issues with the never pip tool as well as multiple packages with new names.
Still to be submitted is updating the various Python packages which are currently outdated, this is waiting for thorough testing.

firmware-packages-opkg

The name of wlanslovenija repository with our custom packages that are used by Nodewatcher like Tunneldigger is firmware-packages-opkg.
A big part of the changes is already merged, in wlanslovenija/firmware-packages-opkg.
This was a first big cleanup after a long time, a lot of packages that were not used and a lot of those that used custom patches were dropped.
I manually verified that those with custom patches had those patches upstreamed before they were dropped, this now enables us to use new iwinfo versions that include many fixes.
This also enables compiling packages such as curl with GCC7.3.
There are currently fixes for packages refusing to compile or dead sources waiting in my tree on Github.

firmware-core and the build process

The name of wlanslovenija repository where all files pertaining to the building of Nodewatcher compatible Docker-based imagebuilders are located is firmware-core.
This repository was the target of the bulk of my effort.
The current code was almost completely dropped or significantly reworked, which in the end resulted in removing 3,964 lines of code while adding only 255.
This significantly reduces the maintenance burden as almost no maintenance is needed except for adding or removing required Ubuntu packages in our Docker images.

Big changes that were made are:

• LEDE and OpenWrt are remerged in our build process
• Building of old OpenWrt versions prior to 17.01 was completely removed (CC 15.05 etc.)
  This was completely unnecessary and only caused the legacy code to stick around.
  There is no explanation for use of OpenWrt Chaos Calmer and even older versions now that OpenWrt and LEDE have merged.
  Those versions have numerous known exploits that have been fixed in 17.01 and now in 18.06.
• Both our build and runtime Docker base images now use Ubuntu 18.04 instead of old 14.04.
  This enables us to fully utilize the fact that OpenWrt uses GCC7.3 as default compiler since Ubuntu 18.04 finally ships with it as default too.
  Size of the base image has reduced due to less unnecessary packages being shipped with it.
• We now use imagebuilders provided by upstream OpenWrt project
  This significantly reduces the build time as most of the packages and the whole toolchain are not built anymore.
  The fact that we can’t patch the sources with custom patches anymore does not matter as we were not using any important patches.
  Unfortunately, due to the fact that most of the packages needed for Nodewatcher to function are custom written and were never upstreamed we still need to custom build them.
  Thankfully upstream OpenWrt provides an SDK next to imagebuilders, those are meant for just what we need, for building packages only.
  They provide already built toolchain and all of the tools needed so that saves a lot of time, but since our packages have a lot of dependencies it still takes some time to build them.
  Then they are simply copied to the imagebuilder, we manually trigger the package index to be regenerated as we use that package index to generate metadata so Nodewatcher knows what and which version of packages are inside each imagebuilder. This enables configuring packages on per version basis.
  Since we can now easily download all of the community packages we dont have to compile them in like we did so far.
  This completely removes the need for us to have package mirrors.In the end, this has reduced the time needed for each target around 3-4 times.
• Configuration of the build process was greatly reduced as well as its complexity.
  No more need for a lot of Dockerfiles and configuration for each of the targets.

Currently, all of these changes been merged into the main repository wlanslovenija/firmware-core

Future

I did not have time to do all of the things I wanted.
This is mainly upstreaming as much of our packages as possible, as they are the biggest time consumer during building.
This will be dealt with after GSoC.

Nodewatcher needs to be updated to merge LEDE and OpenWrt as we have some checks to ensure that more advanced features are only enabled on LEDE as OpenWrt did not have them at that time.
This will be dealt with after GSoC too.

I also wanted to add some new features to our imagebuilders, but since hitting a lot of bugs and unexpected stuff during development I did not have for these, so like previous two points, this will be dealt after GSoC.

So to sum this up, this was a really good experience.
I got to focus on two things I enjoy working on: FOSS software and OpenWrt.
This enabled me to learn a lot on the functioning of our Nodewatcher, OpenWrt imagebuilder and especially OpenWrt SDK.

Thanks to Google for organizing GSoC, Freifunk for enabling me to give back to the community in the usefull way.
And special thanks to my mentor Valent Turković.

Best regards
Robert Marko

Hi,

Last weeks have been spent solely on reworking the build system.

First, it was a matter of rebranding the current LEDE back into OpenWrt and fixing a couple of hard-coded names that would cause issues with OpenWrt name. It also involved dropping the old OpenWrt build system which has not been used for years and most likely never will again, so that removes unnecessary code to maintain.

After rebranding, I spent some time verifying that the whole system still works.
Fortunately, there were only small bugs which were simple to fix.

And then came the main task of this project, to completely rework and massively simplify the whole building the image builder job a lot easier and resource intensive.

Firstly, since I was still gonna use Docker to images for a build environment updating the base image which is the actual build environment was needed from old Trusty 14.04 to fresh 18.04 Bionic. This proved to be mostly trial and error as a lot less of default packages were included in 18.04 so getting all dependencies working. After a while base image is now working fine and is relatively small, actually smaller than 14.04 base image.
This is due to less unnecessary packages.

Once the base image was sorted out I finally got working on dropping the unnecessary scripts, docker files and all of the hardcoded build files.

This proved to be not so hard, so work on a new docker based build system started.

So far it’s broken into only 4 separate scripts:

1. docker-prepare-build system: Like its name hints it builds the base image and installs the needed packages. I am still thinking to maybe pull this from the auto built image on Docker Hub.
2. generate-docker files: Which generates the temporary docker files needed for building inside a Docker 18.04 base image.
3. docker-build: Which actually “builds” the image builder and SDK.
4. build: Main script, which simply calls others to configure and build everything.

Number of scripts will most likely grow by one or two since the built image builder with all of the packages need to be packaged and then deployed in a runtime specific image which will only contain the bare minimum of packages to keep it as lightweight as possible.

Currently, building works fine for most custom packages using SDK, but its stuck at building ncurses with a weird LC_TIME assertion error which I need to fix.

So next period will be strictly for fixing the bugs and finishing the build system.
After that is done I will update the custom packages and try to get them upstreamed.

Since last update I spent most of time on fully understanding current build system and nodewatcher internals.

Build system

During the time spent into looking how the whole system works I belive that I was able to figure out what every step and script does in the current build system.
During that I found a lot of relatively simple improvements that can really reduce custom stuff that we have. Most of it was added 3-5 years ago when OpenWrt wasnt really in the state it is now. Custom mirror for source was added,which is now useless as it was not updated and on some really old custom packages it is really slow. Also,building all packages that are added to the package list during build time is not really efficient as during its building OpenWrt default feeds are replaced with our custom package feeds.
This causes users to be stuck on really old and quite limited number of packages. This will be revorked to only replace the target feed as it contains all of the kernel mods that are tied to specific kernel version.
Other packages have no such requirments and versions in upstream can be used.
Also,wget was used during the build time to pull dependecies instead of curl which is recommended tool. Wget is fine for simple downloads but a lof of packages are pulled from behind CDNs and with lot of redirects from mirrors such as Sourcefourge,those curl can handle fine but wget cant.

I have started dropping unused and old packages as well as those that had custom patches that were upstreamed a long time ago.
Stuff like iwinfo from 2015 and old curl.

Also,I have started to move both the buildsystem docker image and runtime docker image from Ubuntu 14.04 to 18.04 Bionic.
This does not yet fully work as Imagebuilder does not detect GCC and Ncurses in the runtime image as working despite the fact that GCC works fine. This will be hard to diagnose,but I feel that it is quite simple.

Custom Wlan Slovenija packages are being prepared for upstreaming.

This is all for now,next two weeks should be bring solutions to most issues.

Robert Marko