nodewatcher: Build system rework and package upstreaming – Second update

Hi,

Last weeks have been spent solely on reworking the build system.

First, it was a matter of rebranding the current LEDE back into OpenWrt and fixing a couple of hard-coded names that would cause issues with OpenWrt name. It also involved dropping the old OpenWrt build system which has not been used for years and most likely never will again, so that removes unnecessary code to maintain.

After rebranding, I spent some time verifying that the whole system still works.
Fortunately, there were only small bugs which were simple to fix.

And then came the main task of this project, to completely rework and massively simplify the whole building the image builder job a lot easier and resource intensive.

Firstly, since I was still gonna use Docker to images for a build environment updating the base image which is the actual build environment was needed from old Trusty 14.04 to fresh 18.04 Bionic. This proved to be mostly trial and error as a lot less of default packages were included in 18.04 so getting all dependencies working. After a while base image is now working fine and is relatively small, actually smaller than 14.04 base image.
This is due to less unnecessary packages.

Once the base image was sorted out I finally got working on dropping the unnecessary scripts, docker files and all of the hardcoded build files.

This proved to be not so hard, so work on a new docker based build system started.

So far it’s broken into only 4 separate scripts:

  1. docker-prepare-build system: Like its name hints it builds the base image and installs the needed packages. I am still thinking to maybe pull this from the auto built image on Docker Hub.
  2. generate-docker files: Which generates the temporary docker files needed for building inside a Docker 18.04 base image.
  3. docker-build: Which actually “builds” the image builder and SDK.
  4. build: Main script, which simply calls others to configure and build everything.

Number of scripts will most likely grow by one or two since the built image builder with all of the packages need to be packaged and then deployed in a runtime specific image which will only contain the bare minimum of packages to keep it as lightweight as possible.

Currently, building works fine for most custom packages using SDK, but its stuck at building ncurses with a weird LC_TIME assertion error which I need to fix.

So next period will be strictly for fixing the bugs and finishing the build system.
After that is done I will update the custom packages and try to get them upstreamed.

nodewatcher: Build system rework and package upstreaming – First update

Since last update I spent most of time on fully understanding current build system and nodewatcher internals.

Build system

During the time spent into looking how the whole system works I belive that I was able to figure out what every step and script does in the current build system.
During that I found a lot of relatively simple improvements that can really reduce custom stuff that we have. Most of it was added 3-5 years ago when OpenWrt wasnt really in the state it is now. Custom mirror for source was added,which is now useless as it was not updated and on some really old custom packages it is really slow. Also,building all packages that are added to the package list during build time is not really efficient as during its building OpenWrt default feeds are replaced with our custom package feeds.
This causes users to be stuck on really old and quite limited number of packages. This will be revorked to only replace the target feed as it contains all of the kernel mods that are tied to specific kernel version.
Other packages have no such requirments and versions in upstream can be used.
Also,wget was used during the build time to pull dependecies instead of curl which is recommended tool. Wget is fine for simple downloads but a lof of packages are pulled from behind CDNs and with lot of redirects from mirrors such as Sourcefourge,those curl can handle fine but wget cant.

I have started dropping unused and old packages as well as those that had custom patches that were upstreamed a long time ago.
Stuff like iwinfo from 2015 and old curl.

Also,I have started to move both the buildsystem docker image and runtime docker image from Ubuntu 14.04 to 18.04 Bionic.
This does not yet fully work as Imagebuilder does not detect GCC and Ncurses in the runtime image as working despite the fact that GCC works fine. This will be hard to diagnose,but I feel that it is quite simple.

Custom Wlan Slovenija packages are being prepared for upstreaming.

This is all for now,next two weeks should be bring solutions to most issues.

Robert Marko

GSoC 2018 – nodewatcher: Build system rework and package upstreaming

Who I am

My name is Robert Marko, I am 21 year old Computer Science student in Osijek, Croatia. I am active member of Otvorena Mreža, a partner organisation of Wlan Slovenija here in Croatia. Also, active member of OpenWrt project.

Project Overview

Like with most of the projects,mine has the following goals:

  • Simplification of the build system
  • Move from Ubuntu 14.04 as a base image to 18.04 to get the benefits from OpenWrt-s move to GCC7.3
  • Use OpenWrt generated image builders instead of building everything from source
  • Upstreaming and updating crucial project packages
  • Remove a number of hard coded outdated packages that are now provided by core OpenWrt

Bonding Period Experiences

I have already started working on my project by setting up the development environment with a deployment of Nodewatcher. Along the way I found out that installation was unfortunately broken so I fixed that and couple more bugs regarding packages used. I am now testing updates to newer versions of all Python packages used. Some of them were not updated for a couple of years.

First Goal/Milestone

My first goal will be to further analyze the build system as I am sure that some parts could be more optimised than in project proposal.

And then get to pushing packages upstream,that will most likely take the longest.

Best of luck to fellow GSOC students.
Robert Marko