GSoC’23 : Automation tools for LibreMesh firmware build and monitoring – part 2

Hi all!

Previous post: https://blog.freifunk.net/2023/05/27/gsoc23-automation-tools-for-libremesh-firmware-build-and-monitoring/

During this period of writing I’ve been reading up on these projects:

OpenWrt Buildroot

The main openwrt project that allows the greatest level of customization of configurations and packages, this allows to compile directly from source, and among the main features, to build firmware images for all devices, called ‘profiles’ of a given target/subtarget, or for a sub-list of these profiles. It is also the slower solution to build firmware images, compared to the OpenWrt ImageBuilder. It also allows building an OpenWrt ImageBuilder and an OpenWrt SDK.

https://openwrt.org/docs/techref/buildroot

https://openwrt.org/docs/guide-developer/start#using_the_toolchain

https://openwrt.org/docs/guide-user/virtualization/obtain.firmware.docker

Openwrt ImageBuilder (docker)

This tool allows building firmware images from precompiled packages and is also packaged as a docker image.

https://openwrt.org/docs/guide-user/additional-software/imagebuilder

https://github.com/openwrt/docker

https://hub.docker.com/u/openwrt/

OpenWrt SDK (docker)

This tool allows individual packages to be built from source and is also packaged as a docker image.

https://openwrt.org/docs/guide-developer/obtain.firmware.sdk

https://github.com/openwrt/docker

https://hub.docker.com/u/openwrt/

OpenWrt Firmware Selector

This is a GUI for selecting OpenWrt firmware images from the official repository https://downloads.openwrt.org/, and which acts as a client for the Attendedsysupgrade Server, for building custom firmware images.

This works by scanning all device profiles and building overview json files for each version from them. From this information the selection interface is then constructed.

https://gitlab.com/mwarning/firmware-selector-openwrt-org

Attendedsysupgrade Server

This is a server that accepts requests, from different clients, to build firmware images with custom lists of packages and files, these requests are then delivered to an ImageBuilder that will perform the build, the server at this point returns the sysupgrade image produced.

It is packaged as a Python project and as a docker image.

https://openwrt.org/docs/guide-user/installation/attended.sysupgrade?s[]=asu&s[]=attendedsysupgrade

https://github.com/openwrt/asu

Ansible roles to build LibreMesh

For each of these components I wrote an ansible role

• openwrt_buildroot
• openwrt_imagebuilder(_docker)
• openwrt_sdk(_docker)
• openwrt_firmware_selector
• openwrt_asu

Each role basically goes over the steps necessary to:

• prepare the system
• install the component
• configure it
• use it

These roles are based on a file called ‘recipe’ that defines the devices for which to build.

This file is provided to the ansible playbook as a variable, and is by default dependent on which version of libremesh you intend to use and which version of openwrt.

These ‘mandatory’ variables are used to expand the list of known configurations by going for the list of supported devices and known changes needed to install libremesh on them, this information is written to the ‘target’ folders in the collection repository in turn selected based on the chosen versions of libremesh and openwrt.

The mechanism is that of the inclusion of additional variable files, containing partial information. The recipe file provided has the final say by allowing any information gathered in the other files to be overwritten. This part of the code for collecting necessary information for each device still needs to be improved.

Creating device specific configurations

The creation of single device specific configurations goes over these steps:

• creation of devices as ansible hosts
• creation of a definition file for each host
• from the latter generation of a libremesh configuration file `lime-macaddress` which the way libremesh is set up ranks in the hierarchy of main libremesh configuration files found in `/etc/config` as second:
  • 0. lime-autogen: configurations actually applied, not editable
  • 1. lime-node: manual configurations at the device
  • 2. lime-<macaddress>: provisioned configurations
  • 3. lime-community: community configurations
  • 4. lime-defaults: defaults suggested by libremesh
• vpn server upgrade

Adding devices to monitoring

The setup of the monitoring system goes over these steps:

• customizing configurations
• customizing scrape_options
• definition of labels
• definition of monitoring targets
• definition of probing targets
• definition of contact channels: email, telegram
• definition of datasources and dashboards
• generation of target lists
• installation of components
• placement of components in a webserver under dns names

I have added a minimal amount of documentation and published a template for using the project at:

https://gitlab.com/a-gave/libremesh-ansible-playbooks/

The product code for the entire roles collection is available at:

https://gitlab.com/a-gave/libremesh-ansible-collection/

All roles defined in this collection to build LibreMesh and to setup the monitoring system have been written and tested on debian 11 and 12.

The product code for the role corresponding to the single OpenWrt Buildroot is available at:

https://gitlab.com/a-gave/ansible_openwrt_buildroot

Example diagrams

Here is a summary of three main playbooks/roles:

Here are listed some scenarios of possibles use cases:

Example 1: simple build of libremesh for list of devices grouped by targets

Example 2: build of libremesh, and build of firmware images to meet specific devices needs

Example 3: Like the previous but with the generation of a list of targets to monitor with prometheus

Example 4: Introduce a vpn (wireguard) to monitor also devices that aren’t reachables in the same lan of hosts that do monitoring