Author: Samlo

Previous post: https://blog.freifunk.net/2023/07/08/gsoc23-automation-tools-for-libremesh-firmware-build-and-monitoring-part-2/

Project results

These are the repositories with the produced code, where the first is the main:

https://gitlab.com/a-gave/libremesh-ansible-playbooks

https://gitlab.com/a-gave/libremesh-ansible-collection

https://gitlab.com/a-gave/ansible_openwrt_buildroot

Playbooks and roles to build releases

In this final part other than improving the code, I also used it a lot to prepare a list of firmware images with the new releases of LibreMesh: the v2020.3 based on OpenWrt 19.07.10 and the upcoming new relase v2023.1-rc1 that support the latest OpenWrt 22.03.5.

I extended and tested the automation tools to build for all devices of a defined target/subtarget to make the precompiled firmwares for the latest releases of LibreMesh. The list of target/subtarget is based on the advice of one of the last LibreMesh meeting about the architectures most used, that mainly match also those with a lot of low-cost devices.

Building for all devices has meant to encounter these kind of issues:

• the `default` set of lime-packages doesn’t fit in the factory and/or the sysupgrade image of a device and this cause a build failure.
• as the previous but the devices fail silently without interrupting the build
• multicore/parallel compiling randomly fails

For the first problem, there isn’t in the OpenWrt Buildroot a mechanism to predict the resulting firmware size i.e. based on the list of the selected packages. This seems related to the compression tools used to produce the firmware that could change depending on the devices, and leads to have successful builds for a device with an image size of 7168k while another with the same image size will fail. This means to not be able to predict if the produced LibreMesh firmware will fit in the device memory, and the devices that cause a build failure should be identified singularly.

I take notes of devices that cause a build failure for the selected list of target/subtarget for the mentioned OpenWrt and LibreMesh releases using the default set of LibreMesh packages. Firstly providing a mechanism to exclude the devices that throws an error. This doesn’t mean that the other devices are somehow ‘supported’ but simply that the build doesn’t fail. Since this work of mapping of all devices is still in development it is available at the branch dev of the collection of roles:

• Notes about LibreMesh 2020.3 on OpenWrt 19.07.10: https://gitlab.com/a-gave/libremesh-ansible-collection/-/tree/dev/target/libremesh_2020.3/openwrt_19.07.10?ref_type=heads
• Notes about LibreMesh 2023.1-rc1 on OpenWrt 22.03.5: https://gitlab.com/a-gave/libremesh-ansible-collection/-/tree/dev/target/libremesh_master/openwrt_22.03.5?ref_type=heads

Since the support of LibreMesh for OpenWrt 22.03.5 is still in testing, and the amount of time and space needed to reproduce all the images for the selected architectures may be considerable. I put in place two mechanism:

• having a list of `supported_devices`, that can be used to rebuilt only a sublist of devices, ideally those of people/community who can test them.
• I started to setup a set of docker images for differents OpenWrt targets/subtargets to build with the default packages of LibreMesh, that I briefly explain.

Dockerized buildroot for each target/subtarget

There is a set of Dockerfiles that I’m including in the set of ansible playbooks/roles to speed up the process of building and to save space among different LibreMesh releases based on the same OpenWrt releases.

https://github.com/a-gave/libremesh_openwrt_buildroot_docker

It is thinked to be able to rebuild different versions of LibreMesh with a separated environment but avoiding rebuilding the same OpenWrt tools and toolchain more times. For instance if the build of LibreMesh for all devices of the OpenWrt target ath79/generic takes 1 hour and has a size of 28.8GB, a subsequent builds with minor changes will take around 30 min and increase the size of the produced docker image by only 15GB. In a similar way a docker image with pre-compiled tools, toolchain, pre-extracted kernel and precompiled kmods and packages, that build without specifying a device but only the target/subtarget will take 11.2GB of space but allow to build then an image, that has pre-selected the set of LibreMesh suggested packages, within 4 minutes. This time also depends on the amount of additional packages selected and on the availability of computing resources for the builder machine.

Other contributions

In this period I also contributed to LibreMesh project providing metrics for this analysis of the effect of changing the default distance for long wireless links.

https://github.com/ilario/wifi-distance-setting-exploration

This is a kind of regression for outdoor devices, still unfixed due to two fact:

• neither OpenWrt nor LibreMesh, has an easy way to determine if the device is manufactored to be used indoor (tipically a router) or outdoor (an antenna).
• to improve the performances of routers, a LibreMesh choice was to lower the default distance, but so remain disadvantaged the antennas. This means they are more susceptible to have a broken wireless link (as if it were a cut cable) if the device is accidentally reset and so they require a custom build with different defaults.

Conclusion

A thanks to Ilario and Stefca for having been my mentors, to the organizations of FreiFunk and LibreMesh that made this work possible, and to all folks of OpenWrt and Gluon that contribute to free and open source networks.

Hi all!

Previous post: https://blog.freifunk.net/2023/05/27/gsoc23-automation-tools-for-libremesh-firmware-build-and-monitoring/

During this period of writing I’ve been reading up on these projects:

OpenWrt Buildroot

The main openwrt project that allows the greatest level of customization of configurations and packages, this allows to compile directly from source, and among the main features, to build firmware images for all devices, called ‘profiles’ of a given target/subtarget, or for a sub-list of these profiles. It is also the slower solution to build firmware images, compared to the OpenWrt ImageBuilder. It also allows building an OpenWrt ImageBuilder and an OpenWrt SDK.

https://openwrt.org/docs/techref/buildroot

https://openwrt.org/docs/guide-developer/start#using_the_toolchain

https://openwrt.org/docs/guide-user/virtualization/obtain.firmware.docker

Openwrt ImageBuilder (docker)

This tool allows building firmware images from precompiled packages and is also packaged as a docker image.

https://openwrt.org/docs/guide-user/additional-software/imagebuilder

https://github.com/openwrt/docker

https://hub.docker.com/u/openwrt/

OpenWrt SDK (docker)

This tool allows individual packages to be built from source and is also packaged as a docker image.

https://openwrt.org/docs/guide-developer/obtain.firmware.sdk

https://github.com/openwrt/docker

https://hub.docker.com/u/openwrt/

OpenWrt Firmware Selector

This is a GUI for selecting OpenWrt firmware images from the official repository https://downloads.openwrt.org/, and which acts as a client for the Attendedsysupgrade Server, for building custom firmware images.

This works by scanning all device profiles and building overview json files for each version from them. From this information the selection interface is then constructed.

https://gitlab.com/mwarning/firmware-selector-openwrt-org

Attendedsysupgrade Server

This is a server that accepts requests, from different clients, to build firmware images with custom lists of packages and files, these requests are then delivered to an ImageBuilder that will perform the build, the server at this point returns the sysupgrade image produced.

It is packaged as a Python project and as a docker image.

https://openwrt.org/docs/guide-user/installation/attended.sysupgrade?s[]=asu&s[]=attendedsysupgrade

https://github.com/openwrt/asu

Ansible roles to build LibreMesh

For each of these components I wrote an ansible role

• openwrt_buildroot
• openwrt_imagebuilder(_docker)
• openwrt_sdk(_docker)
• openwrt_firmware_selector
• openwrt_asu

Each role basically goes over the steps necessary to:

• prepare the system
• install the component
• configure it
• use it

These roles are based on a file called ‘recipe’ that defines the devices for which to build.

This file is provided to the ansible playbook as a variable, and is by default dependent on which version of libremesh you intend to use and which version of openwrt.

These ‘mandatory’ variables are used to expand the list of known configurations by going for the list of supported devices and known changes needed to install libremesh on them, this information is written to the ‘target’ folders in the collection repository in turn selected based on the chosen versions of libremesh and openwrt.

The mechanism is that of the inclusion of additional variable files, containing partial information. The recipe file provided has the final say by allowing any information gathered in the other files to be overwritten. This part of the code for collecting necessary information for each device still needs to be improved.

Creating device specific configurations

The creation of single device specific configurations goes over these steps:

• creation of devices as ansible hosts
• creation of a definition file for each host
• from the latter generation of a libremesh configuration file `lime-macaddress` which the way libremesh is set up ranks in the hierarchy of main libremesh configuration files found in `/etc/config` as second:
  • 0. lime-autogen: configurations actually applied, not editable
  • 1. lime-node: manual configurations at the device
  • 2. lime-<macaddress>: provisioned configurations
  • 3. lime-community: community configurations
  • 4. lime-defaults: defaults suggested by libremesh
• vpn server upgrade

Adding devices to monitoring

The setup of the monitoring system goes over these steps:

• customizing configurations
• customizing scrape_options
• definition of labels
• definition of monitoring targets
• definition of probing targets
• definition of contact channels: email, telegram
• definition of datasources and dashboards
• generation of target lists
• installation of components
• placement of components in a webserver under dns names

I have added a minimal amount of documentation and published a template for using the project at:

https://gitlab.com/a-gave/libremesh-ansible-playbooks/

The product code for the entire roles collection is available at:

https://gitlab.com/a-gave/libremesh-ansible-collection/

All roles defined in this collection to build LibreMesh and to setup the monitoring system have been written and tested on debian 11 and 12.

The product code for the role corresponding to the single OpenWrt Buildroot is available at:

https://gitlab.com/a-gave/ansible_openwrt_buildroot

Example diagrams

Here is a summary of three main playbooks/roles:

Here are listed some scenarios of possibles use cases:

Example 1: simple build of libremesh for list of devices grouped by targets

Example 2: build of libremesh, and build of firmware images to meet specific devices needs

Example 3: Like the previous but with the generation of a list of targets to monitor with prometheus

Example 4: Introduce a vpn (wireguard) to monitor also devices that aren’t reachables in the same lan of hosts that do monitoring

Introduction

Hi everyone! I’m samlo. I’m a fullstack webdev that live in a rural area in Italy and dedicates part of his time to build an maintain a self-managed community network based on LibreMesh https://antennine.noblogs.org/.

For GSoC’23, I’ll be working on setup a set of ansible playbooks and roles to do common network administration tasks useful for a tech team of a community network based on LibreMesh.

This first blog post intends to cover details on the necessary background to understand the project and its implementation.

What

As state the site https://libremesh.org/, LibreMesh is a modular framework for creating OpenWrt-based firmwares for wireless mesh nodes.

It’s a list of packages (lime-packages) with support for various mesh protocols, to installing and configuring them properly, and offer to the end user a dedicated web interface (lime-app).

It has support for potentially every OpenWrt supported devices, following the documentation you find all the information to build the firmware and configure the main files and start using it.

It also has a list of network configurations used by different communities (network-profiles) that provide the information about how to configure your firmware (installing packages) and your network (e.g. editing main configuration files), or both, to join the community mesh.

Motivations

Libremesh is a set of packages you can include as feeds – via sources or precompiled packages – in a OpenWrt build system and then select those of your choice, but it’s possible to overwrite openwrt default configs only manually, and make backup of produced configurations files per build.

In this scenario is necessary to save configurations and ways to reproduce the same firmware image.

Instead of start writing a list of bash scripts to handle just our community needs I’m interested in exploring the possibility of using a configuration management and automation tool as Ansible https://docs.ansible.com.

This would lead to simplify common needs, in particular:

– automate the build of firmwares for groups of devices with specific configurations, packages, libremesh and openwrt versions.

– build test firmwares by versioning experiments

– build on localhost or on a remote machine

– manage configurations (monitoring, vpn, ssl certificates) in the same system that also build firmwares

– automate the insertion of information that may be synced between networking devices and servers

– share configurations in a set meaningful and reproducible for other people inside and outside the local community network.

An issue

Libremesh doesn’t handle a system to build consistently for every supported devices or to patch openwrt to meet the needs of particular targets or devices.

So every community should understand how to build for devices they are using.

One inspiration for this came from the project Gluon https://gluon.readthedocs.io/ that include a system to keep traces of specific packages related to openwrt targets, subtargets and device.

https://github.com/freifunk-gluon/gluon/blob/master/targets

Deliverables of the project

– Have an ansible set of playbooks and roles to build openwrt firmwares

– Have an ansible set of playbooks and roles to build libremesh firmwares

– Have an ansible role to build libremesh firmwares depending on libremesh version, openwrt version, libremesh default packages, libremesh target’s or device’s specific packages, libremesh community packages, libremesh community set of packages linked to specific list of devices.

– Have an ansible set of playbooks and roles to setup a monitoring/probing/alerting/metric-visualizer system

Concluding Thoughts

In this period of design and blueprint of the project, before starting coding I thinked a lot at use cases, who could want to use it, and to simplify contributions to keep updated the code in the future.

I look forward to publish on https://galaxy.ansible.com/ two collection of roles (openwrt and libremesh) and make available via git repository the set of playbooks to use the roles above.

I’ll update you in July.