Load-correlated distributed bandwidth analysis for LibreMesh networks – #3: Completed test network and broadened scope of the work

The planned test network has been built, employing both fully supported (I just documented them in the tested routers list here) and common home routers (officially unsupported by LibreMesh but supported by OpenWrt).

Employing non supported routers required an expansion of my previous work about making possible an AP-sta (point to multi point access point to clients) network architecture (instead of the default IEEE802.11s mesh). My previous solution relied on BMX6 which will not be included in the next release, in favor of Babeld, so the problem is open again. I provisionally managed to have Babeld on AP and client interfaces adding the following setting in /etc/config/lime on the access point:

config wifi 'radio0'
     list modes 'apname'
     option country 'ES'
     option channel_2ghz '11'
     option apname_ssid 'LibreMesh.org/%H'
     option apname_key 'someAPpassword'
     option apname_encryption 'psk2'
     option distance '100'

 config net 'wirelessap'
     option linux_name 'wlan0-apname'
     list protocols 'babeld:17'

and the following in the /etc/config/lime of the client (taking advantage of the client protocol I added some time ago here):

config wifi 'radio0'
     list modes 'client'
     option country 'ES'
     option channel_2ghz '11'
     option client_ssid 'LibreMesh.org/LiMe-eb7f64'
     option client_key 'someAPpassword'
     option client_encryption 'psk2'
     option distance '100'

 config net 'wirelessclient'
     option linux_name 'wlan0-sta'
     list protocols 'client'
     list protocols 'babeld:17'

For some reason this solution does not propagate the default route obtained from Babeld to the whole network, this does not directly affect my project, anyway I’ll surely manage to fix this in the upcoming days.
In case the usage of such perfectly-working trashware was a blocker, I will receive a few more supported routers in the following days and I will just use those.

Also due to the switch to Babeld, to obtain a complete graph of the network is not yet possible (Babeld being based on the distance vector principle, does not know the whole topography and we’ll have to aggregate it using the new shared-state LibreMesh feature).

During the building of the test network, the planned topography changed a bit resulting in this one (solid lines are cabled connections, directional dotted lines with arrows points from the client to the access point, non-directional dotted lines are proper IEEE802.11s mesh):

All the routers were flashed with OpenWrt 18.06-SNAPSHOT image, which is OpenWrt 18.06.4 with additional fixes appeared in the release branch here compiled locally using OpenWrt buildroot. LibreMesh packages were also compiled in the same process but not included in the compiled image, and installed later using opkg and serving the packages over the local network. This approach showed to be more convenient than expected, additionally, the fallback image is a plain OpenWrt, which decrease the risk of “brikking” the devices.

The complete list of the installed packages from the LibreMesh ones is:

check-date-http first-boot-wizard hotplug-initd-services lime-app lime-debug lime-hwd-ground-routing lime-hwd-openwrt-wan lime-proto-anygw lime-proto-babeld lime-proto-batadv lime-proto-wan lime-system shared-state shared-state-babeld_hosts shared-state-dnsmasq_hosts shared-state-bat_hosts shared-state-persist shared-state-dnsmasq_leases shared-state-pirania

Lately, I got also involved in the development of lime-log-review, which uses liblognorm to decrease the volume of the logs and can be used in my project for storing the key information from the voluminous logs when an incident is detected.

Unittesting LibreMesh GSoC mid-term update

A few weeks passed and I want to share the progress of the project and what are the next steps 🙂

Unit testing tools and ecosystem

As one of the goals is that it must be easy for developers to write, modify and run the tests I created some simple tools to do this:

  • testing image -> Dockerfiles/Dockerfile.unittests
  • testing shell environment -> tools/dockertestshell
  • running the tests -> ./run_tests script

Testing image

In order to run the tests and have a reliable environment without the it’s working on my computer syndrome I created a simple and very small Docker image. This image is based on an image with Lua 5.1 and luarocks made by abaez. And then it just installs busted and luacov frameworks and bash.

FROM abaez/luarocks:lua5.1

WORKDIR /root

RUN luarocks install luacov; \
    luarocks install busted

# TODO: move into a development dockerfile
RUN apk add --no-cache bash bash-completion

Nixio library

It would be good to have nixio available inside the docker image because this library is widely used in libremesh and also it could be very handy to have it available for the testing also.

I did an effort to add it to the image but many problems arised. The luarocks version of nixio 0.3-1 it is not working mainly because some compilation issues with newer versions of gcc. So I tried to work on a rockspec without this problem but I could not finish it because other problems arised I think that related to the Alpine/musl distribution and libc/linux headers. I will try help the author of nixio to publish a new version to luarocks as this will benefit others too.

Testing shell environment

To provide an easy way to develop or test things within the docker image I created a tool that opens a bash shell inside the docker image that has some features that allows easy development:

  • /home/$USER is mounted inside the docker image so each change you do to the code from inside is maintained when you close the docker container
  • the same applies to /tmp
  • you have the same user outside and inside
  • network access is garanted
  • and some goodies like bashrc, some useful ENV variables, PS1 modification, etc.

To enter the shell environment run:

[san@page lime-packages]$ ./tools/dockertestshell 
(docker) [san@page lime-packages]$

You can see that the prompt is changed adding (docker) in the left part so you can easily remember that you are inside the docker container.

This environment is also used by run_tests script.

Running the tests: run_tests bash script

run_tests

This script is what you should be running each time you want to run the tests. As you can see in the image we currently have 19 tests and all are passing 🙂

For the sake of showing you what to expect when a test fails I modified a test condition to be false and here is the output:

run_tests_fail

now 18 tests are good, and there is one failure. The assertion that is failing is on line 11 and the test is Fake uci tests test simple get and set. Also it is shown that the expected result is the number 2 but the actual result is the number 1.

As it is expected, run_tests returns with 0 when all tests pass and != 0 when there is at least one failure.

The script in detail

The idea behind this script is simple:

  • sets the search path of the tests for buster (the unittesting framework)
  • sets the lua library paths, prepending the fake library and adding the paths to the libremesh packages with packages/lime-system/files/usr/lib/lua/?.lua. This doesn’t work automaticaly for every package if the paths does not use the files/path/to/final/destination. So if you want to test some package without the files convention maybe it would be good to move the package to this convention. Also it does not work if the lua module we want to test does not finish with .lua, in this case the path must be explicitly added (I wrote about this in a previous blog post).
  • runs the tests using the dockertestshell

run_tests also passes the first argument as an argument to busted so you can do things like this:

[san@page lime-packages]$ ./run_tests "--list --verbose"
packages/lime-system/tests/test_lime_config.lua:11: LiMe Config tests test empty get
packages/lime-system/tests/test_lime_config.lua:15: LiMe Config tests test simple get
packages/lime-system/tests/test_lime_config.lua:20: LiMe Config tests test get with fallback
packages/lime-system/tests/test_lime_config.lua:24: LiMe Config tests test get with lime-default
packages/lime-system/tests/test_lime_config.lua:30: LiMe Config tests test get precedence of fallback and lime-default
packages/lime-system/tests/test_lime_config.lua:36: LiMe Config tests test get with false value
packages/lime-system/tests/test_lime_config.lua:41: LiMe Config tests test get_bool
packages/lime-system/tests/test_lime_config.lua:54: LiMe Config tests test set
packages/lime-system/tests/test_lime_config.lua:64: LiMe Config tests test set nonstrings
packages/lime-system/tests/test_lime_config.lua:81: LiMe Config tests test get_all
packages/safe-upgrade/tests/test_safe_upgrade.lua:5: safe-upgrade tests test get current partition
tests/test_fake_uci.lua:4: Fake uci tests test simple get and set
tests/test_fake_uci.lua:14: Fake uci tests test multiple cursors
tests/test_fake_uci.lua:31: Fake uci tests test nested get and set
tests/test_fake_uci.lua:49: Fake uci tests test state not preserved between tests
tests/test_fake_uci.lua:54: Fake uci tests test save
tests/test_fake_uci.lua:59: Fake uci tests test delete
tests/test_fake_uci.lua:73: Fake uci tests test foreach
tests/test_fake_uci.lua:87: Fake uci tests test get_all

Here is the code:

$ cat run_tests 
#!/bin/bash

TESTS_PATHS='packages/*/tests/test*.lua  tests/test*.lua'
LIB_PATHS='tests/fakes/?.lua;packages/lime-system/files/usr/lib/lua/?.lua;packages/safe-upgrade/files/usr/sbin/?;;'

./tools/dockertestshell "busted -v ${TESTS_PATHS} --lpath='${LIB_PATHS}'" ${1}

Integration of unittests with Travis CI

LibreMesh has already a github/Travis integration with two objetives:

  • test that the packages can be built (no Makefile errors, etc)
  • build and publish the packages of the master branch to an external server

The build pipeline of LibreMesh has been broken for a couple of months because the docker image that has been in use is not longer available. This is becaouse there is an ongoing effort by aparcar to create canonical docker images for OpenWrt.
So I did an atempt to fix the current LibreMesh build pipeline using the new infraestructure in this pull request. The build is still not passing but it seems it is something easy to fix as the build is passing but then the deploy is failing.

Travis unit testing

Beside fixing the current pipeline and to integrate the unittesting work I did a refactoring of the build steps to have a unittest stage and a build stage. To do this I installed the Github/Travis integration on my lime-packages fork. In the following image you can see that the two stages are green (tests are passing) 🙂

TravisPipeline

And here is the log of the unittests stage. You can see that it takes less than a minute to run the stage, with 15 seconds building the docker image and 0.011133 seconds to run the tests :100:

Next Steps

Now that the framework is in place and in continous integration we should be doing the following:

  • Add documentation on how to write tests
  • Integrate nixio in the docker image
  • Proofread the core LibreMesh code and inform about its testability
  • Provide some mocks for common functionality (uci already done!)

The first weeks of august I will move to Catalunya to work with a core developer of LibreMesh. So with my mentor NicoP we will adapt the schedule to take advantage of this.

Load-correlated distributed bandwidth analysis for LibreMesh networks – #2: Setting up the LibreMesh test network

In order to use the latest version of everything, I merged the latest commits from the LibreMesh community into my forked lime-packages repository.

To set up the test network was more complex than expected.
I managed to collect a very disperse set of routers: 8 routers of 6 different producers and 7 different models.
Two of these are officially supported by LibreMesh (TP-Link TL-WDR3600, Ubiquiti NanoStation Loco M2) and the others which are supported by OpenWrt but not by LibreMesh (Comtrend AR-5387un, Huawei HG556a-C, Observa VH4032N, Comtrend AR-5315u, Astoria ARV7519RW22-A-LT).

The non-LibreMesh-supported routers either cannot do multi-AP or mesh via IEEE802.11s, but this was not expected to be a problem as I took care to add the support to AP-client networks (no need for the routers to support IEEE802.11s mesh, only the last mentioned router does not have support for wifi at all).
My solution was based on BMX6 which seems will be dropped in the next LibreMesh release in favour of Babeld, and this will require an adaptation of the AP-client solution.

As mentioned in the previous post, I started compiling my LibreMesh firmware based on LibreRouter fork of OpenWrt 18.06 repository.
When I flashed my routers and configured the wireless interfaces for using AP or client rather than the default AP+AP+IEEE802.11s, most of them were showing strongly erratic behaviours.

So I decided to flash the routers with plain OpenWrt 18.06.2 without using LibreRouter fork and to install all LibreMesh packages via opkg.
In order to ensure that the compiled packages will be compatible with OpenWrt 18.06.2 release, the LibreMesh packages were compiled in my local buildroot of OpenWrt branch openwrt-18.06.
Then the openwrt/bin/ directory was served via HTTP from my local machine.
In order to have the routers accept my local repositories I had to install usign, create a key pair, sign the Packages file, push the public key to the routers and add the directions of the local repositories to /etc/opkg/customfeeds.conf
So for example, the customfeeds.conf file of the Observa VH4032N router will look like:

src/gz local_base http://192.168.1.3/packages/mips_mips32/base
src/gz local_libremap http://192.168.1.3/packages/mips_mips32/libremap
src/gz local_libremesh http://192.168.1.3/packages/mips_mips32/libremesh
src/gz local_luci http://192.168.1.3/packages/mips_mips32/luci
src/gz local_packages http://192.168.1.3/packages/mips_mips32/packages
src/gz local_routing http://192.168.1.3/packages/mips_mips32/routing
src/gz local_brcm63xx_smp http://192.168.1.3/targets/brcm63xx/smp/packages

Once completely configured, the network structure planned is represented in black in the following scheme.

Planned test network structure.

In order to better test the on a proper mesh, I ordered 3 additional routers fully supported by LibreMesh: YouHua WR1200JS (see here and here) from here.
They come with OpenWrt pre-installed and they fully support multi-AP + IEEE802.11s.
Once I will receive these two additional routers I will be able to add the mesh part of the test network as indicated in the scheme in red.

Regarding the load analysis of the network, the first approach will be to obtain this value from the number of clients currently connected to the network.
This number will be obtained in at least the following ways:

batadv-vis -f jsondoc | sort -u | wc -l

ip neigh show nud reachable | wc -l

In the meanwhile, a minor enhancement has been suggested and two others were accepted.

GSoC 2019 – Unit testing LibreMesh – Update 1

In the last weeks I have been involved in getting deeper into becoming part of the development team of LibreMesh.

During that process, I worked together with NicoPace in writing this blogpost where we build a solid ground for unit testing: https://blog.freifunk.net/2019/06/03/gsoc-2019-evaluating-options-to-do-unit-and-integration-tests-in-libremesh-and-a-first-working-example/

Not covered by the last blog post is the work that I did in a fake/mock implementation of the libuci library in lua. This allows writing a lot of tests for LibreMesh as ucithe most used library in the codebase that make sense to write a mock. The implementation is very small but covers the most used functionality of libuci: cursor(), get(), set(), save(), delete() and foreach(). This was implemented doing TDD with the support of the unittesting framework.

All this work is being done in the following branch of my lime-packages fork: https://github.com/spiccinini/lime-packages/commits/unittest_docker

During the upcoming weeks all this work will be properly released as a PR to the lime-packages repo accompanied by the Travis CI integration in a Docker container to do the tests in a contained environment, and more tests are going to follow 🙂

GSoC 2019 – Evaluating options to do unit and integration tests in LibreMesh (and a first working example)

Prior experience

Some people that have been writing about unit testing in lua, and also about lua for embedded:

  • http://lua-users.org/wiki/UnitTesting
  • https://blog.freifunk.net/2019/05/26/gsoc-2019-unit-testing-libremesh/

Requirements

The set of requirements for the LibreMesh project in regards of testing are the following:

  • Must support lua 5.1, as it is the one packaged in OpenWRT
  • Must provide helpful assert test functions, like showing table diffs or formatters for outputs to understand the difference easily
  • We need mock functionality, because a lot of functions are hardware related and may not be possible to test them on hardware all the time.
  • It is desirable for the library to be a one-file import, so we can use it in the routers and in the continuous integration in the same way we do it in the desktop.

Options

We need to consider unit testing, mocking and coverage tests.

Unit Testing

There is a list of unittesting libraries in the lua package manager, luarocks:
https://luarocks.org/labels/test?non_root=on

These are the ones analyzed:

LuaUnit

URL: https://github.com/bluebird75/luaunit

Upsides:

  • No external dependencies (single file),
  • it’s well maintained,
  • popular (200k downloads luarocks, 200 stars Github),
  • supports multiple versions of Lua.
  • Has TAP support (for CI)
  • It is being used in other OpenWRT based images like OpenWISP: https://github.com/openwisp/openwisp-config/blob/master/openwisp-config/tests/test_utils.lua

Downsides:

  • It doesn’t have mocking helpers: Could be combined with a one file mocking library mockagne

Telescope

They describe themselves as a A highly customizable test library for Lua that allows declarative tests with nested contexts.
URL: https://github.com/norman/telescope/

Last release 2013. Lua 5.1 last release was done in 2012, so it is not that big of a deal… but has not received any updates since, so it might have not evolved since.

Busted

URL:

  • https://github.com/Olivine-Labs/busted
  • http://olivinelabs.com/busted/

Upsides:

  • Very well maintained by olivinelabs and contributors,
  • very popular (900k downloads luarocks, 800 github stars).
  • It has setup/teardowns and also mocks, spies, and matchers.
  • Has TAP support.
  • Has good documentation
  • It is integrated with luacov for test coverage

Downsides:

  • Must be installed using luarocks (it is a lot of files). A question has been posted to luarocks to explore the possibility of creating bundles for a package (one file with all dependencies). That would simplify its use: https://github.com/luarocks/luarocks/issues/1023

Mocking libraries

lua-mock

URL: https://github.com/henry4k/lua-mock

Mach

URL: https://github.com/ryanplusplus/mach.lua/

More or less well maintain, though it is not so popular.

Coverage reports

luacov

URL: https://github.com/keplerproject/luacov

Upsides:

  • Well maintained

Unit testing Architecture for LibreMesh (only for LibreMesh?)

The idea is to allow unit-testing packages and also the integration between them as some of the packages depend on other packages.

Context:

  • LibreMesh enables functionality selecting which packages must be installed and changing enabling/disabling the exposed features in configuration files.
  • In some packages the code is all inside the executable lua file (not like a library)
  • Some packages are independent, provide functionality without depending on lime-system. This packages are in lime-packages for convenience.
  • Packages could (should) be migrated into OpenWrt repositories. This migration may happen steps and when migrated the code may be in an independent repository for the package iteself.
  • many packages are wrappers of bash code, and this complicates the tests as you need a running system to test it out

This context is not an easy one to test as it has a lot of trade offs!

Options

Single and global tests directory

The easiest architecture is to have a global tests directory and some utility functions that allow to “install” a certain module for testing

Directory structure:

lime-packages/package/package1/
lime-packages/package/package1/...
lime-packages/package/package2
lime-packages/package/package2/...
lime-packages/tests/utils.lua
lime-packages/tests/fake_modules/nixio.fs
lime-packages/tests/test_package_1.lua
lime-packages/tests/test_package_2.lua
lime-packages/tests/test_package_1_and_2_integration.lua
lime-packages/run_tests.sh

Example of a (integration) test that uses libraries and fake modules
test_lime_proto_anygw.lua:

utils = require("test.utils")
-- installs required modules in the lua path
utils.install_limesystem_module() -- to allow access to lime.network, etc
utils.install_module("packages/lime-proto-anygw/src/anygw.lua", "lime.proto.anygw")
utils.install_module("tests/fake_modules/nixio.fs", "nixio.fs")

-- now we can load the modules
anygw = require("lime.proto.anygw")

function test_foo()
  assert anygw.foo() == 'bar'
end

Pros:

  • Easy to start with and to understand

Cons:

  • As all tests are together it is not easy to move a package to other repository or even to its own repository.

Tests inside each module and a shared tests directory to test integrations

Directory structure:

lime-packages/package/package1/
lime-packages/package/package1/tests/test_foo.lua
lime-packages/package/package2
lime-packages/package/package2/tests/test_bar.lua
lime-packages/tests/utils.lua
lime-packages/tests/fake_modules/nixio.fs
lime-packages/tests/test_package_1_and_2_integration.lua
lime-packages/run_tests.sh

Pros:

  • Each package has more independence

Cons:

  • ?

Testing in a fully working image with all packages and libraries installed

Tests can be run installing all files of the packages (by some script that parses the Makefiles, or “by hand in a helper script”).

Pros:

  • It requires less boilerplate to test package integration
  • Libraries of the target system can be used directly
  • Other packages may be installed

Cons:

  • Less control over what it is really happening
  • slower than the other options, as it needs to load a full system

Fake modules as library

Testing executable modules

Executable lua modules can be tested with a simple modification in the file creating a main() function and then using something like:

function main()
  --- the main code in here
end

-- detect if this module is run as a library or as a script
if pcall(debug.getlocal, 4, 1) then
  -- Library mode, do nothing
else
  -- Main script mode
  main()
end

Then from a test file it can be loaded like any normal module and all the functions can be accesed without executing main()

Testing environment

A docker environment (or multiple, even using “qemu-user” under docker) with the testing libraries and target lua version is loaded by the “run_tests.sh” executable.
This environment can provide some useful libraries for testing (coverage reporting,

Direct import and testing can be done for unit tests were functions are not using system libraries, or when these are simple enough to be mockable (mocking shell() calls).

luarocks router-local environment

described by this guy, thanks!

We did a trial to run busted inside a router… that would have been useful for in-router tests and also for tests inside a virtual environment.

It used the strategy of installing luarocks dependencies in a separate directory and copying them to the router.

The steps are pretty straightforward:

$ sudo apt install luarocks
$ luarocks install --tree lua_modules busted
$ cat <EOF
require 'busted.runner'()

describe('Busted unit testing framework', function()
  describe('should be awesome', function()
    it('should be easy to use', function()
      assert.truthy('Yup.')
    end)

    it('should have lots of features', function()
      -- deep check comparisons!
      assert.same({ table = 'great'}, { table = 'great' })

      -- or check by reference!
      assert.is_not.equals({ table = 'great'}, { table = 'great'})

      assert.falsy(nil)
      assert.error(function() error('Wat') end)
    end)

    it('should provide some shortcuts to common functions', function()
      assert.unique({{ thing = 1 }, { thing = 2 }, { thing = 3 }})
    end)

    it('should have mocks and spies for functional tests', function()
      local thing = require('thing_module')
      spy.spy_on(thing, 'greet')
      thing.greet('Hi!')

      assert.spy(thing.greet).was.called()
      assert.spy(thing.greet).was.called_with('Hi!')
    end)
  end)
end)

EOF
> test.lua
$ cat <EOF
-- set_paths.lua
local version = _VERSION:match("%d+%.%d+")
package.path = 'lua_modules/share/lua/' .. version .. '/?.lua;lua_modules/share/lua/' .. version .. '/?/init.lua;' .. package.path
package.cpath = 'lua_modules/lib/lua/' .. version .. '/?.so;' .. package.cpath
EOF
> set_paths.lua
$ scp -r test.lua set_path.lua lua_modules root@thisnode.info:~
$ ssh root@thisnode.info 'lua -l set_paths test.lua'

The output of this command though was not what we expected:

$ ssh root@thisnode.info 'lua -l set_paths test.lua'
lua: lua_modules/share/lua/5.1/pl/path.lua:28: pl.path requires LuaFileSystem
stack traceback:
    [C]: in function 'error'
    lua_modules/share/lua/5.1/pl/path.lua:28: in main chunk
    [C]: in function 'require'
    lua_modules/share/lua/5.1/busted/runner.lua:3: in main chunk
    [C]: in function 'require'
    test.lua:1: in main chunk
    [C]: ?

Deeper inspection showed that the library’s dependencies had C bindings that we compiled for a different arquitecture, so tha strategy was not feasable for routers anymore:

find . -name \*.so
./lua_modules/lib/lua/5.1/lfs.so
./lua_modules/lib/lua/5.1/term/core.so
./lua_modules/lib/lua/5.1/system/core.so

where term/core.so and system/core.so are system libs, but lfs.so is from LuaFileSystem.

There is a lua-only implementation of LuaFileSystem: https://github.com/sonoro1234/luafilesystem , but as it doesn’t support luarocks deeper understanding of the platform is needed to attempt to replace the only binary binding with this implementation.

Found a sister library from that one in luarocks: https://luarocks.org/modules/3scale/luafilesystem-ffi based on this repo: https://github.com/spacewander/luafilesystem. So:

$ luarocks install --tree lua_modules luafilesystem-ffi

installed it and then touched the code were the penlight library was imported in busted:

$ grep -r require.\*lfs *
path.lua:local res,lfs = _G.pcall(_G.require,'lfs')
$ pwd
/home/nico/tmp/lua_local_test/lua_modules/share/lua/5.1/pl

but this library, as it depends on ffi (a module of luajit), it depends on a C extension too.

also, luafilesystem exists as a native library in OpenWRT, so it could be included just for the sake of the exercise: https://openwrt.org/packages/pkgdata/luafilesystem … but not this time.

Docker with LuaRocks and Lua 5.1

Some docker images already exist:

  • https://github.com/akornatskyy/docker-library/
  • https://hub.docker.com/r/abaez/luarocks/
  • https://hub.docker.com/r/abaez/lua
  • https://github.com/martijnrondeel/docker-luarocks

A simple Dockerfile whould be:

FROM abaez/luarocks:lua5.1

WORKDIR /root

RUN luarocks install luacov; \
    luarocks install busted

Excellent blog post on handling Lua paths: http://www.thijsschreijer.nl/blog/?p=1025

Example of LUA_PATH to load executables (without ending in .lua): LUA_PATH="packages/safe-upgrade/files/usr/sbin/?;;". The double ;; at the end means append the default paths.

First attempt running tests

I selected safe-uprgade libremesh module to start doing unittests because I know the module as I wrote it so I already know which code would gain value being tested. Also I am confident to refactor the module if needed.

First I start using the busted unittest library with a simple test of the function get_current_partition() that must return the partition number that is currently running. As this is done from reading /proc/mtd I refactored the function so we can pass from the outside the expected content.

Content of lime-packages/safe-upgrade/tests/test_safe_upgrade.lua:

local su = require "safe-upgrade"

describe("safe-upgrade tests", function()

    it("test get current partition", function()

        proc_mtd = [[#!
        dev:    size   erasesize  name
        mtd0: 00020000 00010000 "factory-uboot"
        mtd1: 00020000 00010000 "u-boot"
        mtd2: 00180000 00010000 "kernel"
        mtd3: 00d40000 00010000 "rootfs"
        mtd4: 00b10000 00010000 "rootfs_data"
        mtd5: 000f0000 00010000 "config"
        mtd6: 00010000 00010000 "firmware"
        mtd7: 00ec0000 00010000 "fw2"
        mtd8: 00ec0000 00010000 "ART"
        ]]
        assert.is.equal(su.get_current_partition(proc_mtd), 1)

        proc_mtd = [[#!
        dev:    size   erasesize  name
        mtd0: 00020000 00010000 "factory-uboot"
        mtd1: 00020000 00010000 "u-boot"
        mtd2: 00180000 00010000 "kernel"
        mtd3: 00d40000 00010000 "rootfs"
        mtd4: 00b10000 00010000 "rootfs_data"
        mtd5: 000f0000 00010000 "config"
        mtd6: 00010000 00010000 "fw1"
        mtd7: 00ec0000 00010000 "firmware"
        mtd8: 00ec0000 00010000 "ART"
        ]]
        assert.is.equal(su.get_current_partition(proc_mtd), 2)

    end)
end)

The modifications I did to do to the safe-upgrade module are:

  • refactor get_current_partition() into get_proc_mtd() and get_current_partition(proc_mtd). This way we can inject different /proc/mtd values for testing.
  • return a table containing the module exported functions when running in library mode. In this case we are exporting get_current_partition.
  • move argparse module loading to the parse_args function that only gets executed when the module is run in script mode (not library mode)

This changes may not be the best way of handling testing but for now it allow us to move forward without digging a hole too deep:

[san@jones lime-packages]$ git diff
diff --git a/packages/safe-upgrade/files/usr/sbin/safe-upgrade b/packages/safe-upgrade/files/usr/sbin/safe-upgrade
index 8aeece4..fc6d467 100755
--- a/packages/safe-upgrade/files/usr/sbin/safe-upgrade
+++ b/packages/safe-upgrade/files/usr/sbin/safe-upgrade
@@ -17,7 +17,6 @@
 ]]--

 local io = require "io"
-local argparse = require 'argparse'

 local version = '1.0'
 local firmware_size_bytes = 7936*1024
@@ -114,10 +113,15 @@ function get_current_cmdline()
     return data
 end

-function get_current_partition()
+function get_proc_mtd()
     local handle = io.open('/proc/mtd', 'r')
     local data = handle:read("*all")
     handle:close()
+    return data
+end
+
+function get_current_partition(proc_mtd)
+    local data = proc_mtd or get_proc_mtd()
     if data:find("fw2") == nil then
         return 2
     else
@@ -289,6 +293,7 @@ end


 function parse_args()
+    local argparse = require 'argparse'
     local parser = argparse('safe-upgrade', 'Safe upgrade mechanism for dual-boot systems')
     parser:command_target('command')
     local show = parser:command('show', 'Show the status of the system partitions.')
@@ -338,6 +343,9 @@ end
 -- detect if this module is run as a library or as a script
 if pcall(debug.getlocal, 4, 1) then
     -- Library mode
+    local safe_upgrade = {}
+    safe_upgrade.get_current_partition = get_current_partition
+    return safe_upgrade
 else
     -- Main script mode

To run the test inside the docker container we have to add the module under test to the LUA_PATH (check that it is an executable module that does not ends with .lua so the expresion is ? instead of ?.lua):

(docker) [san@jones lime-packages]$ LUA_PATH="packages/safe-upgrade/files/usr/sbin/?;;" busted packages/safe-upgrade/tests/test_safe_upgrade.lua
●
1 success / 0 failures / 0 errors / 0 pending : 0.001127 seconds

Sum up

  • We did an evaluation of testing libraries and shrinked our selection to busted or luaunit. We are selecting busted as it has more pros than luaunit, mainly integrated mocking and coverage.
  • Some architectural options were proposed as starting point. We discussed them with @nicopace and we will be moving forward iterating with the Tests inside each module and a shared tests directory to test integrations idea.
  • Running tests in a local OpenWrt based device was investigated (thanks @nicopace!)
  • A working Dockerfile is proposed.
  • I did a real world example of unit testing a single function of a simple module. Little but working 🙂

GSoC 2019 – Load-correlated distributed bandwidth analysis for LibreMesh networks

Introduction

Performance tests are key for identifying the bottlenecks and optimize the network topology.
The main indicator is the bandwidth, but also other values can be useful like latency, number of active users for each node, load average and RAM consumption of each node.
The value of these quantities can vary greatly between the peak time and the night time, for this reason some of the measurements should be carried on in both these two moments.
Some other measurements, which can affect the user experience, could instead be run just in the night time.
To identify the night time we can’t relay on the router’s internal clock which could be years away from the actual time.
So a method for getting the network-wide peak time will be sought.
Each router in the network should separately run these tests, and for avoiding to influence each others’ results they should run at different times.
This synchronization should be possible taking advantage of the LibreMesh architecture and the shared-state service.

About me

Here’s Ilario, I studied organic chemistry in Pisa, Italy and I’m currently in a PhD on perovskite solar cells in Tarragona, Catalonia, Spain.
During the university I contributed to the mesh network eigenNet, part of the Italian community network consortium Ninux.
I started the (nowadays stalled) NinuxVerona community and once in Spain I started actively contributing to GuifiCamp and LibreMesh.

Setup of develop environment and initial interactions with LibreMesh community

After proposing a fix, I managed to build the LibreMesh firmware at its current stable release (17.06) using lime-sdk.
Then I built the latest LibreMesh code on top of forked OpenWrt 18.06 buildroot as suggested by the mentors; at first this was not possible on Arch Linux but after contacting with the community they updated the forked OpenWrt repository and it worked, thanks!
Finally, in order to be able to have the most updated OpenWrt code available, I compiled the latest LibreMesh code on top of the trunk (master branch, the unstable version) of OpenWrt buildroot, this was possible after adapting some configuration to the latest OpenWrt.
For pushing my code I forked lime-packages repository and created a gsoc2019 branch which can be accessed here.
Additionally, in case modifications to OpenWrt 18.06 core were needed, I will push them here.
All the buildroot-based compilation methods are already setup with the new branch as a feed, while the possibility of a back port to the stable LibreMesh 17.06 release will be evaluated once the project is completed.

Objectives

  • Flash with LibreMesh 4+ routers (preferably different models with different performances, if needed buy some) and setup a test network;
  • define a set of information to collect, divide it in network safe (e.g. number of clients)/network intensive (e.g. bandwidth test) and understand how to collect this data;
  • understand how a Prometheus exporter works and develop one in lua for the “network safe” quantities;
  • choose a reasonable “network safe” quantity for identifying the usage peak of the whole network (e.g. number of clients);
  • develop a script that locally identifies the peak and the night time;
  • develop the scripts for the network intensive tests, these should also store on the flash memory the results;
  • discuss with the mentors if the previous logs can be overwritten or if they should accumulate on the router for a certain period of time, in the latter case implement it;
  • implement a strategy for avoiding network intensive tests on different routers to happen at the same time;
  • if for achieving this last point a synchronization of the routers’ clocks is unavoidable, find a converging way for doing so or an available tool which does not require internet access (no NTP);
  • write a small Prometheus exporter for serving the last peak and off-peak network intensive tests results;
  • write the init service;
  • create a Makefile for the package;
  • test in a real-world community;
  • adapt the code written for LibreMesh trunk version to run also on LibreMesh 17.06 release;
  • adapt the code to plain OpenWrt, evaluate needed dependencies, if possible push the created package to upstream repository.

GSoC 2019 – Unit testing LibreMesh

Project introduction

LibreMesh as an embedded Operating System depends a lot on the underlying hardware. But, there are some parts of the code that don’t have that dependency, neither they depend on the network, or any particular state that the device could have. Also, there are many other cases were the states that one would like to achieve in order to reproduce a situation are complex or impractical to reproduce with hardware. In this project I will integrate a testing and mocking framework to LibreMesh and provide the functionality needed to easily write new tests for actual or new code. Also I will add tests for the core functions of LibreMesh.

Motivation

Unit testing the LibreMesh code-base will greatly help on approaching this two situations, and help having a much more robust solution for the communities it serves. Having automated unit testing integration test may improve the quality, the development speed, and shorten the release cycles of the LibreMesh software.

Also, having tests that safeguard the core functionality may allow new developers to engage with the code-base with more confidence.

For some developers (like me) having the option of doing test driven development greatly enhance the development experience. For reviewers it is also easier to understand and maintain code that has unit tests.

About me

I am Santiago Piccinini an Electronic Engineering student of University of Buenos Aires, Argentina. In my studies I focused on wireless, communication protocols, signal analysis, electronics prototyping and software development. I am currently finishing my master thesis.

I have been involved in different projects related to Community Networks, lately focused in the LibreRouter project.

This is my first GSoC, I wanted to participate for a long time!

Deliverables

  • Integrate unit testing and mocking framework to LibreMesh, that would allow the code-base to be tested with the specific lua version of the target system
  • Integrate tests with existing testing infrastructure (Travis CI)
  • Incorporate coverage reports and increase the coverage level of LibreMesh code
  • Proofread report of LibreMesh code testability, particularly what needs to be mocked of the code in order to be tested, and a rough idea of the complexity of the refactor needed.
  • At least one pull request of a refactor task for different levels of complexity as examples for the community to follow.
  • Try to write mocks for common functionality, for example:
    • Iwinfo
    • Nixio.fs, etc
    • Uci, lime.config
  • Write tests for some core functionality of lime-system package.
  • Refactor some LibreMesh code if needed for easy testing.
  • Add a device emulation module that provides specific mocking of device details (iwinfo, /sys/class/iee80211/, etc) to allow writing integration tests.

Next steps

I will start doing research about open-source Lua unit testing and mocking frameworks starting from this blog post https://blog.freifunk.net/2017/06/29/tdd-unit-testing-lua-openwrtlede-case/ from @nicopace. Next I will discuss with mentors about pros/cons of each framework and will propose a reasonable architectural solution to integrate this framework with some testing code as example and discuss with LibreMesh developers. If everything goes smooth then I will integrate it with Travis CI.

GSoC 2018 – Ground Routing in LimeApp – 1st update

Overview

In this past month I was working on the update of the lime-app dependencies (it was quite outdated). I also worked on the view and the ubus module that reads and saves ground routing settings in the LiMe config file.

The view: (Github LimeApp branch)

It is the minimum configuration of a plugin for lime-app. It has defined the constants, the store, actions (set and get) and basic epics to obtain the data using uhttp-mod-ubus.

Lime-app uses Preact for rendering the views, redux for state management and rxjs-observable as middleware for asynchronous events. For now you only get the setting as a json and expose it to the user.

Ubus (Github lime-package-ui branch)

Create the lime-groundrouting package that exposes and sets the graound rotuing configuration to lime. For the time being, just expose the settings.

ubus call lime-groundrouting get

To do this I use the LUA library lime.config.

Next step: Save changes.

In the coming weeks I will mount the form and the validation scheme in both the app and the ubus module.

Ground Routing GUI in LiMe App

Introduction of the project

There are many cases in which equipment attached to libremesh routers is used to make point-to-point links. This is known as ground routing. There is a way to configure routers to work through these links in a completely normal way, without splitting the experience of the mesh. Currently, it requires intervention by using uci commands or by typing the file configuration file directly. The idea is to build a simple configuration screen for LiMe app, which allows users to make use of this feature without major complications.

About me

My name is Marcos Gutierrez, I’m from Argentina and I’ve been working with the Libremesh team for two years. I am a user and member of QuintanaLibre, a mesh network where LibreMesh is used.

Project requirements

  • Build a ubus module that writes and tests the ground routing settings in lime.
  • Build a user interface within the LiMe app to configure settings and display alert messages if necessary.
  • Deploy a use case and validate that it works in the real world.
  • Bonus: If there is time left to generate a LuCi view with the same capabilities.