Unittesting LibreMesh GSoC mid-term update

A few weeks passed and I want to share the progress of the project and what are the next steps 🙂

Unit testing tools and ecosystem

As one of the goals is that it must be easy for developers to write, modify and run the tests I created some simple tools to do this:

• testing image -> Dockerfiles/Dockerfile.unittests
• testing shell environment -> tools/dockertestshell
• running the tests -> ./run_tests script

Testing image

In order to run the tests and have a reliable environment without the it’s working on my computer syndrome I created a simple and very small Docker image. This image is based on an image with Lua 5.1 and luarocks made by abaez. And then it just installs busted and luacov frameworks and bash.

FROM abaez/luarocks:lua5.1

WORKDIR /root

RUN luarocks install luacov; \
    luarocks install busted

# TODO: move into a development dockerfile
RUN apk add --no-cache bash bash-completion

Nixio library

It would be good to have nixio available inside the docker image because this library is widely used in libremesh and also it could be very handy to have it available for the testing also.

I did an effort to add it to the image but many problems arised. The luarocks version of nixio 0.3-1 it is not working mainly because some compilation issues with newer versions of gcc. So I tried to work on a rockspec without this problem but I could not finish it because other problems arised I think that related to the Alpine/musl distribution and libc/linux headers. I will try help the author of nixio to publish a new version to luarocks as this will benefit others too.

Testing shell environment

To provide an easy way to develop or test things within the docker image I created a tool that opens a bash shell inside the docker image that has some features that allows easy development:

• /home/$USER is mounted inside the docker image so each change you do to the code from inside is maintained when you close the docker container
• the same applies to /tmp
• you have the same user outside and inside
• network access is garanted
• and some goodies like bashrc, some useful ENV variables, PS1 modification, etc.

To enter the shell environment run:

[san@page lime-packages]$ ./tools/dockertestshell 
(docker) [san@page lime-packages]$

You can see that the prompt is changed adding (docker) in the left part so you can easily remember that you are inside the docker container.

This environment is also used by run_tests script.

Running the tests: run_tests bash script

This script is what you should be running each time you want to run the tests. As you can see in the image we currently have 19 tests and all are passing 🙂

For the sake of showing you what to expect when a test fails I modified a test condition to be false and here is the output:

now 18 tests are good, and there is one failure. The assertion that is failing is on line 11 and the test is Fake uci tests test simple get and set. Also it is shown that the expected result is the number 2 but the actual result is the number 1.

As it is expected, run_tests returns with 0 when all tests pass and != 0 when there is at least one failure.

The script in detail

The idea behind this script is simple:

• sets the search path of the tests for buster (the unittesting framework)
• sets the lua library paths, prepending the fake library and adding the paths to the libremesh packages with packages/lime-system/files/usr/lib/lua/?.lua. This doesn’t work automaticaly for every package if the paths does not use the files/path/to/final/destination. So if you want to test some package without the files convention maybe it would be good to move the package to this convention. Also it does not work if the lua module we want to test does not finish with .lua, in this case the path must be explicitly added (I wrote about this in a previous blog post).
• runs the tests using the dockertestshell

run_tests also passes the first argument as an argument to busted so you can do things like this:

[san@page lime-packages]$ ./run_tests "--list --verbose"
packages/lime-system/tests/test_lime_config.lua:11: LiMe Config tests test empty get
packages/lime-system/tests/test_lime_config.lua:15: LiMe Config tests test simple get
packages/lime-system/tests/test_lime_config.lua:20: LiMe Config tests test get with fallback
packages/lime-system/tests/test_lime_config.lua:24: LiMe Config tests test get with lime-default
packages/lime-system/tests/test_lime_config.lua:30: LiMe Config tests test get precedence of fallback and lime-default
packages/lime-system/tests/test_lime_config.lua:36: LiMe Config tests test get with false value
packages/lime-system/tests/test_lime_config.lua:41: LiMe Config tests test get_bool
packages/lime-system/tests/test_lime_config.lua:54: LiMe Config tests test set
packages/lime-system/tests/test_lime_config.lua:64: LiMe Config tests test set nonstrings
packages/lime-system/tests/test_lime_config.lua:81: LiMe Config tests test get_all
packages/safe-upgrade/tests/test_safe_upgrade.lua:5: safe-upgrade tests test get current partition
tests/test_fake_uci.lua:4: Fake uci tests test simple get and set
tests/test_fake_uci.lua:14: Fake uci tests test multiple cursors
tests/test_fake_uci.lua:31: Fake uci tests test nested get and set
tests/test_fake_uci.lua:49: Fake uci tests test state not preserved between tests
tests/test_fake_uci.lua:54: Fake uci tests test save
tests/test_fake_uci.lua:59: Fake uci tests test delete
tests/test_fake_uci.lua:73: Fake uci tests test foreach
tests/test_fake_uci.lua:87: Fake uci tests test get_all

Here is the code:

$ cat run_tests 
#!/bin/bash

TESTS_PATHS='packages/*/tests/test*.lua  tests/test*.lua'
LIB_PATHS='tests/fakes/?.lua;packages/lime-system/files/usr/lib/lua/?.lua;packages/safe-upgrade/files/usr/sbin/?;;'

./tools/dockertestshell "busted -v ${TESTS_PATHS} --lpath='${LIB_PATHS}'" ${1}

Integration of unittests with Travis CI

LibreMesh has already a github/Travis integration with two objetives:

• test that the packages can be built (no Makefile errors, etc)
• build and publish the packages of the master branch to an external server

The build pipeline of LibreMesh has been broken for a couple of months because the docker image that has been in use is not longer available. This is becaouse there is an ongoing effort by aparcar to create canonical docker images for OpenWrt.
So I did an atempt to fix the current LibreMesh build pipeline using the new infraestructure in this pull request. The build is still not passing but it seems it is something easy to fix as the build is passing but then the deploy is failing.

Travis unit testing

Beside fixing the current pipeline and to integrate the unittesting work I did a refactoring of the build steps to have a unittest stage and a build stage. To do this I installed the Github/Travis integration on my lime-packages fork. In the following image you can see that the two stages are green (tests are passing) 🙂

And here is the log of the unittests stage. You can see that it takes less than a minute to run the stage, with 15 seconds building the docker image and 0.011133 seconds to run the tests :100:

Next Steps

Now that the framework is in place and in continous integration we should be doing the following:

• Add documentation on how to write tests
• Integrate nixio in the docker image
• Proofread the core LibreMesh code and inform about its testability
• Provide some mocks for common functionality (uci already done!)

The first weeks of august I will move to Catalunya to work with a core developer of LibreMesh. So with my mentor NicoP we will adapt the schedule to take advantage of this.