Coverage report integration. Final coverage output
CI integration with Travis. The build was split in two stages: unit testing and package build:
I have writen tests for the following core parts: lime.config, lime.network, lime.wireless, lime.utils. Quality of the tests is diverse, some are just stubs so we can improve them in the future, but some are good.
Tests for packages: firstbootwizard has been improved in order to support unit testing and a first simple test is in place. To write more tests more changes to FBW are needed.
Integration tests: lime-config with device support
iwinfo fake library, with many helper functions to easily fake a device and station connected, etc.
Uci testing environment helpers
Device support: A simple device support was implemented. For the moment this needs the /etc/board.json of the device and the /etc/config/network and /etc/config/wireless that are generated by OpenWRT on the first boot. With this files a testing environment is created using uci and iwinfo so for the tests a device is emulated. Using this infrastructure a lime-config test was implemented. For the moment only LibreRouter-v1 device is supported but it is very easy to add more devices.
Here is a reference PR with all the work I did for this GSoC. In order to have this work merged I created many small PR in the LibreMesh repository: #562, #563, #564, #565, #566, #567 and #568. Some of the work is not yet in PR to LibreMesh to don’t overwhelm the reviewers.
Future work
Add more devices.
Discuss if writing an integration test that uses lime-defaults and lime-defaults-factory with a device and check that the result is what it is expected is helpful, and if it is, write this tests.
Provide a way to test packages that use ubus library.
Explore how to use this testing environment in other openwrt Lua packages outside of LibreMesh. Even C code should be easily tested with automatic Lua bindings.
Lessons learned
Unit testing framework
After working with Busted I think it has been an excellent choice choice as unit testing framework. It is very well documented, very powerful and at the same time is easy to use. I used it for writing very different tests and I never missed something. Mocks and stubs are good and asserts are powerful.
At first my idea was to create a fake library because I thought that this could be easy and at the same time very handy for the tests. I implemented quickly a fake library but it did not behave the same as the uci library in many corner cases. I realized that behaving exactly the same will take a lot of work and if it does not behave exactly the same it will be very annoying because the tests will work differently than in production. And that is a very bad idea.
So I decided to try to use the real UCI library and create a clean environment for each test with helper functions. It was very easy to do it as UCI provides a way to change the config environment.
Docker image
A side effect of basing the testing Docker image in Alpine Linux is that it is ABI compatible with OpenWrt x86_64 packages because both use musl C library. This allow us to easily use some openwrt libraries like luci.ip, uci, etc directly from public OpenWrt packages. This keeps the testing maintenance effort low as we are not having to build this libraries by ourselves.
Lua is powerful
Coming from a Python background I thought I will miss many things but from a language perspective that was not the case!
In order to run the tests and have a reliable environment without the it’s working on my computer syndrome I created a simple and very small Docker image. This image is based on an image with Lua 5.1 and luarocks made by abaez. And then it just installs busted and luacov frameworks and bash.
FROM abaez/luarocks:lua5.1
WORKDIR /root
RUN luarocks install luacov; \
luarocks install busted
# TODO: move into a development dockerfile
RUN apk add --no-cache bash bash-completion
Nixio library
It would be good to have nixio available inside the docker image because this library is widely used in libremesh and also it could be very handy to have it available for the testing also.
I did an effort to add it to the image but many problems arised. The luarocks version of nixio 0.3-1 it is not working mainly because some compilation issues with newer versions of gcc. So I tried to work on a rockspec without this problem but I could not finish it because other problems arised I think that related to the Alpine/musl distribution and libc/linux headers. I will try help the author of nixio to publish a new version to luarocks as this will benefit others too.
Testing shell environment
To provide an easy way to develop or test things within the docker image I created a tool that opens a bash shell inside the docker image that has some features that allows easy development:
/home/$USER is mounted inside the docker image so each change you do to the code from inside is maintained when you close the docker container
the same applies to /tmp
you have the same user outside and inside
network access is garanted
and some goodies like bashrc, some useful ENV variables, PS1 modification, etc.
You can see that the prompt is changed adding (docker) in the left part so you can easily remember that you are inside the docker container.
This environment is also used by run_tests script.
Running the tests: run_tests bash script
This script is what you should be running each time you want to run the tests. As you can see in the image we currently have 19 tests and all are passing 🙂
For the sake of showing you what to expect when a test fails I modified a test condition to be false and here is the output:
now 18 tests are good, and there is one failure. The assertion that is failing is on line 11 and the test is Fake uci tests test simple get and set. Also it is shown that the expected result is the number 2 but the actual result is the number 1.
As it is expected, run_tests returns with 0 when all tests pass and != 0 when there is at least one failure.
The script in detail
The idea behind this script is simple:
sets the search path of the tests for buster (the unittesting framework)
sets the lua library paths, prepending the fake library and adding the paths to the libremesh packages with packages/lime-system/files/usr/lib/lua/?.lua. This doesn’t work automaticaly for every package if the paths does not use the files/path/to/final/destination. So if you want to test some package without the files convention maybe it would be good to move the package to this convention. Also it does not work if the lua module we want to test does not finish with .lua, in this case the path must be explicitly added (I wrote about this in a previous blog post).
runs the tests using the dockertestshell
run_tests also passes the first argument as an argument to busted so you can do things like this:
[san@page lime-packages]$ ./run_tests "--list --verbose"
packages/lime-system/tests/test_lime_config.lua:11: LiMe Config tests test empty get
packages/lime-system/tests/test_lime_config.lua:15: LiMe Config tests test simple get
packages/lime-system/tests/test_lime_config.lua:20: LiMe Config tests test get with fallback
packages/lime-system/tests/test_lime_config.lua:24: LiMe Config tests test get with lime-default
packages/lime-system/tests/test_lime_config.lua:30: LiMe Config tests test get precedence of fallback and lime-default
packages/lime-system/tests/test_lime_config.lua:36: LiMe Config tests test get with false value
packages/lime-system/tests/test_lime_config.lua:41: LiMe Config tests test get_bool
packages/lime-system/tests/test_lime_config.lua:54: LiMe Config tests test set
packages/lime-system/tests/test_lime_config.lua:64: LiMe Config tests test set nonstrings
packages/lime-system/tests/test_lime_config.lua:81: LiMe Config tests test get_all
packages/safe-upgrade/tests/test_safe_upgrade.lua:5: safe-upgrade tests test get current partition
tests/test_fake_uci.lua:4: Fake uci tests test simple get and set
tests/test_fake_uci.lua:14: Fake uci tests test multiple cursors
tests/test_fake_uci.lua:31: Fake uci tests test nested get and set
tests/test_fake_uci.lua:49: Fake uci tests test state not preserved between tests
tests/test_fake_uci.lua:54: Fake uci tests test save
tests/test_fake_uci.lua:59: Fake uci tests test delete
tests/test_fake_uci.lua:73: Fake uci tests test foreach
tests/test_fake_uci.lua:87: Fake uci tests test get_all
LibreMesh has already a github/Travis integration with two objetives:
test that the packages can be built (no Makefile errors, etc)
build and publish the packages of the master branch to an external server
The build pipeline of LibreMesh has been broken for a couple of months because the docker image that has been in use is not longer available. This is becaouse there is an ongoing effort by aparcar to create canonicaldocker images for OpenWrt.
So I did an atempt to fix the current LibreMesh build pipeline using the new infraestructure in this pull request. The build is still not passing but it seems it is something easy to fix as the build is passing but then the deploy is failing.
Travis unit testing
Beside fixing the current pipeline and to integrate the unittesting work I did a refactoring of the build steps to have a unittest stage and a build stage. To do this I installed the Github/Travis integration on my lime-packages fork. In the following image you can see that the two stages are green (tests are passing) 🙂
And here is the log of the unittests stage. You can see that it takes less than a minute to run the stage, with 15 seconds building the docker image and 0.011133 seconds to run the tests :100:
Next Steps
Now that the framework is in place and in continous integration we should be doing the following:
Add documentation on how to write tests
Integrate nixio in the docker image
Proofread the core LibreMesh code and inform about its testability
Provide some mocks for common functionality (uci already done!)
The first weeks of august I will move to Catalunya to work with a core developer of LibreMesh. So with my mentor NicoP we will adapt the schedule to take advantage of this.
Not covered by the last blog post is the work that I did in a fake/mock implementation of the libuci library in lua. This allows writing a lot of tests for LibreMesh as ucithe most used library in the codebase that make sense to write a mock. The implementation is very small but covers the most used functionality of libuci: cursor(), get(), set(), save(), delete() and foreach(). This was implemented doing TDD with the support of the unittesting framework.
During the upcoming weeks all this work will be properly released as a PR to the lime-packages repo accompanied by the Travis CI integration in a Docker container to do the tests in a contained environment, and more tests are going to follow 🙂
Must provide helpful assert test functions, like showing table diffs or formatters for outputs to understand the difference easily
We need mock functionality, because a lot of functions are hardware related and may not be possible to test them on hardware all the time.
It is desirable for the library to be a one-file import, so we can use it in the routers and in the continuous integration in the same way we do it in the desktop.
Options
We need to consider unit testing, mocking and coverage tests.
Unit Testing
There is a list of unittesting libraries in the lua package manager, luarocks:
https://luarocks.org/labels/test?non_root=on
It is being used in other OpenWRT based images like OpenWISP: https://github.com/openwisp/openwisp-config/blob/master/openwisp-config/tests/test_utils.lua
Downsides:
It doesn’t have mocking helpers: Could be combined with a one file mocking library mockagne
Telescope
They describe themselves as a A highly customizable test library for Lua that allows declarative tests with nested contexts.
URL: https://github.com/norman/telescope/
Last release 2013. Lua 5.1 last release was done in 2012, so it is not that big of a deal… but has not received any updates since, so it might have not evolved since.
Busted
URL:
https://github.com/Olivine-Labs/busted
http://olivinelabs.com/busted/
Upsides:
Very well maintained by olivinelabs and contributors,
very popular (900k downloads luarocks, 800 github stars).
It has setup/teardowns and also mocks, spies, and matchers.
Must be installed using luarocks (it is a lot of files). A question has been posted to luarocks to explore the possibility of creating bundles for a package (one file with all dependencies). That would simplify its use: https://github.com/luarocks/luarocks/issues/1023
Mocking libraries
lua-mock
URL: https://github.com/henry4k/lua-mock
Mach
URL: https://github.com/ryanplusplus/mach.lua/
More or less well maintain, though it is not so popular.
Coverage reports
luacov
URL: https://github.com/keplerproject/luacov
Upsides:
Well maintained
Unit testing Architecture for LibreMesh (only for LibreMesh?)
The idea is to allow unit-testing packages and also the integration between them as some of the packages depend on other packages.
Context:
LibreMesh enables functionality selecting which packages must be installed and changing enabling/disabling the exposed features in configuration files.
In some packages the code is all inside the executable lua file (not like a library)
Some packages are independent, provide functionality without depending on lime-system. This packages are in lime-packages for convenience.
Packages could (should) be migrated into OpenWrt repositories. This migration may happen steps and when migrated the code may be in an independent repository for the package iteself.
many packages are wrappers of bash code, and this complicates the tests as you need a running system to test it out
This context is not an easy one to test as it has a lot of trade offs!
Options
Single and global tests directory
The easiest architecture is to have a global tests directory and some utility functions that allow to “install” a certain module for testing
Example of a (integration) test that uses libraries and fake modules
test_lime_proto_anygw.lua:
utils = require("test.utils")
-- installs required modules in the lua path
utils.install_limesystem_module() -- to allow access to lime.network, etc
utils.install_module("packages/lime-proto-anygw/src/anygw.lua", "lime.proto.anygw")
utils.install_module("tests/fake_modules/nixio.fs", "nixio.fs")
-- now we can load the modules
anygw = require("lime.proto.anygw")
function test_foo()
assert anygw.foo() == 'bar'
end
Pros:
Easy to start with and to understand
Cons:
As all tests are together it is not easy to move a package to other repository or even to its own repository.
Tests inside each module and a shared tests directory to test integrations
Testing in a fully working image with all packages and libraries installed
Tests can be run installing all files of the packages (by some script that parses the Makefiles, or “by hand in a helper script”).
Pros:
It requires less boilerplate to test package integration
Libraries of the target system can be used directly
Other packages may be installed
Cons:
Less control over what it is really happening
slower than the other options, as it needs to load a full system
Fake modules as library
Testing executable modules
Executable lua modules can be tested with a simple modification in the file creating a main() function and then using something like:
function main()
--- the main code in here
end
-- detect if this module is run as a library or as a script
if pcall(debug.getlocal, 4, 1) then
-- Library mode, do nothing
else
-- Main script mode
main()
end
Then from a test file it can be loaded like any normal module and all the functions can be accesed without executing main()
Testing environment
A docker environment (or multiple, even using “qemu-user” under docker) with the testing libraries and target lua version is loaded by the “run_tests.sh” executable.
This environment can provide some useful libraries for testing (coverage reporting,
Direct import and testing can be done for unit tests were functions are not using system libraries, or when these are simple enough to be mockable (mocking shell() calls).
We did a trial to run busted inside a router… that would have been useful for in-router tests and also for tests inside a virtual environment.
It used the strategy of installing luarocks dependencies in a separate directory and copying them to the router.
The steps are pretty straightforward:
$ sudo apt install luarocks
$ luarocks install --tree lua_modules busted
$ cat <EOF
require 'busted.runner'()
describe('Busted unit testing framework', function()
describe('should be awesome', function()
it('should be easy to use', function()
assert.truthy('Yup.')
end)
it('should have lots of features', function()
-- deep check comparisons!
assert.same({ table = 'great'}, { table = 'great' })
-- or check by reference!
assert.is_not.equals({ table = 'great'}, { table = 'great'})
assert.falsy(nil)
assert.error(function() error('Wat') end)
end)
it('should provide some shortcuts to common functions', function()
assert.unique({{ thing = 1 }, { thing = 2 }, { thing = 3 }})
end)
it('should have mocks and spies for functional tests', function()
local thing = require('thing_module')
spy.spy_on(thing, 'greet')
thing.greet('Hi!')
assert.spy(thing.greet).was.called()
assert.spy(thing.greet).was.called_with('Hi!')
end)
end)
end)
EOF
> test.lua
$ cat <EOF
-- set_paths.lua
local version = _VERSION:match("%d+%.%d+")
package.path = 'lua_modules/share/lua/' .. version .. '/?.lua;lua_modules/share/lua/' .. version .. '/?/init.lua;' .. package.path
package.cpath = 'lua_modules/lib/lua/' .. version .. '/?.so;' .. package.cpath
EOF
> set_paths.lua
$ scp -r test.lua set_path.lua lua_modules root@thisnode.info:~
$ ssh root@thisnode.info 'lua -l set_paths test.lua'
The output of this command though was not what we expected:
$ ssh root@thisnode.info 'lua -l set_paths test.lua'
lua: lua_modules/share/lua/5.1/pl/path.lua:28: pl.path requires LuaFileSystem
stack traceback:
[C]: in function 'error'
lua_modules/share/lua/5.1/pl/path.lua:28: in main chunk
[C]: in function 'require'
lua_modules/share/lua/5.1/busted/runner.lua:3: in main chunk
[C]: in function 'require'
test.lua:1: in main chunk
[C]: ?
Deeper inspection showed that the library’s dependencies had C bindings that we compiled for a different arquitecture, so tha strategy was not feasable for routers anymore:
where term/core.so and system/core.so are system libs, but lfs.so is from LuaFileSystem.
There is a lua-only implementation of LuaFileSystem: https://github.com/sonoro1234/luafilesystem , but as it doesn’t support luarocks deeper understanding of the platform is needed to attempt to replace the only binary binding with this implementation.
Found a sister library from that one in luarocks: https://luarocks.org/modules/3scale/luafilesystem-ffi based on this repo: https://github.com/spacewander/luafilesystem. So:
but this library, as it depends on ffi (a module of luajit), it depends on a C extension too.
also, luafilesystem exists as a native library in OpenWRT, so it could be included just for the sake of the exercise: https://openwrt.org/packages/pkgdata/luafilesystem … but not this time.
Docker with LuaRocks and Lua 5.1
Some docker images already exist:
https://github.com/akornatskyy/docker-library/
https://hub.docker.com/r/abaez/luarocks/
https://hub.docker.com/r/abaez/lua
https://github.com/martijnrondeel/docker-luarocks
A simple Dockerfile whould be:
FROM abaez/luarocks:lua5.1
WORKDIR /root
RUN luarocks install luacov; \
luarocks install busted
Excellent blog post on handling Lua paths: http://www.thijsschreijer.nl/blog/?p=1025
Example of LUA_PATH to load executables (without ending in .lua): LUA_PATH="packages/safe-upgrade/files/usr/sbin/?;;". The double ;; at the end means append the default paths.
First attempt running tests
I selected safe-uprgade libremesh module to start doing unittests because I know the module as I wrote it so I already know which code would gain value being tested. Also I am confident to refactor the module if needed.
First I start using the busted unittest library with a simple test of the function get_current_partition() that must return the partition number that is currently running. As this is done from reading /proc/mtd I refactored the function so we can pass from the outside the expected content.
Content of lime-packages/safe-upgrade/tests/test_safe_upgrade.lua:
The modifications I did to do to the safe-upgrade module are:
refactor get_current_partition() into get_proc_mtd() and get_current_partition(proc_mtd). This way we can inject different /proc/mtd values for testing.
return a table containing the module exported functions when running in library mode. In this case we are exporting get_current_partition.
move argparse module loading to the parse_args function that only gets executed when the module is run in script mode (not library mode)
This changes may not be the best way of handling testing but for now it allow us to move forward without digging a hole too deep:
[san@jones lime-packages]$ git diff
diff --git a/packages/safe-upgrade/files/usr/sbin/safe-upgrade b/packages/safe-upgrade/files/usr/sbin/safe-upgrade
index 8aeece4..fc6d467 100755
--- a/packages/safe-upgrade/files/usr/sbin/safe-upgrade
+++ b/packages/safe-upgrade/files/usr/sbin/safe-upgrade
@@ -17,7 +17,6 @@
]]--
local io = require "io"
-local argparse = require 'argparse'
local version = '1.0'
local firmware_size_bytes = 7936*1024
@@ -114,10 +113,15 @@ function get_current_cmdline()
return data
end
-function get_current_partition()
+function get_proc_mtd()
local handle = io.open('/proc/mtd', 'r')
local data = handle:read("*all")
handle:close()
+ return data
+end
+
+function get_current_partition(proc_mtd)
+ local data = proc_mtd or get_proc_mtd()
if data:find("fw2") == nil then
return 2
else
@@ -289,6 +293,7 @@ end
function parse_args()
+ local argparse = require 'argparse'
local parser = argparse('safe-upgrade', 'Safe upgrade mechanism for dual-boot systems')
parser:command_target('command')
local show = parser:command('show', 'Show the status of the system partitions.')
@@ -338,6 +343,9 @@ end
-- detect if this module is run as a library or as a script
if pcall(debug.getlocal, 4, 1) then
-- Library mode
+ local safe_upgrade = {}
+ safe_upgrade.get_current_partition = get_current_partition
+ return safe_upgrade
else
-- Main script mode
To run the test inside the docker container we have to add the module under test to the LUA_PATH (check that it is an executable module that does not ends with .lua so the expresion is ? instead of ?.lua):
We did an evaluation of testing libraries and shrinked our selection to busted or luaunit. We are selecting busted as it has more pros than luaunit, mainly integrated mocking and coverage.
Some architectural options were proposed as starting point. We discussed them with @nicopace and we will be moving forward iterating with the Tests inside each module and a shared tests directory to test integrations idea.
Running tests in a local OpenWrt based device was investigated (thanks @nicopace!)
A working Dockerfile is proposed.
I did a real world example of unit testing a single function of a simple module. Little but working 🙂
LibreMesh as an embedded Operating System depends a lot on the underlying hardware. But, there are some parts of the code that don’t have that dependency, neither they depend on the network, or any particular state that the device could have. Also, there are many other cases were the states that one would like to achieve in order to reproduce a situation are complex or impractical to reproduce with hardware. In this project I will integrate a testing and mocking framework to LibreMesh and provide the functionality needed to easily write new tests for actual or new code. Also I will add tests for the core functions of LibreMesh.
Motivation
Unit testing the LibreMesh code-base will greatly help on approaching this two situations, and help having a much more robust solution for the communities it serves. Having automated unit testing integration test may improve the quality, the development speed, and shorten the release cycles of the LibreMesh software.
Also, having tests that safeguard the core functionality may allow new developers to engage with the code-base with more confidence.
For some developers (like me) having the option of doing test driven development greatly enhance the development experience. For reviewers it is also easier to understand and maintain code that has unit tests.
About me
I am Santiago Piccinini an Electronic Engineering student of University of Buenos Aires, Argentina. In my studies I focused on wireless, communication protocols, signal analysis, electronics prototyping and software development. I am currently finishing my master thesis.
I have been involved in different projects related to Community Networks, lately focused in the LibreRouter project.
This is my first GSoC, I wanted to participate for a long time!
Deliverables
Integrate unit testing and mocking framework to LibreMesh, that would allow the code-base to be tested with the specific lua version of the target system
Integrate tests with existing testing infrastructure (Travis CI)
Incorporate coverage reports and increase the coverage level of LibreMesh code
Proofread report of LibreMesh code testability, particularly what needs to be mocked of the code in order to be tested, and a rough idea of the complexity of the refactor needed.
At least one pull request of a refactor task for different levels of complexity as examples for the community to follow.
Try to write mocks for common functionality, for example:
Iwinfo
Nixio.fs, etc
Uci, lime.config
Write tests for some core functionality of lime-system package.
Refactor some LibreMesh code if needed for easy testing.
Add a device emulation module that provides specific mocking of device details (iwinfo, /sys/class/iee80211/, etc) to allow writing integration tests.
Next steps
I will start doing research about open-source Lua unit testing and mocking frameworks starting from this blog post https://blog.freifunk.net/2017/06/29/tdd-unit-testing-lua-openwrtlede-case/ from @nicopace. Next I will discuss with mentors about pros/cons of each framework and will propose a reasonable architectural solution to integrate this framework with some testing code as example and discuss with LibreMesh developers. If everything goes smooth then I will integrate it with Travis CI.
