I am very happy to have participated in this GSoC. I’ve learned many things and I’ve been able to implement stuff that I hope it is useful.
Things that have been achieved
- Busted unit testing framework integration
- Coverage report integration. Final coverage output
CI integration with Travis. The build was split in two stages: unit testing and package build:
- I have writen tests for the following core parts: lime.config, lime.network, lime.wireless, lime.utils. Quality of the tests is diverse, some are just stubs so we can improve them in the future, but some are good.
- Tests for packages: firstbootwizard has been improved in order to support unit testing and a first simple test is in place. To write more tests more changes to FBW are needed.
- Integration tests: lime-config with device support
- iwinfo fake library, with many helper functions to easily fake a device and station connected, etc.
- Uci testing environment helpers
- Device support: A simple device support was implemented. For the moment this needs the
/etc/board.jsonof the device and the/etc/config/networkand/etc/config/wirelessthat are generated by OpenWRT on the first boot. With this files a testing environment is created usinguciandiwinfoso for the tests a device is emulated. Using this infrastructure alime-configtest was implemented. For the moment only LibreRouter-v1 device is supported but it is very easy to add more devices.
Here is a reference PR with all the work I did for this GSoC. In order to have this work merged I created many small PR in the LibreMesh repository: #562, #563, #564, #565, #566, #567 and #568. Some of the work is not yet in PR to LibreMesh to don’t overwhelm the reviewers.
Future work
- Add more devices.
- Discuss if writing an integration test that uses
lime-defaultsandlime-defaults-factorywith a device and check that the result is what it is expected is helpful, and if it is, write this tests. - Provide a way to test packages that use
ubuslibrary. - Explore how to use this testing environment in other openwrt Lua packages outside of LibreMesh. Even C code should be easily tested with automatic Lua bindings.
Lessons learned
Unit testing framework
After working with Busted I think it has been an excellent choice choice as unit testing framework. It is very well documented, very powerful and at the same time is easy to use. I used it for writing very different tests and I never missed something. Mocks and stubs are good and asserts are powerful.
OpenWrt Unified Configuration Interface (UCI) library
At first my idea was to create a fake library because I thought that this could be easy and at the same time very handy for the tests. I implemented quickly a fake library but it did not behave the same as the uci library in many corner cases. I realized that behaving exactly the same will take a lot of work and if it does not behave exactly the same it will be very annoying because the tests will work differently than in production. And that is a very bad idea.
So I decided to try to use the real UCI library and create a clean environment for each test with helper functions. It was very easy to do it as UCI provides a way to change the config environment.
Docker image
A side effect of basing the testing Docker image in Alpine Linux is that it is ABI compatible with OpenWrt x86_64 packages because both use musl C library. This allow us to easily use some openwrt libraries like luci.ip, uci, etc directly from public OpenWrt packages. This keeps the testing maintenance effort low as we are not having to build this libraries by ourselves.
Lua is powerful
Coming from a Python background I thought I will miss many things but from a language perspective that was not the case!
Links to older posts
First post, second post, third post, fourth post.
Beside writing comments to this blog you can also write me to spiccinini _ altermundi _ net if you want to discuss anything.
Best!