OpenWifi – GSoC 2017 final report

Hello everyone!
First things first so here is the code that I’ve written. You can find it in these repositories:
OpenWifiCore (core server application)
OpenWifiFeed (LEDE/OpenWRT feed with boot flasher and boot notifier)
OpenWifiWeb (web frontend)
OpenWifiTemplates (old templating system)
OpenWifiLocation (plugin that detects the location of a node via google location api and nearby wifi aps)

I also worked together with Arne to interact with his SDWN controller and agent. We created a small website that informs you on how to use the two tools together. (As I’m writing this it is still somewhat under construction. But I hope everything will be there soon ­čÖé ) Furthermore there is also specific documentation (WIP) for OpenWifi here.

To be a little more precise here is the list of commits that have been done:

May

OpenWifiCore
OpenWifiFeed
OpenWifiWeb

June

OpenWifiCore
OpenWifiWeb
OpenWifiLocation

July

OpenWifiCore
OpenWifiWeb
OpenWifiFeed

August

OpenWifiCore
OpenWifiFeed
OpenWifiWeb
OpenWifiTemplates

Overview about what has been done

Everything that has been done can be put into 4 categories: authentication/authorization, api, database-model and infrastructure. I want to give you a brief overview of all these categories.

Infrastructure

Docker images are now build automatically via TravisCI and deployed on docker hub. You can find more information in my first evaluation blog post. How to use the docker images is described in the documentation. These images are also used for testing.

database-model

The graph based database model was made a first class citizen. It now automatically converts between the internal representation and the representation needed to sync to the AP. It is also now possible to create new configurations and create links between these. The query format is also used for authentication/authorization.

api

Providing all the new functionality via a REST style API was the focus of this google summer of code (web views are still needed for quite some things). It is used for managing users, services, nodes and changing the graph based configuration model. It is described in the documentation.

The concept of having a service is also something new that I created together with Arne in august. If an external application needs to make changes to configurations of specific nodes it can just subscribe as a service. The service takes a list of database queries, a name and a shell script with a compare string. If the output (stdout) of the shell scripts matches the compare value the node gets the name of the service as it’s capability (also something that was added during this GSoC). When a node has the capability the queries are applied to it’s config in a regular interval by the job-server.

authentication/authorization

This was the biggest part of this GSoC. There are two ways of giving access to a node – either by giving access to a path string of a configuration or by allowing a specific database query. For more details see my second evaluation blog post.

There were quite some challenges as this system allows for some really complex access configurations and there are still some things to improve here. (see below)

What else? Aka the smaller bits

There were also some smaller bits that were done that don’t really fit the other categories. The boot-notifier and boot-flasher scripts were improved, a way to abstract communication with the nodes was started (see documentation), some small UI fixes to the graph view were done. And probably a lot of other smaller fixes I forgot ­čśë

Future – or what needs to be done

I guess the most important thing is to get all of these new features exposed via the web views. That shouldn’t be to hard but wasn’t the highest priority during GSoC. Everything related to the graph should be made aware of all possible path strings that lead to a configuration. Currently just one string is used – this should be lists! The fine grained authorization needs some more testing and I also want to improve the pattern matching by combining the regular expressions you could use to describe a path string. This could be done with greenery. The authorization right now is also just focused on nodes and configuration – it would be nice to restrict access to some views and actions. For example restrict the access to LUCI, sshkeys or executing commands on a node. Furthermore there should be a per node option where “the truth” of a config lies (like if there is a difference between the actual node configuration and the configuration on the server which one is considered the one to go with) – than it would also be nice to disable the sync for some nodes (if manual changes on the node need to be made for example).make

Acknowledgments

A big thanks goes out to Google for organizing something cool as the Google Summer of Code, to Freifunk for letting me do this project with them and last but not least Julius for being my mentor!

OpenWifi status report before 2nd evaluation

Hello everyone,
this is the second milestone blog post about OpenWifi. In this blog post I want to talk a little more about the authorization in detail and give an update on what has been done in the last weeks.

Authentication

Authentication was enhanced with possibility to use a client side certificate or apikey (which can be send to the server with a HTTP-GET like ?key=MY_API_KEY). It uses groups to distinguish between the authentication methods and if you login with username/password it is also possible to mark a user as an admin.

Username/password login uses cookies and client side certificates have to validated by a proxy (like nginx for example) and added to the header. It is you can find the code for authentication and groups.

Authorization

The basic principle of authorization in pyramid is a ACL based authorization that gets principals (basically strings) from authentication. These ACLs are given by so called factories (if a view doesn’t use a custom factory the so called root factory is used).

I decided that it would make sense to have two strategies for authorization – coarse grained and fine grained. I’ll talk about the two in the next part.

Coarse grained authorization

Coarse grained authorization is based on specific views and access to a node. Access to specific views is managed by the root factory which is just a simple ACL that stores which groups have access to which views (like only admins can manage users and access objects, admins and client side certificate can add nodes).

Access to nodes is managed via access objects which are attached to a user or apikey and also attached to a list of nodes. (There is also a special setting that gives access to all nodes) To protect views that operate on a single node (like settings, status, etc.) there is the node context factory which generates an ACL that requests access to the specific node. (a user/apikey is given principals that match their accessible nodes)

To protect views that list nodes there is the get_nodes function that returns all nodes accessible by a user.

It might be a good idea to extend the node access that it can define what can be done with the node (to allow/disable luci access or ssh key access for example).

Fine grained authorization

Fine grained authorization is used in conjunction with the graph-based DB model. It is still WIP and I use a separate branch for it.This uses the data portion of the access object. Since access to the model is given as a rest API provided by cornice I’m using cornice validators give access. Since it shall be possible to share the same master configuration across different nodes it has to determine what are the maximum access for all nodes using the config for this user is.

Access can be granted in two ways – either by specifying a “path string” and an access (like rw, ro or none) or by just a allowing one specific query to the db. A path string has the following format: ConfigName (ConfigType).LinkName.Config2Name (Config2Type) etc. I want that it is possible to use regular expression for matching these names.

To make things easier I started by just comparing the levels of the “path string” to see if they refer to the same string when trying to figure out the maximum rights for a given node set.

Services

Julius, Arne and me started thinking if it is a good a idea to make it a little bit easier for external software to just update nodes with specific configuration options. So that this software doesn’t need to use queries for all the options it wants to modify. (And therefor use polling to update new nodes etc.) After thinking a bit about it I decided it would be a good idea to add something like services a service can register to. This service contains a shell-script and a compare value to determine if the node is desired to be modified by this service and also a list of queries that should be applied to this config.

Arne is using this to set a ONOS Server for the nodes via OpenWifi.

Discovery Script

The discovery script was updated to use client side certificates and advertise if the node is able to communicate via TLS.

Graph-Based DB Model

The GUI for the graph based DB model was updated a bit – now it is sorted with a different algorithm (based on the direction) and also displays information about the vertices (there for the is a small rest API for the vertices).

Also a lot of the code was refactored which will make it easier to add new features here.

Miscellaneous

It was evaluated how a github page could be used to add more documentation for OpenWifi. Some small things in the docker image were fixed as well. (for example discovery not working with nginx image or wrong file system rights)

What is going to be done in the next weeks

Fine grained access proved to be more complicated than I thought. But I want to finish it soon and start working on the communication abstraction. I also hope that there is enough time to improve the graph based db model (since the code is now refactored it should be not to hard).

OpenWifi status report before 1st evaluation

Hello everyone,
this is the status report of the OpenWifi project before the first evaluation. I like to point out in this blog post what has been done and how I like to proceed the next weeks. Looking at my initial schedule I did some things that are not on that list and missed some others. So I also like to revamp the schedule. I did a Google-document for that and I’m happy to receive comments about it ­čÖé

I think a lot of things that have been done are the foundation for everything else to come and a lot of things that have not been done are low hanging fruits.

I think overall my initial schedule contained too many elements and I need to focus more on what is important and what are optional changes. I think I want to focus on testing/docker/https and authentication/authorization until the next evaluation and do the communication plugin-API from then until the end. Everything GUI-related might happen after GSoC.

What has been done

Testing, Docker and CI

That is the biggest part I did but I think also the most important point to get the project to a more professional level and make usage easier for new people. There is a docker hub for openwifi docker images. This way you don’t have to build the image – you just need to pull it. The image is build by TravisCI on new commits – so it is up to date.

I also started to implement tests based on docker images. There is a test that boots up a LEDE docker image and an openwifi docker image and checks if the LEDE container discovers the openwifi image and registers to it. This test is also done by TravisCI.

Just last week I also updated the docker image in the way that it can now also optionally ship with nginx and therefore use TLS! Nginx is also doing the client side certificate validation and sets a header value according to the outcome. (I guess something similar is also possible for apache and other servers)

TravisCI testing has also been added for pyuci.

Discovery Script

The discovery script has been completely revamped and is now a lot more readable. It also serves now as small client that updates the registration every 3 minutes.

pyuci

Pyuci has been refactored and extended. Diffs are now a separate class and can be applied and reverted from a config. Also the testing has been extended somewhat and is done by TravisCI.

Authentication

A basic user management has been added. That is a database entry and password hash storage using passlib and an openwifi authentication scheme. The admin user can add and modify users via a rest API.

I also thought about how to implement rights management – more on that below.

New DB Model

The new database model has not been on my initial schedule so much. But actually there is a lot to do. What has been done is proper syncing via sqlalchemy events of the database representation and the rpcd-json representation.

Revisions

OpenWifi now saves change to a config in a revisions database table.

What is going to happen next

Testing

Since there is now a proper infrastructure for docker based testing it should be used more like for testing https and configuration changes.

More tests are needed for the new database model and pyuci.

New DB Model

The new db model needs testing and methods for adding and removing of links and configurations. It should keep IDs consistent upon updating (to allow configuration sharing) -> so maybe make use of new pyuci diff features.

Also I should get rid of all old configuration access (like old templating and configuration changing). It might be nice to have a gui for the DB configuration but this has not the highest priority.

Authentication

The authentication needs to authenticate API keys and client side certificates.

Authorization

Give users/API-keys access based on node and pattern matching. Add option to restrict registration to client side authentication.

The pattern matching could be in the form the current master-config query or a path string (like network.lan.ipaddr or network.[interface].ipaddr or network.lan.*). It should be possible to set patters for read only, read write and restrict. (For example to have a eveything mathing pattern but restrict access to one specific pattern).

It should also be possible to restrict access luci2 (it doesn’t make sense to restrict the db querys if luci2 access is allowed ;)).

Discovery Script

The discovery script should be updated to setup capabilities and communication protocol (like rpcd, rpcd-TLS, NetJSON, etc.) on registration.

Nice to have features would be that it has an option to stop once a registration was successful and to accept a retry count.

I the future it would be nice to work together with OpenWISP and have and share the discovery mechanism.

GSoC 2017 Project – OpenWifi: LEDE/OpenWRT configuration Mangement

GSoC 2017 – OpenWifi

Hi, my name is Johannes Wegener and I’ll be working on OpenWifi this Google Summer of Code. I’m 27 years old and study computer engineering at TU Berlin. In this blog post I’m going to explain to you what OpenWifi is and what should be done during this summer of code.

What is OpenWifi?

OpenWifi is a OpenWRT/LEDE configuration management system. It is intended to manage a bigger or smaller amount of OpenWRT/LEDE devices.

If a new access point joins a network it is be able to auto detect the management Server and register to it. After the node has been registered its configuration (static configuration like what you’ll typically find in /etc/config/ on OpenWRT/LEDE devices) will be downloaded by the server and stored in a database.

It is now possible to query and modify aspects of the configuration. It also possible to change values depending on other values. This means configuration changes can be applied to a lot of different routers. There is an older templating system which shall be replaced by a new graph-oriented one. It also manages SSH-Keys, has a rudimentary file upload functionality (for uploading new images for example) and extensible API. It is possible for example for a node to ask for an image it should install. (Currently this is not very dynamic – but it is intended that an image could be selected on various parameters)

The Server also regularly pulls the health-state of the node and displays it in an overview. Furthermore the server acts as a luci2 proxy.

It is also extensible via PlugIns and is able to serve as an entry point to other services like icinga or location services. (Both already have a proof of concept PlugIn available)

How does it work? aka what makes it tick?

The main software is written in python and uses the pyramid framework and sqlalchemy as ORM. It uses pyramid-rpc for json-rpc requests (which is used for nodes to register for example) and cornice for a REST-style API (which is intended for the user of the system). The Core-System can be found in this repository.

Most of the tasks operating on nodes are done by a jobserver that uses celery – so that they don’t block the main thread.

All Webviews have been moved to a different repository and are realised in fact as a PlugIn. (If you just want to manage your nodes on CLI/with scripts this will be possible in the near future!)

The nodes use a notification script written as a shell script. It uses a fixed DNS entry (openwifi), a configuration file (/etc/config/openwifi) or mdns (using umdns or avahi) to detect a server and register to it. The packages can be found in the openwifi-feed repository. There is also a boot-flasher which is intended for flashing an alix2-style board from ramfs – since it is also possible to execute commands on the node I’ll integrate an update solution that uses sysupgrade.

The communication between the server and the node is realized currently via rpcd. But one goal during this Google Summer of Code is to abstract that and realize it via a rpcd-communication-PlugIn – this would make it possible to also a NetJSON-communication-PlugIn for example.

The PlugIns are realized with python entry points. There are some special named entry points which will extend the main application. You could have a look at the example plugin to get an idea how it works.

How to try it?

You can easily try the software with docker. Navigate to the Docker directory – there three files that will assist you with the setup. In conf.sh you setup if you want to use LDAP for authentication, avahi for mdns announcement and dnsmasq as dhcp server in the docker container.

With build_image.sh you build the docker image. It tries to detect if you need to use sudo for docker or not. Last but not least you need run_image.sh to start the image.

You can easily add PlugIns to the docker image if you just check them out or copy them inside the Plugins directory. To start off I would recommend to add the Web-Views Plugin and use avahi. Now you can navigate your browser to http://localhost:6543 to see the webviews (you need to restart the container after Plugin install – use docker stop OpenWifi and docker start OpenWifi).

What is needs to be done?

In the next section I’ll explain what should be done during Google Summer of Code. I need to prioritize these things because I’m not sure if it is possible to do everything. I think most important things are Testing, Authentication and the new graph-based database model.

Testing-Infrastructure

Currently just very few things have tests. But because it is possible to start a LEDE container in docker (there are some scripts helping to create a LEDE container in the Docker/LEDEImage directory) a lot of things are possible to test with docker orchestration. I want to provide a small LEDE image with the code to test it.

Before adding big new things I like to transfer the development to true test-driven development and having test for at least 90% of the core component code. And after that continuing development by writing tests first.

For that I would also have a look at coverage.py.

Security and authentication

There is a simplistic authorization for OpenWifi with a LDAP backend. I would like to add authorization based on user/password (without LDAP), API-Key and client side certificate (for the nodes to authenticate theirself against the API).

I would like to have the authorization as granular as possible. Authorization should check whether the authorized identity is able to operate on this node and what kind of action is allowed.

Security is provided by TLS. Last week I added the last bits to also use https for client communication – but right now it needs to be setup up by the user. I would like to add the appropriate hooks to the notification shell script.

Expand new graph-based DB-Model

The new DB-Configuration model is based on a graph – where configurations can be connected by links (the link can contain addition information – like what kind of link it is – currently the addition data is the option name (for example if you have a wifi-iface configuration the “device” links it to a wifi-device)).

The model is implemented and there is a query API to get and set options. But for communication purposes the configuration is stored as uci-json-config. There is code to convert between these two (but the one from uci to graph-model needs quite some more intelligence to detect everything right). There are also some hooks that update each part. But the update process is not consistent – the graph-based config gets a new ID for example. It would be nice to have a consistent conversion.

Furthermore I like nodes to share parts of a configuration. This should be easy to implement with some small DB changes. For this consistency is also very important. By default UCI generates unique strings for every configuration – this could be used for consistency.

The DB-Model also currently lacks creating new configurations and reordering configurations.

API

The API to modify nodes should get expanded and everything should get documented. (see above) This should be done in code and a document generated by sphinx.

I also like to implement a CLI program to use the API.

Versioning

The uci-json parser supports diffs between configurations. I would like to save a diff for each change which is applied – to have roll back configurations or delete a specific change.

There is a simplistic revisioning implemented. But the diff should be updated to a class of its own with json import and export and operations to apply a diff to a UCI configuration. (upgrade and downgrade)

Modularity and expandability

OpenWifi is right now is already quite modular – but I also want to modularize configuration parsing and communication. It would also be nice to have some interoperability with OpenWISP.

I use alembic for tracking database changes it would be nice to find a good way to use alembic for database changes needed by PlugIns as well.

Scheduled Updates

I want to schedule config updates – for example if you have a mesh you probably want to update the deepest nested nodes first and than the ones above etc. – this should be easy with celery (as long as the mesh is known and somehow represented in the database).