VRConfig Final

Hi,

this is the final blog post about my project VRConfig.
VRConfig aims to improve the accessibility and usability especially for inexperienced users of OpenWrt and its Webinterface LuCI.
It achieves this by introducing a graphical configuration option. The users can configure their router by interacting with a picture of the router model they are using, instead of digging through menus full of technical terms they do not understand.
In order to be able to present to every user the correct picture of the more than 1000 different supported router models, the help of the community is needed.
Everyone can take a picture of the backside of their router and annotate the ports on that picture using the annotation-app, I developed (https://vrconfig.gitlab.io/annotator/).

The annotator can be used to mark the location of all ports of the router

 

You could then send in the jpg-file together with the annotation file (which is a json-file) to the luci app via a merge request here: https://gitlab.com/vrconfig/luci-app-vrconfig.
The makefile will automatically choose the right jpg/json file based on their file name during the build process.

The luci application is currently a demo application, which will be improved in the future.
Currently, it looks like this:

You can hover over the different ports, a click will bring you to the corresponding configuration. It also marks those LAN-ports green, which are currently connected with a LAN cable.
For that I developed a lua demon which monitors the corresponding ports in real time and provides the interface with their status.
Also there is list of all currently configured virtual interfaces. Clicking on them will mark the associated physical ports on the image.

Future Plans

In the future I plan to continue to polish the Luci interface. One extension could be to marks those ports, which currently have Internet access. Other extension could revolve around making it possible to configure some setting via drag and drop on the image.

Acknowledgments

Thanks a lot to my mentor Thomas for his excellent support and his long term visions that made this project possible in the first place.
Also thanks to my colleague Benni for his extremely helpful suggestions throughout the project.
Also thanks to Freifunk for letting me work on this project and thanks to Google for organizing GSoC.

The full source code of everything related to this project can be found here: https://gitlab.com/vrconfig

GSoC 2018 – DAWN a decentralized WiFi controller (2st update)

Hi,
I still try to get my patches upstream.
For the libiwinfo patch I had to add the lua bindings. I never used lua so first I had to get comfortable with this. Additionally I wanted to add the channel utilization in the luci statistics app. But suddenly Luci is giving me a null pointer exception in the dev branch.


Additionally I tried to get comfortable with Luci for developing my own app.
Meanwhile another developer created nearly the same patch for iwinfo that add the survey data for the nl802.11 driver… This patch is still not accepted. The only difference is that it returns all survey data for all channels (like iw dev wlan0 survey dump)…
Furthermore, my pull request for the hostapd ubus bindings that add information about the ht and vht capabilities had to be rewritten. (https://github.com/openwrt/openwrt/pull/898). Again I have to wait for some feedback. While rewriting this patch, I had a new idea: If you subscribe to the hostapd via ubus and want to notify on the messages you have to activate it. It would be possible to add flags in the hostapd_ubus_bss to select what information should be published via the ubus bus. Before doing so, I want some feedback if this is a good idea.If somebody is interested why I am interested in the capabilities: I want to create a hearing map for every client. I’m building this hearing map using probe request messages. This probe request messages contain information like (rssi, capabilities, ht capabilities, vht capabilities, …). VHT give clients the opportunity to transfer up to 1,750 Gigabits (theoretical…) If you want to select some AP you should consider capabilities… In the normal hostapd configuration you can even set a flag that forbids 802.11b rates. If you are interested what happens if a 802.11b joins your network search for: WiFi performance anomaly. 🙂

Summarizing, I spent a lot of time waiting for feedback, debugging, modifying my patches or replying on the email lists. It is a bit frustrating.
The cool stuff was that I had my first pull request. 🙂 (it was just a typo ^^) But somebody took the time to fork my project and create a pull request. 😉
Furthermore, it is exam time and I have a lot of stuff to do for the university.

Actually I wanted to go on with more interesting stuff like connecting to the netifd demo to get more information.

Or to look at PLC. There is an interesting paper EMPoWER Hybrid Networks: Exploiting Multiple Paths over Wireless and ElectRical Mediums.

 

VRConfig Update 2

Hi,

I spent the last weeks mainly developing the LuCI Application for VRConfig. As soon as you want to do advanced things with LuCI, it gets cumbersome.
As the API is mostly undocumented, you have to dig through the LuCI’s source code trying out functions which could be useful according to their name.
It’s a bit of a trial and error game.
Currently the LuCI app does the following.
It displays an image of the router and parses the JSON file, which contains the locations of the components.
With this information it can mark the associated physical ports to the currently selected network interface and display those network ports, which are connected to a cable. You can also hover over the components and click on them, which leads you to their respective settings page.

I also improved the annotation app. It now lets you choose the router name from a list of all currently supported router models of OpenWrt. I got that list with a series of grep and sed commands from the OpenWrt git repository.
For your information, there are currently around 1100 different router models supported. 🙂

In the next weeks I will polish the LuCI Application and try to integrate VRConfig into the openwrt build system to be able to select the correct router image and JSON file at build time.

VRConfig Update

Hi,

I have some quick updates about VRConfig for you.
Short recap: VRConfig aims to introduce a graphical configuration mode for OpenWrt’s Webinterface LuCI.
For that need to collect pictures of the backside of all supported routers. The idea is to do this in a crowdsourcing manner. The community can submit pictures of their routers together with a metadata file which contains the locations of the components on the picture.

I spent the last weeks developing a web application to provide the annotation functionality of the router components.
A working prototype is now ready and can be tested at the following URL: https://vrconfig.gitlab.io/annotator/
Source code: https://gitlab.com/vrconfig/annotator

The annotator produces a JSON file which in turn can be parsed by the LuCI Application to provide the graphical configuration mode.

The LuCI application is being developed right now and will be provided shorty under the following URL: https://gitlab.com/vrconfig/luci-app-vrconfig

More info about that in the next blog post.

DAWN – A decentralized WiFi Controller


Hi,
I’m Nick. I study Computer Engineering at the TU Berlin. It is my first time participating in Google Summer of Code. I am realizing a decentralized WiFi controller.

DAWN is the first decentralized WiFi controller for OpenWrt. The controller provides access to valuable information, e.g., all connected stations, their capabilities, and information about all participating nodes. Moreover, DAWN provides load balancing to increase the network performance by controlling the clients association.

What’s missing?
An important aspect of the controller is the simple installation. Everybody, even people with limited technical knowledge, should use this controller to increase their network performance at home. Until now, DAWN requires a special patched OpenWrt to run. So a user needs to compile his own image. The first thing I have to do is to bring the last patches upstream. Some of the patches were rejected and that is why I have to rewrite different functionality and create new pull requests. Furthermore, I have to extend the libiwinfo library to get all necessary informations from the OpenWrt system.
After this, the configuration of the nodes should be simplified. So far, the user has to configure all participating nodes individually. I want to implement some bootstrapping to automatically configure the participating routers.
After simplifying the installation and configuration, I want to visualize the information of the participating nodes with a graphical user interface.
The last step is to improve the controller functionality by adding mechanisms like a channel interference detection and other useful features. Moreover, this step contains to improve the load balancing.

In my next blog post, I will write about why some of my OpenWrt patches were rejected, how I have to extend the libiwinfo. However, if this steps are successful everybody will be able to simply install DAWN without the need to patch OpenWrt.

VRConfig – Visual Router Configuration for OpenWrt

Hi,

I am Tobias, a Computer Science student at the TU-Berlin. This is my second time participating in GSoC for Freifunk.
I am excited about this project as it helps to reduce the entry barrier for inexperienced users of OpenWrt and its web interface LuCI.

When you look at the current LuCI Webinterface you will notice that it looks fairly decent, especially with the Material Theme.
However for an inexperienced user without a technical background it surely looks scary. All the text full of technical terms with few pictures can look like a book with seven seals.

This project aims to introduce a graphical configuration mode.
To make the configuration interface more connected to the actual router the user owns, we want to display an image of the backside of the ports in the web interface.
The user shall be able to interact with this graphical representation of the router by hovering and clicking on the different parts like LAN ports, antenna etc.

What are the necessary steps to archive this goal?

First, we need pictures of the backside of all the different router models. Here the idea is to collect them via crowdsourcing by the community. Everyone can take a picture of their router and upload it to a Git repository. Also, the location of of the router components must be marked on every picture. For that I will develop a small application which allows the user to annotate a router picture and generates a metadata file.
Second, the annotated pictures need to be integrated into the OpenWrt buildsystem.
Third, a LuCI application needs to be developed to display the result as an interactive graphic in the web interface.

In the next blog post I will go into more detail on the individual steps as well as update you about the progress.

PowQuty Live Log GSoC 2017 Final Update

This is the last blog entry in the series of Google Summer of Code project updates. It will describe, what has been done and what is left to improve in the future in the PowQuty project.

Powquty

PowQuty is a power quality monitoring tool, which can be installed on a router running LEDE or OpenWrt. The router can be connected to an USB oscilloscope providing measurements which powqutyd will process and provide to the user in human readable form.
All this was tested on a x86 based LEDE router.

GSoC 2017

During this Google Summer of Code a live log functionality was added to PowQuty to provide information on power quality events. These events are:

  • voltage dip of 10% – 90% of the reference voltage on the measurement signal
  • voltage swell > 110% of the reference voltage on the measurement signal
  • voltage dip < 10% of the reference voltage on the measurement signal
  • > 5% of the measured values of one specific harmonic are over the defined threshold
  • On event occurrence important information like time, duration and event type will be written to a log file and presented in the extended luci app.

    As shown in the above picture, the interface provides a traffic light like color system behind these events, green indicates everything is within the EN50160 power quality norm. Yellow means, that 80% of the maximum time per week is already reached, red means, that the norm was violated during the last week.
    In addition to log writes, notifications are send out with Mosquitto. Mosquitto is a message broker using the MQTT protocol. It provides a publish/subscribe model, which allows a central server to subscribe to a topic and clients to send out messages to the server with a topic. Mosquitto was already in use in powquty but was extended for EN50160 event notifications. This will allow a central logging of bigger power supply networks, monitored by multiple devices.

    As another option Slack messages can be send by powquty now. Slack is a messaging program, using (as one option among many) webhooks for interaction. Everyone with the webhook can send messages to the team. Sending out messages allows a user to react quickly to changing situations, or get immediately informed on power event occurrence.

    Beginning with pull request 20 [https://github.com/thuehn/PowQuty/pull/20] I started to implement these features.
    First an option was developed to read measurements from a file, as most power supply networks are pretty stable and wont provide many opportunities to test event handling in powquty.
    Afterwards slack and MQTT notifications where added.
    During testing of mosquitto event messages, some seemed to be lost on intervals with many En50160 events in a short period(sometimes more than 35 events per seconds). The solution seems to buffer all events before sending.
    Something similar happened with Slack. Slack only allows one message per second(short bursts excluded) [rate-limits].
    Buffering events would resolve this problem as well. An option for live email notification was considered at first, but was dropped as spam protection would stop most of the messages and probably list users as spammers.
    The last step was to add the traffic light system to the luci app, to enable users without knowledge of the norm to get an idea of the power quality of their power supply network.
    In addition a slack library was written [libwebslack] to send slack messages from PowQuty.

    What can be improved
  • As mentioned before event buffering is needed and will be added after GSoC
  • Email notification in form of a weekly summary
  • More Error checking and handling
  • improving libwebslack to not use libcurl to reduce its size
  • provide libwebslack as OpenWrt/LEDE package, for easier future use
  • Finally I have to thank Dr. Thomas Huehn for being my mentor and Freifunk for their work they do and especially for being a mentoring organisation for Google Summer of Code.
    Last but not least I would like to thank Google for making this all possible.

    If you want to review some of my earlier posts:

  • Introduction
  • First Update
  • Second Update
  • Best regards
    Stefan

    GSoC 2017 – Attended Sysupgrade – Final evaluation update

    Hi Freifunk,

    This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.

    tl;dr:
    Direct links to the work I’ve done this GSoC:

    First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.

    Progress

    Server

    At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.

    The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.

    Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.

    To try the the server at https://betaupdate.libremesh.org have a look at the demos I prepared. Continue reading “GSoC 2017 – Attended Sysupgrade – Final evaluation update”

    GSoC 2017 – Milestone 2

    Following the rather stark change of direction after the last milestone (see my last post), I have worked on the integration of my project with Johannes Wegener’s: OpenWifi.

    July Progress

    For my software-defined networking (SDN) agent, I have added support for configuration through a UCI config file as well as process management through procd. This ties the daemon in neatly with the LEDE/OpenWRT system. Now, parameters such as the SDN controller’s address are read from this file and the configuration can be reloaded at runtime without stopping the agent.
    Using the UCI system also exposes the configuration parameters to Johannes’ OpenWifi system. He and I discussed the bootstrapping process with our mentor and we have come up with the following idea: When a newly installed LEDE/OpenWRT access point first boots, it discovers the OpenWifi server via mDNS and fetches its configuration from it. With the address contained in the configuration, my SDN agent on the access point is able to connect to the SDN controller and thus integrate automatically with an existing centrally managed deployment. In the event of a configuration parameter change (e.g. a switch to a different SDN controller), OpenWifi can trigger a configuration reload to quickly update all access points in the network.

    On the controller side, I have implemented a REST client to interface with OpenWifi. Through it, the controller can register with the OpenWifi server. During the registration, it installs its address and OpenFlow listening port in the UCI configuration which later gets sent to access points.
    I have also begun writing a REST interface for my controller to offer more comfortable management of SDN applications and the network state. Right now, I can query the controller about its resources: access points, switches and clients. With the basics in place, expanding the interface to expose more SDN functionality should be pretty straightforward.

    Next Steps

    During the final part of this year’s GSoC, I want to focus on adding functionality, testing and documentation. I will spend the remaining weeks of the project like this:
    1) write an SDN application
    2) expose the app’s functionality via the controller’s command-line and REST interfaces
    3) test it on the university department’s testbed
    4) document its usage
    5) goto 1)

    Since the foundation for the SDN applications is in place and running, I am optimistic about getting a lot done during August. I will start with a client hearing map that keeps track of associated and unassociated clients in the vicinity of the deployment’s access points. Leveraging the hearing map, I want to implement a client load balancer that distributes associated clients evenly across available access points. I will also look into automated channel selection to avoid interference between neighbouring access points.
    Finally, I would like to wrap up the controller in a docker image for easy deployment.

    GSoC 2017 Project – OpenWifi: LEDE/OpenWRT configuration Mangement

    GSoC 2017 – OpenWifi

    Hi, my name is Johannes Wegener and I’ll be working on OpenWifi this Google Summer of Code. I’m 27 years old and study computer engineering at TU Berlin. In this blog post I’m going to explain to you what OpenWifi is and what should be done during this summer of code.

    What is OpenWifi?

    OpenWifi is a OpenWRT/LEDE configuration management system. It is intended to manage a bigger or smaller amount of OpenWRT/LEDE devices.

    If a new access point joins a network it is be able to auto detect the management Server and register to it. After the node has been registered its configuration (static configuration like what you’ll typically find in /etc/config/ on OpenWRT/LEDE devices) will be downloaded by the server and stored in a database.

    It is now possible to query and modify aspects of the configuration. It also possible to change values depending on other values. This means configuration changes can be applied to a lot of different routers. There is an older templating system which shall be replaced by a new graph-oriented one. It also manages SSH-Keys, has a rudimentary file upload functionality (for uploading new images for example) and extensible API. It is possible for example for a node to ask for an image it should install. (Currently this is not very dynamic – but it is intended that an image could be selected on various parameters)

    The Server also regularly pulls the health-state of the node and displays it in an overview. Furthermore the server acts as a luci2 proxy.

    It is also extensible via PlugIns and is able to serve as an entry point to other services like icinga or location services. (Both already have a proof of concept PlugIn available)

    How does it work? aka what makes it tick?

    The main software is written in python and uses the pyramid framework and sqlalchemy as ORM. It uses pyramid-rpc for json-rpc requests (which is used for nodes to register for example) and cornice for a REST-style API (which is intended for the user of the system). The Core-System can be found in this repository.

    Most of the tasks operating on nodes are done by a jobserver that uses celery – so that they don’t block the main thread.

    All Webviews have been moved to a different repository and are realised in fact as a PlugIn. (If you just want to manage your nodes on CLI/with scripts this will be possible in the near future!)

    The nodes use a notification script written as a shell script. It uses a fixed DNS entry (openwifi), a configuration file (/etc/config/openwifi) or mdns (using umdns or avahi) to detect a server and register to it. The packages can be found in the openwifi-feed repository. There is also a boot-flasher which is intended for flashing an alix2-style board from ramfs – since it is also possible to execute commands on the node I’ll integrate an update solution that uses sysupgrade.

    The communication between the server and the node is realized currently via rpcd. But one goal during this Google Summer of Code is to abstract that and realize it via a rpcd-communication-PlugIn – this would make it possible to also a NetJSON-communication-PlugIn for example.

    The PlugIns are realized with python entry points. There are some special named entry points which will extend the main application. You could have a look at the example plugin to get an idea how it works.

    How to try it?

    You can easily try the software with docker. Navigate to the Docker directory – there three files that will assist you with the setup. In conf.sh you setup if you want to use LDAP for authentication, avahi for mdns announcement and dnsmasq as dhcp server in the docker container.

    With build_image.sh you build the docker image. It tries to detect if you need to use sudo for docker or not. Last but not least you need run_image.sh to start the image.

    You can easily add PlugIns to the docker image if you just check them out or copy them inside the Plugins directory. To start off I would recommend to add the Web-Views Plugin and use avahi. Now you can navigate your browser to http://localhost:6543 to see the webviews (you need to restart the container after Plugin install – use docker stop OpenWifi and docker start OpenWifi).

    What is needs to be done?

    In the next section I’ll explain what should be done during Google Summer of Code. I need to prioritize these things because I’m not sure if it is possible to do everything. I think most important things are Testing, Authentication and the new graph-based database model.

    Testing-Infrastructure

    Currently just very few things have tests. But because it is possible to start a LEDE container in docker (there are some scripts helping to create a LEDE container in the Docker/LEDEImage directory) a lot of things are possible to test with docker orchestration. I want to provide a small LEDE image with the code to test it.

    Before adding big new things I like to transfer the development to true test-driven development and having test for at least 90% of the core component code. And after that continuing development by writing tests first.

    For that I would also have a look at coverage.py.

    Security and authentication

    There is a simplistic authorization for OpenWifi with a LDAP backend. I would like to add authorization based on user/password (without LDAP), API-Key and client side certificate (for the nodes to authenticate theirself against the API).

    I would like to have the authorization as granular as possible. Authorization should check whether the authorized identity is able to operate on this node and what kind of action is allowed.

    Security is provided by TLS. Last week I added the last bits to also use https for client communication – but right now it needs to be setup up by the user. I would like to add the appropriate hooks to the notification shell script.

    Expand new graph-based DB-Model

    The new DB-Configuration model is based on a graph – where configurations can be connected by links (the link can contain addition information – like what kind of link it is – currently the addition data is the option name (for example if you have a wifi-iface configuration the “device” links it to a wifi-device)).

    The model is implemented and there is a query API to get and set options. But for communication purposes the configuration is stored as uci-json-config. There is code to convert between these two (but the one from uci to graph-model needs quite some more intelligence to detect everything right). There are also some hooks that update each part. But the update process is not consistent – the graph-based config gets a new ID for example. It would be nice to have a consistent conversion.

    Furthermore I like nodes to share parts of a configuration. This should be easy to implement with some small DB changes. For this consistency is also very important. By default UCI generates unique strings for every configuration – this could be used for consistency.

    The DB-Model also currently lacks creating new configurations and reordering configurations.

    API

    The API to modify nodes should get expanded and everything should get documented. (see above) This should be done in code and a document generated by sphinx.

    I also like to implement a CLI program to use the API.

    Versioning

    The uci-json parser supports diffs between configurations. I would like to save a diff for each change which is applied – to have roll back configurations or delete a specific change.

    There is a simplistic revisioning implemented. But the diff should be updated to a class of its own with json import and export and operations to apply a diff to a UCI configuration. (upgrade and downgrade)

    Modularity and expandability

    OpenWifi is right now is already quite modular – but I also want to modularize configuration parsing and communication. It would also be nice to have some interoperability with OpenWISP.

    I use alembic for tracking database changes it would be nice to find a good way to use alembic for database changes needed by PlugIns as well.

    Scheduled Updates

    I want to schedule config updates – for example if you have a mesh you probably want to update the deepest nested nodes first and than the ones above etc. – this should be easy with celery (as long as the mesh is known and somehow represented in the database).