SWOON: Simultaneous Wireless Organic Optimization within Nodewatcher – in the middle of the road.

Hi everyone,

This is my first update on working on SWOON, an algorithm to automatically allocate channels to nodes in nodewatcher. Not a lot of time has passed since I started working and but the wlanslovenija community was immensely helpful in getting me started. Up to now, I only worked on my own coding projects, where I got to be the main architect behind the codebase. I decided where functions were defined, I decided on the coding style, I did code reviews and I was the one to approve branch merges (of which there were none, because I wrote all the code 🙂 ). Now, GSoC threw me in the middle of a well established open source project where you have to get accustomed to existing coding practices. And accustoming to this was more difficult than I expected. So people from wlanslovenija had to be really patient with me (and I think they’ll have to keep this up for a while 🙂 ). But I slowly got the hang of it. My overarching plan for the algorithm is to first read the data, then process the data and finally issue a warning to those who maintain nodes with inefficient channel allocations. My work on reading the data from nodes is almost done (almost because a new amazing feature was suggested only yesterday!). On a tangent, I think this shows another immense benefit of community oriented projects – everyone’s free to suggest additional features that would make this process more eficient. To get the data about neighboring nodes, one has to temporarily turn the wireless radios off. Because of that, nodewatcher-agent only performs a survey of neighbors every two hours. But we collect data much more regularly. So how do we deal with this? My current implementation simply stores the data into the database every hour and it does not take into account the age of the data. I will now work on comparing the current datapoint with the latest one and only insert it into the datastream if the data has changed. This will reduce the space we’re using and could even be expanded for use by other modules. It’s great to know that you can contribute to any aspect of the project. Mitar, a senior developer of nodewatcher, said that they built all of it themselves, so we can rewrite anything as well.

I also worked on a parallel implementation of the mechanism, which read the data from old dumps. This gave me the insight into the architecture that made writing the actual implementation much easier.

My next steps are to finish the data collection part and create a module to analyze this data. This is exciting for everyone because there’s no documentation about writing such modules so I’ll get to be the first one to do it! I can only hope I’ll document it well enough to make the process easier for those who will follow as new members in the community. The github PR can be found here.

Overview: GSoC projects at freifunk.net

In 2016 we found 9 students doing Google Summer of Code projects for freifunk. This is an overview on all our projects including links to repositories and further information:

Title Student Repository Project information
A schema-based configuration system for OpenWrt Neoraider  https://gitlab.com/neoraider/ece  
DynaPoint – A dynamic access point validator and creator for OpenWrt Asco  https://github.com/thuehn/dynapoint
https://github.com/thuehn/luci-app-dynapoint 
 
Implementing Pop-Routing Gabriel https://github.com/gabri94/poprouting http://firenze.ninux.org/
Monitoring and quality assurance of open wifi networks: the client view Tarek https://git.nordwest.freifunk.net/ffnw-firmware/monitoring-drone   
netifd extension: external device handlers arkap    
OpenWrt – poWquty (poWer quality) Neez   https://github.com/thuehn/powquty https://github.com/neez34/
Provide a cryptographic implementation for Qaul.net spacecookie  https://github.com/spacekookie/qaul.net  https://spacekookie.de/gsoc2016/
Sharable Plugin System for qaul.net – Mesh Network Communication App Anu  https://github.com/anuvazhayil/qaul.net https://github.com/anuvazhayil 
SWOON: Simultaneous Wireless Organic Optimization within Nodewatcher marn 

https://github.com/CdavM/nodewatcher

https://github.com/CdavM/swoon 

 

Blog posts on these projects you can find with the tag GSocC 2016. Use the RSS feed to receive all updates. You can also see our projects from previous summers.

GSoC 2016 Introduction: external device handlers for netifd

HELLO WORLD

It’s time I started blogging about my Google Summer of Code project with Freifunk.

To begin, let me introduce myself. I am Arne, computer science student at TU Berlin and pursuing my Bachelor’s degree. This is my first GSoC — and the first time I contribute to an Open Source software project, so, naturally, I’m pretty excited to get to know the process.

My project aims at extending OpenWRT/LEDE’s network interface daemon (netifd).

I’ve familiarize myself with the inner workings of netifd while I was working on a student project last semester. The result of that student project will assist me in realizing the GSoC project. Moreover, the existing source code will partially build the foundation of the new device handler which will be realized in this project.

 

THE PROJECT

Here’s the general idea: Netifd allows the user — as the name suggests — to create and configure network devices and interfaces. Per device class, there is a hard-coded structure called a ‘device handler’. Each device handler exposes an interface for operations such as creation or re-configuration of device instances of its class.

The point I’m tackling is the structures hard-codedness. Today, whenever someone wants to introduce a new device class, is necessary to alter the netifd code base and maintain these changes. The proposed mechanism allows to realize external device handlers as separate programs communicating via the bus system ubus. Accordingly, to introduce a new device class into netifd, one just needs to implement device handling logic and link it to netifd. Thus, no maintenance — or knowledge of the innermost workings of netifd — is necessary.

Building on my work from the aforementioned class I’ll write a device handler to integrate Open vSwitch support into OpenWRT/LEDE using netifd. The ‘ovsd’ as it is called will relay calls to the device handler interface via the ‘ovs-vsctl’ command line utility to set up Open vSwitch bridges and ports.

 

MY DEVELOPMENT ENVIRONMENT

I am hosting my code in a git repository that’s included in the GitLab of the INET department which is employing me at TU Berlin. Once the code becomes usable, it will appear on GitHub here.

We also have a repository providing an OpenWRT/LEDE feed that bundles the Open vSwitch software with ovsd: (link will follow)

 

Testing is done on a PC Engines Alix 2 board which is part of our testbed:

 

WHERE I AM AT

As of today I have realized a mechanism to generate device handler stubs within netifd when the daemon is first started and before it parses /etc/config/network. The device class’ parameters, such as its name in the ubus system and most importantly the format of its configuration, are stored in JSON files in /lib/netifd/ubusdev-config/. The logic to parse a configuration format from JSON files existed already within netifd but was only used for protocol handlers before. I adapted it to generate device handlers dynamically.

When such a device handler is generated, it subscribes to its corresponding external device handler using ubus. This way, the latter can generate events asynchronously for the former to react to.

In order for this to work, the external device handler must be up and running before netifd attempts the subscription.

I also realized a simple proof-of-concept implementation that works in one simple case: create an ovs-bridge with one or more interfaces like this example /etc/config/network file demonstrates:

 

config interface ‘lan’

    option ifname ‘eth0’

    option type ‘ovs’

    option proto ‘static’

    option ipaddr ‘172.17.1.123’

    option gateway ‘172.17.1.1’

    option netmask ‘255.255.255.0’

This will create an Open vSwitch bridge called ‘ovs-lan’ with the port ‘eth0’ — and that’s about all it does right now. Error handling is missing and more complex options like setting OpenFlow controllers or adding wireless interfaces don’t work yet.

 

THE ROAD AHEAD

Among the first things I’ll tackle when GSoC’s coding phase begins are adding the possibilities to incorporate WiFi-interfaces to the bridges created with external device handlers and introducing the missing parts of the device handler interface. Since I’m dealing with netifd and the external device handler which is an independent process running asynchronously, getting their state to stay in sync will take some work. I need to think about the way the callback system that comes with ubus’ pubsub-mechanism will function. In the end, it is supposed to be abstract enough to work not only for my Open vSwitch use case but for any external device handler.

Then, further into the coding phase, I want to increase feature coverage of the Open vSwitch options, so that users can assign OpenFlow controllers for their bridges like so:

 

config interface ‘lan’

    option ifname ‘eth0’

    option type ‘ovs’

    option proto ‘static’

    option ipaddr ‘172.17.1.123’

    option gateway ‘172.17.1.1’

    option netmask ‘255.255.255.0’

    option of-controller ‘1.2.3.4:5678’

In the end I would like to have the mechanism implemented and documented so that others can build on it. Documenting my work is especially important to me, because I found myself frustrated with the scarcity of the netifd documentation more than once before. I wish for my code to become a part of OpenWRT/LEDE and be extended and improved in the future and I would like to make it easy for anyone insterested to doing that with a helpful manual and commented code.

I’m excited to get started and hope to find the time to blog about my progress here, regularly.

Monitoring and quality assurance of open wifi networks out of client view

Hey everyone,
 
My name is Jan-Tarek Butt and I am in my second term of computer science at Emden University in Lower Saxony (Germany). I am one of the students that are participating for Freifunk in the Google Summer of Code 2016. In the following post I would like to introduce you to my Google Summer of Code project. I split up the project into 3 main subjects: Mainline project, sub-projects and seminars.
 
Before I explain the three mentioned subjects I would like to give you some background information about the project in general. My project mentor is Clemens John. He studies computer science at University OsnabrĂźck in Lower Saxony (Germany) and recently started to write his bachelor thesis. As part of his bachelor thesis he will build a monitoring and administration software for open wireless networks called “Netmon-SC” based on a previosly existing software that will be rewritten to follow a decentralised idea. The coarse structure of Netmon-SC will consist of a core module as a data storage backend that can be queried by using a REST API based on NetJSON. In addition to the REST API the core module will include a plug-In system which developers can use to easily extend the core module to build data storages for creating visualisation tools, quality assurance metrics or any other community specific tools like special administration applications.
 
My mainline GSoC project is to develop a new firmware for routers based on openWRT or LEDE. This firmware will  be the basis for an application to monitor the clients view onto an open wireless network. From now on we will call this firmware the “Monitoring-Drone”. This monitoring firmware will gather quality assurance metrics and send them to Netmon-SC. This metrics will help developers and administrators to enhance the quality of open wireless networks and find bugs more easier. The firmware will include an API package to communicate with Netmon-SC. It will also contain a small LUCI web-interface for basic configuration e.g. wireless network settings etc. Additionally to the basic settings it should be possible to integrate small plugins in form of  .sh or .lua files witch contain custom commands.This will allow communities to gather network specifics metrics without compiling community specific firmware images. The API for communication with Netmon-SC, the web-interface and the plugin system will embedded into a package bases structure. The git repository for this project can be found here.
 
Now to the sub-projects. Sub-projects are small projects adjacent to the mainline project. The first sub-project is the so called hoodselector that I finished reviewing on Mai 21th. The hoodselector creates decentralized, semi automated ISO OSI layer 2 network segmentations based on geostationary fixed quadrants for batman-adv mesh networks. It detects in which quadrant the node is in by using wireless location services and configurates the router using the settings that have been stored for this quadrant. In conclusion the hoodselector enables us to build scaled decentralised mesh-networks. It is a small program on open wireless routers on the Nordwest-Freifunk community network.
 
The second sub-project is, to create a proper workaround for building images on a multicore system instead of a single core system by using Gitlabs continuous-integration (CI) feature. This will fully automate the firmware image building process for the Monitoring-Drone and also for the open wireless firmware images from the local community.
 
The third sub-project are regular seminars. The idea of this seminars is to give earned knowledge to the public and also acquire new developers for open wireless projects. The seminars should have one or two short presentations about technical processes from open wireless networks, for instance how the hoodselector works or how batman-adv works. The Presentations will be video recorded and uploaded to the internet. After thous presentations on the seminars I plan short discussions about the contend of the presentations. Afterwards hack-sessions should do force forward the developing processes for the Google Summer of Code project and illuminate the opensource software development scene of the open wireless project.
 
Last but not least we created a project timeline. Clemens and I, will have regular Mumble meetings during the GSoC, at the middle of every month. On this meetings we will discuss the work already done and what should happen next. Beside the work on the main project we will also use the meetings to plan the next seminar that will always follow at the end of each month.
 
Timeline:
23. May: Community Bonding (3 weeks)
test and deploy hoodselector  ← Done
16. May 6:00 PM: GSoC Mumble ← Done
Refine the roadmap ← Done
23. May – 20. June: Work period 1 (4 weeks)
28. May 2:00 PM: hacker-session
  1. Presentation about the hoodselector
  2. Presentation about the openwifi.su project and the geolocator
Midtermevaluation
20. June – 15. August: Work period 2 (8 weeks)
Tarek & Clemens exams!!!
13. June 6:00 PM: GSoC Mumble
25. June 2:00 PM: hacker-session
  1. Presentation about workaround with git CI processes.
  2. actual unknown
18. July 6:00 PM: GSoC Mumble
30. July 2:00 PM: hacker-session
  1. actual unknown
  2. actual unknown
15. August 6:00 PM: GSoC Mumble
Finalevaluation
Cheers
Jan-Tarek Butt

Sharable Plugin System for qaul.net

Hi everyone,

I’m Anu Vazhayil, exchange student at University of Paderborn from India. I am one of the Google Summer of Code participant for the project qaul.net

Qaul.net is a mesh network communication app which can be run in Linux, OSX, Windows and Android. It can be used to send text messages, file sharing and even make voice calls with people who are connected to this network without being connected to internet or cellular networks. There is also a feature to share your internet with the other users connected to the qaul.net network. All this makes it a cool app which everyone will feel to try atleast onceWink If you have not installed it in your system, you can still connect to the network via web interface after connecting to a wifi.

As part of the GSoC project, we are planning to extend the usability of the messaging system of qaul.net. The messaging system can be used as a communication layer for other purposes like location sharing, sending contacts etc. Such apps/extensions can be downloaded by the user as a zipped file which will automatically get downloaded to the web plugin directory available to the web server. It also proposes an additional feature to share the installed plugins with other users in the network over file sharing.

The following needs to be implemented for the project:

  • Plugin API: to describe the features of a library and how to interact with it. It defines a mechanism to use the messaging system for plugins and give it access to the qaul.net topology.
  • Plugin structure: a zipped folder with a special extension that can be shared with other users in the network over file sharing.
  • Plugin management function: Installation and plugin sharing. Once the Plugin is installed, it is copied and extracted to a directory in the web server.
  • Sample Plugin: Share the location details of the user. It will have access to the user’s location through the API implemented.
Mockup UI
Stay tuned for my next blog posts with the updates regarding my projectLaughing. I will also update my blog: https://captanu.wordpress.com/ in the coming days. Our coding period starts today so, I wish All the best to other GSoC students too. Happy coding everyone!
Cheers!

Implementing Pop-Routing

Hi everyone!

I am Gabriele from the Ninux community. I am participating in GSoC 2016 for the first time and I am very glad I have been accepted as a Student for Freifunk. I am from Florence, Italy. Here I’m studying Computer Science, soon I will graduate and I hope to use the results of this project to write my bachelor thesis.

Four years ago, with others community networks’ enthusiasts we have funded Ninux Firenze[1], the fist Wireless Community Network in Florence where I had the chance to learn how these networks work and to meet many others people interested in this field. The network is constantly growing, and now it counts almost 20 nodes. In May ’14 I have been for the first time to Wireless Battle of the Mesh in Leipzig where I met the Freifunk community. For this GSoC I will work on a project called PopRouting[2]:

OONF (OLSRv2) is a link state routing protocol. It works sending periodical messages to his neighbors with the aim of transmitting information about topology changes. With these information each node of the network is able to calculate the paths to reach any other destination. These messages are periodically generated, based on the configuration parameter that regulates the sending interval. A short period will make the network react rapidly but it will also cause a large overhead due to control messages. Pop Routing is a recent technique that takes advantage of the knowledge of the network topology to find the optimal value for the OONF’s timers. Using Pop Routing every node computes the “betweenness centrality” of every other node and uses it to calculate the optimal trade-off between convergence and overhead for its timers. The algorithm has been developed at the UniTN and the algorithm to compute the BC in C++ is available as free software. My goal is to code a daemon (in C) that is able to calculate autonomously the BC of the network and push it to OONF using the telnet plugin.

In this month of community bonding I have been to Wireless Battle of the Mesh v9 in Oporto(PT). There I met the OONF developers and we discussed how to implement this inside OONF. I also gave a presentation on the project. After the Battlemesh I started working on the algorithm developed by UniTN and I made a C/C++ library out of it [3].

Today I will start coding for the GSoC, stay tuned and I will give you more updates soon.

 

Gabriel

 

[1]: http://firenze.ninux.org/

[2]: https://ans.disi.unitn.it/users/maccari/assets/files/bibliography/INFOCOM2016.pdf

[3]: https://github.com/gabri94/poprouting/tree/master/graph-parser

OpenWrt – poWquty (poWer quality): Computing and providing power quality information on OpenWrt routers.

Dear Freifunkers,

Please allow me to introduce the poWquty Project within Google Summer of Code 2016 at Freifunk.

The big picture behind this project relates to the energy production and consumption. Sustainable energy production and consumption are crucial for a prospering life on earth. The importance of energy led many theorists to even define the level of civilization by its energy profile. With the renewable energies shift the energy production paradigm from centralized to decentralized energy production which poses one of the next big challenges, which will influence the energy production in the future years.

From the big picture we move to the concrete case, increasingly faced when dealing with renewable energies: monitoring and control.
The emerging smart grids include an inherent need for communication for monitoring and control purposes in a more and more dynamic environment. One of the major challenges is monitoring the power quality parameters in a decentralized manner. In such case, decentralized information need to be retrieved and transported with the lowest latency possible. One way to solve this challenge could be to install expensive infrastructure to each point. The better way is to use wireless mesh infrastructure that could also serve this purpose.

Here where Freifunk comes in: The Freifunk mesh network is an outstanding example for a decentralized infrastructure that could be augmented with grid related functionalities to cope with future energy challenges. In order to use wireless mesh networks such as Freifunk for energy monitoring, we could use extra hardware that does the power measurements and use the wireless networks solely for transporting the information. The drawback of this is the need to install separate hardware. But, since all routers run on power, we could integrate the measurements into the router, which is the main goal of this project: to enable power quality measurements on OpenWrt.

Here is the initial plan how to do this. First we need to retrieve voltage samples from the power grid. For the beginning we will rely on an oscilloscope device that delivers the real time samples over a USB interface. This way voltage samples from the electric socket are retrieved at the router. With these voltage samples we can go ahead and calculate the power quality parameters, using moving window algorithms, fourrier transform, and z-transform to get the phase angle, the effective power, the frequency, and the harmonics. This calculation should be time, and memory efficient since it has to run on the OpenWrt embedded devices. Once these values are calculated we need to think about how we want to make them available for retrieval over IP networks.

Now we come to the Code: The goal of the project is to create an OpenWrt package which ensures three functionalities:
1-    Retrieving sample data from the measurement device
2-    Calculating power quality parameters form the retrieved samples
3-    Provisioning of the calculated parameters for retrieval

This project is intended to strengthen the role of open software in the uprising smart grids by providing some essential functionalities, communication devices need to have in the context of smart grids, especially in regard to the future role of the home routers in the future energy solutions.

More updates on this will follow in the next weeks.

Cheers,

Neez

GSoC: A new configuration system for OpenWrt/LEDE

Hi,
I’m Matthias (aka NeoRaider), and this year, I’ll participate in the Google Summer of Code for the Freifunk project.

The goal of my project is to develop an alternative to the UCI configuration system, as UCI has a number of issues that make it cumbersome to use in some situations.

One of the basic issues of UCI that affects many Freifunk (or generally community mesh) firmwares is the upgrade behaviour. Mesh firmwares usually contain elaborate default configurations, which set up network interfaces and other things to allow participating in a mesh without deep knowledge of the setup.

But this setup needs to change from time to time, as the firmware is upgraded. In the Gluon firmware framework, we usually solve this by providing upgrade scripts which modify the configuration after flashing the firmware. Writing these script is often a tedious task, and the scripts easily break when the configuration differs too much from the expected one.

But the ability to change the configuration is important for many Freifunk users: They want to change the role of ethernet ports, WLAN settings, and a lot more. But UCI doesn’t provide information how a setting was changed: if a script encounters an unexpected value, it can’t find out if it is an old default value, or was changed by the user. This often leads to a difficult choice for the script author: either to overwrite the value unconditionally, maybe disregarding voluntary configuration changes, or not to overwrite it, rendering communities unable to change certain configuration during upgrades.

My project aims at solving this by saving the user configuration independently of the defaults provided by packages. This way, a package upgrade can change the default values, but explicit user configuration will not be affected.

Another issue is that the upgrade scripts are usually part of the packages that bring the configuration. Removing a package will leave the configuration behind, which is usually a good thing (for users which know about this and may be interested in the the old config), but for mostly-automatic upgrades, old configuration may accumulate, which can quickly become problematic on devices with very limited flash.

I plan to fix this by basing the new configuration system on schema definitions, which specify which configuration options and values are valid. The schema will probably be based on JSON, as there are already lots of existing systems for defining JSON schemas, which may be used for my project or at least serve as inspiration. This will also finally provide real datatypes for the configuration and make things more consistent (finally no more 1/true/on/yes/0/false/off/no booleans!). If we want too keep the package/section/option organization of UCI, or rather allow defining schemas for any JSON structure, is still a subject of debate.

Instead of going into detail even more in this post, I’ll provide a link to my Gitlab project: https://gitlab.com/neoraider/ece

Design documents, examples for configuration and schemas, and first code will start to appear there very soon Laughing

DynaPoint – A dynamic access point validator and creator

Hi everybody,

 

I am Tobias, a Computer Science student at the TU-Berlin. I am glad to have the opportunity to participate at GSoC for Freifunk this year.

 

My project aims at making the handling with access points in OpenWrt/LEDE easier. The goal is defined as follows: Find an easily configurable solution (with reasonable defaults) for making the wireless access SSID in OpenWrt/LEDE dependent on a set of network conditions.

 

What does that mean? Consider the following example. You have a wireless access point with SSID “Freifunk”. Suddenly for whatever reason the AP looses Internet connectivity without anyone noticing it. When users now connect to this AP, expecting a working Internet connection, they get frustrated, because they can’t check their emails or surf the Internet.

 

With DynaPoint I want to develop a daemon, which regularly monitors the Internet connectivity. When it’s lost, the SSID will automatically be changed. In this example it could be changed to “Freifunk-offline”. When Internet connectivity is re-established, the SSID would automatically be changed back to “Freifunk”.

This way users as well as admins get informed about the state of an access point just by looking at the SSID.

 

To verify Internet connectivity the first obvious step would be to do a ping. For this purpose there already exists a package called pingcheck, which I am planning to use. Further steps could include DNS-Queries and HTTP-Downloads.

 

Speaking about easy configuration and reasonable defaults, I want to require as little configuration steps as possible, but also provide enough configuration options to be adjustable to different kind of setups. Ideally the configuration will also be possible via the LuCI web interface.

 

Until next time,

 

Tobias

SWOON: Simultaneous Wireless Organic Optimization within Nodewatcher

Hi everyone,

I will contribute to one of the Freifunk projects; nodewatcher, via Google Summer of Code this summer and I wanted to keep you updated on my progress as well as exchange thoughts about my ideas.

First of all, nodewatcher is an open source, modular community oriented platform used for network planning, node deployment, node monitoring and maintenance. nodewatcher was initially developed to be primarily used by the wlan slovenija project. With 1336 nodes, it’s really successful and a great example for community networks. As nodewatcher gets deployed elsewhere with even more nodes, it’s natural to ask ourselves if we can be smarter about allocating spectrum to our wireless nodes – these nodes are mostly inexpensive wireless routers but it’s natural to extend the meaning of the term to dedicated wireless access points (i.e. Unifi AP).

The theoretical foundation for this problem is fascinating by itself: Each node has a different amount of noise in each channel (the 2.4GHz band allows 3 non-overlapping channels where each channel is 20MHz wide) and each node wants to maximize its SNR (signal-to-noise ratio). I will term this as the greedy approach, which is already used in enterprise level devices. However, in an urban setting, nodes are close enough to each other for their signal to act as noise to other nodes. The greedy approach is no longer optimal as it bears a high price of anarchy. Instead, our goal is now to maximize the sum of channel capacities (under a power constraint). I will have to devise an algorithm to solve this problem and the algorithm does not seem trivial since the number of combinations is increasing exponentially with the number of nodes in the system. Even with only 10 nodes, we haveover 59000 possible allocations on 2.4GHz band and over 95 trillion on the 5GHz band.

Traditional networking literature tackles this optimization problem with Lagrange multipliers. An alternative is to look at approximate graph coloring schemes and compute chromatic numbers. I hope to experiment across various settings and approaches.
Over the course of the project, I hope to experiment with a real network which consists of at least 10 nodes and measure the improvements. One exciting thing about real life experiments is that nodewatcher was mostly used inside wlan slovenija’s network and I get to run it independently! This will probably allow me to fix some bugs on the way and contribute to nodewatcher in this aspect as well.

The algorithm will initally be developed as a nodewatcher module, but I hope to eventually port it to openwrt (possibly after the summer ends). The main difficulty is that nodewatcher can act as a central level planner, whereas the openwrt scenario requires negotiation among nodes. So it’s harder to convince a node to decrease its TX power to benefit other users. But imagine a network where nodes can communicate and achieve a socially optimal point of spectrum allocation! A glorious future awaits us.