Tag: gsoc

Monitoring and quality assurance of open wifi networks out of client view (final evaluation)

Posted on by Jan Tarek Butt

Hi together,

now the time has come to explain the full Google Summer of Code Project. In both blog posts before I explained the work packages and structure of the Project [0][1]. In the first post I declare the three main subjects. Here is a short overview to remind of the project structure:

sub-projects,

Mainline project

and seminars.

The sub-projects are background work for community projects.

The mainline Google Summer of Code project is to develop a new firmware for routers, based on LEDE [2]. The third point are seminars for enlightment of technical aspects of the Freifunk Community.

 

First I would like to list up all sub-projects and their status.

The first of the sub-projects is the hoodselector. For a final explanation of this construct I would like to explain the following points to give you a good understanding of this concept. On the Nordwest Freifunk community we had one big problem. Due to the batman-adv management traffic, the network setup is not really scalable. This problem also exists on many other communities where they have thousands of mesh routers inside one single network. If there are too many routers inside of one layer 2 network the batman-adv management traffic will flood this network and make it useless.

 

Therefore in the Nordwest Freifunk Community we decided to develop the concept of hood-networking. This concept consists of two main components: The hoodfile [3] and the hoodselector[4]. The hoodfile is a json file containing all informations necessary for a definition of a geobased hood. One hood is defined by a geostationary fixed quadrants, VPN-peering- and wireless configurations. Inside a hoodfile are multiple hoods defined[5]. Following an example of a definition of a hood:

 

{

     “name”: “rastede”,

     “bssid”: “02:00:0A:12:A0:00”,

     “defaulthood”: false,

     “servers”: [

         {

             “host”: “ras01.sn.ffnw.de”,

             “port”: “10000”,

             “publickey”: “ca1b5487ffc20a1f90e0ac14e835d84ab9e52612b5ca62e073d0a13dad98775e”

         }

     ],

     “boxes”: [

         [

             [

                 53.22,

                 8.09

             ],

             [

                 53.36,

                 8.32

             ]

         ]

     ]

},

 

A hood has the following definition:

 

– name

The name describes the region depending on its geo coordinates. For example, if you create a hood over the city Oldenburg(Oldb) (Germany), a good name could be Oldenburg or ol as short. Every name has to be unique inside a hoodfile. Redundant names are not allowed!

 

– bssid

The bssid will be set for the adhoc wireless interface. This is the main part of splitting the layer 2 network. Inside the bssid there is the IPv4 sub-network encoded which is in use inside the hood. In the above json part the following IPv4 sub-network is encoded “10.18.160.0/21” dec to bin => “0000’1010 0001’0010 1010’0000 0000’0000” bin to hex => 0x0A 0x12 0xA0 0x00 hex to mac => 02:00:0A:12:A0:00. Therefore the bssid should also be unique.

 

– defaulthood

The defaulthood boolean is only true on the default hood. The default hood doesn’t have coordinates and is the inverted form of all other hoods with geo coordinates.

 

– servers

Contains an array of VPN connection informations. These informations are:

VPN server address (host)

VPN port

VPN crypto key

One VPN server should be used for one hood only! If two hoods have the same VPN server, batman-adv will loop them over VPN.

 

– boxes

This 3 dimensional array describes the geographical size of the hood. The surface is rectangular. You just need two points per box to reconstruct it. Here is an example:

 

 

                                          53.22,

                                          8.09 #____________

                                                   |                  |

                                                   |                  |

                                                   |                  |

                                                   |____________|

                                                                      # 53.36,

                                                                          8.32

 

 

Each hood can have any number of rectangles inside the boxes array.

 

To make your life a little easier you can use the hoodgen[6] and source[7] to write your json with the required informations. This is a simple web visualization tool to plan hoods and generate the right json format for the hoodfile. This tool has been written by Eike Baran. Big Thanks to him for this helpful tool!

 

Now to the hoodselector.

It is a software that creates decentralized, semi automated ISO OSI layer 2 network segmentation for batman-adv layer 2 routing networks. This program reads the geobased sub-networks called hoods from the above mentioned hoodfile. The decision of choosing the right hood is made on following points: first, the hoodselector checks, if the router has a VPN connection. If it has, the hoodselector then checks, if a static geoposition was set on the router. If not, it tries to get a position using wireless based localization with the so called geolocator. The geolocator [8][9][10] is a software which makes it possible to receive a position based on wireless networks “seen” around. These informations will be sent to the openwifi map project [11]. Knowing the position of the router the hoodselector can find the right hood, because each hood is defined with geocoordinates. If the Router doesn’t have a VPN connection e.g. as a mesh only router, the hoodselector triggers a WIFI scan and searches for neighboured mesh routers in other hoods. If there is an other router with a different BSSID but with the same mesh SSID, the router chooses it’s hood based on the neighboured BSSID. I got much positive feedback from many other Freifunk Communities. Someone even created a integration request issue for gluon [12]. Gluon is a framework based on openWRT[13] and is very popular in the Freifunk community [14]. Before I will send this as a patch to gluon there remains one last urgent issue [15]. The current hoodselector is not able to handle mesh on LAN or WAN connections. So there is still a potential point of failure. Because persons who are not familiar enough with the hood-networking concept can accidentally interconnect hoods over the mesh on cable functions. I plan to fix this problem up to mid of september. When this issue is closed I would like sending patches for integration to gluon. Other issues can be found here[16].

 

On the Nordwest Freifunk community we currently have 10 active hoods including a default hood. That is a special hood where all routers will connect to, if they are not able to choose a hood, including also routers there out of ranges from other real hoods. After the last 3 months we can safely say that the setup works. Commits can be found here[17] All currently active hoods can be see here in this picture.

 

 

 

As next I would like to tell you about the second sub-project as a prework for the mainline project. In that part I work on a proper workaround with the continuous-integration (CI) system of Gitlab[18]. As I explained in the midterm evaluation, on our Nordwest community we started automatically building of nightly testing firmware images for our community firmware. The CI works now with a dynamical multiple core build processes and auto generated architecture targets out of source. At the moment it is not possible for Gitlab to handle high verbose inside the web-engine while the build process. I discussed the problem with the gitlab team and open an issue [19]. The CI builder is very helpful for the developing process of the monitoring drone. Here you can see the result for the local community image[20] and for the monitoring-drone [21].

 

Mainline Project.

The mainline project was to create a new firmware for monitoring and quality assurance of open wireless networks. So I started reading of informatins about openWRT [13] and LEDE [2]. I decided to use LEDE as base system. I know there still no release to use this as a defined base structure (we are all looking forward to this moment) but since july 2016 I am on the developer list and the way where LEDE is growing looks good. Next I looked for a build management script. First I thought about using make and Makefiles but this was not my favourite, so I decided to use the buildscript from the Franken Freifunk community as base, which is written in bash. Now I’ll explain the structure how to work with and use it. The following directories and files are important for basic work:

 

buildscript ← File

     BSP ← Dir

     Community ← Dir

     build_patches ← Dir

     modules ← File

 

The buildscript is mainly a bash script for a humanly working with this buildroot. In other words it is an abstraction from the LEDE build ENV.

 

Inside the BSP directory are all necessary architecture specific informations.

BSP means Board-Support-Package. Also inside this directory are default informations like the shell banner system configs and so on.

 

The community directory includes community specific configurations, similar like the gluon siteconf [22]. Currently there are only two config parameters inside: first the “AP SSID” to set a default SSID with witch WIFI network should the monitoring-drone connect and the second parameter, the “AP BSSID” to set a node specific BSSID in case if more than one router with the same SSID is present. Then the monitoring drone is pinned on one specific node. This config parameter will be dropped in the future because it is not really effectively if a default BSSID is set . In future I plan to configure thous parameters over an extra web interface[23].

 

In the build_patches directory you can put patches for LEDE or if you what you can also put patches for each package repository. Here is a schemata:

 

build_patches

     lede

         0001-this is a patch.patch

         0002-this is another one.patch

     ffnw-packages

         0001-this is a patch.patch

         0002-this is another one.patch

 

The last file is called modules. Inside this file you can add external package repositories and also select specific packages out of this repositories. Following an example:

 

OPENWRT=(ffnw-packages

     $PACKAGEURL

     $PACKAGEREV)

OPENWRT_PKGS=”libwlocate lwtrace ffnw-node-info hoodselector”

 

Clemens and I discussed about the API design[24] regarding the communication between the monitoring drone and the netmon core. So we met together. Here a picture:

Jan-Tarek Butt (left) and Clemens John (right)

 

At last point were the seminars. During the Google Summer of Code we started to gave seminars for technical aspects of Freifunk because we have not enough developers and system administrators. That is mostly a big problem in volunteer activities. On the hacking sessions we follow a simple structure:

 

– two lectures about Freifunk technical aspects.

– discus about the contend of the lectures

– work session on projects

 

On the first hacking session at the 28. may 2:00 PM we created video recordings of the two lectures, you can find them here[25]. The next hacking session were failures because of Clemens and my exams. In future there will follow other streams about tecnical aspects of Freifunk.

 

Last but not least, my future plans:

For the hoodselector, I plan to close up the last urgent issue[15] that I mentored in the above for the migration into gluon I also started prework on gluon for this. One part was the implementation of a sequential code minifying process at compile time [26]. Also some other issues are still open so I will continuing the work on the hoodselector[16].

 

For the monitoring firmware they currently is just configurable over ssh. A web interface should follow soon and also a plugin system for community specific monitoring data requests.

 

On the Kieler Linux information days[27] inside the Kieler Innovations- and technology center I will hold amongst others 4 presantations about Freifunk relevants themes:

 

    Hoodselector – Network segmentation for Layer 2 routing at 11:00 (16.09.2016)[28]

    Wireless-based localization (openwifi.su project) at 13:00 (16.09.2016)[29]

    OpenWRT Embedded Linux distribution at 16:00 (16.09.2016)[30]

    Freifunk Kiel/Nordwest (2016) – year review at 16:00 (17.09.2016)[31]

 

[0] http://blog.freifunk.net/2016/monitoring-and-quality-assurance-open-wifi-networks-out-client-view

[1] http://blog.freifunk.net/2016/monitoring-and-quality-assurance-open-wifi-networks-out-client-view-midterm-evaluation

[2] https://lede-project.org/

[3] https://git.nordwest.freifunk.net/ffnw-firmware/packages/blob/master/hoods/files/lib/ffnw/hoods/hoods.json

[4] https://git.nordwest.freifunk.net/ffnw-firmware/packages/blob/master/hoodselector/luasrc/hoodselector

[5] https://git.nordwest.freifunk.net/ffnw-firmware/packages/commits/master/hoods

[6] http://hood.ffnw/hoodgen.html#

[7] https://git.nordwest.freifunk.net/ffnw-server/hoodgen

[8] https://git.nordwest.freifunk.net/ffnw-firmware/packages/tree/master/libwlocate

[9] https://git.nordwest.freifunk.net/ffnw-firmware/packages/tree/master/lwtrace

[10] https://git.nordwest.freifunk.net/ffnw-firmware/packages/tree/master/ffnw-node-info

[11] http://openwifi.su/

[12] https://github.com/freifunk-gluon/gluon/issues/789

[13] https://openwrt.org/

[14] http://gluon.readthedocs.io/en/latest/

[15] https://git.nordwest.freifunk.net/ffnw-firmware/packages/issues/63

[16] https://git.nordwest.freifunk.net/ffnw-firmware/packages/issues?label_name%5B%5D=hoodselector

[17] https://git.nordwest.freifunk.net/ffnw-firmware/packages/commits/master/hoodselector

[18] https://about.gitlab.com/gitlab-ci/

[19] https://gitlab.com/gitlab-org/gitlab-ce/issues/18039

[20] https://git.nordwest.freifunk.net/ffnw-firmware/siteconf/builds

[21] https://git.nordwest.freifunk.net/ffnw-firmware/monitoring-drone/builds

[22] https://github.com/freifunk-gluon/gluon/tree/master/docs/site-example

[23] https://git.nordwest.freifunk.net/ffnw-firmware/monitoring-drone/tree/master

[24] https://git.nordwest.freifunk.net/groups/netmon-sc

[25] https://www.youtube.com/channel/UCX0nJzimLNF38pfgQIuZLEQ

[26] https://github.com/freifunk-gluon/gluon/commits/master?author=2tata

[27] http://www.kilux.de/

[28] http://www.kilux.de/index.php?seite=programm.html&untermenu=Besucher-Info#248

[29] http://www.kilux.de/index.php?seite=programm.html&untermenu=Besucher-Info#246

[30] http://www.kilux.de/index.php?seite=programm.html&untermenu=Besucher-Info#247

[31] http://www.kilux.de/index.php?seite=programm.html&untermenu=Besucher-Info#263

GSoC: The ECE configuration system – summary

Posted on by NeoRaider

The Google Summer of Code is almost over, so in this blog post I’ll give a overview over the targets I’ve met (and those I haven’t).

Code repositories

  • https://gitlab.com/neoraider/ece/commits/gsoc2016 (daemon, client libraries, CLI client)
  • https://gitlab.com/neoraider/pkg-ece/commits/gsoc2016 (OpenWrt/LEDE package feed)
  • https://gitlab.com/neoraider/uci-ece/compare/gsoc2016-upstream…gsoc2016 (UCI ECE backend)

All code in the first two repositories has been developed by me during the GSoC. The third link shows the work I’ve done to integrate a ECE backend into libuci.

What is working

As described in earlier posts, my GSoC project was a configuration storage system for OpenWrt/LEDE, trying to solve various issues of the UCI config system. The principal points of this new system are

    • ubus-based config daemon maintaining a central storage database file
    • JSON-based configuration data model
    • Validation based on simplified JSON-Schema

The Wiki at https://gitlab.com/neoraider/ece/wikis/home gives a good overview of the design and the usage of ECE and describes many features in detail. The pkg-ece package feed can be used to build and install the different components of ECE on OpenWrt/LEDE easily.

If you’ve worked with OpenWrt/LEDE, you probably know the UCI config system. A UCI config file looks like this:

config system
        option hostname 'lede'
        option timezone 'UTC'

This format is very simple: Each file (called “package”) has a number of sections (named or unnamed) of different types (this example from the “system” package has a single unnamed section of type “system”). These section contain options with single values or lists of values.

Unnamed sections are usually accessed using indices, for example a command to set the hostname would look like this:

uci set system.@system[0].hostname='betterhostname'

With the simplicity of UCI, there come various issues and missing features; these are only a few of them:

      • The fixed data model (package/section/option) makes some kinds of configuration very awkward: In the example above, the index 0 must be given for the system section, but having a second section of this kind would not make sense. In other cases, deeper configuration trees must be flattened to be stored in UCI, making the configuration harder to understand
      • All values in UCI are strings, which often causes inconsistencies (booleans are usually stored as ‘0’/’1′, but several other pairs like ‘false’/’true’ and ‘off’/’on’ are supported as well; different users of UCI sometimes parse numbers differently)
      • UCI doesn’t have built-in validation. Frontends like LuCI usually validate the entered data, but as soon as the CLI client is used, no validation is done.
      • UCI always stored the whole configuration file and not only changes from the defaults, making the storage inefficient on overlay-based filesystem setups as they are common on OpenWrt/LEDE
      • In some situations, upgrades to default values should also affect the effective values; but only if the user didn’t change the values themselves. With UCI, this is not possible, as it doesn’t store the information if a value was changed by a user.
      • UCI allows comments in config files, but they are lost as soon as libuci or the CLI tool is used to modify it

The configuration given above could be represented in ECE as this JSON document:

{
  "system": {
    "hostname": "lede",
    "timezone": "UCI"
  }
}

Note that this is only the external representation of the configuration; internally, it is stored in a more efficient binary format.

JSON gives us a lot of features for free: arbitrary configuration trees with proper data types. Existing standards and standard drafts like JSON Pointer and JSON Schema can be used to reference and validate configuration (the JSON Schema specification is simplified for ECE a bit though to allow more efficient validation on embedded systems).

The command for changing the hostname would look like this in ECE:

ece set /system/hostname '"betterhostname"'

The quoting is currently necessary to make the string a valid JSON document; this may change in a future version.

The whole configuration is saved in a single JSON document, but the specific format is not defined by a single schema; instead, each package can provide a schema, and the configuration tree is validated against a merged schema definition.

The schemas also provide default values for the configuration. Adding documentation for the configuration options to the schemas is planned as a future addition and might be used to support the user in configuration utilities and automatically generate web-based or other interfaces.

This gives only a small example for the usage of ECE, the abovementioned ECE Wiki contains much more information about the usage of ECE and the ideas behind it.

In addition to the daemon and a simple CLI utility, I’ve developed libraries for C, Lua and Shell which allow to access the configuration. While there are still some features missing (some points for future work are given in https://gitlab.com/neoraider/ece/wikis/todo ), I think most of the missing pieces can be added in the near future.

The UCI/ECE bridge

When I proposed my project for the GSoC, I didn’t aim at making it a full replacement for the current UCI system, at least not in the near future. While the possibility to move some of UCI config files into the ECE config database had been my plan from the beginning, my ideas for backwards compatibility didn’t go further than a one-time import from UCI to ECE, and one-way generation of UCI config files from ECE.

After talking to a few LEDE developers and package maintainers, it became clear to me and my mentors that many people are interested in replacing UCI with a better system in the not-too-far future. But for ECE to become this replacement, a real two-way binding between UCI and ECE would be necessary to allow gradual migration, so configuration utilities like LuCI (and many other utilities somehow interacting with UCI) don’t need to be adjusted in a flag-day change.

An incomplete design draft for this UCI/ECE bridge has been outlined in https://gitlab.com/neoraider/ece/wikis/design/uci-bridge . The code found in the UCI ECE backend repository linked above implements a part of this bridge (it can load “static” and “named” bindings from ECE into UCI, and commit “static” bindings back to ECE) and has been implemented as an API- and ABI-compatible extension to libuci. The development of this bridge has taken a lot of time (much more time than I had originally scheduled for UCI compatibility features), as the data models of UCI and ECE are very different.

Future work

Of all points given in https://gitlab.com/neoraider/ece/wikis/todo , finalizing the database format is the most important, as any future change in the storage format will either break compatibility or involve some kind of conversion. When it is clear the format won’t be changed anymore, ECE should be added to the OpenWrt community package repository to make it easily accessible to all OpenWrt and LEDE users.

After that, other points given in the TODO should be dealt with, but none of those seem too pressing to prevent actually using ECE for some software (but some of the points given in the first section of the TODO page would need to be addressed to properly support software that requires more complex configuration).

Last, but no least, I’d like to express my gratitude to my mentors and all people in the OpenWrt, LEDE and Freifunk communities who have helped me develop ECE by giving guidance and lots of useful feedback, and to Google, who allowed me to focus on this project throughout this summer.

Implementing Pop-Routing – Midterm Updates

Posted on by Gabriele Gemmi

 

Hi Everyone!

Today has started the midterm evaluation, the deadline Is next Monday, so I have to show the work I have done ‘till now. It can be resumed in the following parts:


1) Refactoring of graph-parser and C Bindings

During the community bonding period I started working on the code of Quynh Nguyen’s M.Sc. Thesis. She wrote a C++ program capable of calculating the BC of every node of a topology [1]. I re-factored the code, and now it is a C/C++ shared Library [2]. I’ve also applied some OOP principles (Single responsibility and inheritance) and unit tests to make it more maintainable.

The interface of the library Is well defined and it can be re-used to implement another library to perform the same tasks (parsing the json and calculating the BC).


2)Prince Basic functionalities

After I completed the library a started working on the main part of the project. the daemon. We decided to call it Prince in memory of the Popstar.

This daemon connect to the routing protocol using the specific plugin (see below), calculate the BC using graph-parser, computes the timers and then it push them back using again the specific plugin. With this architecture it can be used with any routing protocol.I wrote the specific plugin for OONF and OLSRd. At the moment it has been tested with both, but I need to write a plugin for OLSRd to change the timers at runtime. For OONF I used the RemoteControl Plugin.

With these feature Prince is capable of pulling the topology, calculate the BC and Timers and push them back to the routing protocol daemon.

 

3) Additional Features: Configuration file, Dynamic plugins,

I wrote a very simple reader for a configuration file. Using the configuration file the user can specify: routing protocol host and port, routing protocol (olsr/oonf), heuristic, (un)weighted graphs.

As you can see from this Issue [3], I’m going to use INI instead of this home-made format.

As I said before I moved to a specific plugin all the protocol specific methods (pulling the topology and pushing the timers), to keep the daemon light I decided to load this plugin dynamically at runtime. So if you specify “olsr” in the configuration file just the OLSRd specific plugin will be loaded.

 

 

At the moment I consider this an “alpha” version of Prince. In the next 2 months I’ll be working on it to make it stable and well tested. The next steps will be:

 

  • Close all the Issues [4]
  • Write tests and documentation for Prince.
  • Write a plugin for OLSRd

 

Cheers, Gabriel

 

[1]: https://ans.disi.unitn.it/redmine/projects/quynhnguyen-ms

[2]: https://github.com/gabri94/poprouting/tree/master/graph-parser

[3]: https://github.com/gabri94/poprouting/issues/4

[4]: https://github.com/gabri94/poprouting/issues

GSoC: The ECE configuration system – midterm update

Posted on by NeoRaider

Hi,
the GSoC is nearing its midterm evaluation, so let’s have a look at the current state of the “Experimental Configuration Environment” (working title)!

I started the project with defining models for configuration data, configuration diffs (that are used to modify data and store modifications) and schemas. All of the design documents I wrote can be found at: https://gitlab.com/neoraider/ece/wikis/design

I tried hard not to re-invent things that already exist in a more or less standardized way:

  • My configuration model is just JSON (without support for floating-point numbers right now, as the libubox blobmsg format ubus uses to represent JSON doesn’t support them – I plan to fix this)
  • JSON Pointers (https://tools.ietf.org/html/rfc6901) are used to refer to elements of the configuration tree
  • The schema format uses a subset of JSON Schema (http://json-schema.org/)

Based on this design, I’ve started implementing the configuration management daemon eced. The page https://gitlab.com/neoraider/ece/wikis/implementation gives a good overview of the current features and usage of eced. Quite a lot is already working:

  • Multiple schemas can be loaded and merged
  • Default configuration is generated from the schemas
  • A ubus interface allows to query and modify the configuration
  • Config modifications can be loaded from and stored into a diff file

There’s still a lot missing, so here’s what I plan to do next:

  • Make something use ECE! UCI is well-integrated in OpenWrt/LEDE and can be used from C, Lua and shell scripts; all of these languages should also be supported by an ECE client library. An example schema to replace /etc/config/system does already exist and could be used to experiment with partially replacing UCI with ECE.
  • Support schema upgrades: When schemas are updated, this can partially invalidate config diffs. A way to deal with such inconsistencies must be defined.

Discussions and feedback have also led me to the decision to put a stronger focus to UCI interoperability. While I had only planned import and export of UCI configuration at first, I’ve realized that a two-way binding between UCI and ECE (i.e. an API-compatible libuci replacement/extension that will use ECE as backend) will be necessary for ECE to find acceptance. This will also allow to continue to use existing configuration tools like LuCI with UCI configuration converted to ECE.

I also plan to have OpenWrt/LEDE packages for ECE ready soon, so you can play around with the current implementation yourselves. Maybe I’ll also set up a public host with ECE Laughing

Monitoring and quality assurance of open wifi networks out of client view (midterm evaluation)

Posted on by Jan Tarek Butt
Hey everyone,
 
Now we are on the midterm evaluation. I would like to tell you what I have done so far and what will come next. In the first post [0] I explained the work packages. In this post I will come back to the work packages  and show you what I have done for each package.
 
The first sub-project was the hoodselector. At the beginning of the work period I did some bugfixing for the hoodselector so that we where able to deploy the hoodselector in our live environment. The hoodselector creates decentralized, semi automated ISO OSI layer 2 network segmentations. You can find a detailed discription here [0]. Retrospective I can say that the deploymend of the hoodsystem went without any major problems opposed to my first expectations. Currently we have 4 hoods active. Around Oldenburg(Oldb), Ibbenbüren, Osnabrück and Friesland. More hoods will follow in future. Open Issues can be find here [1].
 
The second sub-project was to create a propper workaround for building images with the continuous-integration (CI) system of Gitlab using make on multiple cores.
The Freifunk Nordwest firmware now has automatically built testing images that are not only build on a single core but can be built on multiple cores. And the architecture targets are also autogenerated out of the sourcecode. This makes it possible to generate images dynamically for all targets also including new targets that may come in the future. I implemented a small algorithm that manages the thread counter of make commands. I use the number of CPUs out of /proc/cpuinfo * 2 this means for each logic core will follow two threads. In example our runner02.ffnw.de server has 8 cores so the CI build process will automatically build with 16 Threads [2]. Here is an example of a passed buildprocess with our CI builder[3]. Actually it is not possible to build images with a high verbose output, because the CI logfiles will get to big. That makes it impossible to use the webfrontend for analyzing the buildprocesses. I opened an issues for this and discussed the problem with the gitlab developers [4].
The CI builder is very helpful for the developing process of the monitoring drone.
 
Following I would like to report about our first hacking seminar.
The first hacking seminar was on 28.05.2016. We started with two presentations. One about Wireless Geo Location and the second one about the Hoodselector. We recorded the presentation with our new recording equipment [7] bought using some of the money for the mentoring organisation and uploaded the recordings to youtube [5].
 
The first presentation was about geolocating with wireles technologies.
Based on the Nordwest Freifunk geolocator [6]

The second presentation was about the function of the Hoodselector

 
After this two presentations we had a smal disscussion about the presentation topics and than we started our hacking session where the developers started coding on their projects.
 
Now all sub-Projects are finnisched and I will continue with the Monitoring Drone Project after I finish my Study exams. Also the date of next hacking seminar is set for 9th of Juli 2016. Again we will have two presentations. One on Gitlab CI and one about how to use our new Puppet Git repositories including the submodule feature. The presentations will be recorded and after the presentation we will have a coding session like last time.
 
Timeline:
23. May: Community Bonding (3 weeks)
test and deploy hoodselector  <- Done
16. May 6:00 PM: GSoC Mumble  <- Done
Refine the roadmap  <- Done
23. May – 20. June: Work period 1 (4 weeks) <- Done
28. May 2:00 PM: hacker-session  <- Done
  1. Presentation about the hoodselector <- Done
  2. Presentation about the openwifi.su project[4] and the geolocator <- Done
13. June 6:00 PM: GSoC Mumble  <- Done
Midtermevaluation
Tarek & Clemens exams!!!
20. June – 15. August: Work period 2 (8 weeks)
9. July 2:00 PM: hacker-session
  1. Presentation about workaround with git CI processes.
  2. Presentation about puppet deployment system
13. June 6:00 PM: GSoC Mumble
25. June 2:00 PM: hacker-session
  1. Presentation about workaround with git CI processes.
  2. Presentation about puppet deployment system
18. July 6:00 PM: GSoC Mumble
30. July 2:00 PM: hacker-session
  1. actual unknown
  2. actual unknown
15. August 6:00 PM: GSoC Mumble
Finalevaluation
 

DynaPoint update

Posted on by Tobias Ilte

Hi everyone,

here are some updates regarding DynaPoint. The idea was to create a daemon, which regularily checks the Internet connection and changes the used access point depending on the result. That way the handling with APs would become easier as you already could tell the status by the AP’s SSID.

A daemon with basic functionality is already working. After installation, there is one configuration step necessary.
You have to choose in /etc/config/wireless which AP should be used, if Internet connectivity is available and which one if the connectivity is lost. You can do that by adding “dynapoint 1” or “dynapoint 0” to the respecive wifi-iface section.

You can also configure dynapoint via LuCI, although it’s not yet complete.
I was struggeling a bit with it, because the documentation of LuCI is a bit minimalistic…
Here is a screenshot of how it currently looks like:

Next steps

To verify Internet connectivity, it is probably better to make a small http download than just ping an IP address.
Using “wget –spider” should be suitable for that.

Also, I will see if I can get rid of the required configuration step in /etc/config/wireless in the next weeks and provide fully automatic configuration.

If you want to test dynapoint for yourself, just go to https://github.com/thuehn/dynapoint.

Overview: GSoC projects at freifunk.net

Posted on by Andreas Bräu

In 2016 we found 9 students doing Google Summer of Code projects for freifunk. This is an overview on all our projects including links to repositories and further information:

Title Student Repository Project information
A schema-based configuration system for OpenWrt Neoraider  https://gitlab.com/neoraider/ece  
DynaPoint – A dynamic access point validator and creator for OpenWrt Asco  https://github.com/thuehn/dynapoint
https://github.com/thuehn/luci-app-dynapoint 		  
Implementing Pop-Routing Gabriel https://github.com/gabri94/poprouting http://firenze.ninux.org/
Monitoring and quality assurance of open wifi networks: the client view Tarek https://git.nordwest.freifunk.net/ffnw-firmware/monitoring-drone   
netifd extension: external device handlers arkap    
OpenWrt – poWquty (poWer quality) Neez   https://github.com/thuehn/powquty https://github.com/neez34/
Provide a cryptographic implementation for Qaul.net spacecookie  https://github.com/spacekookie/qaul.net  https://spacekookie.de/gsoc2016/
Sharable Plugin System for qaul.net – Mesh Network Communication App Anu  https://github.com/anuvazhayil/qaul.net https://github.com/anuvazhayil 
SWOON: Simultaneous Wireless Organic Optimization within Nodewatcher marn 

https://github.com/CdavM/nodewatcher

https://github.com/CdavM/swoon 

  

Blog posts on these projects you can find with the tag GSocC 2016. Use the RSS feed to receive all updates. You can also see our projects from previous summers.

GSoC 2016 Introduction: external device handlers for netifd

Posted on by ArneKappen

HELLO WORLD

It’s time I started blogging about my Google Summer of Code project with Freifunk.

To begin, let me introduce myself. I am Arne, computer science student at TU Berlin and pursuing my Bachelor’s degree. This is my first GSoC — and the first time I contribute to an Open Source software project, so, naturally, I’m pretty excited to get to know the process.

My project aims at extending OpenWRT/LEDE’s network interface daemon (netifd).

I’ve familiarize myself with the inner workings of netifd while I was working on a student project last semester. The result of that student project will assist me in realizing the GSoC project. Moreover, the existing source code will partially build the foundation of the new device handler which will be realized in this project.

 

THE PROJECT

Here’s the general idea: Netifd allows the user — as the name suggests — to create and configure network devices and interfaces. Per device class, there is a hard-coded structure called a ‘device handler’. Each device handler exposes an interface for operations such as creation or re-configuration of device instances of its class.

The point I’m tackling is the structures hard-codedness. Today, whenever someone wants to introduce a new device class, is necessary to alter the netifd code base and maintain these changes. The proposed mechanism allows to realize external device handlers as separate programs communicating via the bus system ubus. Accordingly, to introduce a new device class into netifd, one just needs to implement device handling logic and link it to netifd. Thus, no maintenance — or knowledge of the innermost workings of netifd — is necessary.

Building on my work from the aforementioned class I’ll write a device handler to integrate Open vSwitch support into OpenWRT/LEDE using netifd. The ‘ovsd’ as it is called will relay calls to the device handler interface via the ‘ovs-vsctl’ command line utility to set up Open vSwitch bridges and ports.

 

MY DEVELOPMENT ENVIRONMENT

I am hosting my code in a git repository that’s included in the GitLab of the INET department which is employing me at TU Berlin. Once the code becomes usable, it will appear on GitHub here.

We also have a repository providing an OpenWRT/LEDE feed that bundles the Open vSwitch software with ovsd: (link will follow)

 

Testing is done on a PC Engines Alix 2 board which is part of our testbed:

 

WHERE I AM AT

As of today I have realized a mechanism to generate device handler stubs within netifd when the daemon is first started and before it parses /etc/config/network. The device class’ parameters, such as its name in the ubus system and most importantly the format of its configuration, are stored in JSON files in /lib/netifd/ubusdev-config/. The logic to parse a configuration format from JSON files existed already within netifd but was only used for protocol handlers before. I adapted it to generate device handlers dynamically.

When such a device handler is generated, it subscribes to its corresponding external device handler using ubus. This way, the latter can generate events asynchronously for the former to react to.

In order for this to work, the external device handler must be up and running before netifd attempts the subscription.

I also realized a simple proof-of-concept implementation that works in one simple case: create an ovs-bridge with one or more interfaces like this example /etc/config/network file demonstrates:

 

config interface ‘lan’

    option ifname ‘eth0’

    option type ‘ovs’

    option proto ‘static’

    option ipaddr ‘172.17.1.123’

    option gateway ‘172.17.1.1’

    option netmask ‘255.255.255.0’

This will create an Open vSwitch bridge called ‘ovs-lan’ with the port ‘eth0’ — and that’s about all it does right now. Error handling is missing and more complex options like setting OpenFlow controllers or adding wireless interfaces don’t work yet.

 

THE ROAD AHEAD

Among the first things I’ll tackle when GSoC’s coding phase begins are adding the possibilities to incorporate WiFi-interfaces to the bridges created with external device handlers and introducing the missing parts of the device handler interface. Since I’m dealing with netifd and the external device handler which is an independent process running asynchronously, getting their state to stay in sync will take some work. I need to think about the way the callback system that comes with ubus’ pubsub-mechanism will function. In the end, it is supposed to be abstract enough to work not only for my Open vSwitch use case but for any external device handler.

Then, further into the coding phase, I want to increase feature coverage of the Open vSwitch options, so that users can assign OpenFlow controllers for their bridges like so:

 

config interface ‘lan’

    option ifname ‘eth0’

    option type ‘ovs’

    option proto ‘static’

    option ipaddr ‘172.17.1.123’

    option gateway ‘172.17.1.1’

    option netmask ‘255.255.255.0’

    option of-controller ‘1.2.3.4:5678’

In the end I would like to have the mechanism implemented and documented so that others can build on it. Documenting my work is especially important to me, because I found myself frustrated with the scarcity of the netifd documentation more than once before. I wish for my code to become a part of OpenWRT/LEDE and be extended and improved in the future and I would like to make it easy for anyone insterested to doing that with a helpful manual and commented code.

I’m excited to get started and hope to find the time to blog about my progress here, regularly.

Monitoring and quality assurance of open wifi networks out of client view

Posted on by Jan Tarek Butt
Hey everyone,
 
My name is Jan-Tarek Butt and I am in my second term of computer science at Emden University in Lower Saxony (Germany). I am one of the students that are participating for Freifunk in the Google Summer of Code 2016. In the following post I would like to introduce you to my Google Summer of Code project. I split up the project into 3 main subjects: Mainline project, sub-projects and seminars.
 
Before I explain the three mentioned subjects I would like to give you some background information about the project in general. My project mentor is Clemens John. He studies computer science at University Osnabrück in Lower Saxony (Germany) and recently started to write his bachelor thesis. As part of his bachelor thesis he will build a monitoring and administration software for open wireless networks called “Netmon-SC” based on a previosly existing software that will be rewritten to follow a decentralised idea. The coarse structure of Netmon-SC will consist of a core module as a data storage backend that can be queried by using a REST API based on NetJSON. In addition to the REST API the core module will include a plug-In system which developers can use to easily extend the core module to build data storages for creating visualisation tools, quality assurance metrics or any other community specific tools like special administration applications.
 
My mainline GSoC project is to develop a new firmware for routers based on openWRT or LEDE. This firmware will  be the basis for an application to monitor the clients view onto an open wireless network. From now on we will call this firmware the “Monitoring-Drone”. This monitoring firmware will gather quality assurance metrics and send them to Netmon-SC. This metrics will help developers and administrators to enhance the quality of open wireless networks and find bugs more easier. The firmware will include an API package to communicate with Netmon-SC. It will also contain a small LUCI web-interface for basic configuration e.g. wireless network settings etc. Additionally to the basic settings it should be possible to integrate small plugins in form of  .sh or .lua files witch contain custom commands.This will allow communities to gather network specifics metrics without compiling community specific firmware images. The API for communication with Netmon-SC, the web-interface and the plugin system will embedded into a package bases structure. The git repository for this project can be found here.
 
Now to the sub-projects. Sub-projects are small projects adjacent to the mainline project. The first sub-project is the so called hoodselector that I finished reviewing on Mai 21th. The hoodselector creates decentralized, semi automated ISO OSI layer 2 network segmentations based on geostationary fixed quadrants for batman-adv mesh networks. It detects in which quadrant the node is in by using wireless location services and configurates the router using the settings that have been stored for this quadrant. In conclusion the hoodselector enables us to build scaled decentralised mesh-networks. It is a small program on open wireless routers on the Nordwest-Freifunk community network.
 
The second sub-project is, to create a proper workaround for building images on a multicore system instead of a single core system by using Gitlabs continuous-integration (CI) feature. This will fully automate the firmware image building process for the Monitoring-Drone and also for the open wireless firmware images from the local community.
 
The third sub-project are regular seminars. The idea of this seminars is to give earned knowledge to the public and also acquire new developers for open wireless projects. The seminars should have one or two short presentations about technical processes from open wireless networks, for instance how the hoodselector works or how batman-adv works. The Presentations will be video recorded and uploaded to the internet. After thous presentations on the seminars I plan short discussions about the contend of the presentations. Afterwards hack-sessions should do force forward the developing processes for the Google Summer of Code project and illuminate the opensource software development scene of the open wireless project.
 
Last but not least we created a project timeline. Clemens and I, will have regular Mumble meetings during the GSoC, at the middle of every month. On this meetings we will discuss the work already done and what should happen next. Beside the work on the main project we will also use the meetings to plan the next seminar that will always follow at the end of each month.
 
Timeline:
23. May: Community Bonding (3 weeks)
test and deploy hoodselector  ← Done
16. May 6:00 PM: GSoC Mumble ← Done
Refine the roadmap ← Done
23. May – 20. June: Work period 1 (4 weeks)
28. May 2:00 PM: hacker-session
  1. Presentation about the hoodselector
  2. Presentation about the openwifi.su project and the geolocator
Midtermevaluation
20. June – 15. August: Work period 2 (8 weeks)
Tarek & Clemens exams!!!
13. June 6:00 PM: GSoC Mumble
25. June 2:00 PM: hacker-session
  1. Presentation about workaround with git CI processes.
  2. actual unknown
18. July 6:00 PM: GSoC Mumble
30. July 2:00 PM: hacker-session
  1. actual unknown
  2. actual unknown
15. August 6:00 PM: GSoC Mumble
Finalevaluation
Cheers
Jan-Tarek Butt

Sharable Plugin System for qaul.net

Posted on by AnuV

Hi everyone,

I’m Anu Vazhayil, exchange student at University of Paderborn from India. I am one of the Google Summer of Code participant for the project qaul.net. 

Qaul.net is a mesh network communication app which can be run in Linux, OSX, Windows and Android. It can be used to send text messages, file sharing and even make voice calls with people who are connected to this network without being connected to internet or cellular networks. There is also a feature to share your internet with the other users connected to the qaul.net network. All this makes it a cool app which everyone will feel to try atleast onceWink.  If you have not installed it in your system, you can still connect to the network via web interface after connecting to a wifi.

As part of the GSoC project, we are planning to extend the usability of the messaging system of qaul.net. The messaging system can be used as a communication layer for other purposes like location sharing, sending contacts etc. Such apps/extensions can be downloaded by the user as a zipped file which will automatically get downloaded to the web plugin directory available to the web server. It also proposes an additional feature to share the installed plugins with other users in the network over file sharing.

The following needs to be implemented for the project:

  • Plugin API: to describe the features of a library and how to interact with it. It defines a mechanism to use the messaging system for plugins and give it access to the qaul.net topology.
  • Plugin structure: a zipped folder with a special extension that can be shared with other users in the network over file sharing.
  • Plugin management function: Installation and plugin sharing. Once the Plugin is installed, it is copied and extracted to a directory in the web server.
  • Sample Plugin: Share the location details of the user. It will have access to the user’s location through the API implemented.
Mockup UI
Stay tuned for my next blog posts with the updates regarding my projectLaughing. I will also update my blog: https://captanu.wordpress.com/ in the coming days. Our coding period starts today so, I wish All the best to other GSoC students too. Happy coding everyone!
Cheers!