Upadate on the powquty project

Posted on by Neez El Sayed

Dear Freifunkers,
please allow me to update you on the progress if the powquty Project within Google Summer of Code 2016 at Freifunk.

As mentioned in my last blog the goal of the project is to create a LEDE package which ensures three functionalities:

  • Retrieving sampled voltage data from the power quality measurement device via USB
  • Calculating statistical power quality parameters (such as: avg. voltage, harmonics, ect.) form the retrieved voltage samples
  • Provisioning of the calculated parameters for retrieval and graphical representation

Herein I will give a short update about the progress of these functionalities.
For this project we are using an off-the-shelf USB-based oscilloscope “WeSense” from the company A.Eberle. The oscilloscope provides real time samples of measured voltage from the power plug via USB bus, using a binary protocol and a sample frequency up to 10kHz. Initially the oscilloscopes USB bus supported the host functionality only. Hencethe router would need to act in USB device mode, which is a rather unusual mode to be supported by todays WiFi router platforms. To overcome this limitation, aforementioned company agreed to provide us with another hardware implementation that implements the USB device functionality with optional five volts power feeding functionality. The new hardware is expected on my desk in mid July.
As a counter measure for this delay, we started implementing an emulator, that locally generates a signal-samples, which are then organised in packets as similar to the binary protocol.
Regarding the calculation of the power quality parameters functionality, we successfully ported the power quality library (in Ansi C) from A.Eberle to compile and run under Linux LEDE.  The libraries functionality allows to calculate the frequency, effective voltage, harmonics, and phase shift, from the signal samples in an efficient way. We provide this library as binary blob, since it is basically not open sourced (yet), and originated from the manufacturer himself. Now it is ported for LEDE, and can be used for our purposes.
For the provisioning of the calculated parameters, we intend to implement a luci app that shows the calculated parameters.
The rest of the project timeline is depicted below:
Working phase: June 20th – July 10th

  • Finalize the emulator
  • Integration with the power quality library.
  • first prototype of the provisioning functionality

Working phase: July 11th – July 24th

  • Finalize the provisioning functionality
  • prototype implementation of sample retrieval (given that the hardware is delivered)

Working phase: July 25th – August 7th

  • Integration of the three functionalities into a working implementation
  • Software Testing and bug fixing
  • Documentation of the integrated  functionalities

Working phase: August 8th – August 21st

  • Buffer for possible delays

More updates in the upcoming weeks.
BR,
Neez

Implementing Pop-Routing – Midterm Updates

Posted on by Gabriele Gemmi

 

Hi Everyone!

Today has started the midterm evaluation, the deadline Is next Monday, so I have to show the work I have done ‘till now. It can be resumed in the following parts:


1) Refactoring of graph-parser and C Bindings

During the community bonding period I started working on the code of Quynh Nguyen’s M.Sc. Thesis. She wrote a C++ program capable of calculating the BC of every node of a topology [1]. I re-factored the code, and now it is a C/C++ shared Library [2]. I’ve also applied some OOP principles (Single responsibility and inheritance) and unit tests to make it more maintainable.

The interface of the library Is well defined and it can be re-used to implement another library to perform the same tasks (parsing the json and calculating the BC).


2)Prince Basic functionalities

After I completed the library a started working on the main part of the project. the daemon. We decided to call it Prince in memory of the Popstar.

This daemon connect to the routing protocol using the specific plugin (see below), calculate the BC using graph-parser, computes the timers and then it push them back using again the specific plugin. With this architecture it can be used with any routing protocol.I wrote the specific plugin for OONF and OLSRd. At the moment it has been tested with both, but I need to write a plugin for OLSRd to change the timers at runtime. For OONF I used the RemoteControl Plugin.

With these feature Prince is capable of pulling the topology, calculate the BC and Timers and push them back to the routing protocol daemon.

 

3) Additional Features: Configuration file, Dynamic plugins,

I wrote a very simple reader for a configuration file. Using the configuration file the user can specify: routing protocol host and port, routing protocol (olsr/oonf), heuristic, (un)weighted graphs.

As you can see from this Issue [3], I’m going to use INI instead of this home-made format.

As I said before I moved to a specific plugin all the protocol specific methods (pulling the topology and pushing the timers), to keep the daemon light I decided to load this plugin dynamically at runtime. So if you specify “olsr” in the configuration file just the OLSRd specific plugin will be loaded.

 

 

At the moment I consider this an “alpha” version of Prince. In the next 2 months I’ll be working on it to make it stable and well tested. The next steps will be:

 

  • Close all the Issues [4]
  • Write tests and documentation for Prince.
  • Write a plugin for OLSRd

 

Cheers, Gabriel

 

[1]: https://ans.disi.unitn.it/redmine/projects/quynhnguyen-ms

[2]: https://github.com/gabri94/poprouting/tree/master/graph-parser

[3]: https://github.com/gabri94/poprouting/issues/4

[4]: https://github.com/gabri94/poprouting/issues

Sharable Plugin System for qaul.net – Midterm Updates

Posted on by AnuV

Hi everyone, 

As we are on the midterm evaluation process, I would like to share what I have done so far. I created a small HTML5 application to share the location of a user, it uses geolocation API for this purpose. This will work online but, if we are offline it will work only if the device had a browser which communicated directly with the GPS hardware. This is how the application looks like when the location is found out:

As per my plan I was supposed to create a JSON API the application can use to send the request to the web server of qaul.net and a new message type for all the plugins in general. But it is not yet completed, so I have created a button in the message page of qaul.net GUI which will invoke the Javascript function to get the location of the user and it will be sent using the existing message type. There’s more work to be done for the proper working of the plugin mechanism in the qaul.net. Currently the location of the user is sent as longitude and latitude manner in the messaging system as shown below.

Cheers,

Anu

Provide a cryptographic implementation for Qaul.net – midtern update

Posted on by spacekookie

It’s been almost a month since official coding started for Google Summer of Code 2016. And it’s definately been an interesting experience.

When I was told to write this article some sugestions for content were “pictures” and to “provide links to prototypes”. Unfortunately with my subject (especially at this point in time) that’s not as easily possible and I’m not sure how many of you want to see source code in the form of glorious macros in a blog post.

But I will try 🙂

Qaul.net is building it’s crypto on a small, embeddable crypto library maintained and mostly developed by ARM called “mbedtls”. Unfortunately the project just went through some massive refactoring and the API has completely changed. Thus a lot of documentation online is out of date or just plain out doesn’t work. Most of my time so far I’ve spend crawling through the source code of the library components I need and learning how they link together in the background to understand which files I need to compile into the project and which are optional. The goal here is to make the end binary and the memory footprint of qaul.net as small (and as embeddable) as possible.

Next up was the general structure of the crypto module. In libqaul I added a new folder and thus function/ field namespace called “qcry” which is short for Qaul CRYpto. Following is a quick graphic of the layout of the module.

qaul crypto structure

All orange modules are part of the crypto-package, yellow modules show heavy modification to their code base where as grey modules will require very little modification. As you can see the crypto module is structured into five parts: Arbiter, Dispatcher, Context, Targets and Utilities. Following a quick description of what they do, if you’re interested in more in-depth documentation, check out my wiki.

  • Arbiter: Provides abstraction layer to the rest of libqaul and handles “disputes” between different threads or application states.
  • Dispatcher: Acts as a load balancer between the non-threadsafe functions of the crypto and the multithreaded environment of libqaul. Also optimises CPU overhead for context switching.
  • Contexts: Contains raw data for crypto tasks (private keys, mode of operation, targets, etc..). Is created by the arbiter but most handled by the dispatcher.
  • Targets: Contain data required for public key crypto. Mostly contains metadata and public keys of other users. Get created by the arbiter.
  • Utilities: Base64 encodings, hash functions, verifications, creating user avatars, lots of stateless functions that are used throughout libqaul go here.

One thing I have started to work on too (but haven’t gotten far enough yet to show something off in the UI) is user verification. So…users have private keys and with the fingerprint of that key it is possible to tell if they really are who they say they are. But how to communicate that to users? Showing people a long number is very unintuitive and will just cause people not to care.

So instead of a longer number (or maybe additionally), on a new user profile screen we will show an image generated from the username of the person, their fingerprint and a signature (public key encrypted with private key). This way users will be able to quickly see if the person they are talking to is who they say they are.

To do so there are several libraries:

 – https://github.com/Ansa89/identicon-c
 – https://github.com/stewartlord/identicon.js
 – https://github.com/spacekookie/librobohash (My C re-implementation of https://robohash.org)

 Which of these solutions will be used is not clear yet (Personally I like robots, who doesn’t like robots? Wink ). But that should give you an idea of what is to come in the UI department.

 Coming up in the next week is the arbiter and a rudimentary dispatcher. The arbiter needs to be hooked into the rest of libqaul and provide access to all the low level functions the crypto module provides. From there on out test with the rest of qaul.net can be conducted.

 I hope I didn’t ramble on for too long. And I’m looking forward to the next few weeks of coding.

 Cheers,
 spacekookie

GSoC: The ECE configuration system – midterm update

Posted on by NeoRaider

Hi,
the GSoC is nearing its midterm evaluation, so let’s have a look at the current state of the “Experimental Configuration Environment” (working title)!

I started the project with defining models for configuration data, configuration diffs (that are used to modify data and store modifications) and schemas. All of the design documents I wrote can be found at: https://gitlab.com/neoraider/ece/wikis/design

I tried hard not to re-invent things that already exist in a more or less standardized way:

  • My configuration model is just JSON (without support for floating-point numbers right now, as the libubox blobmsg format ubus uses to represent JSON doesn’t support them – I plan to fix this)
  • JSON Pointers (https://tools.ietf.org/html/rfc6901) are used to refer to elements of the configuration tree
  • The schema format uses a subset of JSON Schema (http://json-schema.org/)

Based on this design, I’ve started implementing the configuration management daemon eced. The page https://gitlab.com/neoraider/ece/wikis/implementation gives a good overview of the current features and usage of eced. Quite a lot is already working:

  • Multiple schemas can be loaded and merged
  • Default configuration is generated from the schemas
  • A ubus interface allows to query and modify the configuration
  • Config modifications can be loaded from and stored into a diff file

There’s still a lot missing, so here’s what I plan to do next:

  • Make something use ECE! UCI is well-integrated in OpenWrt/LEDE and can be used from C, Lua and shell scripts; all of these languages should also be supported by an ECE client library. An example schema to replace /etc/config/system does already exist and could be used to experiment with partially replacing UCI with ECE.
  • Support schema upgrades: When schemas are updated, this can partially invalidate config diffs. A way to deal with such inconsistencies must be defined.

Discussions and feedback have also led me to the decision to put a stronger focus to UCI interoperability. While I had only planned import and export of UCI configuration at first, I’ve realized that a two-way binding between UCI and ECE (i.e. an API-compatible libuci replacement/extension that will use ECE as backend) will be necessary for ECE to find acceptance. This will also allow to continue to use existing configuration tools like LuCI with UCI configuration converted to ECE.

I also plan to have OpenWrt/LEDE packages for ECE ready soon, so you can play around with the current implementation yourselves. Maybe I’ll also set up a public host with ECE Laughing

Monitoring and quality assurance of open wifi networks out of client view (midterm evaluation)

Posted on by Jan Tarek Butt
Hey everyone,
 
Now we are on the midterm evaluation. I would like to tell you what I have done so far and what will come next. In the first post [0] I explained the work packages. In this post I will come back to the work packages  and show you what I have done for each package.
 
The first sub-project was the hoodselector. At the beginning of the work period I did some bugfixing for the hoodselector so that we where able to deploy the hoodselector in our live environment. The hoodselector creates decentralized, semi automated ISO OSI layer 2 network segmentations. You can find a detailed discription here [0]. Retrospective I can say that the deploymend of the hoodsystem went without any major problems opposed to my first expectations. Currently we have 4 hoods active. Around Oldenburg(Oldb), Ibbenbüren, Osnabrück and Friesland. More hoods will follow in future. Open Issues can be find here [1].
 
The second sub-project was to create a propper workaround for building images with the continuous-integration (CI) system of Gitlab using make on multiple cores.
The Freifunk Nordwest firmware now has automatically built testing images that are not only build on a single core but can be built on multiple cores. And the architecture targets are also autogenerated out of the sourcecode. This makes it possible to generate images dynamically for all targets also including new targets that may come in the future. I implemented a small algorithm that manages the thread counter of make commands. I use the number of CPUs out of /proc/cpuinfo * 2 this means for each logic core will follow two threads. In example our runner02.ffnw.de server has 8 cores so the CI build process will automatically build with 16 Threads [2]. Here is an example of a passed buildprocess with our CI builder[3]. Actually it is not possible to build images with a high verbose output, because the CI logfiles will get to big. That makes it impossible to use the webfrontend for analyzing the buildprocesses. I opened an issues for this and discussed the problem with the gitlab developers [4].
The CI builder is very helpful for the developing process of the monitoring drone.
 
Following I would like to report about our first hacking seminar.
The first hacking seminar was on 28.05.2016. We started with two presentations. One about Wireless Geo Location and the second one about the Hoodselector. We recorded the presentation with our new recording equipment [7] bought using some of the money for the mentoring organisation and uploaded the recordings to youtube [5].
 
The first presentation was about geolocating with wireles technologies.
Based on the Nordwest Freifunk geolocator [6]

The second presentation was about the function of the Hoodselector

 
After this two presentations we had a smal disscussion about the presentation topics and than we started our hacking session where the developers started coding on their projects.
 
Now all sub-Projects are finnisched and I will continue with the Monitoring Drone Project after I finish my Study exams. Also the date of next hacking seminar is set for 9th of Juli 2016. Again we will have two presentations. One on Gitlab CI and one about how to use our new Puppet Git repositories including the submodule feature. The presentations will be recorded and after the presentation we will have a coding session like last time.
 
Timeline:
23. May: Community Bonding (3 weeks)
test and deploy hoodselector  <- Done
16. May 6:00 PM: GSoC Mumble  <- Done
Refine the roadmap  <- Done
23. May – 20. June: Work period 1 (4 weeks) <- Done
28. May 2:00 PM: hacker-session  <- Done
  1. Presentation about the hoodselector <- Done
  2. Presentation about the openwifi.su project[4] and the geolocator <- Done
13. June 6:00 PM: GSoC Mumble  <- Done
Midtermevaluation
Tarek & Clemens exams!!!
20. June – 15. August: Work period 2 (8 weeks)
9. July 2:00 PM: hacker-session
  1. Presentation about workaround with git CI processes.
  2. Presentation about puppet deployment system
13. June 6:00 PM: GSoC Mumble
25. June 2:00 PM: hacker-session
  1. Presentation about workaround with git CI processes.
  2. Presentation about puppet deployment system
18. July 6:00 PM: GSoC Mumble
30. July 2:00 PM: hacker-session
  1. actual unknown
  2. actual unknown
15. August 6:00 PM: GSoC Mumble
Finalevaluation
 

DynaPoint update

Posted on by Tobias Ilte

Hi everyone,

here are some updates regarding DynaPoint. The idea was to create a daemon, which regularily checks the Internet connection and changes the used access point depending on the result. That way the handling with APs would become easier as you already could tell the status by the AP’s SSID.

A daemon with basic functionality is already working. After installation, there is one configuration step necessary.
You have to choose in /etc/config/wireless which AP should be used, if Internet connectivity is available and which one if the connectivity is lost. You can do that by adding “dynapoint 1” or “dynapoint 0” to the respecive wifi-iface section.

You can also configure dynapoint via LuCI, although it’s not yet complete.
I was struggeling a bit with it, because the documentation of LuCI is a bit minimalistic…
Here is a screenshot of how it currently looks like:

Next steps

To verify Internet connectivity, it is probably better to make a small http download than just ping an IP address.
Using “wget –spider” should be suitable for that.

Also, I will see if I can get rid of the required configuration step in /etc/config/wireless in the next weeks and provide fully automatic configuration.

If you want to test dynapoint for yourself, just go to https://github.com/thuehn/dynapoint.

SWOON: Simultaneous Wireless Organic Optimization within Nodewatcher – in the middle of the road.

Posted on by David Marn

Hi everyone,

This is my first update on working on SWOON, an algorithm to automatically allocate channels to nodes in nodewatcher. Not a lot of time has passed since I started working and but the wlanslovenija community was immensely helpful in getting me started. Up to now, I only worked on my own coding projects, where I got to be the main architect behind the codebase. I decided where functions were defined, I decided on the coding style, I did code reviews and I was the one to approve branch merges (of which there were none, because I wrote all the code 🙂 ). Now, GSoC threw me in the middle of a well established open source project where you have to get accustomed to existing coding practices. And accustoming to this was more difficult than I expected. So people from wlanslovenija had to be really patient with me (and I think they’ll have to keep this up for a while 🙂 ). But I slowly got the hang of it. My overarching plan for the algorithm is to first read the data, then process the data and finally issue a warning to those who maintain nodes with inefficient channel allocations. My work on reading the data from nodes is almost done (almost because a new amazing feature was suggested only yesterday!). On a tangent, I think this shows another immense benefit of community oriented projects – everyone’s free to suggest additional features that would make this process more eficient. To get the data about neighboring nodes, one has to temporarily turn the wireless radios off. Because of that, nodewatcher-agent only performs a survey of neighbors every two hours. But we collect data much more regularly. So how do we deal with this? My current implementation simply stores the data into the database every hour and it does not take into account the age of the data. I will now work on comparing the current datapoint with the latest one and only insert it into the datastream if the data has changed. This will reduce the space we’re using and could even be expanded for use by other modules. It’s great to know that you can contribute to any aspect of the project. Mitar, a senior developer of nodewatcher, said that they built all of it themselves, so we can rewrite anything as well.

I also worked on a parallel implementation of the mechanism, which read the data from old dumps. This gave me the insight into the architecture that made writing the actual implementation much easier.

My next steps are to finish the data collection part and create a module to analyze this data. This is exciting for everyone because there’s no documentation about writing such modules so I’ll get to be the first one to do it! I can only hope I’ll document it well enough to make the process easier for those who will follow as new members in the community. The github PR can be found here.

Google Summer of Code 2016: External netifd Device Handlers – Milestone 1

Posted on by ArneKappen

OVERVIEW OF THE LAST WEEKS
During the last 5 to 6 weeks I have implemented the possibility to include wireless interfaces in Open vSwitch bridges, rewritten a lot of the code for creating bridges with external device handlers and brought my development environment up to speed. I am now working with an up-to-date copy of the LEDE repository.
I have also implemented the possibility for users of external device handlers to define what information and statistics of their devices look like and to query the external device handler for that data through netifd.

CHANGES TO THE DEVELOPMENT ENVIRONMENT
So far, I have been using quilt to create and manage patches for my alterations of the netifd code.
One day they were completely broken. I still do not know what and how it happened but it was not the first time and this time recovery would have been way too tedious.
This is why I switched to a git-only setup. I now have a clone of the netifd repo on my development machine that is symlinked into the LEDE source tree using the nice ‘enable package source tree override’ option in the main Makefile. I used the oppportunity to update both the LEDE source tree and the netifd repository to the most recent versions.
Before, I was working on an OpenWRT Chaos Calmer tree, because of a bug causing the Open vSwitch package to segfault with more recent kernels.
Now, everything is up-to-date: LEDE, netifd and Open vSwitch.

MY PROGRESS IN DETAIL

More Dynamic Device Creation
In actual coding news, I have refined the callback mechanism for creating bridges with external device handlers and the way they are created and brought up.
Previously, a bridge and its ports were created and activated immediately when /etc/config/network was parsed. Now, the ubus call is postponed until the first port on the bridge is brought up.
Because of the added asynchronicity, I had to add a ‘timeout and retry’-mechanism to keep track of the state of the structures in netifd and the external device handler.

A few questions have come up regarding the device handler interface. As I have explained in my first blog post, I am working on Open vSwitch integration into LEDE writing an external device handler called ovsd. Obviously, this is very useful for testing as well.
I have come across the issue of wanting to disable an bridge without deleting it. This means bringing down the bridge and removing the L2 devices from it. The device handler interface that I mirror for my ubus methods doesn’t really have a method for this. The closest thing is ‘hotplug remove’, which using feels a bit like a dirty hack to me.
I have reached out no netifd’s maintainer about this issue. For the meantime, I stick to (ab)using the hotplug mechanism.

On the ovsd side I have added a pretty central feature: OpenFlow controllers. Obviously, someone who uses Open vSwitch bridges is likely to want to use its SDN capabilities.
Controllers can be configured directly in /etc/config/network with the UCI option ‘ofcontrollers’:

config interface ‘lan’
        option ifname ‘eth0’
        option type ‘Open vSwitch’
        option proto ‘static’
        option ipaddr ‘1.2.3.4’
        option gateway ‘1.2.3.1’
        option netmask ‘255.255.255.0’
        option ofcontrollers ‘tcp:1.2.3.4:5678’
        option controller_fail_mode ‘standalone’

The format in which the controllers are defined is exactly the one the ovs-vsctl command line tool expects.
The other new UCI option below ofcontrollers configures the bridge’s behavior in case the configured controller is unreachable. It is a direct mapping to the ovs-vsctl command ‘set-fail-mode’. The default behavior in case of controller absence is called ‘standalone’ which makes the Open vSwitch behave like a learning switch. ‘secure’ disables the adding of flows if no controller is present.

Function Coverage: Information and Statistics
Netifd device handlers have functions to dump information and statistics about devices in JSON format: ‘dump_info’ and ‘dump_stats’. Usually, these just collect data from the structures in netifd and the kernel but with my external device handlers, it is not as simple. I have to relay the query to an external device handler program and parse the response. Since the interface is generic, I cannot hard-code the fields and types in the response. This is why I relied once more on the JSON data type description mechanism that I have already used for dynamic creation of device configuration descriptions.
In addition to the mandatory ‘config’ description, users can now optionally provide ‘info’ and/or ‘stats’ fields. Just like the configuration descriptions they are stored in the stub device handler structs within netifd where they are available to serve as blueprints for how information and statistics coming from external device handlers have to be parsed.

For my Open vSwitch setup, it currently looks like this in /lib/netifd/ubusdev-config/ovsd.json:
{
    “name” : “Open vSwitch”,
    “ubus_name” : “ovs”,
    “bridge” : “1”,
    “br-prefix” : “ovs”, 
    “config” : [
        [“name”, 3],
        [“ifname”, 1],
        [“empty”, 7],
        [“parent”, 3],
        [“ofcontrollers”, 1],
        [“controller_fail_mode”, 3],
        [“vlan”, 6]
    ],
    “info” : [
        [“ofcontrollers”, 1],
        [“fail_mode”, 3],
        [“ports”, 1]
    ]
}

This is how it looks when I query the Open vSwitch bridge ‘ovs-lan’:

THE NEXT STEPS

During the weeks to come I want to look into some issues which occurred sometimes when I disabled and re-enabled a bridge: Some protocol-realated configuration went missing. This could mean that sometimes the configured IP address was gone. Something which could help me overcome the problem is also in need of some work: reloading/reconfiguring devices.
Along with this, I want to get started with the documentation to prepare for the publication of the source code.

Overview: GSoC projects at freifunk.net

Posted on by Andreas Bräu

In 2016 we found 9 students doing Google Summer of Code projects for freifunk. This is an overview on all our projects including links to repositories and further information:

Title Student Repository Project information
A schema-based configuration system for OpenWrt Neoraider  https://gitlab.com/neoraider/ece  
DynaPoint – A dynamic access point validator and creator for OpenWrt Asco  https://github.com/thuehn/dynapoint
https://github.com/thuehn/luci-app-dynapoint 		  
Implementing Pop-Routing Gabriel https://github.com/gabri94/poprouting http://firenze.ninux.org/
Monitoring and quality assurance of open wifi networks: the client view Tarek https://git.nordwest.freifunk.net/ffnw-firmware/monitoring-drone   
netifd extension: external device handlers arkap    
OpenWrt – poWquty (poWer quality) Neez   https://github.com/thuehn/powquty https://github.com/neez34/
Provide a cryptographic implementation for Qaul.net spacecookie  https://github.com/spacekookie/qaul.net  https://spacekookie.de/gsoc2016/
Sharable Plugin System for qaul.net – Mesh Network Communication App Anu  https://github.com/anuvazhayil/qaul.net https://github.com/anuvazhayil 
SWOON: Simultaneous Wireless Organic Optimization within Nodewatcher marn 

https://github.com/CdavM/nodewatcher

https://github.com/CdavM/swoon 

  

Blog posts on these projects you can find with the tag GSocC 2016. Use the RSS feed to receive all updates. You can also see our projects from previous summers.