GSoC 2018 – DAWN a decentralized WiFi controller (2st update)

Hi,
I still try to get my patches upstream.
For the libiwinfo patch I had to add the lua bindings. I never used lua so first I had to get comfortable with this. Additionally I wanted to add the channel utilization in the luci statistics app. But suddenly Luci is giving me a null pointer exception in the dev branch.


Additionally I tried to get comfortable with Luci for developing my own app.
Meanwhile another developer created nearly the same patch for iwinfo that add the survey data for the nl802.11 driver… This patch is still not accepted. The only difference is that it returns all survey data for all channels (like iw dev wlan0 survey dump)…
Furthermore, my pull request for the hostapd ubus bindings that add information about the ht and vht capabilities had to be rewritten. (https://github.com/openwrt/openwrt/pull/898). Again I have to wait for some feedback. While rewriting this patch, I had a new idea: If you subscribe to the hostapd via ubus and want to notify on the messages you have to activate it. It would be possible to add flags in the hostapd_ubus_bss to select what information should be published via the ubus bus. Before doing so, I want some feedback if this is a good idea.If somebody is interested why I am interested in the capabilities: I want to create a hearing map for every client. I’m building this hearing map using probe request messages. This probe request messages contain information like (rssi, capabilities, ht capabilities, vht capabilities, …). VHT give clients the opportunity to transfer up to 1,750 Gigabits (theoretical…) If you want to select some AP you should consider capabilities… In the normal hostapd configuration you can even set a flag that forbids 802.11b rates. If you are interested what happens if a 802.11b joins your network search for: WiFi performance anomaly. 🙂

Summarizing, I spent a lot of time waiting for feedback, debugging, modifying my patches or replying on the email lists. It is a bit frustrating.
The cool stuff was that I had my first pull request. 🙂 (it was just a typo ^^) But somebody took the time to fork my project and create a pull request. 😉
Furthermore, it is exam time and I have a lot of stuff to do for the university.

Actually I wanted to go on with more interesting stuff like connecting to the netifd demo to get more information.

Or to look at PLC. There is an interesting paper EMPoWER Hybrid Networks: Exploiting Multiple Paths over Wireless and ElectRical Mediums.

 

VRConfig Update 2

Hi,

I spent the last weeks mainly developing the LuCI Application for VRConfig. As soon as you want to do advanced things with LuCI, it gets cumbersome.
As the API is mostly undocumented, you have to dig through the LuCI’s source code trying out functions which could be useful according to their name.
It’s a bit of a trial and error game.
Currently the LuCI app does the following.
It displays an image of the router and parses the JSON file, which contains the locations of the components.
With this information it can mark the associated physical ports to the currently selected network interface and display those network ports, which are connected to a cable. You can also hover over the components and click on them, which leads you to their respective settings page.

I also improved the annotation app. It now lets you choose the router name from a list of all currently supported router models of OpenWrt. I got that list with a series of grep and sed commands from the OpenWrt git repository.
For your information, there are currently around 1100 different router models supported. 🙂

In the next weeks I will polish the LuCI Application and try to integrate VRConfig into the openwrt build system to be able to select the correct router image and JSON file at build time.

GSoC 2018 – Easily Expandable WIDS – Second Update

In this blog post I’d like to present the recent progress made for Eewids. This time, our main focus was the performance of the current setup. Besides we did some minor improvements, mostly adding some more dashboards to Grafana to visualize the data captured. This blog post focuses the results regarding Kismet as a component of Eewids. Continue reading “GSoC 2018 – Easily Expandable WIDS – Second Update”

OpenWLANMap App: Update 2

Hi,

In the last weeks I was working on  the storing process as described in the architecture in the last blog post [0].

Storage Handler:

Old app: the old app saves the data as byte in a file. A data entry is 28 bytes of MAC-address(12 bytes for 12 characters) and latitude(8 bytes for double) and longitude(8 bytes for double). An entry could be saved more than once in the file. There are 2 files, one for data which should be updated and one for data which should be deleted from backend.

New app: Firstly I wanted to adapt the structure from the old app. But since I saw some unreasonable points such as saving redundant data, flash workload, maintenance problem and unstructured storage, I decided for a standard database with more structure and easy to maintain: sqlite. Also I am using the new persistence lib, which provides an abstract layer for database: Room, newly released last year, as a part of android architecture components, with a lot of bug fixed since then. A lib with a lot of advantage when working with sqlite database: verify queries at compile time, reduce a lot of duplicate code in comparison with the last approach with DbHelper etc. In order to store the access point in the database, I implemented a seperate thread, which reads data from a blocking queue and saves it in the database, which works parallel with the scan thread and will be interrupted if there is nothing in the queue to store. Also to save energy and not force the store thread to run the whole time, a list of access points will be put into the blocking queue as an element. To pretend redundant data in storage, a data entry with BSSID will not be saved many times as in the old app but only once. The BSSID is used as primary key in the sqlite table. It will be updated the next time if the received signal strength is better than the last entry in the database. An explicit transaction is implemented to solve this case since the lib Room has only supported annotation for standard update/insert. To decide if a access point should be deleted or updated from backend, a flag is set.

Upload Handler:

The WifiUploader is in process. I did take a look at the uploading format in the old app and how it communicates with the current backend. Also the upload sequence is already defined, mean the scanning thread will be interrupted, all the rest of access point will be stored, the store thread will be interrupted to pretend conflict while 2 threads try to access same database at the same time before the uploading process is started. Also the WifiUploader will read maximum a number of data entries from the database and upload it, not the whole database like old app but one after another,  in order to pretend out-of-memory problem at device with small ram. (see more in the below diagram)

flowchart of uploading process

 

But since I am in the middle of my final exam period, there will be a small delay until this weekend for the WifiUploader to be published. Also from next week I will spend full time making the other features done includes implementing all saving resource features such as adaptive scanning, implementing all settings option. A clean and full documentation will be provided at the end as well.

Available issue: Permission request and handling

[0] https://blog.freifunk.net/2018/06/10/openwlanmap-app-update-1/

WiFi Direct and Bluetooth Meshing

As hinted in my last blog article, for us to really be able to move forward we needed to do some experimentation with the new technologies we have to adapt. The primary candidate in the beginning of that phase was WiFi Direct, a type of WiFi mode setting which is an official standard published by the WiFi consortium which is meant to replace the ad-hoc wifi mode. But only partially: WiFi Direct was mostly designed to make integration with IoT products easier. As such, using it for the meshing applications is a bit outside of it’s primary use case. The idea behind it is to make two WiFi devices talk to each other without needing a router to be the middleman for negotiation and frequency selection. Even groups of devices are possible create, electing a group leader that then manages the group.

Unfortunately…that sounded a lot better in theory than it turned out to be in practice.

Also as hinted in my last blog post we built some little prototype applications to test WiFi direct between multiple devices and ran into some issues. The APIs that are provided by Android are okay to use, but not super convenient. But most of the issues come from bugs that we haven’t exactly been able to trace down yet. The system WiFi Direct interface (System Settings > WiFi > Advanced > WiFi Direct) detects all devices in the vicinity whereas our application, using the WiFi Direct interface in the Android SDK would sometimes (nondeterministically) fail to detect devices or open sessions between them. We also had some bad experience creating groups between the devices.

All in all…it was underwhelming. WiFi Direct really wasn’t meant to do the kind of networking we’re trying to do with it and even if we can figure out the bugs we encountered, there are other concerns to work out. Debugging these issues aren’t easy but there are a few things we can do. For one, there are other (open source) applications that exist (serbal, briar, …) that use this technology and we can study to see how they solved these issues. There is also the option of wireshark-ing packets that are being transmitted between the two devices to get a better understanding of where handshakes are going wrong. Simple debugging via Android/ Java debugger unfortunately hasn’t yielded many useful results.

We need a convenient way for people to be able to join the network. We need to figure out a way to create a captive portal for people just connecting without the software. The handoff between a WiFi Direct network section and a legacy ad-hoc section that might be created between infrastructure nodes that don’t support WiFi Direct. The last week or so I’ve had my head in the WiFi Direct specification, trying to answer these questions. And while I think we have solved most problems, there’s still a few left to answer.

The second technology we are investigating to complement WiFi Direct wherever it isn’t applicable is Bluetooth P2P Meshing. In contrast to WiFi Direct, it was actually developed for devices to mesh with each other which makes the adaptation of it easier for us in the long run. So far we’ve only done some simple experiments with 2 devices (due to a lack of Android devices in one location 😉 ) but these have been a lot more promising than what WiFi Direct has offered.

The biggest take-away from the last 2 weeks of experimentation is that we can’t dedicate the routing core to a single networking backend.

In the design of the actual code interface that I’ve built in the first few weeks of GSoC this means that there are some adjustments to be made before writing more code. This includes being more generic when binding interfaces and allowing a client to use multiple backends at the same time. This was not intended to be used in the initial design. But for the time being those interfaces will simply be mocked by some stub methods or maybe a simple simulation so we can test the actual routing algorithms. This is an interesting challenge because so many parts of qaul.net will have to change in lockstep with each other to make it all work.

There are some corner cases to test when it comes to bluetooth mesh networking such as groups and how they handle devices joining in and out of them

The Turnantenna – Second evaluation update

Time is passing, and work is proceeding.

Last month I reported a problem concerning speed of our beloved Turnantenna: the acceleration was not constant during movement of the stepper engine, as I wanted. The error was caused by implementation of a bad algorithm. A constant acceleration is important to provide a smoother movement, and is needed to reduce the load on the engine. Force is equal to mass times the acceleration; if the acceleration is constant, so is the force; but if the acceleration grows, the stepper’s force grows as well, as long as it can keep up. Uncontrollable acceleration lead to unpredictable forces (or better, toques).

To understand the issue, a brief summary should be given: the way to control the stepper’s speed consists of changing the time between two consecutive steps. The shorter the time, the faster the movement. The previous (and wrong) algorithm, is documented in the older post. It wasn’t a good way to control a torque-limited engine because, as said before, the acceleration was not constant. In the previous algorithm, speed was taken like this:

vn = vn-1 + const

namely, at time tn the speed was a fixed amount more than tn-1.
Time between two steps was

dt = (n – n-1)/vn = 1 / vn

It may appear correct, but the resulting graph was the following:

As can be seen, the speed is not linear. This mean that the acceleration is not constant, but increasing.

I found a solution thanks to a document written by Atmel Corporation. It made me think about the relationships between speed (v), space (s), time (t) and acceleration (a) that comes from physics laws:

s = a ½  t² + v₀ * t + s₀

this equation is always true, when accelerating, when the speed is constant, and even when decelerating. Quantities change inside the formula, but it always remain true.

Now, to keep it simple, let’s consider the first phase: the acceleration. A the beginning of its movement, the engine is still (v0 = 0), and it starts without having already done one single step (s0 = 0). The resulting equation is evaluated at v₀ =0  and s₀= 0:

s = a ½  t² + 0 * t + 0
s = a ½  t²

Now, let’s think about what is known: the acceleration a, that it is constant (because I want it so), s and t; s is the number of steps already done at time t. If I know how many steps -s- I have to do I can find how much time I have to wait –t-, and vice versa.

To find the time between two steps (the step #n and the step #n+1) the formula is:

s = a ½  t²
==>  t = sqrt(2 * s / a)

# at the step number ‘n’
tn = sqrt(2 * n / a)

# time between step ‘n’ and ‘n+1’
dt = tn-1 – tn = sqrt(2 / a) * (sqrt(n+1)-sqrt(n))

Using this calculation, acceleration is constant, and speed increases linearly, as it can be seen in the graph below:

AAAAAH.. a perfect blue line! 😀

Problem: Solved!

Working on tests

Now, more progresses have been done in tests. For those who don’t knows, I’ve started my programming adventure with with this project. Everything for me is anexciting discovery, and during this month I learned and implemented the “argparse” and “logging” libraries. Now it is possible to execute the tests with three verbose levels: the first is silent, the second shows debugging informations and the third shows the info level.
It could appear trivial, but I’d never done it before, and now tests are smarter!

It is not all: I reviewed all the tests, fixed problems and improved their reliability. They’re still not perfect, but I’m working on them daily to get details right.

Fly across borders

It was time to go outside the boundaries, and to think about an interface that bring into communication the web interface and the driver. This is what I’m working on in these days.

To achieve that goal, the problem has to be studied starting from an high level. The main process, which is constantly running, float between a small number of determined states: initialising, still, moving and error handling. Together with the Ninux Florence developers community I built the following state machine graph:

This was realized with the GraphMachine module of the “transmissions” library. Now I’m working on the full representation of this map in code lines. But there is something more.. In fact, at this point, multiprocessing became necessary to provide a safe environment: when the engine is in the MOVING state, for example, and a new command is sent to make a new different rotation, the main process should have the possibility to manage simultaneously the ongoing movement and newer requests.

That’s why we choose to keep the main process always active and make it decide when to run the movement procedure in a dedicated process, like a traffic light.

Turnantenna.me

The greatest effort, this month, was done writing down a full, detailed documentation of the project. 80 pages on what is the Turnantenna, how it works, and when and why to use it.

Many people expressed their interest in the project, someone has offered to support us but, without a complete documentation available, it is difficult to provide a starting point.

The whole doc will be soon available, and this post will be updated with the dedicated link. So, if you are interested in the project, let us know! For the moment, GitHub repository is available here.

See you next month!

— UPDATE —

The link of the full documentation of the project is provided below.
turnantenna.readthedocs.io

Meshenger – P2P local network messenger – Update 2

Just a few days after the first update I figured out how to use WebRTC, which I shortly after implemented into Meshenger.

To shortly describe the way signalling is working in my app:

  1. On phone A a call to phone B is issued by tapping the contact, hence phone A already knows the destination address
  2. A sends a call request to B, which B may accept or decline. This is done through a custom JSON-based protocol designed by me
  3. In case of success A creates a Session Description, also called SDP offer, which it transmits to B, using the same protocol
  4. B creates an SDP answer which it transmits back to A, still using my JSON-based protocol
  5. Using the exchanged offer and answer a RTCPeerconnection is established, where the data is transmitted through separate DataChannels/Streams. Upon this moment, WebRTC handles all the communication
  6. Finally, the phones have a RTCPeerconnection which they use to send Audio/Video as well as service messages, e.g. when a camera is connected

 

Besides the sheer implementation of WebRTC the front/back camera can be turned on and even switched on the fly.

The app has undergone some graphical improvements, all buttons containing text has been replaced with Image Buttons,

many of them even show a animation depending on their effect.

 

I even tried to cover the case of the user switching a network, if the link-local address loses its reachability.

Instead of simply trying to reach the last known address, the app now examines every address the phone has

and tries to replace its own MAC address in those addresses with the MAC of the target.

This leads to a higher chance of re-finding a contact even if the phones have switched networks.

 

As you can see in the following screenshot, bidirectional video-transmission is enabled, and the buttons at the bottom now have cute icons.

 

Phone A
Phone B

 

 

 

 

 

 

 

 

 

 

 

From now on I will focus on polishing the app, cleaning the source code, finding and resolving bugs and,

last but not least, writing sufficient documentation.

I may even write a blog post somewhere, explaining how to archieve a serverless WebRTC connection,

since the documentation i have found so far was not really helpful and mainly focused on JavaScript.

Routing and WiFi experimentation

The beginning of my work period was pretty busy, not always with Summer of Code things. My mentor math and I had already talked about a lot of the things that needed to happen in order to move qaul.net away from an OLSR based routing protocol and make it extendable as well.

As previously hinted, we are using Rust for the protocol implementation, allowing for easy integrating into the existing C code as well as giving us the option to bit-by-bit rewrite the entire software in Rust, a much more modern and forgiving language. The first thing I tackled was to design a common API for the qaul.net library (libqaul) to use to talk to any networking backend. The routing code holds the state of the network and allows the sending of direct and flooded messages into a network (regardless of implementation under the hood). But to do that we also had to define some common characteristics for nodes and messages.

In the end, a lot of the work was sitting down, going through old notes and determining what our protocol was supposed to do. We looked at existing protocols a lot, thinking about extendability and backwards compatibility. The protocol itself will be binary encoded, although not yet sure which format. There is msgpack, cpnproto/ protobuf as well as some Rust specifics (Rust Object Notation to mention one) to look at. But that shouldn’t actually matter for now. All the versioning and extend-ability are being done in the struct level of the protocol, meaning that we could even switch binary encoding half way through. Keeping the encoding and decoding written in Rust, this is actually incredibly easy with the `From` traits. But I digress…

The protocl we ended up designing can handle any type that already exists in qaul.net, as well as allows for custom user extentions – messages that have a type field and a random binary message payload which allows plugins on both sides to interact with it.

 

So…so far the routing core isn’t doing much routing. But that’s okay, that comes later 🙂 With the networking API in place, we actually have something what we’ve wanted for the last 2 years: a hardware abstraction over any networking backend. The API is implemented in Rust as a trait (think like a Java interface), which makes implementations and even implementation specific code very easy.

The next thing on our todo list is working out how WiFi direct behaves. This is kinda disconnected from the rest of the project, but it’s something that has to happen. For this purpose I’ve written a small demo app (still WIP at the time of this writing), which will let us explore the way that WiFi direct works, how to build mesh groups, etc. These experiments are still ongoing, and we hope to have something to show until the end of the week. I will probably publish a small article on my blog about it – check it out (if you’re reading this in the future 😉 )

All in all, the amount of code written in the first section of GSoC2018 is medium. We have however answered a lot of open questions, have a good plan on how to continue and hope to have more to show off by the time of the next evaluation.

If you’re curious about the progress being made, check out the github repository.

 

Until next time,
Katharina

nodewatcher: Build system rework and package upstreaming – First update

Since last update I spent most of time on fully understanding current build system and nodewatcher internals.

Build system

During the time spent into looking how the whole system works I belive that I was able to figure out what every step and script does in the current build system.
During that I found a lot of relatively simple improvements that can really reduce custom stuff that we have. Most of it was added 3-5 years ago when OpenWrt wasnt really in the state it is now. Custom mirror for source was added,which is now useless as it was not updated and on some really old custom packages it is really slow. Also,building all packages that are added to the package list during build time is not really efficient as during its building OpenWrt default feeds are replaced with our custom package feeds.
This causes users to be stuck on really old and quite limited number of packages. This will be revorked to only replace the target feed as it contains all of the kernel mods that are tied to specific kernel version.
Other packages have no such requirments and versions in upstream can be used.
Also,wget was used during the build time to pull dependecies instead of curl which is recommended tool. Wget is fine for simple downloads but a lof of packages are pulled from behind CDNs and with lot of redirects from mirrors such as Sourcefourge,those curl can handle fine but wget cant.

I have started dropping unused and old packages as well as those that had custom patches that were upstreamed a long time ago.
Stuff like iwinfo from 2015 and old curl.

Also,I have started to move both the buildsystem docker image and runtime docker image from Ubuntu 14.04 to 18.04 Bionic.
This does not yet fully work as Imagebuilder does not detect GCC and Ncurses in the runtime image as working despite the fact that GCC works fine. This will be hard to diagnose,but I feel that it is quite simple.

Custom Wlan Slovenija packages are being prepared for upstreaming.

This is all for now,next two weeks should be bring solutions to most issues.

Robert Marko

VRConfig Update

Hi,

I have some quick updates about VRConfig for you.
Short recap: VRConfig aims to introduce a graphical configuration mode for OpenWrt’s Webinterface LuCI.
For that need to collect pictures of the backside of all supported routers. The idea is to do this in a crowdsourcing manner. The community can submit pictures of their routers together with a metadata file which contains the locations of the components on the picture.

I spent the last weeks developing a web application to provide the annotation functionality of the router components.
A working prototype is now ready and can be tested at the following URL: https://vrconfig.gitlab.io/annotator/
Source code: https://gitlab.com/vrconfig/annotator

The annotator produces a JSON file which in turn can be parsed by the LuCI Application to provide the graphical configuration mode.

The LuCI application is being developed right now and will be provided shorty under the following URL: https://gitlab.com/vrconfig/luci-app-vrconfig

More info about that in the next blog post.