In the phase 2 period, we set up an emulation environment in CORE, then we tested PRINCE with iperf client/server (https://github.com/pasquimp/prince/tree/iperf) in CORE. We built a simple three nodes topology (n1, n2, n3) where n1 is directly connected (in the wireless coverage) to n2 and n2 is directly connected to n3. n1 is connected to n3 through OLSR. The neighbor estimated throughput at IP level is of about 43 Mbps on a physical link of 54 Mbps (in the figure the throughput estimated from n2 towards n1).
We tested in CORE the initial version of the OONF plugin too (https://github.com/pasquimp/OONF/tree/neighbor-throughput). So, the plugin now is able to send a couple of probe packets towards each neighbor and is able to read the time of reception of packets. I’m now exploring a problem in the reception of the probe.
In the next weeks, we will perform other tests with PRINCE with iperf and with the OONF plugin to resolve the problems in the reception phase and then we will perform timestamp based throughput estimation in order to compare the results obtained with PRINCE with iperf and OONF with the plugin. We will update you in the coming weeks!
Last month I introduced my test setup intended for fast kernel trials and network development. After that updated my shadowsocks-libev fork for the latest 3.2.0 version which is the latest upstream stable version. This fork dont do any encryption which is not so secure but faster – and in our new approach: we can put the data plane into the kernel (because we cant do any data modification in the userspace).
The problem emerged in a different environment recently: at the cloud/datacenter scope. In the cloud transmission between containers (like Docker) happens exactly like in our SOCKS proxy case: from user to kernel, than back to user (throught the proxy) than back to kernel, and to user. Lots of unnecessary copy. There was an attempt to solve that: kproxy .This solution is working pretty well, butthere are two drawbacks: not merged into the kernel (the main part is a module, but also modifies kernel headers) and in my testsit is slower than the regular proxy with the extra copies. Sadly I dont know the exact problem, but with my loopback tests on a patched 4.14 kernel were about ~30% slower than a regular proxy.
The kproxy is currently AFAIK not in development anymore, because featuring TCP zero-copy there is a better solution with zproxy, but its not released yet. But some part of the original kproxy code is already merged into the kernel part of the eBPF socket redirect function: https://lwn.net/Articles/730011/
This is nice because its standard, already in the vanilla 4.14 kernel, but a bit more complicated to instrument it, so I will test it later.
The backup solution if none of them works the I will try it with netfilter hook with the skb_send_sock function, but that version is very fragile and hacky.
Hello in this past month I was working on the validation of the configuration in both the front-end and backend.
Basically it is to confirm that the minimum parameters to generate the basic configuration are selected and are of the corresponding type. The double validation is because the ubus module can be used in the future by other applications, and in this way its good use is guaranteed, while validation in the frontend allows a faster response to the user.
View for LuCi
While doing all this I started to develop the basic view for LuCi, although the goal of GSoC is to develop the view for LimeApp I can do both by reusing much of the code. In the next few days I will upload some screenshots.
I am very happy to say that since my last update I was able to implement most of the features that I have talked about and was able to test them with real data.
In the last update I talked about how I started my own local leaflet map with which I wanted to test every feature before implementing them. While doing that I also need to go through most of the nodewatcher code to see how the map is being generated. The problem here was that nodewatcher uses Django templates and many custom scripts that were placed in multiple locations. It took some time to figure out what each part was doing because the map was made at the start of nodewatcher and wasn’t documented well. So this took most of my time, but after I figured out where everything was I was able to start implementing most of my code.
The implementation went surprisingly fast, so I was able to test everything on my own nodewatcher server that I started at the beginning of GSoC. The only problem here was that I didn’t have any nodes to see on my map. I was able to workaround this by redirecting my API call to gather node data from the nodes.wlan-si.net server which is the wlan slovenija nodewatcher server. It has over 300+ active nodes. In the pictures below you are able to see the things that I have currently implemented that are:
The fullscreen map option
Popup with some general information about the node that you get when you click on it. And also by clicking the name in the popup you can go to that nodes website
Sidebar that gives you a list of all currently online nodes with a search bar and the ability to show each one on the map.
Next thing for me is to try to implement one more feature which is the ability to see nodes that have gone offline in the past 24 hours. I say try because I have already looked into it and the problem with this is that the current API doesn’t have a filtering option so I can’t get only the nodes that have the location parameter set. I will also mostly focus on writing good documentation because that is something that nodewatcher is currently lacking and it would have really helped me a lot.
This is an quick update on my work on LibreNet6 and LibreMesh within the last weeks. The exam period in Tokyo started and I had a cold which slowed me a bit down, once both passed I will focus with doubled concentration on the project again!
The approach of using Tinc allows the usage of more then one IPv6 server, allowing to connect the servers of multiple communities with different IPv6 subnets. Babeld automatically detects where to route traffic when using external subnetworks. This is fortunate as it is possible that there is a high latency between mesh gateway and IPv6 server which would slow down traffic. However, using Tinc and babeld I ran a setup with two mesh gateways both using two different IPv6 subnets. While pings to the other network had high latencies at first (me in Tokyo, one IPv6 server in London and one in Argentina), Tinc automatically exchanged the IPv6 addresses of the mesh gateways which then could connect directly, lowering the latencies. Summarizing this experiment, using Tinc makes the network independent of the public IPv6 addresses.
No lime-app plugin
Initially I though of creating a lime-app plugin which allows to easily requests access to a Tinc mesh. However, after an evolution with my mentor and reading more about Tinc, we decided against it: The new 1.1 release of Tinc not only simplifies joining a mesh by offering the invite and join commands, but also allows to do all configuration automatically with the help of an invitation file. These new features simplify the project much more then I though, following the Spanish documentation on Altermundi.
Adding some security
As mentioned above some parts where easier as excepted, I though of looking into additional tasks for the project. Currently the usage of babeld requires all users of the mesh to fully trust one another as babeld does not provide any security (I could find) regarding announced routes. Mesh routing with security is offered by BMX7, which introduces a model to set (dis)trust between nodes. For this reason I’ve been in contact with Axel Neumann, the developer of BMX7, to fix an long standing error in OpenWrt which lead to false MTU rates in BMX7. The fix was merged upstream and thereby allows to test BMX7 over Tinc as a secure babeld alternative.
Beneath the experiments I’ve started to translate (and simplify) the Spanish documentation of LibreNet6 and will upload it to the GitHub repository once finished. Important part is also how to configure 6to4 tunnels as surprisingly few VM providers offer any IPv6 connectivity per default but only a single public IPv4 address.
Last weeks have been spent solely on reworking the build system.
First, it was a matter of rebranding the current LEDE back into OpenWrt and fixing a couple of hard-coded names that would cause issues with OpenWrt name. It also involved dropping the old OpenWrt build system which has not been used for years and most likely never will again, so that removes unnecessary code to maintain.
After rebranding, I spent some time verifying that the whole system still works. Fortunately, there were only small bugs which were simple to fix.
And then came the main task of this project, to completely rework and massively simplify the whole building the image builder job a lot easier and resource intensive.
Firstly, since I was still gonna use Docker to images for a build environment updating the base image which is the actual build environment was needed from old Trusty 14.04 to fresh 18.04 Bionic. This proved to be mostly trial and error as a lot less of default packages were included in 18.04 so getting all dependencies working. After a while base image is now working fine and is relatively small, actually smaller than 14.04 base image. This is due to less unnecessary packages.
Once the base image was sorted out I finally got working on dropping the unnecessary scripts, docker files and all of the hardcoded build files.
This proved to be not so hard, so work on a new docker based build system started.
So far it’s broken into only 4 separate scripts:
docker-prepare-build system: Like its name hints it builds the base image and installs the needed packages. I am still thinking to maybe pull this from the auto built image on Docker Hub.
generate-docker files: Which generates the temporary docker files needed for building inside a Docker 18.04 base image.
docker-build: Which actually “builds” the image builder and SDK.
build: Main script, which simply calls others to configure and build everything.
Number of scripts will most likely grow by one or two since the built image builder with all of the packages need to be packaged and then deployed in a runtime specific image which will only contain the bare minimum of packages to keep it as lightweight as possible. Currently, building works fine for most custom packages using SDK, but its stuck at building ncurses with a weird LC_TIME assertion error which I need to fix.
So next period will be strictly for fixing the bugs and finishing the build system. After that is done I will update the custom packages and try to get them upstreamed.
I still try to get my patches upstream.
For the libiwinfo patch I had to add the lua bindings. I never used lua so first I had to get comfortable with this. Additionally I wanted to add the channel utilization in the luci statistics app. But suddenly Luci is giving me a null pointer exception in the dev branch.
Additionally I tried to get comfortable with Luci for developing my own app.
Meanwhile another developer created nearly the same patch for iwinfo that add the survey data for the nl802.11 driver… This patch is still not accepted. The only difference is that it returns all survey data for all channels (like iw dev wlan0 survey dump)…
Furthermore, my pull request for the hostapd ubus bindings that add information about the ht and vht capabilities had to be rewritten. (https://github.com/openwrt/openwrt/pull/898). Again I have to wait for some feedback. While rewriting this patch, I had a new idea: If you subscribe to the hostapd via ubus and want to notify on the messages you have to activate it. It would be possible to add flags in the hostapd_ubus_bss to select what information should be published via the ubus bus. Before doing so, I want some feedback if this is a good idea.If somebody is interested why I am interested in the capabilities: I want to create a hearing map for every client. I’m building this hearing map using probe request messages. This probe request messages contain information like (rssi, capabilities, ht capabilities, vht capabilities, …). VHT give clients the opportunity to transfer up to 1,750 Gigabits (theoretical…) If you want to select some AP you should consider capabilities… In the normal hostapd configuration you can even set a flag that forbids 802.11b rates. If you are interested what happens if a 802.11b joins your network search for: WiFi performance anomaly. 🙂
Summarizing, I spent a lot of time waiting for feedback, debugging, modifying my patches or replying on the email lists. It is a bit frustrating.
The cool stuff was that I had my first pull request. 🙂 (it was just a typo ^^) But somebody took the time to fork my project and create a pull request. 😉
Furthermore, it is exam time and I have a lot of stuff to do for the university.
Actually I wanted to go on with more interesting stuff like connecting to the netifd demo to get more information.
I spent the last weeks mainly developing the LuCI Application for VRConfig. As soon as you want to do advanced things with LuCI, it gets cumbersome.
As the API is mostly undocumented, you have to dig through the LuCI’s source code trying out functions which could be useful according to their name.
It’s a bit of a trial and error game.
Currently the LuCI app does the following.
It displays an image of the router and parses the JSON file, which contains the locations of the components.
With this information it can mark the associated physical ports to the currently selected network interface and display those network ports, which are connected to a cable. You can also hover over the components and click on them, which leads you to their respective settings page.
I also improved the annotation app. It now lets you choose the router name from a list of all currently supported router models of OpenWrt. I got that list with a series of grep and sed commands from the OpenWrt git repository.
For your information, there are currently around 1100 different router models supported. 🙂
In the next weeks I will polish the LuCI Application and try to integrate VRConfig into the openwrt build system to be able to select the correct router image and JSON file at build time.
In this blog post I’d like to present the recent progress made for Eewids. This time, our main focus was the performance of the current setup. Besides we did some minor improvements, mostly adding some more dashboards to Grafana to visualize the data captured. This blog post focuses the results regarding Kismet as a component of Eewids. Continue reading “GSoC 2018 – Easily Expandable WIDS – Second Update”→
In the last weeks I was working on the storing process as described in the architecture in the last blog post .
Old app: the old app saves the data as byte in a file. A data entry is 28 bytes of MAC-address(12 bytes for 12 characters) and latitude(8 bytes for double) and longitude(8 bytes for double). An entry could be saved more than once in the file. There are 2 files, one for data which should be updated and one for data which should be deleted from backend.
New app: Firstly I wanted to adapt the structure from the old app. But since I saw some unreasonable points such as saving redundant data, flash workload, maintenance problem and unstructured storage, I decided for a standard database with more structure and easy to maintain: sqlite. Also I am using the new persistence lib, which provides an abstract layer for database: Room, newly released last year, as a part of android architecture components, with a lot of bug fixed since then. A lib with a lot of advantage when working with sqlite database: verify queries at compile time, reduce a lot of duplicate code in comparison with the last approach with DbHelper etc. In order to store the access point in the database, I implemented a seperate thread, which reads data from a blocking queue and saves it in the database, which works parallel with the scan thread and will be interrupted if there is nothing in the queue to store. Also to save energy and not force the store thread to run the whole time, a list of access points will be put into the blocking queue as an element. To pretend redundant data in storage, a data entry with BSSID will not be saved many times as in the old app but only once. The BSSID is used as primary key in the sqlite table. It will be updated the next time if the received signal strength is better than the last entry in the database. An explicit transaction is implemented to solve this case since the lib Room has only supported annotation for standard update/insert. To decide if a access point should be deleted or updated from backend, a flag is set.
The WifiUploader is in process. I did take a look at the uploading format in the old app and how it communicates with the current backend. Also the upload sequence is already defined, mean the scanning thread will be interrupted, all the rest of access point will be stored, the store thread will be interrupted to pretend conflict while 2 threads try to access same database at the same time before the uploading process is started. Also the WifiUploader will read maximum a number of data entries from the database and upload it, not the whole database like old app but one after another, in order to pretend out-of-memory problem at device with small ram. (see more in the below diagram)
But since I am in the middle of my final exam period, there will be a small delay until this weekend for the WifiUploader to be published. Also from next week I will spend full time making the other features done includes implementing all saving resource features such as adaptive scanning, implementing all settings option. A clean and full documentation will be provided at the end as well.