VRConfig Update

Hi,

I have some quick updates about VRConfig for you.
Short recap: VRConfig aims to introduce a graphical configuration mode for OpenWrt’s Webinterface LuCI.
For that need to collect pictures of the backside of all supported routers. The idea is to do this in a crowdsourcing manner. The community can submit pictures of their routers together with a metadata file which contains the locations of the components on the picture.

I spent the last weeks developing a web application to provide the annotation functionality of the router components.
A working prototype is now ready and can be tested at the following URL: https://vrconfig.gitlab.io/annotator/
Source code: https://gitlab.com/vrconfig/annotator

The annotator produces a JSON file which in turn can be parsed by the LuCI Application to provide the graphical configuration mode.

The LuCI application is being developed right now and will be provided shorty under the following URL: https://gitlab.com/vrconfig/luci-app-vrconfig

More info about that in the next blog post.

OpenWLANMap App: Update 1

Hi,

As I mentioned in the last blog post [0], the first step I did is defining the app’s functionalities[1] and designing the app architecture.

Basically the app contains 1 service, which runs in the background and communicates with UI thread per broadcast (public-subscriber pattern). Since as default, the service will run in the main thread, which is not wanted, I created a ScanThread to handle the scanning. It sends every 2s as default (and should be adapted with user’s speed etc. later) a scan request to the WifiLocator and gets a scan result back from it asynchronous. The WifiValidator then validates the scan result as well as the returned location, and puts the valid wifi access point in a WifiQueue. The WifiStorer will take anything from the WifiQueue and writes it to local disk (simple Consumer-Producer pattern). Based on the user’s upload mode setting, the WifiReader will be triggered if uploading is wanted, reads the local disk files in wanted format for uploading and passes it to the WifiUploader. It then uploads the data to any supported project api and as soon as the uploading is successful, the data will be deleted and ranking will be updated.

The next step I did is designing the new UI, got some feedback from mentor and changed it appropriately. Also in the process I defined all the user’s setting options. I spent a lot of time reading the android documentation for parallel processing and made decision for each functionalities, which is relevant for the next part. (WifiStorer, WifiReader: normal Thread, WifiUploader: AsynTaskLoader etc.) I write more about it in the next post.

Finally I jumped into implementing. I started with the demo mockup and then slowly implemented the logic part. I have finished the scan service and a part of the WifiValidator.The WifiLocator uses gps for defining location if available, otherwise it makes a request to openwifi.su with the surrounding wifis. I provided methods to do it with both new and old openwifi.su api in case we want to use any of them in the future. I ran into an android bug, where the wifi scan result is always 0 if the user disables GPS, even the location permission is granted.(Tested on Android 6). It is kind of weird because scanning wifi does not have anything to do with the gps and turning on gps the whole time will cost phone a lot of energy. Still it’s kind of wanted feature from Android to make users aware that their location information is being accessed when they use kind of app. Because the location of user’s phone could be defined based on the collected wifis. Since it is OS design, I pop up users a message with those information to ask them to turn on their GPS if they turn it off. I also implemented a part of WifiValidator, the WifiFilterer to check if an access point is openwifi, from freifunk or mobil hotspot or marked with _nomap (which should not be collected).

What’s now?

If you want to check the app, feel free to download install file .apk from [2].

If you as usual do not want to install an unknown source, I also provide a short demo video

 

What’s next?

In the next time, I will finish the WifiValidator, which should not only filter the access point but also validate the location to provide scan service a better scan period to save energy (in case the location is not changed for a long time, the scan service should be stopped etc.) and then other parts as shown in architecture image above.

Links

[0] https://blog.freifunk.net/2018/05/14/introduction-openwlanmap-app/

[1] https://github.com/openwifi-su/OpenWLANMap-App

[2] https://androidsmyadventure.wordpress.com/2018/06/03/openwlanmap/

 

VRConfig – Visual Router Configuration for OpenWrt

Hi,

I am Tobias, a Computer Science student at the TU-Berlin. This is my second time participating in GSoC for Freifunk.
I am excited about this project as it helps to reduce the entry barrier for inexperienced users of OpenWrt and its web interface LuCI.

When you look at the current LuCI Webinterface you will notice that it looks fairly decent, especially with the Material Theme.
However for an inexperienced user without a technical background it surely looks scary. All the text full of technical terms with few pictures can look like a book with seven seals.

This project aims to introduce a graphical configuration mode.
To make the configuration interface more connected to the actual router the user owns, we want to display an image of the backside of the ports in the web interface.
The user shall be able to interact with this graphical representation of the router by hovering and clicking on the different parts like LAN ports, antenna etc.

What are the necessary steps to archive this goal?

First, we need pictures of the backside of all the different router models. Here the idea is to collect them via crowdsourcing by the community. Everyone can take a picture of their router and upload it to a Git repository. Also, the location of of the router components must be marked on every picture. For that I will develop a small application which allows the user to annotate a router picture and generates a metadata file.
Second, the annotated pictures need to be integrated into the OpenWrt buildsystem.
Third, a LuCI application needs to be developed to display the result as an interactive graphic in the web interface.

In the next blog post I will go into more detail on the individual steps as well as update you about the progress.

Introduction: OpenWLANMap App

Hi,

My name is Lilli and I am studying technical computer science at the 6.th semester in Hamburg, Germany. In this summer, i will work on a new wardriving app for openwifi.su

OpenWifi.su is working on wifi positioning system. It uses an android app called OpenWLANMap App as the wardriving tool to collect wifi access points and sends it with the geolocation of the phone to the backend. The backend stores the data in database and also offers an API for non-GPS devices to request their positions based on the surrounding wifis.The backend currently uses triangulation technique on the numbers of access points it receives from the request device to calculate it’s geolocation.

A data entry from the wardriving app is currently BSSID + LATITUDE + LONGITUDE, which is stored temporarily effectively in local disk with 28 bytes (12 bytes for 12 characters of the MAC Address, 8 bytes for each latitude and longitude) before being uploaded to backend. The wardriver can do it manual or automatically. The app respects _nomap Wifi APs and does filter out some mobile hotspots on public transportation in europe. Unfortunately the app is hardly out of date. There are no developers working on the app and no updates for years. It does not run in new android devices. The OpenWifi.su has an amazing community of wardriver, over four thousands people. They have to keep very old phones to be able to run the app. But this community began to shrink in fact because peoples buy new phones which are not able to run the old app anymore.

I am so glad to get involved in the project and can spend my Google Summer of Code rewriting the app. Thank to freifunk community I was allowed to participate in the Wireless Community Weekend in Berlin in the community bonding period and had the chance to present my project, as well as talk to people about it and possible solutions for many problems. I spent the last weeks analysing the old app code and was talking a lot with my mentor about the old app performance and functionalities and ended up with many important decision about the design for new app. Here are some of them

  1. Wlocator is a service of getting GPS either from the device itself. In worst case where the GPS is undefined, it will send a request with the surrounding wifis to the Openwifi.su backend to ask for it’s geolocation. The service will run every n seconds
  2. Wifiscan is a service of scanning wifi access points. All the data of the AP such as BSSID, SSID, RSSI, frequency, channel , encrypted method, scan timestamp etc. will be scanned and display as user’s option. Necessary data will then be stored temporally local and later on uploaded to the database. The service will be stopped if the GPS of the device isn’t changed after n seconds in order to save device’s battery.
  3. Wififilter helps filter out: _nomap, mobile hotspot ( Call for help to collect mobile hotspot from different countries), ad_hoc network (I am working on it), collect open wifi to automatic connect and upload data if possible.
  4. Use trilateration or other techniques to define the location of the scanned AP better
  5. WifiUpload let users upload the data manual, automatic if internet available, automatic if only wifi available
  6. Extension: upload data to different APIs
  7. Saving resouces: stop scan service after n second of not changing GPS, put app in standby/doze mode if GPS does not change after n second, reduce brightness, kill app if battery critic
  8. Different languages available
  9. etc.

The plan for the next three months will be:

  1. Design new architecture for new app functionalities
  2. Design new UI
  3. Implement all logic functionalities
  4. JUNIT test + documentation

Stay tuned, I will update it soon.

L3I2

GSoC 2018 – RetroShare Web Interface Modifications/Improvements

Project Introduction

RetroShare, the encrypted peer-to-peer social networking system, has a web interface which makes it easier to use from multiple devices without running multiple nodes. The web interface works, but it is in need of improvements in its API, appearance, and user friendliness.

The web interface is particularly important to those who wish to use RetroShare heedlessly on a server, so they especially will benefit from improvements to it.

This project will mainly involve building on and redesigning the web interface API, libresapi.

Who I am

My name is Kevin Froman, I am a 19 year old American computer science student. I am passionate about online privacy, information security, and decentralized networking. I mainly have worked in web and mobile development, however I have some experience in standard desktop development as well.

More info about me, as well as a mirror of blog posts written here, is available at my website.

Project Overview

As with all projects, mine is made up of several main goals:

  • Switching the RetroShare web server library to Restbed (https://github.com/corvusoft/restbed)
  • Properly serializing all data in and out of the API as JSON
  • Refactoring and/or migrating existing code where applicable
  • Maintain existing supported RetroShare features in the UI (do not lose any in migration)
  • Add authentication (likely sha256-HMAC) to the interface to protect against certain browser attacks
  • Re-designing the front end, both style and functionality, likely using VueJS.
    • I am not a web designer, however I can definitely improve the appearance

Bonus goals include:

  • Adding more RetroShare features to the UI (the features the web UI supports is incomplete)
  • Making the web interface easier to setup on headless servers
  • External HTTP API mechanism with CORS (Cross Origin Resource Sharing)
The web interface’s appearance is currently not a work of art.

 

Bonding Period Experiences

Leading up to and during the Community Bonding Period, I got to know my mentors Cyril and Gio better and explored the RetroShare code base, especially libretroshare and libresapi. We discussed details about the project and the way project goals will be implemented.

As a way to dive into RetroShare, I worked on a backup system for RetroShare profile data to XML files. Unfortunately, I did not get a lot done on this, mainly do to final exams and projects for my classes which overlapped with most of the bonding period, but also due to roadblocks I ran into involving my development environment, personal issues, and less than optimal time management on my part. I felt I needed to spend time looking at libresapi and studying the code base in general, in order to be better prepared for the main coding period. The issues I had have been resolved.

At first, I found diving into this a large, multi-author code base intimidating, but I learned more of its structure and feel more confident for the main project.

First Goal/Milestone

My first project milestone is to fully analyze libresapi and to look for any code that should be kept/refactored and to finalize exact details of the new API system (white-boarding, prototyping, writing unittests). If all goes as planned, the first milestone will be done before May 28th.

I expect this goal to be accomplished early, so I will likely also be starting on the second milestone during this time, which is to get the base API system setup (both for the front-end and back-end).

 

I look forward to a summer working on RetroShare, and I believe my project will be a valuable contribution.

I wish the best of luck to fellow Freifunk GSoC students!

geolocator (Software defined GPS) final evaluation

Hi everyone,

with this blog post I would like to explain the full Google Summer of Code Project as a final post. For people who haven’t read over the geolocator (Software defined GPS) project before, it might be interesting to read these three blog posts at first:

– geolocator (Software defined GPS) (english)[1] and (german)[2]

– geolocator (Software defined GPS) first evaluation (english)[3] and (german)[4]

– geolocator-software-defined-GPS-second-evaluation (english)[5] and (german)[6]

Otherwise I will give in the following a short overview about the project structure to remind you of it. I structured the Google Summer of Code project into 3 main subprojects:

web backend,

– The web backend named sgps-core is a service, which should give requested clients their geo position.

gps-share,

– The idea of gps-share is to create an udev device, which provides NEMA-formata protocols over tty addicted on information, which is received from the above mentioned backend.

LEDE Package,

– The intention behind this subproject is to develop a new package for LEDE called geolocator, which should provide the geo position of LEDE devices.

Now I would like to give you a full state of each above mentioned subproject. Firstly I will explain about the web backend and finally a peroration including my valediction as a Student by Google Summer of Code.

web backend

Generally the backend service receives over the OpenWLANMap[7] App from mobile phones the mac addresses of surrounding wireless networks linked to GPS positions. This information will be stored into a database. If a device like a WiFi router requests its position, it will send surrounding Wireless mac addresses to the backend and get back a geo position, which is calculated from these information in the database.

The new web backend called sgps-core[8] is an API-core, which should replace the old openwifi.su backend. The old one consists of a collection of different programs in different program languages. sgps-core includes a fully backwards compatibility to the old openwifi API[9] for requesting a position. sgps-core is written in Golang, which processes a lot faster than the old API, which is written in Ruby. sgps-core is more secure because it checks and take only requested strings, which contain only comma separated macaddresses with 12 hex characters.

As a fallback feature, sgps-core is able to receive coordinates from unknown WIFIs by requesting them on Mozilla Location Service (MLS)[10] if there are no db entries for that WIFIs. The position for clients will be returned in form of latitude and longitude. As a quick reminder, here is the schemata from the first post, which represented the the functionality of sgps-core:

The sgps-core solved a problem about calculating the position. The old method counts the average of all latitude values. Analogous for longitude. The new method calls geographic midpoint calculation and needs 4 parameters lat0, lon0, lat1, lon1 (give two take one) which will be explained in detail in following:

deg have to be replace with latitude or longitude value.

rad = deg *π / 180 <- Generally conversion from degrees to radiant.

dlon = (lon1 – lon0) * pi / 180

lat0 = lat0 * π / 180 <- lat0 from degrees to radiant.

lat1 = lat1 * π / 180 <- lat1 from degrees to radiant.

lon0 = lon0 * π / 180 <- lon0 from degrees to radiant.

Converting into Cartesian coordinate system.

Bx = cos(lat1) * cos(dlon)

By = cos(lat1) * sin(dlon)

Calculate new position reference to sphere and Converting back from Cartesian coordinate system into new latitude and longitude:

lat2 = atan2(sin(lat0) + sin(lat1), (cos(lat0) + Bx)² + By²)^(1/2))

lon2 = lon0 + atan2(By, cos(lat0) + Bx)

On this point it is also possible to use a ellipsoid to increase the accuracy of positions. This may be interesting for long distances. For short ones like from seen wireless networks, it is not really relevant.

Converting back to degrees:

deg = rad / pi * 180 <- Generally conversion from radiant to degrees.

lat2 = lat2 / pi * 180 <- lat2 from radiant to degrees .

lon2 = lon2 / pi * 180 <- lon2 from radiant to degrees.

In the last few weeks, I spent a lot of time on discussing with the current server administrator of openwifi.su to deploy the sgps-core on the server for a test environment. But he did not have much time, so we decided to migrate the openwifi.su to our Nordwest Freifunk infrastructure to make the administration more accessible for other people. In my last report I wrote “I will release in the next few days a first version”. This could not be done because of the above mentioned discussion. After the migration I can test that backend on huge databases and compatibility to the DBS. The current code can be found here [11]. For people who want to try the sgps-core please check out the following URL[12].

gps-share

The Idea at the beginning of GSoC17 was to write a program to provide GPS NEMA-formats over a tty udev device. The information for the GPS NEMA-formats should come from the above mentioned sgps-core. As I told in the first blog post I discussed with some people from the Mozilla Location Service Malinglist and it turned out that something similar was already exist called geoclue. To avoid developing redundant software I decided to drop this idea. Instead of it the new plan was to build support for native GPS in gps-share[13], which is an add-on for geoclue. But during the Google Summer of code I had to focus more on the both other subprojects because they are more important, especially for Freifunk. In my peroration I will tell about the future plans, especially for gps-share.

LEDE Packages

The third subproject was to create some opkg packages for LEDE[14] and similar Frameworks. The main package called geolocator provides the geo position of the device via UCI[15]. Positions should be received from the above explained sgps-core. The 4 other packages are only for Gluon[16], which add the configuration options of the geolocator to the Web-interface.

This month I mainly worked on the LEDE Packeges. At the beginning of the august I sent a merge request to Gluon for integrating the Gluon-geolocator[17]. The containing geolocator program was written in ash shell code. While reviewing and discussing about the merge request, I realized that I had to rewrite the program from shell to lua code because Gluon mainly work with in lua written programs. You can find the shell code here[18] and the Lua version here[19]. At the moment I am waiting here for another review and subsequently merging.

The other packages for the Gluon Web-interface are also already in process. The first idea was to create a detection of installed packages to show related configuration options on the Web-interface. This idea was dropped because I found a better solution. The problem is detecting packages on runtime, which means many extra code on Routers, which only have 4MB Flash for example. So I decided to generate the package with their options on compile time. These packages are:

gluon-config-mode-geo-location,

gluon-config-mode-geo-location-with-geloc-map,

gluon-config-mode-geo-location-with-geloc,

gluon-config-mode-geo-location-with-map

The main package is gluon-config-mode-geo-location, which is already exist in gluon, but with a difference Web-interface. Each package should either integrate an open street map or the geolocator options. Integrating both are also possible. For communities which would like to stay on the currently variety of functionalities, it is also no problem not integrate any of these extra options.

Here is how the new packages look like:

I wrote some C++ programs, which generate me the Lua code for the Gluon Web-interface, which is written in Lua. Base on preprocessor variables, the amount of options for each package will be included into the Lua output from the C++ program. These preprocessor variables will be set by selecting one of the above packages. Also PO files for the translation will be generated in the same way. A merge request of the above new packages can be found here[20] I am still working on it.

Peroration and Future plans

Now I am coming to my peroration.The last 3 months were really awesome, just like last year as a student on the Google Summer of Code. I would love to recommend this great opportunity for not only students but also for open source organizations. Students can not only learn a lot of new things but also meet new great people, make new friends and take part in many events. For example: at the beginning of august I was on the SHA2017[21] (Still Hacking Anyway) and had a meetup with some Freifunk communities there. We had a great discussion about a lot of technical stuffs and a nice time for socializing. The SHA2017 took place in Netherland nearby Amsterdam. Another example is : this week I flew to Spain to start my exchange semester. Coincidentally a student from Germany who I met at the beginning of the GSoC17 in Berlin on the WCW[22] (Wireless Community Weekend) is also doing an exchange semester in Spain. We have already emailed each other and planned to meet up in the next months, probly in Barcelona or any other places. As I said above, this is my second time as a student on the GSoC, which means this is also my last time and sadly I have to say goodbye to GSoC as a student now. But maybe next year I can work as mentor to support other students in their great opportunities.

Back to the projects, as i said I’m still working on it. I will finish the Integration into Gluon and LEDE and continue developing sgps-core integrate new features and migrate the infrastructure to a better server. I would like to contact Zeeshan Ali, the maintainer of gps-share and try to help on this project as well. Also I am still working on the hoodselector which is my Google Summer of Code project from the last yeah. You can read about it here[23]. The hoodselector should also integrated into Gluon but it requires for sure a few weeks of work to integrate VXLAN on it. A merge request for can be found here[24].

Also I would like to say thank you to my Mentors Clemens John from the Google Summer of code 2016 and Johannes Rudolph from 2017 and especially to Andreas Bräu who works so hard on the Freifunk Org for many years to give students these opportunities to be a part of the Freifunk Community.

[1] https://blog.freifunk.net/2017/05/29/geolocator-software-defined-gps/

[2] https://ffnw.de/geolocator-software-defined-gps/

[3] https://blog.freifunk.net/2017/06/28/geolocator-software-defined-gps-first-evaluation/

[4] https://ffnw.de/geolocator-software-defined-gps-erste-evaluation/

[5] https://blog.freifunk.net/2017/07/26/geolocator-software-defined-gps-second-evaluation/

[6] https://ffnw.de/geolocator-software-defined-gps-zweite-evaluation/

[7] https://f-droid.org/packages/com.vwp.owmap/

[8] https://github.com/openwifi-su/sgps-core/blob/master/README.md

[9] https://sourceforge.net/p/libwlocate/code/ci/master/tree/master/

[10] https://location.services.mozilla.com/

[11] https://github.com/openwifi-su/sgps-core

[12] http://runner01.ffnw.de:8082/api/v1/bssids/64700274945A,0018E7DC21BB,30FC681F37A6,F81A673626E8,EC086BA8ED18,8416F9A8E2FA,E894F6A230E4,98DED020D00A,EC086B3349B2,F81A677F5CD8,A0F3C1654BA6http://runner02.ffnw.de/64700274945A,0018E7DC21BB,30FC681F37A6,F81A673626E8,EC086BA8ED18,8416F9A8E2FA,E894F6A230E4,98DED020D00A,EC086B3349B2,F81A677F5CD8,A0F3C1654BA6

[13] https://github.com/zeenix/gps-share

[14] https://lede-project.org/

[15] https://wiki.openwrt.org/doc/uci

[16] https://gluon.readthedocs.io/en/latest/

[17] https://github.com/freifunk-gluon/gluon/pull/1201

[18] https://git.ffnw.de/ffnw-firmware/packages/blob/ba8007e5a6b99306068847ba98f2bb72b7fb2745/ffnw-node-info/files/lib/ffnw/geolocator/geolocator.sh

[19] https://github.com/2tata/gluon/blob/fa935d83a64c27dd23f133bd8cecf5d72ef9281b/package/gluon-geolocator/luasrc/lib/gluon/geolocator/geolocator

[20] https://github.com/freifunk-gluon/gluon/pull/1211

[21] https://sha2017.org/

[22] https://wiki.freifunk.net/Wireless_Community_Weekend_2017

[23] https://blog.freifunk.net/2016/08/22/monitoring-and-quality-assurance-open-wifi-networks-out-client-view-final-evaluation/

[24] https://github.com/freifunk-gluon/gluon/pull/997

 

PowQuty Live Log GSoC 2017 Final Update

This is the last blog entry in the series of Google Summer of Code project updates. It will describe, what has been done and what is left to improve in the future in the PowQuty project.

Powquty

PowQuty is a power quality monitoring tool, which can be installed on a router running LEDE or OpenWrt. The router can be connected to an USB oscilloscope providing measurements which powqutyd will process and provide to the user in human readable form.
All this was tested on a x86 based LEDE router.

GSoC 2017

During this Google Summer of Code a live log functionality was added to PowQuty to provide information on power quality events. These events are:

  • voltage dip of 10% – 90% of the reference voltage on the measurement signal
  • voltage swell > 110% of the reference voltage on the measurement signal
  • voltage dip < 10% of the reference voltage on the measurement signal
  • > 5% of the measured values of one specific harmonic are over the defined threshold
  • On event occurrence important information like time, duration and event type will be written to a log file and presented in the extended luci app.

    As shown in the above picture, the interface provides a traffic light like color system behind these events, green indicates everything is within the EN50160 power quality norm. Yellow means, that 80% of the maximum time per week is already reached, red means, that the norm was violated during the last week.
    In addition to log writes, notifications are send out with Mosquitto. Mosquitto is a message broker using the MQTT protocol. It provides a publish/subscribe model, which allows a central server to subscribe to a topic and clients to send out messages to the server with a topic. Mosquitto was already in use in powquty but was extended for EN50160 event notifications. This will allow a central logging of bigger power supply networks, monitored by multiple devices.

    As another option Slack messages can be send by powquty now. Slack is a messaging program, using (as one option among many) webhooks for interaction. Everyone with the webhook can send messages to the team. Sending out messages allows a user to react quickly to changing situations, or get immediately informed on power event occurrence.

    Beginning with pull request 20 [https://github.com/thuehn/PowQuty/pull/20] I started to implement these features.
    First an option was developed to read measurements from a file, as most power supply networks are pretty stable and wont provide many opportunities to test event handling in powquty.
    Afterwards slack and MQTT notifications where added.
    During testing of mosquitto event messages, some seemed to be lost on intervals with many En50160 events in a short period(sometimes more than 35 events per seconds). The solution seems to buffer all events before sending.
    Something similar happened with Slack. Slack only allows one message per second(short bursts excluded) [rate-limits].
    Buffering events would resolve this problem as well. An option for live email notification was considered at first, but was dropped as spam protection would stop most of the messages and probably list users as spammers.
    The last step was to add the traffic light system to the luci app, to enable users without knowledge of the norm to get an idea of the power quality of their power supply network.
    In addition a slack library was written [libwebslack] to send slack messages from PowQuty.

    What can be improved
  • As mentioned before event buffering is needed and will be added after GSoC
  • Email notification in form of a weekly summary
  • More Error checking and handling
  • improving libwebslack to not use libcurl to reduce its size
  • provide libwebslack as OpenWrt/LEDE package, for easier future use
  • Finally I have to thank Dr. Thomas Huehn for being my mentor and Freifunk for their work they do and especially for being a mentoring organisation for Google Summer of Code.
    Last but not least I would like to thank Google for making this all possible.

    If you want to review some of my earlier posts:

  • Introduction
  • First Update
  • Second Update
  • Best regards
    Stefan

    netjsongraph.js – Google Summer of Code (GSoC) 2017 summary

    Throughout the last three months, I was quite fortunate to work for Freifunk on netjsongraph.js under the guidance of my mentor Federico Capoano. Thanks for this invaluable experience that I learned a lot of knowledge and use them in a practical project. Here is a summary of the work I have done during the Google Summer of Code (GSoC) 2017.

    Google Summer of Code project page

    Goal

    netjsongraph.js is a visualization library for NetJSON, a network topology data format. The main goal of netjsongraph.js may be concluded in below three lines (more details you can see in GSoC 2017-netjsongraph.js: visualization of NetJSON data):

    • Apply the modern front-end development tools and add tests workflow (#1, #45)
    • Rewrite it with WebGL (#11, #29, #39, #42, #47)
    • Improve the performance (#41, #44, #46)

    Achievement

    Github Repository : https://github.com/netjson/netjsongraph.js

    Examples on GitHub pages: https://netjson.github.io/netjsongraph.js/

    You can browse all examples on GitHub pages. Some screen shots of the application:
    basic example
    performance example
    The force-directed layout is usually used to visualize network data. It offers insights on the relationships between nodes and links. The previous version of netjsongraph.js is implemented by d3 and it’s rendered using SVG. It would be very slow if there were thousands or ten of thousands nodes or links. So I have to embrace the WebGL speeded up by GPU to have a better performance.

    I have recorded my work in the blog every milestone:

    BTW, It’s a great management method to make members submit weekly reports and blog posts in Freifunk.

    During the three months, there have been 116 commits from me. I created a big Pull Request include them:
    netjsongraph.js #48
    netjsongraph.js project panels
    Almost all goals have achieved:

    • Published a minor version
    • Improved development workflow
    • Tests Added
    • Refactored visualization by Three.js and d3-force
    • Added more interaction like hover (show nodes tooltips), click (show nodes or links information panel), pan and zoom
    • Improved performance

    Especially on performance aspect, it runs efficiently on Chrome reached 60FPS under 5k nodes and 10k links. And if you don’t wanna animation, you can choose the static rendering layout.

    Challenges

    I also encounter some challenges I never met before.

    Event binding and handling

    As you know, WebGL renders all objects in one canvas tag. How to bind events on every geometry? You should use the Ray casting. Raycasting is used for mouse picking (working out what objects in the 3d space the mouse is over) amongst other things. So you can know which geometry your mouse over and add some interaction effect.
    There are thousands of objects and every object has several events you should handle, I had to develop an event controller to manage it.

    Performance

    The bottleneck in this visualizer is performance(#41). I tried many methods to improve it include:

    Reuse geometry and material

    However, the color of every node is different and the one link should highlight itself when it hovered, so the material should be independent and can not use in common.

    Combine the mesh

    Same problem with above. It’s not flexible to combine them to one mesh, different nodes and links should have different positions.

    Static rendering

    Make calculation before rendering, so there is no animation and repaint.

    Using Web Worker

    Web Workers is a simple means for web content to run scripts in background threads. The worker thread can perform tasks without interfering with the user interface. So put static layout calculation into it will be efficient.

    Force-directed algorithm

    There are different complexity and cost in the different force-directed algorithm. The Force-Atlas2 algorithm has some benefits over the force layout implemented in d3-force. So current version may be refactored by an advanced algorithm in the future.

    What is left to be done

    • Add optional geographic map (#40)
    • Using Force-Atlas2 algorithm

    More interactions and features should be added, and performance may be optimized by using new algorithm. I’d like to continue developing this project after GSoC.

    In the end, thanks for the great patience and guidance from my mentors. Thanks for Google to provide me with this rare chance to contribute to an open source community together with awesome members from all over the world. I really appreciate this invaluable experience accumulated this summer and I believe it will have the profound impact on my career and life.

    GSoC 2017 – Attended Sysupgrade – Final evaluation update

    Hi Freifunk,

    This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.

    tl;dr:
    Direct links to the work I’ve done this GSoC:

    First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.

    Progress

    Server

    At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.

    The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.

    Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.

    To try the the server at https://betaupdate.libremesh.org have a look at the demos I prepared. Continue reading “GSoC 2017 – Attended Sysupgrade – Final evaluation update”

    GSoC: Improving nodewatcher data representation capability (update 3)

    I have been working on improving the wlan-slo main page. This page has not been updated in a long time and it is crucial at introducing new people to mesh networking.

    Continue reading “GSoC: Improving nodewatcher data representation capability (update 3)”