GSoC 2017 – RetroShare mobile improvements – Final evaluation update

Hi Freifunk and GSoC communities!

Finally we arrive at the end of the summer and also at this third final evaluation of GSoC!
I’m proud I have participated to this edition of GSoC and I am thankful for this opportunity.

This past month I did lot of new work and lot of interesting improvements.

One of the funniest and visible things are the implementation of the 8biticon library that provides a reliable avatar face generated using GXS id as input, it is reliable in the sense that having the same GXS id as input will generate always the same avatar.

The nice thing about this is that it can be used as an user friendly way to verify identities, the user doesn’t need anymore to check long fingerprints but she can do a quick verification just by looking at the generated face.

Another interesting thing I have implemented is sending little files and images via chat using data URL. This works encoding a file to base64 and send it with a header specifying the mime type. The maximum file size is 60KB, and for images, if it’s bigger than that or if it’s size is more than 160×160 pixel, the program resizes it to fit.

This branch results very interesting for me because it uses all the things that I learned this summer: call C++ code from QML, call Android Java code from C++ interact with Android OS, create custom QML components from C++ code…

 

Of course, a big part of work has been to find and fix bugs, on Android and on desktop. At the moment the last commit has little known bugs that I’m going to solve as soon as possible.

One of this bugs causes program crash on some Android devices when send or receive an emoji. This happen because I implement an alternative emoji font if no native compatible emoji font is found. On Android, the Android Emoji font is who provides the typography of emojis, but in some devices its not shown. So I decided to integrate the OpenSansEmoji project. That way, when the program doesn’t detect a compatible emoji font, it uses the OpenSansEmoji.

But the implementation of that is not working as good as expected and causes the crash mentioned above.


Another typical work has been to improve the UI aesthetics in little things: correct a height in one place, change the color somewhere else, draw a line separating the contacts. I create a custom button component with animation and the possibility to attach to them an icon. This is done thinking about to also centralize the application style sheets.

And thinking about the end user, I tried to improve how the user share or import the nodes identities. I put a shortcut on the menu to share identity, and also refactor a little bit the import/export trusted nodes view, making an informative popup that shows you a little bit of information about each option.

 

Also I have done some little attempt to implement key exchange using the device sound card using libquiet available in C, Java, JavaScript and more languages.

This is a very interesting functionality that would improve the key exchange process and make it user friendly.

The functionality is not implemented yet as I believe it would take more time then the GSoC as it seemed more difficult than we thought initially, but I would like to continue working on this topic in the future.

In latest days I have been working with my mentors to do the final polishing and integrate most of the code in RetroShare official repository master branch, most of the code have already been merged as you can see in this commit e06bffdb by one of my mentors.

As always you can find all the code and activity track on my gitlab repository, I also suggest to take a look to the detailed technical report and if you are eager to test the results on your phone you can download the Android Package too.

Finally I want to say goodbye, and again appreciate the opportunity to participate to this exciting GSoC!

GSoC: Improving nodewatcher data representation capability (update 3)

I have been working on improving the wlan-slo main page. This page has not been updated in a long time and it is crucial at introducing new people to mesh networking.

Continue reading “GSoC: Improving nodewatcher data representation capability (update 3)”

GSoC 2017-netjsongraph.js milestone 2

After a month of efforts, netjsongraph.js has been greatly improved.
First, I added some API and options are compatible with previous versions, except those that can not be added to the WebGL element. Followed by updating the documentation and some examples so that users can better understand and try out our projects. Finally, I have optimized interaction and performance.

netjsongraph dark theme
netjsongraph dark theme

Interaction

  • added highlight effect on nodes and links when hovered.
  • added tooltips with node information
  • added pan and zoom interaction
  • added node and link information panel when clicked
  • adjust the size of the canvas when the window resized

Performance

The large part of the performance of force-directed layout is consumed in each element position calculation in every tick. So if the position calculation of the elements is before canvas rendering, it is effective to reduce the performance loss.
So I made a static layout https://github.com/netjson/netjsongraph.js/issues/44, there is no animation when rendering, directly render a static force-directed layout in the page.

Other

Besides, I have refactored the event controller, in order to make zoom and pan or other events added easily.
But now the overall rendering results are not good as our expected. There is a little lag in animation, I currently have no idea about the reason, perhaps because this is not best practice of three.js. In the next stage, I will focus on solving performance problems and improving the visual effect.

Next Plan

The links below may be useful:

  • http://taligarsiel.com/Projects/howbrowserswork1.htm
  • http://igoro.com/archive/what-really-happens-when-you-navigate-to-a-url/
  • https://github.com/alex/what-happens-when
  • https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – July progress

What we have so far

After lots of experiments, I figured out the way which enables the transparent MPTCP operation for both the client and the server (with the regular kernel).
I tested the configurations at my virtualized environment, check the video below. For configuration infos, please read the video description. (the post continues)

Continue reading “GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – July progress”

geolocator-software-defined-gps-second-evaluation

Hi everyone,

 

I am now taking you over my second evaluation. A German version of this blog post can be found here [0]. For people who did not read over the geolocator (Software defined GPS) project before, here is a link to the project introducing (english)[1] and (german)[2]. The first evaluation from last month can be found here (english)[3] and (german)[4]. First of all I would like to tell you about the web backend:

 

Web Backend

As I said it in the blogpost before, the new web backend should replace the old one.
The current openwifi.su web backend consists of a collection of different programs in different programlanguages. The new one is written in Golang it has a fully backward compatibility. The new web backend is more secure.  It checks all requested strings at the end of the url that they contain only comma separated macaddresses with 12 hex characters, as a validation check.

 

Also the new web backend fix an issue by calculating the requested position. The method before just adds all latitude values and devides them over the amount of positions. Same for longitude. The new way is a geographic midpoint calculation. First the lat/lon values will be converted into radius and then the radius will be converted into a Cartesian coordinate system, after that the midpoint calculation will be done and will gave us a lat/lon in the form of radius and the values will be converted back into degrees, that is all.

 

The web backend has a new feature which should help to enhance the position quality. If there are BSSIDs unknown in the database of openwifi.su, the new backend sends an extra request for the unknown BSSIDs to Moziller location service [5] and brings that information into the calculation of the response.

 

At the moment the new web backend is being tested. In the next few days a first version of the new web backend will be released. The current code can be found here [6].

 

LEDE Package

The last project is to build packages for LEDE and send merge requests to Gluon. The idea behind that is, for WiFi routers and other WiFi devices to be able to automatically receive their own position. This could be a nice feature for mobile Freifunk Routers. It could also provide a good alternative instead of setting positions by hand. A first version of that package is here [7].

 

For the integration in Gluon I plan 4 new packages. The first one is the gluon-geolocator, which depends on gluon-node-info and provides the programme with the capability of receiving geopositions based on surrounding WiFis.

 

The next 3 packages are just for the Gluon configmode. The configmode is a state that begins at the initial start of a Gluon router and the owner of that router can configure it over a LAN connection.
I would like to integrate the following 3 packages within the gluon-config-mode-geo-location.
They are named:
gluon-config-mode-geo-location-with-geloc-map,
gluon-config-mode-geo-location-with-geloc,
gluon-config-mode-geo-location-with-map

 

As their names indicate, the package gluon-config-mode-geo-location-with-geloc-map should integrate the options of the geolocator and an OSM map into the configmode. The other 2 packages, gluon-config-mode-geo-location-with-map and gluon-config-mode-geo-location-with-geloc should either integrate only an OSM map or just the configuration options of the geolocator. Also the package gluon-config-mode-geo-location should be able to and will show the current location interface without any map or geolocator.

 

A merge request of the above new packages will follow soon. Some of the gluon-config-mode-geo-location extensions are currently not finish yet.

 

In the next few weeks I would like to finish the integration into Gluon, as well as continuing to develop the new web backend and releasing the first version of it. There are many ToDo points currently open e.g. Multicore functionality, optionally other geographic midpoint calculations. At the beginning of August there is the SHA2017[8], which I will use for a marathon session.

 

With best wishes
Jan-Tarek Butt

 

Luci2 on Libremesh – July

At the moment the development of luci2 is found rethinking some decisions and advancing in others. The thread of the discussion can be seen in
https://github.com/jow-/luci-ng/pull/42 mainly pushed by @ianchi and commented on by @jow.
For my part I started looking for a mid-point between the LiMe App (developed in preact for LibreMesh) and Luci2. At this moment LiMe App uses orangerpcd in the backend and the same api is being migrated to be consumed from ubus.
The plug-in test for “luci2” in preact can be seen at https://github.com/gmarcos87/preact-luci, although the main development will continue from now on in Agular, but trying to maintain an aprouch where views/apps are Completely defined in json.
I hope to get to the next post with the libremesh api on ubus ready to be consumed by Luci2, whatever their final destination.

GSoC 2017 – RetroShare mobile improvements – Second evaluation update

Hello everybody!
This month of coding on RetroShare has been very productive again, with many user experience improvements and bug fixes to the mobile app.
The user can now pick it’s own avatar in an integrated manner. On Android when the user attempt to change her avatar she is directly prompted to select between the available image sources like the gallery or the camera as you can see in the screenshot.
To implement it I had to write some part of the code using Android Java API to create an intent with a chooser, and then handle the answer. After that, using Qt the image is encoded in PNG format and then
to Base64 to handle it to the retroshare JSON API.
Moreover avatars are now cached so the can now be showed the avatars in the contacts view without high resource consumption.
Another improvement is the Unicode emoji input support, adapted from the QMLemoji project, that uses the default system unicode icons.
This enable the user to use emojis whitout heavy and tricky text substitution and extra image bundling.
For platforms where Unicode emoji are not supported yet we plan to just ship a font with Unicode emoji support so the user experience will be coherent accross platforms.
Some more examples of improvements happened this month: chat bubbles now recognizes links and if clicked they are handled to the system with that will open the proper application depending on the URI schema, buttons are now styled and supports icons, importing and exporting of trusted nodes public key is more integratend with the system and more intuitive.
In a separated branch on my repository I have implemented showing to the user the tunnel connection status, but after talking with my mentors, we decided to not merge it yet at least until the new chat system backend is ready because having this now would provide partial information that may confuse the user in some cases.
As always you can get/comment/contribute to the code and followe the discussions on my public Gitlab repo.

one server to sysupgrade them all – second report

During the last weeks I’ve been continuously working on the “image-on-demand” server and the Luci interface. The packages have been split and renamed to be as modular as possible. Developing has been split in the following four packages:

luci-app-attended-sysupgrade

The web interface.  Uses JavasScripts XMLHttpRequest to communicate with ubus and the upgrade server.  Shows basic information to the user.

rpcd-mod-attended-sysupgrade

Has a single sysupgrade function called sysupgrade. It’s a copy of to the Luci sysupgrade function but callable by ubus. This package may be replaced by a native ubus call like ubuscall system sysupgrade

rpcd-mod-packagelist

Returns all currently installed packages, ignoring packages that has been installed as a dependency. It works without the need of opkg installed using /usr/lib/opkg/status. The packages hence requires the package lists to be included in the existing image, ie. keepinto unset CLEAN_IPKG on image build. This module is important for routers with less than 4MB of flash storage. The standard package selection does not install opkg for such small flash devices.

sysupgrade-image-server

The package will install the upgrade server with all dependencies. This especially useful for the planed setup of a single web server and a dynamic amount of workers building the requested images.

In the current implementation a single queue builds one requested image after another. A new approach will separate the we main sever from the building workers. The database is reachable by every worker and they setup ImageBuilders automatically depended on requested images. This could make the service easily scale able distributing the load to zero-conf clients.

The server handles the following tasks:

  • Receiving update requests
    • Validate request
    • Check if new release exists
    • If not, check if packages got updates
  • Receive image requests
    • Validate request
    • Check if image already exists
    • If not, add image to build queue
    • Return status information for the router to show in Luci
  • Forward download requests to a file server
  • Manage workers
    • Starts and stop  workers
    • Receive created images
  • Count downloads

The workers handle the following tasks:

  • Request needed Imagebuilders from the main server
  • Download, validate and setup Imagebuilder
  • Build images and upload to main server

Requesting Images

Do demonstrate how the image requests work I setup a few small scripts. Once called they request the image specified in the json file. Don’t hesitate to modify the files to perform whatever request you’re interested in.

Depending on the request you receive different status codes:

201 imagebuilder is setting up, retry in a few seconds
206 image is currently building, retry in a few seconds
200 image created

The requests repetitions are handled by the luci-app-attended-sysupgrade but must be manually retried by the current demo scripts. Eventually a CLI will follow.

Currently the distributions LEDE and LibreMesh are supported.

Future

Soon I’ll create a Pull Request for all three router packages to merge into the OpenWrt community package repo. At that point current installations could install the packages for further testing. All three packages are tested with a virtual x86 machine and a wr841 router.

Currently Secure HTTP is used for image distribution, to improve the security I’ll look into usign to create signatures for all images.

To enable reproducible builds I’ll create an auto generated info page with all information about the image, especially include the Imagebuilders manifest file listing all installed packages with versions.

The Luci view should be more verbose and also able to notify on updates directly after login (opt-in). Currently a simple “update successful” message will notify the user after an update. This could be expanded to a report function to let the update server know which images fail and which work. All gathered information would be helpful to create different release channels.

PowQuty Live Log Second Update

It’s been nearly a month since my last update on the PowQuty Live Log project and i would like to tell you, what has been done
so far and provide information on what will be done in the next month.

PowQuty got updated, to support Slack and mqtt event notification and can already be used in the current PowQuty version.
In addition to this, there have been some bug fixes during the last month and some new features were added.
On event occurrence the event gets stored in a csv file and each entry is displayed in the luci-app. To increase the usability,
a traffic light system will be added, which will show for each event type its occurrence time and show if the current values
are in violation of EN50160.

  • Green: Everything is ok, no violation
  • Yellow: Close to a violation
  • Red: This event time is in violation of the norm
  • The event messages contain a times stamp, the duration of the event and updated event Type information, as well as event type related
    information and GPS data.
    .

    As receiving notifications or emails on every event occasion can get noisy, we decided to provide a weekly summary of the events in
    addition to the regular notifications.
    The user will be able to decide if he wants to receive this summary, every event, or both. We consider using the traffic light system
    here as well, to increase the readability and enable users to understand the quality of their power supply network, without a lot of
    knowledge of the EN50160 norm.
    We discussed individual intervals and keep it in mind as a possible later feature.

    Best regards
    Stefan

    OpenWifi status report before 2nd evaluation

    Hello everyone,
    this is the second milestone blog post about OpenWifi. In this blog post I want to talk a little more about the authorization in detail and give an update on what has been done in the last weeks.

    Authentication

    Authentication was enhanced with possibility to use a client side certificate or apikey (which can be send to the server with a HTTP-GET like ?key=MY_API_KEY). It uses groups to distinguish between the authentication methods and if you login with username/password it is also possible to mark a user as an admin.

    Username/password login uses cookies and client side certificates have to validated by a proxy (like nginx for example) and added to the header. It is you can find the code for authentication and groups.

    Authorization

    The basic principle of authorization in pyramid is a ACL based authorization that gets principals (basically strings) from authentication. These ACLs are given by so called factories (if a view doesn’t use a custom factory the so called root factory is used).

    I decided that it would make sense to have two strategies for authorization – coarse grained and fine grained. I’ll talk about the two in the next part.

    Coarse grained authorization

    Coarse grained authorization is based on specific views and access to a node. Access to specific views is managed by the root factory which is just a simple ACL that stores which groups have access to which views (like only admins can manage users and access objects, admins and client side certificate can add nodes).

    Access to nodes is managed via access objects which are attached to a user or apikey and also attached to a list of nodes. (There is also a special setting that gives access to all nodes) To protect views that operate on a single node (like settings, status, etc.) there is the node context factory which generates an ACL that requests access to the specific node. (a user/apikey is given principals that match their accessible nodes)

    To protect views that list nodes there is the get_nodes function that returns all nodes accessible by a user.

    It might be a good idea to extend the node access that it can define what can be done with the node (to allow/disable luci access or ssh key access for example).

    Fine grained authorization

    Fine grained authorization is used in conjunction with the graph-based DB model. It is still WIP and I use a separate branch for it.This uses the data portion of the access object. Since access to the model is given as a rest API provided by cornice I’m using cornice validators give access. Since it shall be possible to share the same master configuration across different nodes it has to determine what are the maximum access for all nodes using the config for this user is.

    Access can be granted in two ways – either by specifying a “path string” and an access (like rw, ro or none) or by just a allowing one specific query to the db. A path string has the following format: ConfigName (ConfigType).LinkName.Config2Name (Config2Type) etc. I want that it is possible to use regular expression for matching these names.

    To make things easier I started by just comparing the levels of the “path string” to see if they refer to the same string when trying to figure out the maximum rights for a given node set.

    Services

    Julius, Arne and me started thinking if it is a good a idea to make it a little bit easier for external software to just update nodes with specific configuration options. So that this software doesn’t need to use queries for all the options it wants to modify. (And therefor use polling to update new nodes etc.) After thinking a bit about it I decided it would be a good idea to add something like services a service can register to. This service contains a shell-script and a compare value to determine if the node is desired to be modified by this service and also a list of queries that should be applied to this config.

    Arne is using this to set a ONOS Server for the nodes via OpenWifi.

    Discovery Script

    The discovery script was updated to use client side certificates and advertise if the node is able to communicate via TLS.

    Graph-Based DB Model

    The GUI for the graph based DB model was updated a bit – now it is sorted with a different algorithm (based on the direction) and also displays information about the vertices (there for the is a small rest API for the vertices).

    Also a lot of the code was refactored which will make it easier to add new features here.

    Miscellaneous

    It was evaluated how a github page could be used to add more documentation for OpenWifi. Some small things in the docker image were fixed as well. (for example discovery not working with nginx image or wrong file system rights)

    What is going to be done in the next weeks

    Fine grained access proved to be more complicated than I thought. But I want to finish it soon and start working on the communication abstraction. I also hope that there is enough time to improve the graph based db model (since the code is now refactored it should be not to hard).