Building an app for network capability

This image has an empty alt attribute; its file name is AlterMundiInicio.png

Building an app for network capability

Hi! I’m Tomás. This post is a brief of the work that we did in the last few weeks. The prototype of a network capability app was achieved, and we’re starting to test it on communities. The app is still a prototype: it has only three functions (connect to a webpage using the WiFi, check if you’re in a LibreMesh network, and check the private IP of the device) and the front-end consists of only these three buttons, but it has now all the logic that was needed to start working on the rest of the app.

Basic functions

The first approach was to check if the user was able to connect to the LibreMesh local address by checking it with a ping, and then we decided to move forward to an HTTP GET instead. With this idea in mind, we prepared a new version of the application that sends a command to the device (a curl command) instead of a Java method with a previously developed android interface (for the ping version).

public boolean httpGetToLibreMesh() throws InterruptedException, IOException {
    //FIXME: modificar google por la IP de LibreMesh
    String[] cmdLine = {"sh", "-c", "curl --head --silent --fail google.com"};
    Process p1 = java.lang.Runtime.getRuntime().exec(cmdLine);
    int returnVal = p1.waitFor();
    return returnVal == 0;
}

This simple code solves the problem. It returns true if the HTTP GET to google.com worked, and false if it didn’t. It can be easily modified with the LibreMesh IP Address.

The next objective was to inform the user if the device wasn’t connected to the WiFi. In order to do so, we have to get the WifiManager from the ApplicationContext, and then check if the wifi is working.

public boolean verifyLibreMeshConnection() {
    WifiManager wm = (WifiManager) getApplicationContext().getSystemService(WIFI_SERVICE);
    if(wm.isWifiEnabled()) {
        return (wm.getConnectionInfo().getNetworkId() == -1) ? false : true;
    }
    return false;
}

Then, we needed a web navigator (WebView) inside the app with the capability to run the LibreMesh router website (On the first approach, to a google.com website).

Using a WebView object with the shouldOverrideUrlLoading overridden we can show a webpage in the app without the requirement of showing an external navigator (Android provides the Android WebView App that does this inside the LibreMesh app).

So with this simple code, we can configure the WebView to enter to a site inside the app.

WebView navegador;
navegador = (WebView) findViewById(R.id.navegadorLibreMesh);
navegador.setWebViewClient(new WebViewClient() {

@Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
    view.loadUrl(url);
    return true;
}
});
navegador.loadUrl("http://www.google.com");

Choosing through which network interface to send data to

Once having the WebView, the next step was to control through which network interface the application sends the network requests. In order to do that we have to access the ConnectivityManager. It was created as a class variable and defined on the function “onCreate” of the activity that holds the WebView. The connectivityMaganer isn’t a new instance but a reference to the object that controls the connections in the context of the App.

connectivityManager = (ConnectivityManager) getApplicationContext().getSystemService(Context.CONNECTIVITY_SERVICE);

Then we needed a function that can request to use the WiFi. The idea is to make a NetworkRequest and send it to the connectivityManager, but it also needed a NetworkCallback to specify what to do when the Network was available to accomplish the request. So as a second parameter of the request there’s an anonymous class that overrides the methods needed.

@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
private void requestWifi() {
    final NetworkRequest networkRequest = new NetworkRequest.Builder()
           .addTransportType(NetworkCapabilities.TRANSPORT_WIFI)
           .build();

    connectivityManager.requestNetwork(networkRequest, new ConnectivityManager.NetworkCallback() {
        @Override
        public void onAvailable(Network network) {
            if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.M)
                connectivityManager.bindProcessToNetwork(network);
            else
                ConnectivityManager.setProcessDefaultNetwork(network);
        }

        @Override
        public void onLost(Network network) {
            if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.M)
                connectivityManager.bindProcessToNetwork(null);
            else
                ConnectivityManager.setProcessDefaultNetwork(null);
        }

        @Override
        public void onUnavailable() {
            super.onUnavailable();
        }
    });
}

The last thing that I needed to do was a function that runs the WebView.

private void iniciarNavegador() {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) requestWifi();

    WebView navegador;
    navegador = (WebView) findViewById(R.id.navegadorLibreMesh);
    navegador.setWebViewClient(new WebViewClient() {

       @Override
        public boolean shouldOverrideUrlLoading(WebView view, String url) {
            view.loadUrl(url);
            return true;
        }
    });
    navegador.loadUrl("192.168.0.2");
}

Getting the LibreMesh address

The next step was to move forward with getting the LibreMesh IP address. On the other hand, that’s not more than just an algorithm or a gateway, either way, does the same results. This can give us an alternative way to see if the user is connected or not to the LibreMesh server (we get the IP through the algorithm and compare the gateway version).

The idea was pretty simple and only required a int to ip auxiliar function. So we decided to collect all the methods that returned wifi information and send them to a new WifiInformationManager class. So, this class sends all the information that we need from the WiFi:

public class WifiInformationManager extends AppCompatActivity {
    private static String intToIp(int addr) {
        return  ((addr & 0xFF) + "." +
                ((addr >>>= 8) & 0xFF) + "." +
                ((addr >>>= 8) & 0xFF) + "." +
                ((addr >>>= 8) & 0xFF));
    }

    public static String getPrivateIp(WifiManager wm) {
        int ip = wm.getConnectionInfo().getIpAddress();
        return intToIp(ip);
    }

    public static boolean verifyWifiConnection(WifiManager wm) {
        if (wm.isWifiEnabled()) {
            return wm.getConnectionInfo().getNetworkId() != -1;
        }
        return false;
    }

    public static String getGateway(WifiManager wm) {
        return intToIp(wm.getDhcpInfo().gateway);
    }

}

The function getGateway solves in an elegant way the problem of the LibreMesh Local-Address. The rest of the job was simply to change the address of the WebView to this one.

Using logcat to find bugs

The logic step then was to try the application and test if it worked okay, but when we did that the WebView that shows the Lime-App showed a white screen instead. Using the logcat inside the Android Studio we were able to easily find the error, showing the importance of using this type of debugging tools.

Using the logs it’s easy to see that there’s a TypeError when trying to get the property ‘getVoices’. The problem comes with the plugin ‘window.speechSynthesis’ that isn’t available for some browsers.

The Lime-App is the graphical interface that LibreMesh uses for the configuration of community networks. We found the .js that was calling the function:

let synth = window.speechSynthesis;
let voices = synth.getVoices();

export const speech = (text, lang) => {
let utterThis = new SpeechSynthesisUtterance(text);
utterThis.pitch = 0.9;
utterThis.rate = 1.2;
utterThis.voice = voices.filter(x => x.lang === lang)[0];
synth.cancel();
synth.speak(utterThis);
};

It can be seen that in the line 2 the variable voices is set to a synth.getVoices, but if synth is undefined, then that line will not succeed.
The solution was pretty simple, with a control structure we check if the speechSynthesis was available or not. So the fixed code is:

let synth = window.speechSynthesis;

export const speech = (text, lang) => {
if(synth != "undefined") {
let voices = synth.getVoices();
let utterThis = new SpeechSynthesisUtterance(text);
utterThis.pitch = 0.9;
utterThis.rate = 1.2;
utterThis.voice = voices.filter(x => x.lang === lang)[0];
synth.cancel();
synth.speak(utterThis);
}
};

I sent a pull request to the Lime-App repository fixing this problem and it’s currently waiting to be merged.

Next steps

With the functions of detecting and configuring a libremesh network, we plan to add some features to the app the next weeks:

  • A better graphical interface with the integrations of all the planned functions of the app.
  • Support other services in addition to Lime-App.
  • Add the app to the LibreMesh operating system, giving the posibility to the user to obtain the app directly from the router.

Video

Github project

[GSoC’21] Irdest Android Client – Coding Phase I

NOTE: You can read the same post in LaTeX here

Prelude

Hello! Good to see you here : ) This blog is mostly a summary of work done till now under the first coding phase of summer of code of ’21. Picking from the end of previous blog post, we planned implementing chat feature in the application module, but due to the aforementioned massive refactor in the entire codebase and upgradation of existing modules to support modern hardware the chat API has been deprecated, and some components, because of not being part of CI, got broken : ( To implement features in the module the very first step was to get the project build properly, previously (maybe) due to migration from different version control system to GitLab and that massive refactor there were some unidentified problems that did not let the application codebase build properly, also the main lead of the picture `android-support` crate, not being a part of our GitLab CI workspace too, wasbroken. We fixed all this entire stuff in multiple different steps, each solving a mini problem and writing CI for each missing component so that we or anyone joining the project never encounter similar problem(s) in the future.

I. Fixing the Android Application Codebase

The application codebase was considerably broken and for the very first time when I built the application, it instantly said build failed in10ms, which is really very weird as when for the very first time you build an android application, it takes noticeable time(~6 minutes), this time is for fetching the dependencies that are declared in the dependency management file of android codebase(the ones with the build.gradle name) followed by compiling the android project codebase. It was quite astonishing at first sight,but on closer look to the files present in the android codebase the cause was observable. It was the presence of dependency archives in the android codebase and their corresponding XML files too, and since these dependencies were already present, studio didn’t take the pain of fetching them from maven. So the question arises, When the dependencies were already present still then application didn’t compile, why? Actually what happens is that these dependencies’ XML files are editable so even the slightest edit in them renders them useless and studio doesn’t even report any kind of problem with them, another thing that happens under the hood is that studio stores these dependencies in its local cache so that when user re-compiles the application, no time is taken in fetching the dependencies and it can perform the real build. Also, by time these caches get corrupted and usage of very old cache does not let the project work in the way it should.
Okay, now let’s come to the point how we fixed it. As by now it should’ve been clear that the problem was the existence of binaries and XMLs of dependencies present in the android codebase, so the solution that I anticipated was the deletion of these files. It was not sufficient. After doing this dependency management did go as expected, but still the build failed : ( After some more inspection, I found there was some problem with gradle executable scripts as well and the gradle-wrapper.properties too. So I just referred these scripts from my previous working projects and it finally started working, a moment of joy 🥳, after many days + nights of pain ; ) As this problem was fixed, after working on some other crucial matter(see next section, II one), we wrote a CI pipeline specially for our android application codebase so that it doesn’t break again ever in the mainstream. The android-application pipeline comprises of the 3 stages, in which its lint is checked, followed by build and then the tests are run. In the upcoming coding phase we plan to make this CI pipeline even more robust and enforce stricter formatting rules, introduce Static Analysis and run android Integration Tests on GitLab CI, well we’ll discuss it the next time we meet, leaving some topics for then 👀
You can find the corresponding MRs here:
* Fixing the android application codebase: we/irdest/!21
* CI for android application codebase: we/irdest/!23

II. Fixing the FFI Layer

After the previously mentioned refactor, everything was working fine, only android specific components of the codebase were broken. A part of which, was our FFI layer, the android-support crate. This layer still held references of several deleted and deprecated APIs, therefore compiling this crate too gave a bunch of errors. Fixing them took much more time as this crate was written in Rust and then I was not that fluent with it. So fixing it included updating/modifying existing functions or we had to remove functions as well because of the deprecations. A nice challenge that we encountered was saving the state across multiple platforms(supported hardware), because the crate we used for saving state provided support for almost all the operating systems other than Android. So what we did was using the knowledge of android that an application has access to its own private storage which no other application/service can see, so all we needed to do now was to find this directory in android device file-system, this path we achieved using the ADB, now we investigated where our crate went wrong, so for this we dived deep into the crate’s API and read how it achieved similar behavior for other platforms, which was that, it first of all found the HOME(environment variable) for the OS and then located corresponding path(s) for saving state in dir/file(s), turned out, that the crate was identifying HOME wrong only for android file-system. After this was diagnosed, we wrote a custom API that found HOME env var on all platforms irrespective of their OS (see this patch), by this API we were able to access the app-specific private directory and save state there. It was quite challenging, but we figured it out! Everything related to FFI layer, after this fix was quite easy. We eliminated the problems that existed in FFI layer via refactors and some modifications in functions and then, it built green! After fixing the FFI layer we wrote the CI for it that makes sure it builds each time a commit is pushed to any MR or branch and we can see the build status in pipelines too. Writing the CI for android-support crate was not a cakewalk,actually the application needs cross-compilation of our Rust library in order to function, so we need to make sure that the library which application is going to use on android devices is really compatible with android platform and by virtue we compile it on our PCs directly so that doesn’t work quite, to make expected behavior happen there are two options:
* Compiling the Rust library codebase on android-device(less feasible), or
* Cross-compiling the library on our PC/CI runner via providing support tools for android components
So quite obviously we went with the second option, for this we installed rust and android compatible components in our runners during the CI runtime and then compiled the library via checking out to the correct directory. But since, for compiling Rust library in each CI run we had to install the components and this specific pre-compilation part(or better to say setup portion) consumed a considerable portion of our CI script(and made it look a bit daunting too), so we packaged these components to our custom docker image and pulled it each time in our CI runs, this made our life easy and scripts beautiful : )

NOTE: If you don’t have much idea of cross-compilation, then you can have a look at this awesome blog-post by Milan✨. It gives a clear understanding to the reader what cross-compilation is, irrespective of their previous knowledge on the same(yes, but basic knowledge on compilers is needed a bit). Spoiler alert: That someone in the opening of blog post is me 😛

Also, since the refactor was incomplete in our android application codebase and the android-support crate so between these to big tasks, I fitted this small refactoring, as a light break 😛
You can see the MRs for them here:
* Fixing the FFI Layer: we/irdest/!31
* Refactoring the android-components: we/irdest/!32
* Adding the android-support crate in our CI Pipelines: we/irdest/!33

III. UI Improvements

After fixing issues in android application and our Rust library, and writing a robust end-to-end CI for them we moved forward towards improving the UI of the application. Previously, the application used legacy design components and ideology, under this task we modernized these UI components and followed the material design guidelines(material.io), that improved the overall look of our Authentication screens. There is nothing much to explain in it as it was really quite easy to achieve and also we didn’t encounter any problems.
You can see the related MR here: we/irdest/!26

Acknowledgements

Well, by far it has been the most exciting summer for me and I had interesting experiences working on the project. Fixing components, was very difficult at the beginning due to many reasons one I would mention is the huge codebase which we have and it is not easy to learn about the functionality of each component present in it in short time, and also everything is intertwined too at many places. Going through it and fixing issues would definitely not have been possible without the immense support from my mentor, Spacekookie, who was always there to help me out and direct what to do. Their advice greatly helped in speeding up the development process and they are also a source of inspiration to me. Most importantly, Milan, who is not officially my mentor but has helped me a ton of times in technicalities of CI and setting up Nix environment(which I initially used for cross-compilation) about which I knew nothing, and many more instances. It won’t have been easy for me to accomplish aforementioned tasks without direction and help from Spacekookie & Milan.
Thanks again to both of them : ) and I’m more excited to work on the project with them further!

Cheers Until next time we meet!
~Ayush Shrivastava

Android native app for network selection capability in LibreMesh routers – Overview

Hello! I’m Tomás Assenza. I work on the “Android native app for network selection capability in LibreMesh routers” project with the Altermundi association. I’ll talk about what Libremesh is and why we want to make an app for network selection.

Introduction

LibreMesh is an operating system that works on some Tp-Link routers and LibreRouters, and It’s a practical solution to provide networks to social organizations. These social organizations usually give access to the web to users that only had mobile data networks before. The most practical object that they generally use is a smartphone to connect to the LibreMesh routers.

The issue

Android usually does a “network switch” between Wi-Fi and mobile data considering if the first one provides or not the Internet. The problem with this feature is that if the users have an internet problem, they would not have the possibility to access some LibreMesh internal address to know the trouble and report it to the organization.

We have the idea to solve this problem through an application with the capability To select from which network it sends and receives data from, making possible the connection between the smartphone and the router even when there isn’t an Internet connection.

Who am I? What are my motivations for the project?

I’m a System Engineering student at the Santa Fe Regional of the National Technological University (Universidad Tecnológica Nacional – Facultad Regional Santa Fe). I work as a young teacher of the subject “Algorithms and Data Structures” and in two R+D projects, one about making software tools for programmers with visual disabilities and the other one about developing online simulators for the subject of ‘Chemistry’ by helping the students to experience chemical experiments even during the Covid-19 pandemic.

I always liked to learn how digital communications worked, but they also seemed far away from my profession and career. Considering this, I thought that the GSoC was an opportunity to learn new concepts and help communities with open-source software.

First meeting and objectives for this week

During the last week, we did a meeting to establish objectives for this week. These are:

  • Install the Android Studio IDE.
  • Make an application that can show the local address of the device.
  • Make an application that can show if the device is connected or not to the LibreMesh network (simulating through a ping to a random address).

And the results are:

Github Project

Next objectives:

  • Change the ping-based detection.
  • Define a strategy to connect to the router in case that the app detects that the device is or not connected to the LibreMesh network.
  • Define the scope of the app

[GSoC’21] Irdest Android Client – Overview

Hello everyone, I am Ayush Shrivastava, one of the students selected for Google Summer of Code 2021 for Irdest sub-organization under the umbrella organization freifunk. If you’re wondering which topic/project the blog is focused on, then you may give it a look here!

Irdest

Well, if you don’t know, then let me first introduce what Irdest(irde.st) is and what it does/supports. So, Irdest is a software suite that allows users to create an internet-independent, decentralized, wireless, adhoc mesh network. It removes all the dependencies of the user from a specific single service and enables users to create a mesh of their own. In this network mesh, users can communicate to each other via messaging(both, individual and Room) and placing voice calls. Irdest network mesh service also allows users to share files between them without relying on the internet. And also when a user enters the Irdest mesh network(note that user needs not connected to the internet i.e., without having an IP address, it can be a P2P connection over Wi-Fi or Bluetooth), their IP address is completely hidden thereby maintaining privacy andreducing possible breaches that may occur. In the irdest network mesh, as soon as a user enters the network, they are assigned an ID, which is a unique cryptographic key that helps identifying the users. All the messages/calls/file transfers made in Irdest network mesh are completely end to end encrypted, again a positive sign from security & Privacy perspective. Currently, Irdest is supported for Linux & Android devices. For android, it is pretty much in incubation state and there are a bunch of points & directions where we can improve. This summer, I aim to implement some features on android client from the irdest upstream.

WARNING: This blog has a tons of mentions of FFI, you may get overwhelmed by the term, so it stands for Foreign Function Interface.

Current Progress & FFI Overview

So this was what Irdest is all about and a higher level overview of how things really work in it. As mentioned previously, the android client is pretty much in its incubation state, so we wish to implement it in a clean & modern way. So, since we want to make use of features supported by Irdest(in upstream) on the android application, therefore we need to maintain/create a binding or kind of a link between core Irdest code that makes all this internet independent network mesh possible & application codebase(to call those methods from). The point is that the core Irdest code that supports these functionalities is written natively in Rust, and we we want to utilize this pre-written library on android, without writing it again(in Kotlin)for android. So to make it possible, we’re going to write an FFI Layer(I’ll write in detail about FFI in some other post, but for the time being it is sufficient to know that by FFI we can call functions written in a specific language(mostly native languages like C++, C, Rust) from some other languages). This FFI layer currently exists in the module, but it is a bit old-fashioned, so in the initial phase of my coding period I plan and propose to rewrite this layer with best possible modern day FFI practices and following some standard references. The most crucial part of this project is this FFI layer, because it is the fundamental building block for the android project, once it is set and ready, then we’ll be able to use the functions and services provided in the native Irdest library written in Rust and extend our further development process via writing the application.

FFI Implementation – A Blueprint

The FFI already is quite notorious for the undefined behavior and on top of it, there is very less official documentation available on Rust-Kotlin FFI so this makes it more tricky & challenging to implement, but for the course of our development process, we’ll be following and taking inspiration from how Mozilla have implemented FFI integration between their existing rust library and their firefox android client. Another instance where they use Rust-Kotlin FFI is their Glean project(a telemetry service). I’ll be taking most of the inspiration by and practices being followed in Glean, as a result of which we’ll be writing FFI bindings on our own. After once we’ve written FFI layer, my next step will be to test it again thoroughly and make sure it works perfectly and does not has any bugs residing.

Further Development:

Once we have a robust FFI layer ready, then I’ll move to core android development process, that’ll entail writing UI for the application, architecting it properly, modularizing the application codebase and following the Modern Android Dev practices. Well this will be a relatively easier job to do in comparison to that of writing FFI layer xD
My next step after this would be implementing the chat service in the application. We’ll focus in its detailing after we’re done with the very first milestone, the ”FFI Layer”. We’ll see the plan of action for chat service in a separate blogpost, after the first coding phase : )

Thanks for reading!
Cheers, Until next time we meet!

~Ayush Shrivastava

Our Google Summer of Code Projects

Google Summer of Code Logo

This year, we were finally accepted back as an organization at the Google Summer of Code. In the meantime, the application and selection phase is over. Google has given us 9 project slots. We didn’t make the decision easy and chose the best ones out of the applications.

Organizations

Freifunk manages projects for different initiatives as an umbrella organization. In this Google Summer of Code we have Retroshare, irdest, OpenWrt, LibreMesh and freifunk itself on board.

Timeline

Currently we are in the community bonding period. During this time, all preparations are made so that the students can do their tasks. Also, they should dive into the communities and get to know people and tools.

Coding officially starts on June 7. All projects must be completed by August 23.

Our Google Summer of Code projects in 2021

TitleOrganizationStudent
OpenWRT PPAOpenWrtNeelaksh Singh
Irdest Android ClientirdestAyush Shrivastava
Android native app for network selection capability in LibreMesh routersLibreMeshTomás Assenza
RetroShare WEBUIRetroShareAvinash
Freifunk Digital Twin – test on your virtual mesh before going productiveOpenWrtpschreiber
OpenWrt Device PageOpenWrtAditi-Singh
Freifunk Radio Ressource Management with IEEE 802.11vOpenWrtValerius_B
Updation of the Json Schema to latest version (2020-12) along with the form generation and validation with the updated schema of the toolsfreifunksh15h4nk
LibreMesh Pirania UILibreMeshAngie Ortiz Giraldo

You can find more details for every projects on our GSoC dashboard.

geolocator (Software defined GPS) final evaluation

Hi everyone,

with this blog post I would like to explain the full Google Summer of Code Project as a final post. For people who haven’t read over the geolocator (Software defined GPS) project before, it might be interesting to read these three blog posts at first:

– geolocator (Software defined GPS) (english)[1] and (german)[2]

– geolocator (Software defined GPS) first evaluation (english)[3] and (german)[4]

– geolocator-software-defined-GPS-second-evaluation (english)[5] and (german)[6]

Otherwise I will give in the following a short overview about the project structure to remind you of it. I structured the Google Summer of Code project into 3 main subprojects:

web backend,

– The web backend named sgps-core is a service, which should give requested clients their geo position.

gps-share,

– The idea of gps-share is to create an udev device, which provides NEMA-formata protocols over tty addicted on information, which is received from the above mentioned backend.

LEDE Package,

– The intention behind this subproject is to develop a new package for LEDE called geolocator, which should provide the geo position of LEDE devices.

Now I would like to give you a full state of each above mentioned subproject. Firstly I will explain about the web backend and finally a peroration including my valediction as a Student by Google Summer of Code.

web backend

Generally the backend service receives over the OpenWLANMap[7] App from mobile phones the mac addresses of surrounding wireless networks linked to GPS positions. This information will be stored into a database. If a device like a WiFi router requests its position, it will send surrounding Wireless mac addresses to the backend and get back a geo position, which is calculated from these information in the database.

The new web backend called sgps-core[8] is an API-core, which should replace the old openwifi.su backend. The old one consists of a collection of different programs in different program languages. sgps-core includes a fully backwards compatibility to the old openwifi API[9] for requesting a position. sgps-core is written in Golang, which processes a lot faster than the old API, which is written in Ruby. sgps-core is more secure because it checks and take only requested strings, which contain only comma separated macaddresses with 12 hex characters.

As a fallback feature, sgps-core is able to receive coordinates from unknown WIFIs by requesting them on Mozilla Location Service (MLS)[10] if there are no db entries for that WIFIs. The position for clients will be returned in form of latitude and longitude. As a quick reminder, here is the schemata from the first post, which represented the the functionality of sgps-core:

The sgps-core solved a problem about calculating the position. The old method counts the average of all latitude values. Analogous for longitude. The new method calls geographic midpoint calculation and needs 4 parameters lat0, lon0, lat1, lon1 (give two take one) which will be explained in detail in following:

deg have to be replace with latitude or longitude value.

rad = deg *π / 180 <- Generally conversion from degrees to radiant.

dlon = (lon1 – lon0) * pi / 180

lat0 = lat0 * π / 180 <- lat0 from degrees to radiant.

lat1 = lat1 * π / 180 <- lat1 from degrees to radiant.

lon0 = lon0 * π / 180 <- lon0 from degrees to radiant.

Converting into Cartesian coordinate system.

Bx = cos(lat1) * cos(dlon)

By = cos(lat1) * sin(dlon)

Calculate new position reference to sphere and Converting back from Cartesian coordinate system into new latitude and longitude:

lat2 = atan2(sin(lat0) + sin(lat1), (cos(lat0) + Bx)² + By²)^(1/2))

lon2 = lon0 + atan2(By, cos(lat0) + Bx)

On this point it is also possible to use a ellipsoid to increase the accuracy of positions. This may be interesting for long distances. For short ones like from seen wireless networks, it is not really relevant.

Converting back to degrees:

deg = rad / pi * 180 <- Generally conversion from radiant to degrees.

lat2 = lat2 / pi * 180 <- lat2 from radiant to degrees .

lon2 = lon2 / pi * 180 <- lon2 from radiant to degrees.

In the last few weeks, I spent a lot of time on discussing with the current server administrator of openwifi.su to deploy the sgps-core on the server for a test environment. But he did not have much time, so we decided to migrate the openwifi.su to our Nordwest Freifunk infrastructure to make the administration more accessible for other people. In my last report I wrote “I will release in the next few days a first version”. This could not be done because of the above mentioned discussion. After the migration I can test that backend on huge databases and compatibility to the DBS. The current code can be found here [11]. For people who want to try the sgps-core please check out the following URL[12].

gps-share

The Idea at the beginning of GSoC17 was to write a program to provide GPS NEMA-formats over a tty udev device. The information for the GPS NEMA-formats should come from the above mentioned sgps-core. As I told in the first blog post I discussed with some people from the Mozilla Location Service Malinglist and it turned out that something similar was already exist called geoclue. To avoid developing redundant software I decided to drop this idea. Instead of it the new plan was to build support for native GPS in gps-share[13], which is an add-on for geoclue. But during the Google Summer of code I had to focus more on the both other subprojects because they are more important, especially for Freifunk. In my peroration I will tell about the future plans, especially for gps-share.

LEDE Packages

The third subproject was to create some opkg packages for LEDE[14] and similar Frameworks. The main package called geolocator provides the geo position of the device via UCI[15]. Positions should be received from the above explained sgps-core. The 4 other packages are only for Gluon[16], which add the configuration options of the geolocator to the Web-interface.

This month I mainly worked on the LEDE Packeges. At the beginning of the august I sent a merge request to Gluon for integrating the Gluon-geolocator[17]. The containing geolocator program was written in ash shell code. While reviewing and discussing about the merge request, I realized that I had to rewrite the program from shell to lua code because Gluon mainly work with in lua written programs. You can find the shell code here[18] and the Lua version here[19]. At the moment I am waiting here for another review and subsequently merging.

The other packages for the Gluon Web-interface are also already in process. The first idea was to create a detection of installed packages to show related configuration options on the Web-interface. This idea was dropped because I found a better solution. The problem is detecting packages on runtime, which means many extra code on Routers, which only have 4MB Flash for example. So I decided to generate the package with their options on compile time. These packages are:

gluon-config-mode-geo-location,

gluon-config-mode-geo-location-with-geloc-map,

gluon-config-mode-geo-location-with-geloc,

gluon-config-mode-geo-location-with-map

The main package is gluon-config-mode-geo-location, which is already exist in gluon, but with a difference Web-interface. Each package should either integrate an open street map or the geolocator options. Integrating both are also possible. For communities which would like to stay on the currently variety of functionalities, it is also no problem not integrate any of these extra options.

Here is how the new packages look like:

I wrote some C++ programs, which generate me the Lua code for the Gluon Web-interface, which is written in Lua. Base on preprocessor variables, the amount of options for each package will be included into the Lua output from the C++ program. These preprocessor variables will be set by selecting one of the above packages. Also PO files for the translation will be generated in the same way. A merge request of the above new packages can be found here[20] I am still working on it.

Peroration and Future plans

Now I am coming to my peroration.The last 3 months were really awesome, just like last year as a student on the Google Summer of Code. I would love to recommend this great opportunity for not only students but also for open source organizations. Students can not only learn a lot of new things but also meet new great people, make new friends and take part in many events. For example: at the beginning of august I was on the SHA2017[21] (Still Hacking Anyway) and had a meetup with some Freifunk communities there. We had a great discussion about a lot of technical stuffs and a nice time for socializing. The SHA2017 took place in Netherland nearby Amsterdam. Another example is : this week I flew to Spain to start my exchange semester. Coincidentally a student from Germany who I met at the beginning of the GSoC17 in Berlin on the WCW[22] (Wireless Community Weekend) is also doing an exchange semester in Spain. We have already emailed each other and planned to meet up in the next months, probly in Barcelona or any other places. As I said above, this is my second time as a student on the GSoC, which means this is also my last time and sadly I have to say goodbye to GSoC as a student now. But maybe next year I can work as mentor to support other students in their great opportunities.

Back to the projects, as i said I’m still working on it. I will finish the Integration into Gluon and LEDE and continue developing sgps-core integrate new features and migrate the infrastructure to a better server. I would like to contact Zeeshan Ali, the maintainer of gps-share and try to help on this project as well. Also I am still working on the hoodselector which is my Google Summer of Code project from the last yeah. You can read about it here[23]. The hoodselector should also integrated into Gluon but it requires for sure a few weeks of work to integrate VXLAN on it. A merge request for can be found here[24].

Also I would like to say thank you to my Mentors Clemens John from the Google Summer of code 2016 and Johannes Rudolph from 2017 and especially to Andreas Bräu who works so hard on the Freifunk Org for many years to give students these opportunities to be a part of the Freifunk Community.

[1] https://blog.freifunk.net/2017/05/29/geolocator-software-defined-gps/

[2] https://ffnw.de/geolocator-software-defined-gps/

[3] https://blog.freifunk.net/2017/06/28/geolocator-software-defined-gps-first-evaluation/

[4] https://ffnw.de/geolocator-software-defined-gps-erste-evaluation/

[5] https://blog.freifunk.net/2017/07/26/geolocator-software-defined-gps-second-evaluation/

[6] https://ffnw.de/geolocator-software-defined-gps-zweite-evaluation/

[7] https://f-droid.org/packages/com.vwp.owmap/

[8] https://github.com/openwifi-su/sgps-core/blob/master/README.md

[9] https://sourceforge.net/p/libwlocate/code/ci/master/tree/master/

[10] https://location.services.mozilla.com/

[11] https://github.com/openwifi-su/sgps-core

[12] http://runner01.ffnw.de:8082/api/v1/bssids/64700274945A,0018E7DC21BB,30FC681F37A6,F81A673626E8,EC086BA8ED18,8416F9A8E2FA,E894F6A230E4,98DED020D00A,EC086B3349B2,F81A677F5CD8,A0F3C1654BA6http://runner02.ffnw.de/64700274945A,0018E7DC21BB,30FC681F37A6,F81A673626E8,EC086BA8ED18,8416F9A8E2FA,E894F6A230E4,98DED020D00A,EC086B3349B2,F81A677F5CD8,A0F3C1654BA6

[13] https://github.com/zeenix/gps-share

[14] https://lede-project.org/

[15] https://wiki.openwrt.org/doc/uci

[16] https://gluon.readthedocs.io/en/latest/

[17] https://github.com/freifunk-gluon/gluon/pull/1201

[18] https://git.ffnw.de/ffnw-firmware/packages/blob/ba8007e5a6b99306068847ba98f2bb72b7fb2745/ffnw-node-info/files/lib/ffnw/geolocator/geolocator.sh

[19] https://github.com/2tata/gluon/blob/fa935d83a64c27dd23f133bd8cecf5d72ef9281b/package/gluon-geolocator/luasrc/lib/gluon/geolocator/geolocator

[20] https://github.com/freifunk-gluon/gluon/pull/1211

[21] https://sha2017.org/

[22] https://wiki.freifunk.net/Wireless_Community_Weekend_2017

[23] https://blog.freifunk.net/2016/08/22/monitoring-and-quality-assurance-open-wifi-networks-out-client-view-final-evaluation/

[24] https://github.com/freifunk-gluon/gluon/pull/997

 

netjsongraph.js – Google Summer of Code (GSoC) 2017 summary

Throughout the last three months, I was quite fortunate to work for Freifunk on netjsongraph.js under the guidance of my mentor Federico Capoano. Thanks for this invaluable experience that I learned a lot of knowledge and use them in a practical project. Here is a summary of the work I have done during the Google Summer of Code (GSoC) 2017.

Google Summer of Code project page

Goal

netjsongraph.js is a visualization library for NetJSON, a network topology data format. The main goal of netjsongraph.js may be concluded in below three lines (more details you can see in GSoC 2017-netjsongraph.js: visualization of NetJSON data):

  • Apply the modern front-end development tools and add tests workflow (#1, #45)
  • Rewrite it with WebGL (#11, #29, #39, #42, #47)
  • Improve the performance (#41, #44, #46)

Achievement

Github Repository : https://github.com/netjson/netjsongraph.js

Examples on GitHub pages: https://netjson.github.io/netjsongraph.js/

You can browse all examples on GitHub pages. Some screen shots of the application:
basic example
performance example
The force-directed layout is usually used to visualize network data. It offers insights on the relationships between nodes and links. The previous version of netjsongraph.js is implemented by d3 and it’s rendered using SVG. It would be very slow if there were thousands or ten of thousands nodes or links. So I have to embrace the WebGL speeded up by GPU to have a better performance.

I have recorded my work in the blog every milestone:

BTW, It’s a great management method to make members submit weekly reports and blog posts in Freifunk.

During the three months, there have been 116 commits from me. I created a big Pull Request include them:
netjsongraph.js #48
netjsongraph.js project panels
Almost all goals have achieved:

  • Published a minor version
  • Improved development workflow
  • Tests Added
  • Refactored visualization by Three.js and d3-force
  • Added more interaction like hover (show nodes tooltips), click (show nodes or links information panel), pan and zoom
  • Improved performance

Especially on performance aspect, it runs efficiently on Chrome reached 60FPS under 5k nodes and 10k links. And if you don’t wanna animation, you can choose the static rendering layout.

Challenges

I also encounter some challenges I never met before.

Event binding and handling

As you know, WebGL renders all objects in one canvas tag. How to bind events on every geometry? You should use the Ray casting. Raycasting is used for mouse picking (working out what objects in the 3d space the mouse is over) amongst other things. So you can know which geometry your mouse over and add some interaction effect.
There are thousands of objects and every object has several events you should handle, I had to develop an event controller to manage it.

Performance

The bottleneck in this visualizer is performance(#41). I tried many methods to improve it include:

Reuse geometry and material

However, the color of every node is different and the one link should highlight itself when it hovered, so the material should be independent and can not use in common.

Combine the mesh

Same problem with above. It’s not flexible to combine them to one mesh, different nodes and links should have different positions.

Static rendering

Make calculation before rendering, so there is no animation and repaint.

Using Web Worker

Web Workers is a simple means for web content to run scripts in background threads. The worker thread can perform tasks without interfering with the user interface. So put static layout calculation into it will be efficient.

Force-directed algorithm

There are different complexity and cost in the different force-directed algorithm. The Force-Atlas2 algorithm has some benefits over the force layout implemented in d3-force. So current version may be refactored by an advanced algorithm in the future.

What is left to be done

  • Add optional geographic map (#40)
  • Using Force-Atlas2 algorithm

More interactions and features should be added, and performance may be optimized by using new algorithm. I’d like to continue developing this project after GSoC.

In the end, thanks for the great patience and guidance from my mentors. Thanks for Google to provide me with this rare chance to contribute to an open source community together with awesome members from all over the world. I really appreciate this invaluable experience accumulated this summer and I believe it will have the profound impact on my career and life.

GSoC: Improving nodewatcher data representation capability (update 3)

I have been working on improving the wlan-slo main page. This page has not been updated in a long time and it is crucial at introducing new people to mesh networking.

Continue reading “GSoC: Improving nodewatcher data representation capability (update 3)”

GSoC 2017-netjsongraph.js milestone 2

After a month of efforts, netjsongraph.js has been greatly improved.
First, I added some API and options are compatible with previous versions, except those that can not be added to the WebGL element. Followed by updating the documentation and some examples so that users can better understand and try out our projects. Finally, I have optimized interaction and performance.

netjsongraph dark theme
netjsongraph dark theme

Interaction

  • added highlight effect on nodes and links when hovered.
  • added tooltips with node information
  • added pan and zoom interaction
  • added node and link information panel when clicked
  • adjust the size of the canvas when the window resized

Performance

The large part of the performance of force-directed layout is consumed in each element position calculation in every tick. So if the position calculation of the elements is before canvas rendering, it is effective to reduce the performance loss.
So I made a static layout https://github.com/netjson/netjsongraph.js/issues/44, there is no animation when rendering, directly render a static force-directed layout in the page.

Other

Besides, I have refactored the event controller, in order to make zoom and pan or other events added easily.
But now the overall rendering results are not good as our expected. There is a little lag in animation, I currently have no idea about the reason, perhaps because this is not best practice of three.js. In the next stage, I will focus on solving performance problems and improving the visual effect.

Next Plan

The links below may be useful:

  • http://taligarsiel.com/Projects/howbrowserswork1.htm
  • http://igoro.com/archive/what-really-happens-when-you-navigate-to-a-url/
  • https://github.com/alex/what-happens-when
  • https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

geolocator-software-defined-gps-second-evaluation

Hi everyone,

 

I am now taking you over my second evaluation. A German version of this blog post can be found here [0]. For people who did not read over the geolocator (Software defined GPS) project before, here is a link to the project introducing (english)[1] and (german)[2]. The first evaluation from last month can be found here (english)[3] and (german)[4]. First of all I would like to tell you about the web backend:

 

Web Backend

As I said it in the blogpost before, the new web backend should replace the old one.
The current openwifi.su web backend consists of a collection of different programs in different programlanguages. The new one is written in Golang it has a fully backward compatibility. The new web backend is more secure.  It checks all requested strings at the end of the url that they contain only comma separated macaddresses with 12 hex characters, as a validation check.

 

Also the new web backend fix an issue by calculating the requested position. The method before just adds all latitude values and devides them over the amount of positions. Same for longitude. The new way is a geographic midpoint calculation. First the lat/lon values will be converted into radius and then the radius will be converted into a Cartesian coordinate system, after that the midpoint calculation will be done and will gave us a lat/lon in the form of radius and the values will be converted back into degrees, that is all.

 

The web backend has a new feature which should help to enhance the position quality. If there are BSSIDs unknown in the database of openwifi.su, the new backend sends an extra request for the unknown BSSIDs to Moziller location service [5] and brings that information into the calculation of the response.

 

At the moment the new web backend is being tested. In the next few days a first version of the new web backend will be released. The current code can be found here [6].

 

LEDE Package

The last project is to build packages for LEDE and send merge requests to Gluon. The idea behind that is, for WiFi routers and other WiFi devices to be able to automatically receive their own position. This could be a nice feature for mobile Freifunk Routers. It could also provide a good alternative instead of setting positions by hand. A first version of that package is here [7].

 

For the integration in Gluon I plan 4 new packages. The first one is the gluon-geolocator, which depends on gluon-node-info and provides the programme with the capability of receiving geopositions based on surrounding WiFis.

 

The next 3 packages are just for the Gluon configmode. The configmode is a state that begins at the initial start of a Gluon router and the owner of that router can configure it over a LAN connection.
I would like to integrate the following 3 packages within the gluon-config-mode-geo-location.
They are named:
gluon-config-mode-geo-location-with-geloc-map,
gluon-config-mode-geo-location-with-geloc,
gluon-config-mode-geo-location-with-map

 

As their names indicate, the package gluon-config-mode-geo-location-with-geloc-map should integrate the options of the geolocator and an OSM map into the configmode. The other 2 packages, gluon-config-mode-geo-location-with-map and gluon-config-mode-geo-location-with-geloc should either integrate only an OSM map or just the configuration options of the geolocator. Also the package gluon-config-mode-geo-location should be able to and will show the current location interface without any map or geolocator.

 

A merge request of the above new packages will follow soon. Some of the gluon-config-mode-geo-location extensions are currently not finish yet.

 

In the next few weeks I would like to finish the integration into Gluon, as well as continuing to develop the new web backend and releasing the first version of it. There are many ToDo points currently open e.g. Multicore functionality, optionally other geographic midpoint calculations. At the beginning of August there is the SHA2017[8], which I will use for a marathon session.

 

With best wishes
Jan-Tarek Butt