Category: Allgemein

In my previous blog post, I had achieved the authentication at the initial handshake in the app. Since then there has been quite a lot of progress in upgrading the Meshenger app.

Progress Till Now

1.) Refactoring of the codebase

Initially, I started with the refactoring of the codebase in order to allow different means of connection to the client. For e.g. contacting a client over the server or Internet, enabling direct calls in layer 3 networks with the help of multicast groups and pim6sd. etc. I am working on enabling the calls over the Internet.

Firstly, I started by removing the “challenge” from the entire codebase. Challenge was used as a security parameter but now as authentication has been developed, it became redundant. Secondly, I refactored the Contact (Client data) and AppData (User data) class to hold different connection data like mac address, port and the hostname. I moved the identifier and address in the “connection_data” ArrayList and stored the data structure serialized as a string in the Contacts database which needed to hold different contact data of the form List<ConnectionData>. I added this data to the QR-Presenter Activity ‘s QR-Code and parsed in the QR-Scan Activity. I followed the same procedure for the AppData database also. Lastly, I also removed “username” and “identifier” from the call JSON.

2.) Implementing client online/offline detection over the Internet

For implementing the client online/offline detection over the Internet, I needed to hold a persistent TCP/IP connection to a signalling server. So I started a thread at the start of the app and that thread opened a persistent TCP/IP socket for each SignalingServer object in the connection_data. The sockets were held open for as long as the app is running. I used a signalling server made on node.js and ran it on the laptop. Then after connecting the phone A and phone B to the laptop’s hotspot, I ran the app on the phone A and the server displayed that the user is online. After that, I checked for the client status by running the app on phone B while still keeping the app on phone A on. The result was displayed that the client was online and the client’s status was detected over the Internet.

Next Steps

The next phase i.e. the Final Phase of GSoC 2019, will be about achieving the call over the Internet, adding other features and some code polishing.

After 4 more weeks, there has been progress regarding the agenda. The agenda for the last phase was to create the web interface using which users could download appropriate images for their devices and build custom images too.

What has been achieved

A refined version of the web application has been achieved. The functionality of creating custom images has been added to the application.

The application has been moved to ReactJS. The finished application looks like this:

Note: There are minor bugs and issues in the app which will be rectified in the later versions.

You can look at the code for the interface here and can use the interface here (currently in beta).

Next Steps

In the next phase, an openwrt tool has to be engineered which can be used for upgrading OpenWrt automatically.
A interface for the tool has to be created for LuCI which will house all the settings and preferences of the user. The tool will periodically check if there is a new version of OpenWrt, if so, then it will download and apply the upgrade package automatically.

The foundation to check if there is a new version was laid during the first phase with the JSON metadata.

It’s been a month since the last post, so it’s time to summarize this period. In the previous post I showed the beginnings of my adventure with implementing the designed look of the application. That version looked as I intended, but it lacked interactivity, data management and connection to the main retroshare-service application.

The current version of the application has added animations for transitions between screens or tabs. Example animations (due to limited upload size, video is badly compressed 🙁 ):

Also, screens have been added to help various side activities such as creating a new identity or to make it easier to create an account, if it doesn’t already exist.:

And the next thing I spent some time on was the correct handling of queries for retroshare-service and data processing. Thanks to that, in the application we can already perform basic activities such as logging in or managing our identities.

In addition to the activities presented here, I devoted some time to just building a retroshare-service application, which exceeded my expectations about the ease of this task.

Next steps

In the application there are still a few minor modifications to be done. One of the key changes is to ensure proper storage and generation of the authorization token. Nevertheless, the main task of the upcoming period will be to add new functionalities and minor fixes to the Retroshare itself.

Due to the fact that visually the application is already very close to the final product, I would be happy to hear some criticism and hints what needs to be changed.

Cheers,
Konrad Dębiec

During this coding period I added automatic processing of the data
collected by the wireless testing framework conTest, as well as the graphical
representation of the data.

While processing the data I found some issues in the controller software
as well as the physical testbed.

Most of the tests since the last Blog post were done with the attenuation values
you can see in in the lists below. I changed timing values between the experiments,
to validate the behavior.

Attenuator 1 uses the following values for its attenuation settings:

• 10 s -> 0 dB
• 10 s -> 60 dB
• 10 s -> 0 dB
• 10 s -> 60 dB
• …

Attenuator 2:

• 10 s -> 60 dB
• 10 s -> 0 dB
• 10 s -> 60 dB
• 10 s -> 0 dB
• …

My initial experiment to test functionality was not the best one to validate the
testbed (fig. 2). As both paths were basically set to the same attenuation
I only checked the signal in Wireshark as seen in figure 2, which matches mostly with
the black line in the closer look in figure 3.
The second figure was created with my evaluation scripts, while the first one was
created with wireshark during the first tests.

In the testbed I am using two analogous attenuator in addition to the programmable
attenuator. Figure 4 shows, that one path is stronger dampened. Without the
analogous blocks both paths are nearly equally attenuated, which could mean,
that one of them is broken, or not meant for the frequency range, while both of them
should be. It will need some further investigation, but as I had most of my exams
during this coding phase, I was a little short on time. For now I will continue
without these two devices.

In figure 4 you can see a section where the one of
the two paths is not following the values presented in the lists at the beginning of this post.

This is a problem in the control software of the digital attenuator. I was able to work around
it and fix two more problems in the software. The actual problem needs more time
researching putting into it, but more than I could afford at this time.

The evaluation script will check for our default files, filter out data, we want
to plot, like signal strength, selected MCS rate, throughout rate and packet
numbers. Afterwards the plots are constructed with python and matplotlib.

For most figures, places with more intense red and blue colour are values, which
were encountered more often. The black line represents the overall signal strength
provided by tcpdump.

I Realized that the visual appearance of the application felt very bland and uninteresting, so I decided to shift some of my focus to the design and visual aspect of the UI. I did plenty of reading about UI/UX design principles and modern best practices during this time. And looks like it turned out pretty well, and is definitely a good improvement from the previous appearance. Also, since this is my first attempt at doing professional-level UX design, there is probably room for improvement, so feedback and suggestions are always welcome.

The general theme has been redone from scratch. I chose this soft blue color palette by taking inspiration from the main app’s look:

The home tab, along with displaying the user certificate, now also allows to add friends by using their certificates. It is possible to add friends by copying the certificate contents, dragging and dropping the file, or simply selecting it from the file manager.

Implemented modal messages within the browser that can be used as a popup dialogue box to display any kind of information (here showing information extracted from a Retroshare certificate):

As you can see, the navbar has also been revamped. And the best thing about it? Icons! I along with my mentors agreed on using the Font Awesome icon library, which is open source (licensed with a combination of MIT, CC 4.0 & OFL 1.1 licenses). I can now utilize icons across the whole app.

The downloads tab has also been redesigned. Now showing all downloads in a slightly different way. This layout was chosen with extensibility in mind, it can easily be extended to contain a additional file-related settings and chunk views by having an expandable options box for each file.

The config tab can now be used to change a lot of the setting options similar to the main app. Network, node, services, files and people sections from the app have been implemented. I will shortly finish the remaining sections too.

Also notice the tooltip icon. Which when hovered on, gives a brief description about the option. Just like in Retroshare:

Next steps

Now that the design is steadily making way for a more detailed and specialized variety of widgets and components, I am working on creating tabs for Network, People, Chats, Mail, Channels & Forums so that the Web Interface can finally become a fully usable alternative to the main client app.

You can try out the Web Interface by cloning it from the repository: https://github.com/RetroShare/RSNewWebUI, and my fork: https://github.com/rottencandy/RSNewWebUI. Again, I am always happy to receive feedback and suggestions for improving the Web Interface.

Here again!

This evaluation I spend the work creating an automatically generated wrapper for the API. This wrapper is generated analyzing the Doxygen XML files generated when Retroshare is build. 

Creating the API wrapper

First of all, I modified the python script (made by @sehraf ) that generates the C++ API files, to create a python wrapper for the API. Analyzing the script and the XML files I get my script working generating a first version of the wrapper. Then I test the wrapper, giving support to async functions also. Some features of the wrapper are:

• Document the code using DocString convention.
• Parse also ‘manualwrappers’ like attempt login.
• Requests with authentication and without.
• Support basic authorization or token auth via ‘Authentication: basic bas64Token’ header.
• Async methods support and callback implementation.

Here an example of the API wrapper generated: https://gitlab.com/snippets/1877207 . Some tests for the wrapper can be found here:

class TestMultiple(TestCase):
    def test_login(self):
        res = wrapper.RsLoginHelper.isLoggedIn()
        print(res)
        # Do login
        if not res['retval']:
            res = wrapper.RsLoginHelper.attemptLogin(ACCOUNT, PASSWORD)
            print(res)
            self.assertEqual(res['retval'], 0, "CANT LOG IN")
            return
        self.assertEqual(res['retval'], True, "is not loged in")

    def test_authorizedMethod(self):
        res = wrapper.RsGxsChannels.getChannelsSummaries()
        print(res)
        self.assertEqual(res['retval'], True, "Can't get channel summaries")

class TestAsyncMethods(TestCase):
    def cb(self,res):
        print("cb", res)
    def test_asyncMeth(self):
        wrapper.RsGxsChannels.turtleSearchRequest("XRCB", 300, wrCallback=self.cb, wrTimeout=4)

Creating Retroshare Classes wrapper

After that, the problem was that a lot of functions need to have Retroshare classes as parameters. For example, to create a Retroshare forum, are needed classes like RsGxsForumGroup that at the same time need other inner classes like RsGroupMetaData. With the first version of the wrapper all this classes was passed in JSON format that was really annoying to assemble. 
So the next step was parse also this Retroshare classes recursively from the XML files to a Retroshare classes wrapper. On this step it was difficult to parse it correctly: differentiating the different types of classes, and class attributes, translating the types to python, if they are enums, primitive types etc… Finally I created this second class wrapper so when you need to pass a RsGxsForumGroup to the API wrapper you can just instantiate it and the wrapper do all the necessary to transform it to python and call the API. Some features:

• Parse “compound” classes (structs on C++) recursively.
• Parse “enums” and get their values.
• Parse “typedef” and “using” classes and translate it to the appropriate type on Python .
• Document it using Docstring convention .

Here an example for the class wrapper: https://gitlab.com/snippets/1875153 . Some tests can be found here:

    def test_createChannel(self):
        channelMetadata = RsClass.RsGroupMetaData(mGroupName="TestChdddannelCreation2", mGroupFlags=4, mSignFlags=520)
        channel = RsClass.RsGxsChannelGroup(mMeta=channelMetadata, mDescription="Channel Test")
        res = wrapper.RsGxsChannels.createChannel(channel)
        print(res)
        self.assertEqual(res['retval'], True, "Can't create channel")

For “v2” methods I opened an issue because I can’t communicate with the API. It was resolved as my “retroshare-service” wasn’t updated at all.

Next steps

This script will be adaptable to generate the wrappers for the language needed, for example for an OpenAPI format, TypeScript… Making much easy to other developers to start developing over Retroshare network.

Also it will be very easy to update when new feature is added to the API because it can be generated each time Retroshare is built.

Now will be time to apply the wrapper to the scripts that will import the public datasets!

It’s a wrap for phase one! We have managed to keep up with the goals we set for ourselves and along with my mentor Paul Spooren we have shaken up BMX7 development all around.

Documentation Refactor

One of this phase’s goals was to understand BMX7 functionality, usage and plugins. With a lot of reading (both in-project documentation and with external sources) this was achieved and I was much more synced. While doing so I refactored some pieces of docs to help me and I found a branch I had created with some generic changes that I made in the Community Bonding phase.

These acted as inspiration and I got together a big batch of changes commited and PR’d to the bmx7 upstream.

The PR can be found here and its purpose is to achieve generic changes to ease the introduction of new developers and users to the project, as well as ease the learning curve of usage.

More info one can found in the following blogpost that describes things in detail:

• https://blog.luserx0.cc/GSoC-GSoC-2019-Week-3-Documentation-and-Instrumentation/

It’s worth to be noted that this was not a goal per-se of the phase and that changes did take place, but they sparked other changes that are still WIP on the PR.

Testing Infrastructure

Establish the following Testing setups for testing and later on continuous integration:

• LXC running OpenWrt bridged mesh.
• Qemu OpenWrt bridged mesh networks

The idea was to establish Linux Containers and Qemu infrastructures to do testing and experimentation. Along with aparcar my mentor “we salvaged from oblivion” a very interesting and well-set project that axn had created named Mesh Linux Containers (MLC) and which takes on its own the creation, configuration and command and control (CnC) of a virtualized mesh network on a Linux host, supporting an abundance of community mesh protocols. Paul took on him the modernization of this project producing the new generation of MLC, which switches mlc’s functionalities from the shell scripting era to LXC and distrobuilder to autocreate and populate an OpenWRT mesh mother and OpenWRT clients.

On the above I also based my testing infrastructure, but being a Qubes OS user and relatively confused with LXC versions (a newb), I kept coming across issues.

The situation was able to be relaxed by investing time on QEMU for love and support (as one may find here). Again though problems like libvirt on Qubes, multiple Qemu interfaces and tricky bridging tricks on a standalone VM didn’t allow me to have the results I was hoping for.

Hence, armboards were chosen. I borrowed boards from the hackerspace.gr common sharing stash of toys and I was able to interconnect a Raspberry Pi 1B, one Raspberry Pi 3B and a BeagleBone Black.

This approach is optimal for me to establish decent OpenWrt routers to test and play, that can painlessly run BMX7 (and Wireguard) and leave to the local community things to use like mesh boxes and on-the-go VPN-configured APs.

(The bmx7 luci app works like clockwork btw.)

BMX7 wg_tun Plugin

The initial goal was to implement wrapper functions to WG and establish a secure tunnel that way. This adds an overhead in the development phase, because making wg binary calls is both costly and requires to place code in places where no further improvements can be achieved.

For this reason, the embeddable wireguard implementation was chosen and we switched from addition(to the current tun src) to the creation of the wg_tun plugin earlier on than what was planned.

The idea is that adhering to the current implementation of the tunneling plugin of BMX7 we can reuse some minimum parts and establish:

• The creation of a public and a private wg key associated with the current bmx7 session (or persist across sessions).
• The advertisement of these keys and the network(IP) associated with our interface through descriptive updates for our peers to find us.
• The establishment of connections with peers that have advertised their public keys and networks (and we’ve caught them), a la tunnel announcements style.

This goal is still WIP as more testing needs to take place and optimal ways and options for bmx7 node administrators to use it. In general, for this part and until further work has been put in, the paradigm of danrl for the luci-proto-wireguard package is being followed in the sense that we want the establishment of bmx7 wg tunnel to be that easy and the static configuration or the randomized UDP ports can help get there faster.

Progress can be found on the implementation’s branch on my fork (https://github.com/luserx0/bmx7/tree/wg_tun_plugin)

Developer Misc

My dotfiles repo has seen a lot of action this first period. The refactoring of my vimrc took place to help me in reading through code (a lot of code). The configs are managed through GNU Stow. If something catches your eye, clone, run the install script, tweak and share back.

My personal Blog saw also some pretty intense developments too to make progress report sharing and tutorial posting humane. It’s based on hexo and hexo-next, it’s static and hosted on gh-pages. Perfect combo for a GSoC student. Feel free to grab it from here and use it under CC.

If you have questions on how BMX7 operates and/or is different from other mesh routing protocols feel free to post them here.

Next Steps

• Additions of features and options of the wg_plugin. Getting it merged on the upstream is a good milestone.
• Cleanup and research on reusing cryptographic and networking primitives common to Wireguard and BMX7.
• Creation of the bmx7 Debian Package.
• Port mlc functionality to mlc-ng.
• Finish the second phase of the documentation refactor.
• Attending Battemesh V12 😉
• Order hardware to play with