Tag: Irdest

Note: You can read the same post in LaTeX here

Prelude

Hi super happy to see you here! It has been an exciting and productive summer from which I learnt a bunch of new stuff and irdest plus GitLab have been gracious on me. The project went through many ups and downs but we made our way fixing the bugs and making things work as expected. I hope I’ll be able to convey some information of the work done by me on apiece of this magnificent software over the course of the summer. Let’s begin!

Irdest!?

Okay, so first things first. Let me introduce you with Irdest and what it does to make sure we are on the same page and irde.st doesn’t sound completely (we)ird to you, and then we’ll progress on the title afterwards. First, I’ll try explaining it in a single line,

"Irdest... is a beast!"

Okay no jokes this time(that was no joke btw), going to explanation for real xD, irde.st is a software suite that allows users to create an internet-independent, decentralized & ad-hoc wireless mesh network. It removes all the dependencies of a user from a specific service and enables users to create a local network mesh of their own. It does not expose data or information of the user. Even the IPs of the peers present in the mesh are not known, they communicate via routers and the entire communication is end-to-end encrypted between the users, thereby increasing privacy in user data. As of now, Irdest supports various functionalities to users like sharing files over the network created, call between users, and messaging.

A Gist

This summer was focused on building the FFI Layer to implement the features supported by the library in the upstream. So if you have been following the initial three posts by me on the same topic, then you must be aware of the fact that the biggest challenge being encountered is compiling library properly and linking it to the application in the compile time itself.Apart from this, considerable challenges were about maintaining the robust CI which makes sure we don’t break stuff at any point of development process, and the very sensitive FFI layer. We got through these challenges and finally implemented some of the upstream features in the application, but not all. Because with this sophisticated setting of the components we need to move forward carefully in order to not break stuff, and with limited time in our hands we decided to implement some of the very basic features in the application and write an unbreakable CI for them, from which we can make use of the build artifacts and can keep track where things break.

Work Done

Without going into too much depth of the concepts/thought process and discussion,let’s quickly touch upon the work done in the course of this summer of code. You can refer to the previous posts if find yourself interested in detailing of the changes made/steps taken and why and stuff like that.

I. Compiling the Rust Library

The very first thing I did as a part of the summer of code was fixing the rust library compilation. Initially, the rust library was broken due to the massive refactor and some portion of the huge codebase being left.Due to compilation errors in the rust library(and I being beginner to Rust back then) it took some time to fix the errors, refactor the remaining portion of code accordingly and make it build green. As soon as the rust library was up, the target was to make the application compile and link the library to the application in the compile time.With all these changes being made, a challenge was to write CI for all this cross-compilation setting, which I had never done before.

II. Writing the CI

Writing the CI for android components including our FFI bridge wasn’t that tricky, but it did require some good knowledge of cross-compilation, Cargo and obviously android :P. But we ended up implementing that too, and with the current state of CI, nothing can break easily and we have awesome and strict checks that compile the components as per the need. We made use of GitLab’s one of the greatest and finest works, which is their CI,how they organize and define Pipelines, Jobs, triggering mechanisms and artifacts handling in subsequent and post Jobs. We combined the power of GitLab-CI and our own custom docker image irdest-android-build-env. This made our CI run lightning fast, Jobs that took 11mins to run without any Hi-Fi image being used now finished in 3 to 4 minutes, this was a huge gain and we were able to optimize our CI runs even more via redefining Pipelines, Jobs flow and via introducing the concept of Child Pipelines, another great piece of work by GitLab.

III. Implementing the Features Supported by Library

After all this CI and basic stuff being done, we moved ahead with implementing the functions supported by our rust library in the application. So I implemented the login and registration features, both in the single MR and due to very less time left in hands, I had to make major UI changes in the same MR, thereby increasing its size, the UI changes were not stellar, but they made the application layouts very responsive and with almost zero dimension hardcodings, everything works like springs, other ones get adjusted automatically, if the change is observed/experienced by any one of all present(for a particular layout).

IV. Some UI Fixes

Also there was a very nasty UI bug that I can remember of, in the Login/Registration screen, in which the screen got split into two components,the login one and the registration one, so in this I setup the optimal fragment transactions and created an abstract layout in the root screen which is empty by default and sets the desired layout file as per the requirements, e.g., it shows the Registration one if clicked on registration button and similar for the others.

V. Codebase Modernization

In the final days, we moved towards modernizing the application codebase via following some best practices in it and removing the old/deprecated ones : P , but sadly this couldn’t be merged because of the changes made in the NDK v23 API,which made our cross-compiler plugin incompatible with the project and thereby leading to CI failures, although all of this has now been fixed locally at my fork, but we wish to implement a stable and elegant solution after pondering on the problem for some time. So, along the lines for codebase modernization, the opened MRs included the migration from ol’ school Groovy Gradle files for dependency management to human readable Kotlin DSLs, along with some tool version bumps(out of which one was our NDK which I bumped to v23from v21 xD, yeah I can see ‘ya a bit sad, it hurts ; ( ) and some changes in Kotlin scripts we were able to compile the library directly from the Android-Studio itself, which previously was a great PITA and we had to manually compile the library. The next MR targeted the migration from legacy view scans to ViewBinding, increasing the application performance!

Ah, I am not going to list all the MRs opened by me in the summer here, but if interested you can give ’em a look here:
* we/irdest/merge_requests?author=s-ayush2903

Further Possible Improvements

Well there are really a bunch of improvements that can be made in the existing codebase! Let me help you think of few:

* Writing Unit tests for the features implemented by far
* Writing Instrumented tests for UI flow implemented by far
* Making the application support many/some more functions that the library supports
* Running instrumentation tests on CI
* Fixing the NDK v23 incompatibility with our cross-compiler plugin

last entry was a joke(that was no joke btw), ignore it xD

Acknowledgements : )

Well we finally arrive here. A huge thanks to my amazing mentor, Spacekookie ❤️, who was always there to help me out when stuck and shared their valued thoughts on what directions we need to take for the project. Discussions with them have always been super super insightful and let me ponder for a while about their thought process in figuring out the solutions. A big thanks to you again! Nextly, this project would never have been possible without the organization Freifunk where I got accepted as a GSoC’21 student to work on one of their project. It was a truly amazing experience where I learnt a lot of new stuff and met people having similar interests, which made the project and discussions more involved, productive and helpful. Thanks to all. Although I’m a bit disappointed about the very limited time we had to work on the project and couldn’t make it to the level we thought at a point of time.

But anyways, super happy after working on Irdest!

Btw you can find me on GitHub with username: s-ayush2903 👀

Cheers Until next time we meet 🥂
~Ayush Shrivastava

Note: You can read the same post in LaTeX here

Prelude

Hell yeah, we paved our way to the conclusion of summer of code while working on this magnificent piece of software, Irdest and I’m super excited that you too are here! It is super happy to see if you’ve been following the series of the trailing blogs where I shared the progress the project made with the time of the course of the summer and the proposed timeline. Okay, so in the final phase of summer of code I focused on implementing the features supported in the upstream by Irdest(in the Rust library) in the android application, along with implementing better CI(will touch upon it later) and revisiting how we used our pipelines, Jobs and our custom docker image for CI, also, easing the cross-compilation for developers and modernizing the application codebase via using the best practices, although we faced blockers due to internal changes in NDK v23 and could not go ahead with all the changes, yeah quite sad : (

Okay, so now let’s see in detail and discuss the work done on the each component and brief thought process behind decisions made. This document first contains the work in final phase of summer of code

I. Implementation of Features Supported by Irdest

This was the crux of the project and quite a tricky and technical task to implement, all the work done on fixing and rewriting the FFI layer, whether it be from android application side or the android-support crate from the rust library, in previous phase comes into action here. Considering the time available to us and keeping in mind about not being overwhelmed or too excited to write a bunch of core-library functions, we decided to implement the basic functionality of user Registration and Login in the Application, and manually test these functionalities work fine and wrote a CI for them as well, to not let regressions creep in our codebase again. Yeah, so for implementing the Registration feature in the application, I fixed the FFI layer(again : P) and correctly set the wrap/unwrap functions in the rust side of FFI layer, fixing package name along with mentioned tweaks resulted in correct functioning of the Registration feature. So you can now create a new user and get a cryptographic ID assigned to it and use the credentials to login to the application. Making similar changes in the Login function of library, fixed stuff. With these library functions being fixed, the Auth began to function, theoretically. I had to change and fix the UI/Navigation setup in the application, how screens are changed/exchanged etc, in order to make Auth work, from point of view of an end-user.

II. Redefining & Re-architecting the App Navigation

Previously, the Register screen wasn’t being displayed properly, it was nested or better word to use, split the screen in two parts, Login one and Registration one(see we/irdest/#21 for more context and a clear picture). The problem turned out to be how Fragment transactions in the application were being handled and how we exchanged layout files on-the-fly along with the aforementioned Transactions. So previously everything was handled inside a single root file only, the layout(of login screen) was already present there by default and on going to registration did not entirely remove the Login layout instead split the screen, and some hardcoded dimensions too were present, making the problem persist more and less easy to fix . So what I did was creating an abstraction in the root layout file and keeping that abstract layout empty by default, with proper dimensions, which made sure the entire space is occupied by the concerned screen. So that root layout in the main file was essentially a FrameLayout which spanned screen accordingly and that FrameLayout held exactly which layout is going to be displayed on the screen. So you can consider this FrameLayout as a container which showed layouts as per requirement and initially contains nothing. Yep, you never get to see an empty screen, which is because we dynamically set the layout to be displayed in the FrameLayout via Kotlin files in the order the screens are supposed to appear. Well that’s enough discussion on the topic.
I made all the changes discussed in the previous two sections in a single MR : P , so here it goes
* we/irdest/!38

III. Revisiting the Project CI

Okay, so by then we had our Rust library being compiled in our CI pipelines, but we wanted more than that, the usability of the components/artifacts that were being produced as a result of builds. So we decided to publish the rust library to GitLab CI directly from the pipelines and use those artifacts as per need.Also, we used to publish the APK but since no cross-compilation was taking place in the CI, hence the APK being published from there was pretty much useless, so we enabled the cross-compilation in the CI and continued the APK uploading, as a result of which, the application installed using the APK from CI pipelines was running properly on the device. Next were some productivity related changes madein the CI, e.g., by design the APK obtained from application build is stored deep down in the app/build/.../.../debug/app-debug.apk and was being uploaded to same path in the artifacts archive from GitLab CI, I removed this Matryoshka dolls style hierarchy and moved the needed build files/reports to the top level directory.

You can find the corresponding MRs below:
* Enabling the Cross-compilation: we/irdest/!34
* Uploading Rust Library as CI artifact: we/irdest/!35
* Removing Matryoshka dolls style artifacts archive hierarchy: we/irdest/!40
* Uploading Lint Reports on Failure: we/irdest/!42

IV. Modernizing the Application Codebase

In the final days of the summer of code, we took active and fast steps to migrate chunks of our application codebase to follow Modern Android Development practices. Although, due to some NDK version incompatibility with the cross-compiler plugin we were unable to merge these changes and unable to fix our docker image too.But anyways, since the CI was green previously with optimized build time and our exhaustive docker image, so it’ll have to work again this time too! Okay, so coming back to the topic, the first MR I created in this direction was the migration from Groovy Gradle files to Kotlin DSLs, this migration already as numerous and obvious benefits over conventional Groovy Gradle files, but the cherry on the top was that with these commits in the MR, the cross-compilation was automatically being triggered on hitting the build button/icon only! Previously we had to compile the library first and then the application, to link the library to the application, but this MR saved us a huge time and PITA : )

The next step was regarding improvement of application performance via reducing Memory consumption while it is running. To achieve it we first eradicated all the findViewById() calls and the not so recommended Kotlin Synthetics as well, you can learn about the reason for the change in the linked issue(s). We instead used ViewBinding to bind and reference the views in runtime without worrying about the application crashes, this was a huge asset and since no view scans were being performed in the application runtime, the application runtime speed also increased and resulted in a decrease of memory consumption. But sadly,we couldn’t go ahead with the merging of these MRs because of the mentioned NDK version and cross-compilation plugin incompatibilities : (
We’ll be able to merge these as soon as we fix the docker image. Although, there is a way to fix it but that is not elegant, also, we want to do it for once and all, like no need to touch that CI file again unless we have to introduce some entirely new Job.
Find the corresponding linked MRs here:
* Migration from Groovy Files to Kotlin DSLs: we/irdest/!36
* Using ViewBinding & Remove Slow stuff: we/irdest/!41

And, the issue:
* Using ViewBinding instead old methods: we/irdest/#22

Cheers Until next time we meet and hope to see ‘ya in the final report!
~Ayush Shrivastava

NOTE: You can read the same post in LaTeX here

Prelude

Hello! Good to see you here : ) This blog is mostly a summary of work done till now under the first coding phase of summer of code of ’21. Picking from the end of previous blog post, we planned implementing chat feature in the application module, but due to the aforementioned massive refactor in the entire codebase and upgradation of existing modules to support modern hardware the chat API has been deprecated, and some components, because of not being part of CI, got broken : ( To implement features in the module the very first step was to get the project build properly, previously (maybe) due to migration from different version control system to GitLab and that massive refactor there were some unidentified problems that did not let the application codebase build properly, also the main lead of the picture `android-support` crate, not being a part of our GitLab CI workspace too, wasbroken. We fixed all this entire stuff in multiple different steps, each solving a mini problem and writing CI for each missing component so that we or anyone joining the project never encounter similar problem(s) in the future.

I. Fixing the Android Application Codebase

The application codebase was considerably broken and for the very first time when I built the application, it instantly said build failed in10ms, which is really very weird as when for the very first time you build an android application, it takes noticeable time(~6 minutes), this time is for fetching the dependencies that are declared in the dependency management file of android codebase(the ones with the build.gradle name) followed by compiling the android project codebase. It was quite astonishing at first sight,but on closer look to the files present in the android codebase the cause was observable. It was the presence of dependency archives in the android codebase and their corresponding XML files too, and since these dependencies were already present, studio didn’t take the pain of fetching them from maven. So the question arises, When the dependencies were already present still then application didn’t compile, why? Actually what happens is that these dependencies’ XML files are editable so even the slightest edit in them renders them useless and studio doesn’t even report any kind of problem with them, another thing that happens under the hood is that studio stores these dependencies in its local cache so that when user re-compiles the application, no time is taken in fetching the dependencies and it can perform the real build. Also, by time these caches get corrupted and usage of very old cache does not let the project work in the way it should.
Okay, now let’s come to the point how we fixed it. As by now it should’ve been clear that the problem was the existence of binaries and XMLs of dependencies present in the android codebase, so the solution that I anticipated was the deletion of these files. It was not sufficient. After doing this dependency management did go as expected, but still the build failed : ( After some more inspection, I found there was some problem with gradle executable scripts as well and the gradle-wrapper.properties too. So I just referred these scripts from my previous working projects and it finally started working, a moment of joy 🥳, after many days + nights of pain ; ) As this problem was fixed, after working on some other crucial matter(see next section, II one), we wrote a CI pipeline specially for our android application codebase so that it doesn’t break again ever in the mainstream. The android-application pipeline comprises of the 3 stages, in which its lint is checked, followed by build and then the tests are run. In the upcoming coding phase we plan to make this CI pipeline even more robust and enforce stricter formatting rules, introduce Static Analysis and run android Integration Tests on GitLab CI, well we’ll discuss it the next time we meet, leaving some topics for then 👀
You can find the corresponding MRs here:
* Fixing the android application codebase: we/irdest/!21
* CI for android application codebase: we/irdest/!23

II. Fixing the FFI Layer

After the previously mentioned refactor, everything was working fine, only android specific components of the codebase were broken. A part of which, was our FFI layer, the android-support crate. This layer still held references of several deleted and deprecated APIs, therefore compiling this crate too gave a bunch of errors. Fixing them took much more time as this crate was written in Rust and then I was not that fluent with it. So fixing it included updating/modifying existing functions or we had to remove functions as well because of the deprecations. A nice challenge that we encountered was saving the state across multiple platforms(supported hardware), because the crate we used for saving state provided support for almost all the operating systems other than Android. So what we did was using the knowledge of android that an application has access to its own private storage which no other application/service can see, so all we needed to do now was to find this directory in android device file-system, this path we achieved using the ADB, now we investigated where our crate went wrong, so for this we dived deep into the crate’s API and read how it achieved similar behavior for other platforms, which was that, it first of all found the HOME(environment variable) for the OS and then located corresponding path(s) for saving state in dir/file(s), turned out, that the crate was identifying HOME wrong only for android file-system. After this was diagnosed, we wrote a custom API that found HOME env var on all platforms irrespective of their OS (see this patch), by this API we were able to access the app-specific private directory and save state there. It was quite challenging, but we figured it out! Everything related to FFI layer, after this fix was quite easy. We eliminated the problems that existed in FFI layer via refactors and some modifications in functions and then, it built green! After fixing the FFI layer we wrote the CI for it that makes sure it builds each time a commit is pushed to any MR or branch and we can see the build status in pipelines too. Writing the CI for android-support crate was not a cakewalk,actually the application needs cross-compilation of our Rust library in order to function, so we need to make sure that the library which application is going to use on android devices is really compatible with android platform and by virtue we compile it on our PCs directly so that doesn’t work quite, to make expected behavior happen there are two options:
* Compiling the Rust library codebase on android-device(less feasible), or
* Cross-compiling the library on our PC/CI runner via providing support tools for android components
So quite obviously we went with the second option, for this we installed rust and android compatible components in our runners during the CI runtime and then compiled the library via checking out to the correct directory. But since, for compiling Rust library in each CI run we had to install the components and this specific pre-compilation part(or better to say setup portion) consumed a considerable portion of our CI script(and made it look a bit daunting too), so we packaged these components to our custom docker image and pulled it each time in our CI runs, this made our life easy and scripts beautiful : )

NOTE: If you don’t have much idea of cross-compilation, then you can have a look at this awesome blog-post by Milan✨. It gives a clear understanding to the reader what cross-compilation is, irrespective of their previous knowledge on the same(yes, but basic knowledge on compilers is needed a bit). Spoiler alert: That someone in the opening of blog post is me 😛

Also, since the refactor was incomplete in our android application codebase and the android-support crate so between these to big tasks, I fitted this small refactoring, as a light break 😛
You can see the MRs for them here:
* Fixing the FFI Layer: we/irdest/!31
* Refactoring the android-components: we/irdest/!32
* Adding the android-support crate in our CI Pipelines: we/irdest/!33

III. UI Improvements

After fixing issues in android application and our Rust library, and writing a robust end-to-end CI for them we moved forward towards improving the UI of the application. Previously, the application used legacy design components and ideology, under this task we modernized these UI components and followed the material design guidelines(material.io), that improved the overall look of our Authentication screens. There is nothing much to explain in it as it was really quite easy to achieve and also we didn’t encounter any problems.
You can see the related MR here: we/irdest/!26

Acknowledgements

Well, by far it has been the most exciting summer for me and I had interesting experiences working on the project. Fixing components, was very difficult at the beginning due to many reasons one I would mention is the huge codebase which we have and it is not easy to learn about the functionality of each component present in it in short time, and also everything is intertwined too at many places. Going through it and fixing issues would definitely not have been possible without the immense support from my mentor, Spacekookie, who was always there to help me out and direct what to do. Their advice greatly helped in speeding up the development process and they are also a source of inspiration to me. Most importantly, Milan, who is not officially my mentor but has helped me a ton of times in technicalities of CI and setting up Nix environment(which I initially used for cross-compilation) about which I knew nothing, and many more instances. It won’t have been easy for me to accomplish aforementioned tasks without direction and help from Spacekookie & Milan.
Thanks again to both of them : ) and I’m more excited to work on the project with them further!

Cheers Until next time we meet!
~Ayush Shrivastava

Hello everyone, I am Ayush Shrivastava, one of the students selected for Google Summer of Code 2021 for Irdest sub-organization under the umbrella organization freifunk. If you’re wondering which topic/project the blog is focused on, then you may give it a look here!

Irdest

Well, if you don’t know, then let me first introduce what Irdest(irde.st) is and what it does/supports. So, Irdest is a software suite that allows users to create an internet-independent, decentralized, wireless, adhoc mesh network. It removes all the dependencies of the user from a specific single service and enables users to create a mesh of their own. In this network mesh, users can communicate to each other via messaging(both, individual and Room) and placing voice calls. Irdest network mesh service also allows users to share files between them without relying on the internet. And also when a user enters the Irdest mesh network(note that user needs not connected to the internet i.e., without having an IP address, it can be a P2P connection over Wi-Fi or Bluetooth), their IP address is completely hidden thereby maintaining privacy andreducing possible breaches that may occur. In the irdest network mesh, as soon as a user enters the network, they are assigned an ID, which is a unique cryptographic key that helps identifying the users. All the messages/calls/file transfers made in Irdest network mesh are completely end to end encrypted, again a positive sign from security & Privacy perspective. Currently, Irdest is supported for Linux & Android devices. For android, it is pretty much in incubation state and there are a bunch of points & directions where we can improve. This summer, I aim to implement some features on android client from the irdest upstream.

WARNING: This blog has a tons of mentions of FFI, you may get overwhelmed by the term, so it stands for Foreign Function Interface.

Current Progress & FFI Overview

So this was what Irdest is all about and a higher level overview of how things really work in it. As mentioned previously, the android client is pretty much in its incubation state, so we wish to implement it in a clean & modern way. So, since we want to make use of features supported by Irdest(in upstream) on the android application, therefore we need to maintain/create a binding or kind of a link between core Irdest code that makes all this internet independent network mesh possible & application codebase(to call those methods from). The point is that the core Irdest code that supports these functionalities is written natively in Rust, and we we want to utilize this pre-written library on android, without writing it again(in Kotlin)for android. So to make it possible, we’re going to write an FFI Layer(I’ll write in detail about FFI in some other post, but for the time being it is sufficient to know that by FFI we can call functions written in a specific language(mostly native languages like C++, C, Rust) from some other languages). This FFI layer currently exists in the module, but it is a bit old-fashioned, so in the initial phase of my coding period I plan and propose to rewrite this layer with best possible modern day FFI practices and following some standard references. The most crucial part of this project is this FFI layer, because it is the fundamental building block for the android project, once it is set and ready, then we’ll be able to use the functions and services provided in the native Irdest library written in Rust and extend our further development process via writing the application.

FFI Implementation – A Blueprint

The FFI already is quite notorious for the undefined behavior and on top of it, there is very less official documentation available on Rust-Kotlin FFI so this makes it more tricky & challenging to implement, but for the course of our development process, we’ll be following and taking inspiration from how Mozilla have implemented FFI integration between their existing rust library and their firefox android client. Another instance where they use Rust-Kotlin FFI is their Glean project(a telemetry service). I’ll be taking most of the inspiration by and practices being followed in Glean, as a result of which we’ll be writing FFI bindings on our own. After once we’ve written FFI layer, my next step will be to test it again thoroughly and make sure it works perfectly and does not has any bugs residing.

Further Development:

Once we have a robust FFI layer ready, then I’ll move to core android development process, that’ll entail writing UI for the application, architecting it properly, modularizing the application codebase and following the Modern Android Dev practices. Well this will be a relatively easier job to do in comparison to that of writing FFI layer xD
My next step after this would be implementing the chat service in the application. We’ll focus in its detailing after we’re done with the very first milestone, the ”FFI Layer”. We’ll see the plan of action for chat service in a separate blogpost, after the first coding phase : )

Thanks for reading!
Cheers, Until next time we meet!

~Ayush Shrivastava