Category: Allgemein

1) Introduction

If you haven’t already done so, I highly recommend reading the introductory blog article on the user space resource allocator for Linux-based OpenWRT access points. The blog gives a thorough introduction to rate and power regulation in IEEE 802.11 devices, which will be beneficial for understanding the work covered later. Happy reading!

Before extending the py-minstrel-ht package, it is crucial to ensure that the user space rate control mirrors the behavior of the kernel Minstrel-HT algorithm. This alignment will establish a reliable basis to evaluate the inclusion of power control in the Minstrel-HT algorithm through performance comparison with the kernel variant.

2) Passive Minstrel-HT

To conduct a precise evaluation of py-minstrel-ht alongside the kernel Minstrel-HT, a passive version of the user space Minstrel-HT was developed. The passive py-minstrel-ht runs alongside the kernel Minstrel-HT where the kernel algorithm performs rate control while py-minstrel-ht passively reports all of its rate selection. In order to compare the rate selection between the Minstrel-HT algorithms, the rate statistics for both rate controls must be identical. As such, he py-minstrel-ht obtains the packet counts via the “stats” API lines from the kernel rate control.

In an ideal scenario, during each rate selection, the statistics on both algorithms would be the same yielding identical rate setting. With the help of this passive variant, it has been easier to evaluate the behavior of the user space rate control and investigate the implementation errors in py-minstrel-ht. The WPCA API through minstrel-rcd can send parsable traces to the user space which can be saved. In the output traces, the WPCA also provides information on the rate selection via “best_rates” line which triggers the rate-setting process in the passive py-minstrel-ht. Similar to the API traces, the passive Minstrel-HT saves its own trace which is later analyzed in conjunction with the kernel traces. The following code section consists an example of such API trace:

wl1;174a4f945a7a9aa0;txs;a0:78:17:74:c2:5f;1;1;1;226;2;233;2;ffff;0;ffff;0
wl1;174a4f945bcf1660;txs;a0:78:17:74:c2:5f;1;1;1;265;2;273;2;ffff;0;ffff;0
wl1;174a4f945bcfaad0;stats;a0:78:17:74:c2:5f;136;0;0;0;7c;0;1b2
wl1;174a4f945bd00750;stats;a0:78:17:74:c2:5f;226;0;0;0;7c;b;436
wl1;174a4f945bd02c80;stats;a0:78:17:74:c2:5f;234;11e;344;31;31;31;53d
wl1;174a4f945bd041c0;stats;a0:78:17:74:c2:5f;233;1b0;385;7c;f8;17e1;3371
wl1;174a4f945bd05d90;stats;a0:78:17:74:c2:5f;265;0;0;0;2;290;7d6
wl1;174a4f945bd07eb0;stats;a0:78:17:74:c2:5f;273;272;592;90;11e;1e7a;3af0
wl1;174a4f945bd0bec0;best_rates;a0:78:17:74:c2:5f;273;1b3;233;231;1f2
wl1;174a4f945bd0e0d0;sample_rates;a0:78:17:74:c2:5f;274;1b7;1f6;226;234;237;266;1f7;227;238;176;1a9;1b5;0;0

The key factors that determine the rate selection are the estimated transmission success probability and estimated throughput of each MCS rate. Hence, before comparing the rate selection, the averaging, success probability, and throughput calculation must match in both algorithms. As a side note, both the Minstrel-HT implementations use the Low Pass Butterworth filter to average success probabilities over time.

The following sub-sections cover an overview of the results obtained from several runs of the passive experiments, each running for 20mins.

2.1) Probability Estimation in User space vs Kernel space

The section shows the comparison of the transmission success probability between the kernel Minstrel-HT and py-minstrel-ht.

From the analysis, it is evident that the internals of the Butterworth filter in user space acts exactly the same as the kernel algorithm. However, due to floating point restriction, the kernel Minstrel-HT scales all the floating point numbers to integers which can lose precision. This can be observed as the difference between the success probabilities is sometimes up to 0.2%.

2.2) Throughput Estimation in User space vs Kernel space

Similar to the success probability calculation and discounting in py-minstrel-ht, the estimated throughput for all rates seems to match almost exactly the kernel Minstrel-HT. However, as previously stated, there are negligible errors due to the difference in floating point precision between the two Minstrel-HT algorithms.

2.3) Rate Setting in User space vs Kernel space

Ultimately, the selection of transmission rates determines the comparability of py-minstrel-ht with the kernel Minstrel-HT. In order to compare the rate setting chosen by both algorithms, I’ve developed an analysis script that goes over the saved traces and compares the “best_rates” lines. The following sub-sections show the results of some experiments.

Experiment 1

The Minstrel-HT algorithm, for MT76 chips, utilizes the 5 rates in the Multi-rate Retry (MRR) chain. The results show the number of errors in each MRR index and its relative error percentage between the kernel and user space Minstrel-HT. The MRR indices from 0 to 3 are filled by the rates with the highest throughput estimate and the MRR index 4 is filled with the maximum probability rate for robustness.

MRR Setting info:
0 {'correct_instances': 15831, 'incorrect_instances': 1, 'percent_error': 0.0063163213744315315}
1 {'correct_instances': 15818, 'incorrect_instances': 14, 'percent_error': 0.08842849924204144}
2 {'correct_instances': 15810, 'incorrect_instances': 22, 'percent_error': 0.13895907023749368}
3 {'correct_instances': 15760, 'incorrect_instances': 72, 'percent_error': 0.4547751389590703}
4 {'correct_instances': 15783, 'incorrect_instances': 49, 'percent_error': 0.309499747347145}

Furthermore, the following plot shows the total number of errors in the MRR chain when compared with the kernel algorithm.

Experiment 2

MRR Setting info:
0 {'correct_instances': 12633, 'incorrect_instances': 13, 'percent_error': 0.10279930412778744}
1 {'correct_instances': 12626, 'incorrect_instances': 20, 'percent_error': 0.15815277558121146}
2 {'correct_instances': 12628, 'incorrect_instances': 18, 'percent_error': 0.1423374980230903}
3 {'correct_instances': 12623, 'incorrect_instances': 23, 'percent_error': 0.18187569191839315}
4 {'correct_instances': 12429, 'incorrect_instances': 217, 'percent_error': 1.7159576150561442}

2.4) Remarks

After running experiments with the passive py-minstrel-ht and the active kernel Minstrel-HT, a lot of minor errors and bugs in the user space algorithm were resolved. I’ve listed some of the issues that were solved after investigating the results from the passive experiments:

• The probability calculation in py-minstrel-ht did not take into account the packet counts from the last update interval, unlike the kernel Minstrel-HT.
• During the selection of the maximum probability rate, the Minstrel-HT algorithm calculates a threshold airtime value. This value acts as the minimum airtime a maximum probability rate must have. In the kernel algorithm, it is set to 18% higher than the maximum airtime between the rates in MRR₀ and MRR₁. However, in py-minstrel-ht, this value is erroneously set to 18% higher than the minimum airtime between the top two throughput rates.
• Additionally, during the selection of the maximum probability rate, py-minstrel-ht mistakenly considers only rates that had at least 1 packet attempt in the historical counts. However, there are unused rates that derive their probability estimate based on higher rates in the same minstrel rate group.
• As mentioned earlier, the unused rates also derive their probability estimate based on higher rates in the same rate group. After investigating the high error in MRR₄, it was found that py-minstrel-ht was only updating the estimated success probability and not the estimated throughput, which has now been fixed.

In summary, the behavior of py-minstrel-ht aligns with the kernel Minstrel-HT, with observed errors likely stemming from minor precision discrepancies in the kernel space and potential inconsistencies in rate statistics at the start. For example, if MRR₁ and MRR₂ have very close expected throughput and their order is swapped in either kernel or user space Minstrel-HT, then it is extremely likely that the chosen maximum probability rate is also incorrect due to difference in the airtime threshold. Nevertheless, the Butterworth filter’s role in discounting the estimated probability average over time suggests that with longer experiment runs, the relative error in the MRR chain is expected to decrease.

It’s important to note that despite these efforts, small errors and discrepancies may still exist, and continuous evaluation and refinement of the py-minstrel-ht algorithm will be necessary to enhance its performance and accuracy further, especially with the changing kernel Minstrel-HT code base.

3) Joint Power and Rate Control Algorithm

As the power control extension in the kernel has only been properly implemented on the ath9k devices, I had to wait for the arrival of the ath9k routers as I only have access to MediaTek devices. However, due to an unexpected delay and problems in the shipping of the routers, I have mostly been working on the passive py-minstrel-ht and its analysis on MT76 chips. As such, the py-minstrel-ht hasn’t been implemented with the power control extension, however, thanks to Arne, the py-minstrel-ht has been already refactored such that it works with the new WPCA version and the updated Python-WiFi-Manager.

Nevertheless, I would like to introduce the concept of my proposed power control extension to the Minstrel-HT rate adaptation algorithm, especially the “max_tp” mode. For the max throughput mode, the implementation needs to be well thought-out for every part of the user space Minstrel-HT, not to hamper the optimal throughput

3.1) Initialisation

For the max throughput mode, initially, the py-minstrel-ht would initialize all the rates to use the maximum power level or the static power level that kernel Minstrel-HT uses.

3.2) Best Rate Selection

The set of best rates for the MRR chain could also be selected by the py-minstrel-ht as it already provides the sort_rates_by_tp function to find the best-performing MCS rates. However, the power annotation for each rate in the MRR, except the max probability rate, would be carried out by the power controller. As the max probability rate is used as the last fallback option, it could only use a static high power level.

3.3) Power Sampling and Fallback

As Minstrel-HT already probes every 20ms, in order to get the most out of power sampling, it could be only done on a small subset of sorted rates by throughput. This way the algorithm can better gather ideas about the effect of different power levels for the most used rates instead of gathering less information on more MCS rates. For example, the algorithm could only consider the top 5 throughput rates for power level sampling. However, this could easily be added as a configurable parameter in the rc_opts structure.

For each MCS rate that is considered for power sampling, the algorithm will keep track of two power levels, namely “safe” and “optimal“. The power level which gives the best transmission probability is stored in the “safe” parameter for reference. On the other hand, the “optimal” parameter will store the lowest power level which still provides performance comparable to the maximum throughput. In the beginning, the “safe” and “optimal” parameters will start at the highest power level. The power level stored in these parameters will depend on 𝛿 and 𝜎 in the following way:

where 𝛿, 𝜎 ∈ [0, 1], 𝑃𝑅(𝑋) is the success probability of rate 𝑅 at power level 𝑋, and max_prob is the highest success probability observed across the power levels. The 𝛿 constant specifies the minimum success probability that the “safe” power level needs to satisfy.

In case equation 3.1 is satisfied, then we explore a lower power level to check if it can still satisfy the “safe” success probability. All of the power levels to be sampled with an MCS rate are stored in a 𝑆_𝑇𝑋𝑃 set.

Similarly, if equation 3.2 is satisfied:

In case 3.1 is not satisfied, we increase the “safe” power level considering the tolerance factor Ψ. We allow a tolerance around the 𝛿 limit which still denotes that the performance at the power level hasn’t completely deteriorated and hence, we simply increase the safe level by Δ_𝐼 . If the 𝑃_𝑅(𝑠𝑎𝑓𝑒) falls down even beyond this tolerance factor then we simply switch the safe level to the maximum power level supported.

Similarly, in case 3.2 is no longer satisfied, we also increase the “optimal” power level considering the same tolerance factor Ψ.

3.4) Updating Rate Statistics

The rate statistics used by Minstrel-HT will be updated at every update interval, i.e. 50ms, but we could update the per power level statistics independently. In order to not influence Minstrel-HT too much, the rate control algorithm could only consider 𝑜𝑝𝑡𝑖𝑚𝑎𝑙 and 𝑠𝑎𝑓𝑒 packet counts for statistics.

As using the probe API command may not generate enough packet counts to judge the actual performance of an MCS rate at a certain power level, the per power level statistics could be updated when the curr_attempts reach a certain update threshold. However, if we fix a power sample frequency that could build up enough packet counts, then we can also use a time-based per-power level interval.

3.5) Power Sampling Frequency and Sequence

Since the joint controller would mostly be working on the subset of the best throughput rates for probing power levels, having a higher power sampling frequency in comparison to rate probing may not have an adverse effect on the throughput. For instance, we could set the power sampling frequency to every millisecond or lower for time-based per power level update. However, while power sampling, the algorithm should be able to quickly identify power levels that do not work and reduce the sampling of such power levels so as to not decrease the link throughput.

I have formulated a sequence for the power sampling such that priority is set in the following order: 𝑏𝑒𝑠𝑡_𝑟𝑎𝑡𝑒₁ > 𝑏𝑒𝑠𝑡_𝑟𝑎𝑡𝑒₂ > 𝑟𝑒𝑚𝑎𝑖𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒𝑠. As at any point in time, for a single MCS rate, the 𝑆_𝑇𝑋𝑃 set will only consist of two sample power levels: one derived from 𝑠𝑎𝑓𝑒 and the other derived from 𝑜𝑝𝑡𝑖𝑚𝑎𝑙. Hence, during the power sampling of a rate, both the power levels will be sent for sampling together.

3.6) Rate Probing

Judging from the transmit power vs throughput graph in the introductory blog, testing out whether an MCS rate works at all is best at high power levels. As Minstrel-HT already has a robust probing mechanism with three distinct sampling types, I propose not to change it in any way. However, a feature can be added where the user can also specify how high the static power level of sample rates should be. By default, it could be set to the power level that the kernel Minstrel-HT uses for sampling packets or the “safe” power level.

3.7) Configurable Parameters

4) Concluding Thoughts

The next half of the GSoC ’23 coding period will mostly be testing and extending the py-minstrel-ht with power tuning. The foundations of py-minstrel-ht rate control have been extensively tested out and the passive experiments show that the kernel and user space algorithm are comparable. If you want to know more about my project, please feel free to reach out. Thanks for reading!

Hello there ! I have already published a blog post where you can learn about what is Qaul and why currently it needs a matrix bridge ? You can read about it here : GSoC Project blog Qaul Matrix bridge

Getting Started

Initially we thought that we would run matrix bridge as a daemon process and use Go Lang to create the bridge. But, My mentor has a friend working in element, which is a matrix client, suggested us that since the world is taking on for the Rust, Matrix-SDK are actively written in Rust and there is something called as RuMa which stands for Rust Matrix. RuMa is an amazing work and thus, We have decided to do it in rust only because our Qaul has its entire backend in Rust.

Knowing the toolkits

I had read and researched through the RuMa and Matrix-SDK-Rust projects since this is now the part which we will be using for our project instead of GoLang.

Mentor suggested that I should duplicate the qaul-cli binary and tweak it in whatever way I would need to in order to work on it. Here is the main reason why we chose the qaul-cli specifically for implementing the bridge concept.

• It already has two workers set in place which will check for any activity on entire qaul network each 10ms.
• We have access to CLI which we use to interact with RPC protocol and protobuff messaging.

I was reading through the documentation of the matrix-sdk crate and what I found was a beautifully commented codes for creating the bot inside examples/ directory. They were really helpful for me to initiate the coding part.

Planning the bridge

Version 0

[On Matrix]

1. Create a bot account for Qaul and specify a server to work on.
2. Invite the bot to the testing matrix room.

[On Qaul]

3. Create a binary copy of qaul-cli
4. Code the logic to login our bot into the matrix room as soon as the qaul-cli binary is running.
5. Also Code a basic testing functionality (For Eg : Call it !ping command)

[On Matrix]

6. Login with our personal account [@harshil1] and send a message with !ping in the room.
7. In response we should receive all the nodes connected to the network.

Version 1

Instead of just an echo as response, We should pick the messages from both the ends. Send “Hi” from qaul and it should first sense in matrix without our personal human activity that there is some event triggered in qaul. Once event is detected, the message should show up into the matrix room.

Next we can reverse engineer the above feature and do the same in qaul by sending a message in matrix room.

Version 2

This just follows the Version 1 functionality wise but this should be implemented for 1-on-1 direct messages. In matrix and qaul both, private DMs are nothing but a group with only two members. We need to create a use case where we can send the message in groups by inviting a bot and the bot invite the user on other application and rest remains same.

Version 2+

After the above completions, We can think of double puppeting the bot so now our bot is not just qaul-bridge but a real username from the qaul node.

Progress till Mid Evaluation

We have built end to end Matrix to Qaul bridge working as expected for Version 1 and will be achieving the Version 2 within next week. Speaking in-depth about version 2, We already have a functionality to check if there is any new group requiring to connect with a matrix user and accordingly the Qaul-Bot opens Matrix room and invites wanter Matrix user. Then Matrix user is able to send messages into the qaul group. We are supposed to close the part where a message goes from Qaul into the Matrix.

Resources

If you are interested in learning our code for the bridge, I am writing a book where I have explained an approach to integrate the bridge in qaul world and it has organized chapters with snippets. For more, You can also refer to my raised Pull Request and It will give more clear insights.

Link to Book : GSoC 2.0 Journey Book – By Harshil Jani

Link to Pull Request : qaul/qaul.net/pull/563

Hi all!

Previous post: https://blog.freifunk.net/2023/05/27/gsoc23-automation-tools-for-libremesh-firmware-build-and-monitoring/

During this period of writing I’ve been reading up on these projects:

OpenWrt Buildroot

The main openwrt project that allows the greatest level of customization of configurations and packages, this allows to compile directly from source, and among the main features, to build firmware images for all devices, called ‘profiles’ of a given target/subtarget, or for a sub-list of these profiles. It is also the slower solution to build firmware images, compared to the OpenWrt ImageBuilder. It also allows building an OpenWrt ImageBuilder and an OpenWrt SDK.

https://openwrt.org/docs/techref/buildroot

https://openwrt.org/docs/guide-developer/start#using_the_toolchain

https://openwrt.org/docs/guide-user/virtualization/obtain.firmware.docker

Openwrt ImageBuilder (docker)

This tool allows building firmware images from precompiled packages and is also packaged as a docker image.

https://openwrt.org/docs/guide-user/additional-software/imagebuilder

https://github.com/openwrt/docker

https://hub.docker.com/u/openwrt/

OpenWrt SDK (docker)

This tool allows individual packages to be built from source and is also packaged as a docker image.

https://openwrt.org/docs/guide-developer/obtain.firmware.sdk

https://github.com/openwrt/docker

https://hub.docker.com/u/openwrt/

OpenWrt Firmware Selector

This is a GUI for selecting OpenWrt firmware images from the official repository https://downloads.openwrt.org/, and which acts as a client for the Attendedsysupgrade Server, for building custom firmware images.

This works by scanning all device profiles and building overview json files for each version from them. From this information the selection interface is then constructed.

https://gitlab.com/mwarning/firmware-selector-openwrt-org

Attendedsysupgrade Server

This is a server that accepts requests, from different clients, to build firmware images with custom lists of packages and files, these requests are then delivered to an ImageBuilder that will perform the build, the server at this point returns the sysupgrade image produced.

It is packaged as a Python project and as a docker image.

https://openwrt.org/docs/guide-user/installation/attended.sysupgrade?s[]=asu&s[]=attendedsysupgrade

https://github.com/openwrt/asu

Ansible roles to build LibreMesh

For each of these components I wrote an ansible role

• openwrt_buildroot
• openwrt_imagebuilder(_docker)
• openwrt_sdk(_docker)
• openwrt_firmware_selector
• openwrt_asu

Each role basically goes over the steps necessary to:

• prepare the system
• install the component
• configure it
• use it

These roles are based on a file called ‘recipe’ that defines the devices for which to build.

This file is provided to the ansible playbook as a variable, and is by default dependent on which version of libremesh you intend to use and which version of openwrt.

These ‘mandatory’ variables are used to expand the list of known configurations by going for the list of supported devices and known changes needed to install libremesh on them, this information is written to the ‘target’ folders in the collection repository in turn selected based on the chosen versions of libremesh and openwrt.

The mechanism is that of the inclusion of additional variable files, containing partial information. The recipe file provided has the final say by allowing any information gathered in the other files to be overwritten. This part of the code for collecting necessary information for each device still needs to be improved.

Creating device specific configurations

The creation of single device specific configurations goes over these steps:

• creation of devices as ansible hosts
• creation of a definition file for each host
• from the latter generation of a libremesh configuration file `lime-macaddress` which the way libremesh is set up ranks in the hierarchy of main libremesh configuration files found in `/etc/config` as second:
  • 0. lime-autogen: configurations actually applied, not editable
  • 1. lime-node: manual configurations at the device
  • 2. lime-<macaddress>: provisioned configurations
  • 3. lime-community: community configurations
  • 4. lime-defaults: defaults suggested by libremesh
• vpn server upgrade

Adding devices to monitoring

The setup of the monitoring system goes over these steps:

• customizing configurations
• customizing scrape_options
• definition of labels
• definition of monitoring targets
• definition of probing targets
• definition of contact channels: email, telegram
• definition of datasources and dashboards
• generation of target lists
• installation of components
• placement of components in a webserver under dns names

I have added a minimal amount of documentation and published a template for using the project at:

https://gitlab.com/a-gave/libremesh-ansible-playbooks/

The product code for the entire roles collection is available at:

https://gitlab.com/a-gave/libremesh-ansible-collection/

All roles defined in this collection to build LibreMesh and to setup the monitoring system have been written and tested on debian 11 and 12.

The product code for the role corresponding to the single OpenWrt Buildroot is available at:

https://gitlab.com/a-gave/ansible_openwrt_buildroot

Example diagrams

Here is a summary of three main playbooks/roles:

Here are listed some scenarios of possibles use cases:

Example 1: simple build of libremesh for list of devices grouped by targets

Example 2: build of libremesh, and build of firmware images to meet specific devices needs

Example 3: Like the previous but with the generation of a list of targets to monitor with prometheus

Example 4: Introduce a vpn (wireguard) to monitor also devices that aren’t reachables in the same lan of hosts that do monitoring

Hello again folks 👋
If this is your first time here, then consider reading Part 1 of this blog, GSoC’23: Implementation of Web Interface of Retroshare.

Now, let’s continue.
The first phase of GSoC has been amazing, up until now. I was able to implement a lot of features, fixed bugs and issues, faced difficulties, got stuck and learned a lot. I am going to discuss my journey so far, so buckle up your seat belts.

Progress on the WebUI

I have been able to improve the Web Interface of Retroshare very much in terms of usability and a better UI. All thanks to my mentors and the community members.

File Search feature

During the start of the Coding Period, I was already working on implementing the File Search feature, so naturally It was completed in the first week of GSoC itself. This was a very difficult feature to implement as there were only a few options on how to do it correctly.

Previously, the file search results data was sent as a response to the request made to the endpoint /rsFiles/turtleSearchRequest. So, I thought of using Event Source, as It would create a persistent connection to get the data in stream. But it needed its own auth headers to be used properly. Also, there was some issue in the format of response coming from the backend, which was causing webui to crash. I had a discussion upon this with the mentors in the issue I created on GitHub.

Later the mentors discussed and decided that It would be better to make the results of file search as a stream of events which would be then captured at the frontend side and displayed. As there was already a way to handle events, It was not much of a problem. But the real issue was to update them in the UI after capturing them, as I was not sure how to do that in Mithril.js, and this was a perfect opportunity to learn something new.

And after some research, I found out that Proxy in JavaScript can be used to check for any change in the objects made using the handler defined in eventQueue. And, then I could manually trigger a re-render using `m.redraw()` in mithril.js to update the results in the UI.

// Custom Proxy code
const createArrayProxy = (arr, onChange) => {
  return new Proxy(arr, {
    set: (target, property, value, reciever) => {
      const success = Reflect.set(target, property, value, reciever);
      if (success && onChange) {
        onChange();
      }
      return success;
    },
  });
};

const createProxy = (obj, onChange) => {
  return new Proxy(obj, {
    get: (target, property, reciever) => {
      const value = Reflect.get(target, property, reciever);
      return typeof value === 'object' && value !== null
        ? Array.isArray(value)
          ? createArrayProxy(value, onChange)
          : createProxy(value, onChange)
        : value;
    },
    set: (target, property, value, reciever) => {
      const success = Reflect.set(target, property, value, reciever);
      if (success && onChange) {
        onChange();
      }
      return success;
    },
  });
};

const proxyObj = createProxy({}, () => {
  m.redraw();
});

So, after some head banging and googling, I was able to implement this feature. Furthermore, I improved the small issues in next iterations. You can see the merged PR here.

Config Mail

After that, I started working on the config section, in which I implemented the feature to create and manage custom tags for mails in the mail config. The PR I raised is here.

Config Network

Then, I implemented some missing features in existing section of Config section. In the Network config, some options weren’t working. So, after discussing with my mentor, I implemented the Hidden Network Configuration for TOR/I2P Socks Proxy here.

Config Files

I implemented the missing options as well as fixed the options which were not working in the config files section also for this, I implemented the getQueueSize() function in libretroshare.

By this time, I was also getting a hang of How things were working internally in the libretroshare. And, my mentor Cyril Soler motivated me and supported me to start writing code in libretroshare for Doxygen to generate the JSON API for the required endpoint. I raised three PRs in total, where I also implemented an endpoint in the actual C++ code.

Honestly, this was more satisfying to be able to do this rather than writing code for the webui. You can view all of my libretroshare PRs here.

Fix UI of whole Interface

This was a very big task but didn’t take that long, I did it gradually in small bits. Improved the whole UI of the web interface, and it looks much better now. See all the changes made in this PR here.
I would not include any screenshots here as It would be too much.

Fix working and UI of mail Composer

Of all the features I have implemented so far, none of this were this confusing where I was fighting with the code. I was stuck trying this on my own for two days, since the code I wrote was fine on its own. But, there was something which was causing an issue, and I was not able to understand what it was. So finally, I asked for help from a community member, M. Saud. He is also the code reviewer of all of my PRs, since as far as I know he is the only JS dev in RSWebUI team.

He found the issue and told me some ways It could be done. Those seemed too complex to me, I read some mithril.js docs and tried to figure out ways to do it. And In the end, I found a very easy way to do it. You can read the discussion here on GitHub.

This PR is related to the mail reply as well, so this is still in the Draft version, And I am currently working on it to implement the features. You can see it here. Other than this, I have been doing some minor fixes and refactoring in the whole webui along with completing the major goals in the timeline.

So, this is the progress made in the WebUI until now. And, I have learned and still learning many new things while working on it.

What’s Next?

First, I will finish the PR#80 where I have to implement the Reply feature in the Mail Section, and then will work on the next set of goals as mentioned in the timeline which are :

• Forums Section
  • Implement the remaining features from the Qt app in the forum section.
  • Implement a better layout of the forum and posts view, comment and reply feature etc.
• Boards and Channel Section
  • Implement the remaining features and make them more user-friendly and easy to use.
• Implement a feature for Configuring and Visualizing own shared files with features such as managing shared directories, user permissions etc.

Apart from these goals, I have also discussed with my mentor to work on more important issues and necessary features for the upcoming release of the webui.

Now, I will see you in the next and final blog of this GSoC series.

Thanks for reading and have a great day 🙂

GSoC ’23 OpenWrt PPA Part 2: GitLab packaging

Previous post: https://blog.freifunk.net/2023/05/13/gsoc-23-documenting-the-openwrt-compilation-process-to-set-up-a-ppa

When working on this project we’ve encountered the networking troubles: there is no networking on OpenBuildService builds. This would require packaging everything into tarballs and copying these over to the source files before building, which would not be user-friendly at all. (The purpose of this project is to make packaging easier, not harder!) This seemed impractical.

GitLab’s packaging system solves this issue cleanly. It provides a full CI interface, and allows for networking as well. It works cross-architecture too! The key thing to resolve is to get OpenWrt tools in, as these do not get official status like Debian or Ubuntu, so they need to be implemented in a way that follows GitLab conventions.

Zoobab helped a lot by producing an automated build system: https://gitlab.com/zoobab/openwrtsdkbuild. This takes in a git repository and exports binaries as artifacts. This is really useful as this is essentially the input and output of this process: some code to run on OpenWrt becomes an architecture-specific binary.

I have experimented with different ways to store these binaries (OpenWrt’s opkg expects a very specific file structure, not unlike most other package managers). Over time, I was able to port this into a repository structure: https://gitlab.com/ndren/openwrtsdkbuild/-/blob/7d92349d53befe8e2cc5ce1a89919b68050b9ce2/.gitlab-ci.yml#L50. This uses GitLab’s feature where it integrates an internal package repository with HTTP requests to read and write data. In this case it was very useful so that there is long-term storage for the package binaries. (This uses GitLab’s generic package repositories.)

It would be very useful if the CI system could verify that the binaries actually work! I’ve looked into OpenWrt’s docker repository and I found they provide a root filesystem (rootfs) exactly made for CI testing for different architectures: https://gitlab.com/ndren/openwrtsdkbuild/-/blob/7d92349d53befe8e2cc5ce1a89919b68050b9ce2/.gitlab-ci.yml#L72. With a little bit of tweaking to prepare the SDK, it worked great and meant that we could keep up with any upstream updates to the SDK or demo rootfs.

We have now a dropdown menu where we can choose the SDK and the git repo. We have to check whether a Gitlab Page could provide a better UI (with or without login page in front).

Future plans: Because the main CI system is setup this way, it easily allows for improvements. For example, exploring things like adding support for new architectures takes editing a line of YML and testing. Therefore I can start looking at self-hosted GitLab, and testing more architectures.

TODO list:

1. Try github Actions (see if it is portable)

2. Problems with gitlab package repo URLs (requires authenticated downloads which is not ideal)

3. Problem with the SDK that outputs lots of warnings about missing packages (can this be fixed without breaking the build?)

4. Check whether other OpenWRT SDK images are usable, so the build can be done in different environments.

Finally, a message from our hello world package, running in CI in Docker in Docker in aarch64 in amd64:

Hello world!
MyClass::MyClass()
MyClass::printMessage()
This is my message to print

Introduction

Hello everyone!

I’m Prashiddha, a former GSoC contributor with Freifunk in 2022 during which I extended the Py-Minstrel-HT rate control to make it further comparable with its kernel counterpart, allowing for better experimentation between the rate controls in user space and kernel space. If you would like to know more about WiFi rate control and my previous project, please feel free to start with the introduction blog from 2022.

For GSoC ’23, I’ll be working on the research and development of a resource allocation algorithm that can select the optimum transmission rates in conjunction with the optimum power level. The joint power and rate control algorithm is intended to work on OpenWRT routers, capable of making resource allocation decisions for each station connected with it.

Overview of Joint Power and Rate Control

A rate control algorithm, such as Minstrel-HT, determines the best transmission rates that are promising in providing the maximum throughput for the given link condition. These algorithms usually assign a high static power level which could potentially cause interference, especially in a highly dense network. It is already evident that, for a transmission rate, even though a higher transmission power implies a higher signal-to-noise ratio (SNR), it doesn’t necessarily mean higher throughput. Hence, it could be best to use the lowest transmission-power level that is still capable of providing the optimum throughput. As such, this could allow for better management of interference along with an increase in spatial reuse.

The graph presented in a dissertation from Prof. Thomas Hühn shows the relation between the power level and measured throughput where the throughput stops increasing after a certain power level.

WiFi Resource Allocation in Userspace

As part of the SupraCoNeX research, the development of WiFi Parameter Control API (WPCA) for OpenWrt access points, has enabled WiFi resource allocation from the user space. The API exposes relevant information from the mac80211 kernel subsystem, such as supported Modulation Coding Scheme (MCS) rates and packet counts (ACKs), that could be required by resource allocation algorithms to make decisions. Previously, the WPCA API could be used to only set the MCS rates for wireless transmission, however, with the recent extension, it allows the MCS rates to be set in conjunction with power levels. Consequently, it is now possible to develop a joint rate and power controller in user space.

In order to further facilitate resource allocation, a Python-based package called “WiFi-Manager” has also been developed which utilizes the minstrel-rcd to concurrently operate on multiple access points and parse the exposed kernel information from the API. The package is implemented such that the resource allocation algorithms can be executed through it while also providing them with the parsed kernel information for decision-making.

Extending Py-Minstrel-HT with power control

Since a rate control algorithm in user space already exists, namely “py-minstrel-ht”, I plan on extending the user space Minstrel-HT algorithm with an additional capability for transmit power tuning, also making it convenient to test the effects of power tuning on a rate control algorithm. The main idea behind the joint controller is to let Minstrel-HT decide the set of the best rates while a power tuning module tweaks the power levels to an optimal value. With the addition of power control, the user space Minstrel-HT can be executed with different power settings to achieve various goals. For instance, three different power modes could be realized: fixed power, maximum throughput, and power ceiling.

The fixed power and power ceiling modes are straightforward to understand and implement. The fixed power mode, as the name suggests, sets the power level of all the transmission rates to the specified value. Similarly, the power ceiling mode can be used to specify the maximum power level that can be used for wireless transmission. However, the maximum throughput mode is a bit complicated as the wireless channel is highly dynamic in nature and the controller needs to accurately assess the quality of the link in real-time. Hence, the implementation needs to be well thought-out for every part of the user space Minstrel-HT so as to not hamper the optimal throughput. As the addition of power control adds another depth to the sampling parameter, the set of possible sampling candidates will grow tremendously. However, as Minstrel-HT already probes with a frequency of 50 Hz or 20 ms, sampling too much can greatly degrade the overall performance of the link.

Deliverables

• Extension of py-minstrel-ht with a power controller with complete documentation and execution guide.
• Ready to run demo scripts to showcase the potential of the joint rate and power control.
• Evaluation of the joint controller by comparing it at different modes and with different rate controls.

What’s Next?

At the beginning of the GSoC ’23 coding period, I’ll start by modifying the WiFi-Manager package such that the rate statistics dictionary is properly structured to relay successes and attempts statistics per power level per rate. Consequently, I will modify the Py-Minstrel-HT to accommodate the change in the rate statistics structure. This would allow algorithms to better assess the performance of an MCS rate at different power levels. Furthermore, I will extend the rate setting and probing functions from Py-Minstrel-HT to enable power annotation for a desired rate.

Initially, the power ceiling and fixed power modes will be implemented in order to make testing out the power tuning easier. For this, the Py-Minstrel-HT will also be extended to parse the power setting specified by the user in the rc_opts dictionary. If possible, the following questions could also be investigated before the implementation of the max throughput mode:

• Is the power setting completely static with kernel Minstrel-HT? Does the driver play any role in independent power adjustment?
• In general, is the throughput vs tx-power graph strictly non-decreasing? Is it possible that an MCS rate works at power level 𝑇𝑋𝑃₁ but not at 𝑇𝑋𝑃₂ where 𝑇𝑋𝑃₁ < 𝑇𝑋𝑃₂?
• In a Minstrel-HT rate group, let 𝑅₁ and 𝑅₂ be two rates where 𝑅₂ is a higher rate than 𝑅₁. If 𝑅₂ works at 𝑇𝑃₁, does it imply that 𝑅₂ also works at 𝑇𝑃₁?

With this, I’d like to conclude the first blog on the joint power and rate controller in user space. Thanks for reading! Please feel free to reach out and connect with me 🙂

Introduction

Hi everyone! I’m samlo. I’m a fullstack webdev that live in a rural area in Italy and dedicates part of his time to build an maintain a self-managed community network based on LibreMesh https://antennine.noblogs.org/.

For GSoC’23, I’ll be working on setup a set of ansible playbooks and roles to do common network administration tasks useful for a tech team of a community network based on LibreMesh.

This first blog post intends to cover details on the necessary background to understand the project and its implementation.

What

As state the site https://libremesh.org/, LibreMesh is a modular framework for creating OpenWrt-based firmwares for wireless mesh nodes.

It’s a list of packages (lime-packages) with support for various mesh protocols, to installing and configuring them properly, and offer to the end user a dedicated web interface (lime-app).

It has support for potentially every OpenWrt supported devices, following the documentation you find all the information to build the firmware and configure the main files and start using it.

It also has a list of network configurations used by different communities (network-profiles) that provide the information about how to configure your firmware (installing packages) and your network (e.g. editing main configuration files), or both, to join the community mesh.

Motivations

Libremesh is a set of packages you can include as feeds – via sources or precompiled packages – in a OpenWrt build system and then select those of your choice, but it’s possible to overwrite openwrt default configs only manually, and make backup of produced configurations files per build.

In this scenario is necessary to save configurations and ways to reproduce the same firmware image.

Instead of start writing a list of bash scripts to handle just our community needs I’m interested in exploring the possibility of using a configuration management and automation tool as Ansible https://docs.ansible.com.

This would lead to simplify common needs, in particular:

– automate the build of firmwares for groups of devices with specific configurations, packages, libremesh and openwrt versions.

– build test firmwares by versioning experiments

– build on localhost or on a remote machine

– manage configurations (monitoring, vpn, ssl certificates) in the same system that also build firmwares

– automate the insertion of information that may be synced between networking devices and servers

– share configurations in a set meaningful and reproducible for other people inside and outside the local community network.

An issue

Libremesh doesn’t handle a system to build consistently for every supported devices or to patch openwrt to meet the needs of particular targets or devices.

So every community should understand how to build for devices they are using.

One inspiration for this came from the project Gluon https://gluon.readthedocs.io/ that include a system to keep traces of specific packages related to openwrt targets, subtargets and device.

https://github.com/freifunk-gluon/gluon/blob/master/targets

Deliverables of the project

– Have an ansible set of playbooks and roles to build openwrt firmwares

– Have an ansible set of playbooks and roles to build libremesh firmwares

– Have an ansible role to build libremesh firmwares depending on libremesh version, openwrt version, libremesh default packages, libremesh target’s or device’s specific packages, libremesh community packages, libremesh community set of packages linked to specific list of devices.

– Have an ansible set of playbooks and roles to setup a monitoring/probing/alerting/metric-visualizer system

Concluding Thoughts

In this period of design and blueprint of the project, before starting coding I thinked a lot at use cases, who could want to use it, and to simplify contributions to keep updated the code in the future.

I look forward to publish on https://galaxy.ansible.com/ two collection of roles (openwrt and libremesh) and make available via git repository the set of playbooks to use the roles above.

I’ll update you in July.

Hello folks 👋
I am Sumit Kumar Soni, a frontend developer who loves Linux, design and doing open source contributions. Being a part of GSoC fills me with excitement. Furthermore, I hope to learn, expand my knowledge and do some impactful contributions in implementation of webui of retroshare throughout this summer.

Project Context

RetroShare provides a decentralized, encrypted connection with maximum security between nodes where they can chat, share files, mail, etc. It uses GXS (Generic eXchange System) that provides asynchronous distribution, authentication, privacy, security of generic data. RetroShare is a C++ software program that comprises a headless lib called libretroshare. So, this lib helps in making a headless server (retroshare-service), a standalone app with a user interface built using Qt, an android client and more.

Moreover, a web interface has started being developed that allows users to control the headless server from their web browsers. The web interface uses an automatically generated JSON API. And, It includes all necessary functions to send and receive data from the software, communicating with libretroshare.

Goals and Deliverables

The previous GSoC contributors have accomplished astonishing work on the WebUI, yet there remain many functionalities to implement and bugs to fix. This summer, I intend to implement some of the most important features and make the design more appealing and user-friendly as well.

The main deliverables during the GSoC period will be :

• Config Section
  • Implement the panels which have not yet been implemented from the Qt application.
  • Enhance the already existing panels and fix the existing inconsistencies.
• Mail Section
  • Implement the Reply, Reply All and Forward feature in the mail view.
  • Fix the Compose Mail popup and make it usable.
• Forums Section
  • Implement the remaining features from the Qt app in the forum section.
  • Implement a better layout of the forum and posts view, comment and reply feature etc.
• Boards and Channel Section
  • Implement the remaining features and make them more user-friendly and easy to use.
• Implement a feature for Configuring and Visualizing own shared files with features such as managing shared directories, user permissions etc.

Previous Contributions

During and before the community bonding period, I contributed and added features for the implementation of webui of retroshare. And, The most recent contribution I did was the implementation of file search in Files section, which wouldn’t have been possible without the help of my mentor Cyril Soler.

I have also implemented various other features such as Attachment view for Mail section to view all the mail attachments in one place, improving the sidebar etc. Likewise, I have also tried to reduce the overall size of the WebUI by minifying the files. The PRs I raised until now can be seen here.

What’s Next?

Currently, The community bonding period is going on, and I have made myself a bit familiar with my mentors and some fellow members. The mentors are really supportive, and I couldn’t have made it to GSoC without my mentor Cyril Soler and one more amazing person, Defnax. In addition, I actively participate in weekly meetings where I report my progress, discuss different approaches.

So, for the first phase which is until the midterm evaluation I have planned to work towards the implementation of the first two features listed in the Goals and Deliverables section of this blog. It has been an amazing learning experience, and I am looking forward to achieving more amazing things this summer with Freifunk and Retroshare!

Thanks for reading and Have a great day 😃