GSoC ’23: Joint Power and Rate Control in Userspace for Freifunk OpenWrt Mesh & Access Networks

Introduction

Hello everyone!

I’m Prashiddha, a former GSoC contributor with Freifunk in 2022 during which I extended the Py-Minstrel-HT rate control to make it further comparable with its kernel counterpart, allowing for better experimentation between the rate controls in user space and kernel space. If you would like to know more about WiFi rate control and my previous project, please feel free to start with the introduction blog from 2022.

For GSoC ’23, I’ll be working on the research and development of a resource allocation algorithm that can select the optimum transmission rates in conjunction with the optimum power level. The joint power and rate control algorithm is intended to work on OpenWRT routers, capable of making resource allocation decisions for each station connected with it.

Overview of Joint Power and Rate Control

A rate control algorithm, such as Minstrel-HT, determines the best transmission rates that are promising in providing the maximum throughput for the given link condition. These algorithms usually assign a high static power level which could potentially cause interference, especially in a highly dense network. It is already evident that, for a transmission rate, even though a higher transmission power implies a higher signal-to-noise ratio (SNR), it doesn’t necessarily mean higher throughput. Hence, it could be best to use the lowest transmission-power level that is still capable of providing the optimum throughput. As such, this could allow for better management of interference along with an increase in spatial reuse.

The graph presented in a dissertation from Prof. Thomas Hühn shows the relation between the power level and measured throughput where the throughput stops increasing after a certain power level.

WiFi Resource Allocation in Userspace

As part of the SupraCoNeX research, the development of Open-source Resource Control API (ORCA) for OpenWrt access points, has enabled WiFi resource allocation from the user space. The API exposes relevant information from the mac80211 kernel subsystem, such as supported Modulation Coding Scheme (MCS) rates and packet counts (ACKs), that could be required by resource allocation algorithms to make decisions. Previously, the ORCA API could be used to only set the MCS rates for wireless transmission, however, with the recent extension, it allows the MCS rates to be set in conjunction with power levels. Consequently, it is now possible to develop a joint rate and power controller in user space.

In order to further facilitate resource allocation, a Python-based package called “Rateman” has also been developed which utilizes the minstrel-rcd to concurrently operate on multiple access points and parse the exposed kernel information from the API. The package is implemented such that the resource allocation algorithms can be executed through it while also providing them with the parsed kernel information for decision-making.

Extending Py-Minstrel-HT with power control

Since a rate control algorithm in user space already exists, namely “py-minstrel-ht”, I plan on extending the user space Minstrel-HT algorithm with an additional capability for transmit power tuning, also making it convenient to test the effects of power tuning on a rate control algorithm. The main idea behind the joint controller is to let Minstrel-HT decide the set of the best rates while a power tuning module tweaks the power levels to an optimal value. With the addition of power control, the user space Minstrel-HT can be executed with different power settings to achieve various goals. For instance, three different power modes could be realized: fixed power, maximum throughput, and power ceiling.

The fixed power and power ceiling modes are straightforward to understand and implement. The fixed power mode, as the name suggests, sets the power level of all the transmission rates to the specified value. Similarly, the power ceiling mode can be used to specify the maximum power level that can be used for wireless transmission. However, the maximum throughput mode is a bit complicated as the wireless channel is highly dynamic in nature and the controller needs to accurately assess the quality of the link in real-time. Hence, the implementation needs to be well thought-out for every part of the user space Minstrel-HT so as to not hamper the optimal throughput. As the addition of power control adds another depth to the sampling parameter, the set of possible sampling candidates will grow tremendously. However, as Minstrel-HT already probes with a frequency of 50 Hz or 20 ms, sampling too much can greatly degrade the overall performance of the link.

Deliverables

  • Extension of py-minstrel-ht with a power controller with complete documentation and execution guide.
  • Ready to run demo scripts to showcase the potential of the joint rate and power control.
  • Evaluation of the joint controller by comparing it at different modes and with different rate controls.

What’s Next?

At the beginning of the GSoC ’23 coding period, I’ll start by modifying the Rateman package such that the rate statistics dictionary is properly structured to relay successes and attempts statistics per power level per rate. Consequently, I will modify the Py-Minstrel-HT to accommodate the change in the rate statistics structure. This would allow algorithms to better assess the performance of an MCS rate at different power levels. Furthermore, I will extend the rate setting and probing functions from Py-Minstrel-HT to enable power annotation for a desired rate.

Initially, the power ceiling and fixed power modes will be implemented in order to make testing out the power tuning easier. For this, the Py-Minstrel-HT will also be extended to parse the power setting specified by the user in the rc_opts dictionary. If possible, the following questions could also be investigated before the implementation of the max throughput mode:

  • Is the power setting completely static with kernel Minstrel-HT? Does the driver play any role in independent power adjustment?
  • In general, is the throughput vs tx-power graph strictly non-decreasing? Is it possible that an MCS rate works at power level 𝑇𝑋𝑃1 but not at 𝑇𝑋𝑃2 where 𝑇𝑋𝑃1 < 𝑇𝑋𝑃2?
  • In a Minstrel-HT rate group, let 𝑅1 and 𝑅2 be two rates where 𝑅2 is a higher rate than 𝑅1. If 𝑅2 works at 𝑇𝑃1, does it imply that 𝑅2 also works at 𝑇𝑃1?

With this, I’d like to conclude the first blog on the joint power and rate controller in user space. Thanks for reading! Please feel free to reach out and connect with me 🙂

GSoC’23 : Automation tools for LibreMesh firmware build and monitoring

Introduction

Hi everyone! I’m samlo. I’m a fullstack webdev that live in a rural area in Italy and dedicates part of his time to build an maintain a self-managed community network based on LibreMesh https://antennine.noblogs.org/.

For GSoC’23, I’ll be working on setup a set of ansible playbooks and roles to do common network administration tasks useful for a tech team of a community network based on LibreMesh.

This first blog post intends to cover details on the necessary background to understand the project and its implementation.

What

As state the site https://libremesh.org/, LibreMesh is a modular framework for creating OpenWrt-based firmwares for wireless mesh nodes.

It’s a list of packages (lime-packages) with support for various mesh protocols, to installing and configuring them properly, and offer to the end user a dedicated web interface (lime-app).

It has support for potentially every OpenWrt supported devices, following the documentation you find all the information to build the firmware and configure the main files and start using it.

It also has a list of network configurations used by different communities (network-profiles) that provide the information about how to configure your firmware (installing packages) and your network (e.g. editing main configuration files), or both, to join the community mesh.

Motivations

Libremesh is a set of packages you can include as feeds – via sources or precompiled packages – in a OpenWrt build system and then select those of your choice, but it’s possible to overwrite openwrt default configs only manually, and make backup of produced configurations files per build.

In this scenario is necessary to save configurations and ways to reproduce the same firmware image.

Instead of start writing a list of bash scripts to handle just our community needs I’m interested in exploring the possibility of using a configuration management and automation tool as Ansible https://docs.ansible.com.

This would lead to simplify common needs, in particular:

– automate the build of firmwares for groups of devices with specific configurations, packages, libremesh and openwrt versions.

– build test firmwares by versioning experiments

– build on localhost or on a remote machine

– manage configurations (monitoring, vpn, ssl certificates) in the same system that also build firmwares

– automate the insertion of information that may be synced between networking devices and servers

– share configurations in a set meaningful and reproducible for other people inside and outside the local community network.

An issue

Libremesh doesn’t handle a system to build consistently for every supported devices or to patch openwrt to meet the needs of particular targets or devices.

So every community should understand how to build for devices they are using.

One inspiration for this came from the project Gluon https://gluon.readthedocs.io/ that include a system to keep traces of specific packages related to openwrt targets, subtargets and device.

https://github.com/freifunk-gluon/gluon/blob/master/targets

Deliverables of the project

– Have an ansible set of playbooks and roles to build openwrt firmwares

– Have an ansible set of playbooks and roles to build libremesh firmwares

– Have an ansible role to build libremesh firmwares depending on libremesh version, openwrt version, libremesh default packages, libremesh target’s or device’s specific packages, libremesh community packages, libremesh community set of packages linked to specific list of devices.

– Have an ansible set of playbooks and roles to setup a monitoring/probing/alerting/metric-visualizer system

Concluding Thoughts

In this period of design and blueprint of the project, before starting coding I thinked a lot at use cases, who could want to use it, and to simplify contributions to keep updated the code in the future.

I look forward to publish on https://galaxy.ansible.com/ two collection of roles (openwrt and libremesh) and make available via git repository the set of playbooks to use the roles above.

I’ll update you in July.

GSoC’23 : LuCI Migrate to JavaScript-Based Framework

Project Details

LuCI is an open-source framework that is widely used to build web interfaces for embedded devices such as WiFi routers. In the CBI based old system, pages were rendered on the router and delivered as HTML to the browser, which causes a higher load on the embedded devices. This makes the system less efficient and can lead to performance issues.

To facilitate this migration, LuCI provides LuCI-JavaScript API that will be used to build web interfaces that can be rendered in the browser. Additionally, data will be provided via RPCD and UBUS. The project will involve writing new RPCD services to provide data to the client side that was formerly used directly on the router.

Project Goals

The migration of LuCI to a JavaScript-based framework will bring numerous advantages to the OpenWrt community and other users of OpenWrt-based devices. One of the primary benefits is enhanced performance and reduced load on embedded devices, such as WiFi routers. By shifting the rendering of pages to the client-side using JavaScript, instead of on the router, the workload on the router will be decreased, resulting in a better user experience, particularly for users with lower-specification routers.

Another benefit of the new system is increased flexibility for developers. The utilization of a client-side JavaScript framework provides developers with more options for customization and extension of the LuCI web interface in the future. It also establishes a standardized approach for developers to interact with the router’s services, retrieve or set configuration data, and facilitate the development and maintenance of LuCI-based applications.

Community networks, which often rely on lower-specification devices, can greatly benefit from these improvements. The improved performance and reduced load on devices will make it easier for community networks to manage and maintain their networks using LuCI-based tools.

In summary, the migration of LuCI to a JavaScript-based framework will bring significant benefits to the community and users of OpenWrt-based devices. These benefits include improved performance, increased flexibility for developers, and potentially easier management of LuCI-based applications for community networks.

Project Progress

I successfully migrated luci-app-uhttpd to JavaScript, gaining valuable experience and insights from the process which will help to migrate more advanced applications. It improved performance, enhanced user experience, and provided with greater flexibility as a developer. I’m excited to continue contributing to the growth of LuCI and further advancing OpenWrt.

I am currently working on the migration of luci-app-olsr to a JavaScript-based framework. It has been an engaging and exciting experience so far. By leveraging JavaScript, I aim to enhance the performance, usability, and customization options of the web interface for olsrd. I am excited to contribute to the improvement of this essential tool for mesh routing and network management on OpenWrt-based devices.

Community Bonding Period

During the GSoC community bonding period, I have had an incredible learning experience. I have been fortunate to have constant communication and guidance from my mentor, Andreas Bräu, who has been exceptionally supportive throughout the process. Whenever I face challenges or got stuck, my mentor is there to provide valuable insights and assistance. Additionally, this journey has allowed me to become more familiar with the OpenWrt and Freifunk communities, providing me with a broader understanding of the ecosystem and related technologies. The community bonding period has been instrumental in preparing me for the successful migration of future applications and has fostered valuable connections within the community.

GSoC’23 : Implementation of Web Interface of Retroshare

Hello folks 👋
I am Sumit Kumar Soni, a frontend developer who loves Linux, design and doing open source contributions. Being a part of GSoC fills me with excitement. Furthermore, I hope to learn, expand my knowledge and do some impactful contributions in implementation of webui of retroshare throughout this summer.

Project Context

RetroShare provides a decentralized, encrypted connection with maximum security between nodes where they can chat, share files, mail, etc. It uses GXS (Generic eXchange System) that provides asynchronous distribution, authentication, privacy, security of generic data. RetroShare is a C++ software program that comprises a headless lib called libretroshare. So, this lib helps in making a headless server (retroshare-service), a standalone app with a user interface built using Qt, an android client and more.

Moreover, a web interface has started being developed that allows users to control the headless server from their web browsers. The web interface uses an automatically generated JSON API. And, It includes all necessary functions to send and receive data from the software, communicating with libretroshare.

Goals and Deliverables

This is the homepage of Retroshare’s web interface.

The previous GSoC contributors have accomplished astonishing work on the WebUI, yet there remain many functionalities to implement and bugs to fix. This summer, I intend to implement some of the most important features and make the design more appealing and user-friendly as well.

The main deliverables during the GSoC period will be :

  • Config Section
    • Implement the panels which have not yet been implemented from the Qt application.
    • Enhance the already existing panels and fix the existing inconsistencies.
  • Mail Section
    • Implement the Reply, Reply All and Forward feature in the mail view.
    • Fix the Compose Mail popup and make it usable.
  • Forums Section
    • Implement the remaining features from the Qt app in the forum section.
    • Implement a better layout of the forum and posts view, comment and reply feature etc.
  • Boards and Channel Section
    • Implement the remaining features and make them more user-friendly and easy to use.
  • Implement a feature for Configuring and Visualizing own shared files with features such as managing shared directories, user permissions etc.

Previous Contributions

During and before the community bonding period, I contributed and added features for the implementation of webui of retroshare. And, The most recent contribution I did was the implementation of file search in Files section, which wouldn’t have been possible without the help of my mentor Cyril Soler.

I have also implemented various other features such as Attachment view for Mail section to view all the mail attachments in one place, improving the sidebar etc. Likewise, I have also tried to reduce the overall size of the WebUI by minifying the files. The PRs I raised until now can be seen here.

What’s Next?

Currently, The community bonding period is going on, and I have made myself a bit familiar with my mentors and some fellow members. The mentors are really supportive, and I couldn’t have made it to GSoC without my mentor Cyril Soler and one more amazing person, Defnax. In addition, I actively participate in weekly meetings where I report my progress, discuss different approaches.

So, for the first phase which is until the midterm evaluation I have planned to work towards the implementation of the first two features listed in the Goals and Deliverables section of this blog. It has been an amazing learning experience, and I am looking forward to achieving more amazing things this summer with Freifunk and Retroshare!

Thanks for reading and Have a great day 😃

GSoC’23 Qaul : An Internet Independent communication application

Preface

In this blog post, We will discuss about a P2P chat application Qaul.net and look at the importance of protecting our communication in true sense. Also, I will throw some light on what we are planning as a Google Summer of Code 2023 project for this application to make it more accessible, robust and independent from any form of services.

Qaul.net is completely internet independent and Peer to Peer chat application where within a local network, Devices can be connected and the communication can take place. There are no chances of being wiretapped since, It does not work on the internet. Also if you get caught up in situations where the internet services are down deliberately or indeliberately then too you can become part of the network using Qaul. All you need is the working device and the application itself.

Use case

There are multiple examples of places where the governments cuts down the communication links over a particular region due to riots or suspicious activities or sometimes due to political playings. With Qaul the aim is to provide local links of communication so no one feels lefted behind. For Example, In India-Pakistan border, there is always tension of riots between the nations for accquiring regions of Kashmir. Due to this, The internet services are blocked and also the communication infrastructure is damaged at times. Here, With qaul the issue can be addressed and people can still utilize communication since it falls under very basic rights of human beings. Another use case arises in a huge crowd events. Recently, My university organized the Techo-Cultural Fest with over 7000+ student gathered in one ground. Due to this large number of people gathered at a single place, We were hardly able to use daily communication services like texting on whatsapp or calling. The base stations went crazy to handle such a huge number of traffic over a particular cell. So, What if we students used qaul to create a network of our devices. This could have helped us in texting and finding lost friends.

Enough of use cases, But you might wonder : How does it works ? Well it uses mix and match of various communication protocols and cryptographic encryptions using which the application is decentralized and internet independent.

Implementation

Each device is called node and it has a cryptographic id called qaul-id. Now for discovering the peer devices, It used mDNS (Multicast Domain Name System) using which you can get the IP addresses of the peer devices without reaching or using the root DNS server (like .com or .edu or .uk etc) and no data is transfered to anyone outside the local network. The messages stays end-to-end encrypted because of the cryptographic keys used to sign them while sending and are verified on the receiving end. For routing the messages the Distance Vector Routing protocol is used which is based on the Round trip time per connection request. So under the hoods, Every 5 secsonds each device sends ping to neighbour nodes to measure RTT and every 10 seconds each node sends the routing information to all neighbouring nodes. For routing you can use any route. It could be LAN or INTERNET or even BLE (in progress). The protocol will choose the best route and send the message over the network using that route. The device id remains throughout the time even when device is not connected to the network until the application is uninstalled. So, You can easily go offline and come back and get connected over the same local network.

Project Details

We are going to implement the Matrix Bridge for the Qaul.net this summer. So If you wonder, Why would we need that at first place ? We need it because it will help us to broadcast messages over many mediums of communication which are supported by Matrix. This would allow the messages to be shared from local network to other networks and store it based on user consent. Let’s say, My government forces to not to keep any such application like qaul but luckily they don’t spy on my slack and allows me to keep it. I can simply transfer my messags from qaul to slack using the matrix server. If you remember earlier I said, You can get connected to same local network using qaul-id which get’s lost if you uninstall application. So, If there is a bridge, You will receive the message now in any communication medium. You can communicate from Slack as well. Relay Bridge is the appropriate name for this. But only relay bridge won’t be effective solution because we are using cryptographic encryptions and decryptions for each node or device. So, How would I know your real identity ? There is another kind of bridge which is puppetting. So, You can puppet yourself as old qaul user on the Slack or Telegram instead of a ghost username and then there can be fledged two ways communication. So, In total we need Relay and Double Puppetting (both ended) bridges with Matrix. Further from Matrix other bridges with other applications are already implemented and won’t be an issue.

There is a recent news where Indian Government is banning 14 Messaging Applications. One among the list is Element which is a matrix client which I am using for all my matrix chats. The government don’t understand the difference of banning the decentralized application due to their misconceptions about the technology.

I wish, We can bring qaul to greater reaches and get it working. Even if the government bans qaul from playstore or any downloading mediums, It can still be sent from one device to another by using open networks or application file transfer. But government can never block it from its functioning since it is completely internet independent. It just spreads like a virus and can be used wherever needed.

I would like to thank mentor Mathias Jud for helping very much in explaning and getting the concepts clear about the internal working of the applications, reviewing the proposal and helping in getting the IPv6 addresses to be launced in one of the new beta versions. Looking very much forward to work on this project with lots of enthusiasm and knowledge.

GSoC ’23: Documenting the OpenWrt compilation process to set up a PPA

GSoC ’23: Documenting the OpenWrt compilation process to set up a PPA

Getting the OpenWrt PPA set up will require understanding and documenting the current approaches for portable compilation of OpenWrt packages. This is my first impression of this task.

When I verified the “hello world” program provided and verified by Zoobab correctly compiled, I have looked to see how exactly this process works. The Docker build script depends on the toolchain compilation script. They both require an internet connection, and I will have to keep this in mind when porting this to make an OpenBuildService (OBS) package: “Mentioning repositories directly is not allowed (using obsrepositories:/ is ok)“.

Interesting technical details I found while researching how to transfer the current Docker approach to OBS:
– At first I did not know what the flag V was in `make V=s`. It turns out it turns on verbose compilation output on console on the OpenWrt build system. (I was surprised when I saw this was undocumented in the new version of the guide.)
– The package manager for OpenWrt uses ipk files. The helloworld package (that has been compiled under the OpenWrt SDK) has been successfully installed inside an openwrt rootfs and runs great!
– The current system uses cascading Docker images (multi-stage builds). This is done by producing the OpenWrt SDK container image first and then a new Docker image compiles the actual system. This is useful because it means the actual package being compiled is kept separate from the source itself.
– It turns out that obs-build from OpenBuildService assumes a working chroot so one must be created manually. The actual container must be merged in separately into the upstream project, outside of obs-build.

A message from our working hello world package:

root@localhost:/# helloworld
Hello world!
MyClass::MyClass()
MyClass::printMessage()
This is my message to print


I can’t wait to learn more and get up to speed on how to approach this project. I’ll see you later!

Six exciting projects at Google Summer of Code

Google Summer of Code Logo

Freifunk as umbrella organization unites wireless communities like Ninux, qaul.net, Guifi.net, and Evernet e.G. Our communities extensively rely on OpenWRT Linux, OLSR, BATMAN, libremesh, or retroshare

We are proud to announce our participation in the Google Summer of Code (GSoC) program. This year, Freifunk has six exciting projects that will contribute to the development of the Freifunk firmware, mesh networking protocols, and user-friendly tools.

Google Summer of Code Logo

The six projects and their respective mentors are:

Project TitleContributorMentors
Automation tools for LibreMesh firmware build and monitoringsamloIlario Gelmetti, stefca
Joint Power and Rate Control in User space for Freifunk OpenWrt Mesh & Access NetworksPrashiddhaThomas Hühn, Julius Schulz-Zander
LuCI Migration to JavaScript based Framework: Improved UX and Performance on OpenWrt-based DevicesAyushmanPaul Spooren, Andi Bräu
Implementation of Web Interface of RetroshareSumit Kumar SoniCyril Soler, G10h4ck
Qaul Matrix BridgeHarshil JaniMathias Jud
OpenWRT PPA Part 2Mr. AndreiZoobab
Our table of projects

These projects are all aligned with Freifunk’s mission to build a decentralized, community-owned network that is free from corporate control and censorship. By participating in GSoC, Freifunk is able to tap into the talent and creativity of the wider open source community and accelerate its development efforts.

We are excited to see what our GSoC contributors will achieve this summer and we look forward to sharing their progress with the wider Freifunk community. Stay tuned for updates on our blog and social media channels!

In conclusion, Freifunk’s participation in Google Summer of Code is a great opportunity to advance the development of its mesh networking technology and tools. We are excited to see the impact these projects will have on our communities.

Our history of 11 successful summers of code during the last 15 years can be found in this blog.

GSoC’22 Final Report : Videoodyssee

Introduction

Hello folks 👋

The past 4 months has been pretty exciting and challenging for me . Me and my mentor Andi Bräu together worked on a cool project called Videoodyssee , with GSoC’22 almost coming to an end in this blog post I wanna share all of my accomplishments , results and little bit about the project.

About Videoodyssee

In 2017 we started our video portal for freifunk at media.freifunk.net. To get videos online we have a more or less complicated upload workflow:

  1. Upload video file to a processing server.
  2. Correct video file settings.
  3. Convert video to target formats.
  4. upload to CDN and Youtube.
  5. Publish the video to the media portal.

The previous Videoosyssee system has some problem like

  • With current lambdacd pipeline we can’t have multiple workers to enable parallel video processing.
  • The current lambdacd pipeline isn’t maintainable anymore and it has got a lot of dependency issues.
  • Currently we don’t have any video verification steps in our publishing workflow which leads to publishing of unwanted videos.

So we decided to re-implement the pipeline using a new CI/CD tool and we chose to use the GoCD tool to build the pipeline as it suits better for the video processing pipeline we are looking to build. Later in the project we also decided to re-implement the Videoodyssee uploader using React.

Systems Involved in the project:

1 . Videopipeline : A GoCD server with processing pipeline to process and publish the video.

2. Videoodyssee Frontend : A React frontend application for the users to submit the video data and admin dashboard for the admins.

3.Videoodyssee API : A Node.js REST API implemented using Express framework.

Video upload workflow:

  1. User submits the details of the video using the video details submission form.
  2. Admin gets a notification email about the newly submitted video.
  3. Admins logs in to the Videoodyssee admin dashboard reviews the video and approves it.
  4. When the admin approves the video it will trigger the video processing pipeline.
  5. The processing pipeline will download the video , fixes the video meta-data , encodes the video into webm and h264 formats , generates the thumbnail and timelens data , uploads all the files into the CDN and publishes the video to Voctoweb.
  6. After the video is published to Voctoweb user will a notification email saying that his got published to the video portal successfully.

Video processing pipeline

The pipeline is implemented using GoCD , a CI/CD tool as it suits better for the video processing pipeline we are looking to build. The GoCD follows master-slave architecture which consists a Go-Server and multiple Go-Agents.

Go-Server

Go-server acts as a master and is the one that controls everything, assign the jobs to Go-agents and provides the user interface to users of the system.This is where we define the pipelines and trigger the pipelines.Go-server assigns the jobs to the agents which are free and stores the artifacts produced by Go-agents.

Go-Agents

The agents are the ones that do the actual work like processing videos , generating thumbnails and uploading processed videos. We can have N number of agents which we can scale up or scale down according to our needs. So in future if we need to process more number videos at the same time we can increase the number of agents to get the job done.

You can have a look at the video processing pipeline code here. And here is the Videpipeline GitHub repo https://github.com/freifunk/videopipeline .

Tasks accomplished:

  • We created a config repo in GitHub to store the pipeline code so that whenever we change the pipeline code GoCD server will automatically pull the changes and builds the new pipeline.
  • Implemented the video processing pipeline.
  • Modified the pipeline processing scripts to make them work with the new GoCD pipeline.
  • Automated the provisioning and deployment of Go-Server and Go-Agents using Ansible. You can have a look at the Ansible playbooks here.

Remaining tasks:

  • Automating the pipeline deployment from the GitHub config repo.
  • Publish the video to YouTube.
  • Notify the admins with an email when a new video is submitted by someone.
  • Notify the publisher when his video got approved , processed and published to Voctoweb.

Videoodyssee Frontend

For the users to submit the video details we need a frontend application which takes the data from the user and sends it to the REST API which will eventually trigger the pipeline using the GoCD API to start the video upload workflow.

We chose to use the React to implement the frontend application as it is quick and easy. The frontend application will have a upload form for normal users and admin dashboard for the admins for all administration tasks like approving/rejecting videos , updating video details etc.

Task Accomplished:

  • Implemented the video upload form so a user can submit the details of a new video.
  • Implemented the Videodyssee admin dashboard where admins can review the videos to accept or reject them.
  • Implemented Authentication so that only admins can access the videoodyssee admin dashboard.

Remaining Tasks:

  • Implement the functionality to update the details of already published videos.
  • Implement the functionality to update video details before approving a video.
  • Implement the user management.
  • Implement profile management.

Videoodyssee API

We used Node.js Express framework to implement the REST API which will handle the requests from the Videoodyssee frontend. We chose Express as it is quick and easy to implement a REST API using Express framework. We have used systemd as the process manager for the API server which will have some benefits like the server will restart on its own when it crashes.

Tasks Accomplished :

  • Implemented a route which will take the video details from the frontend and triggers the GoCD processing pipeline.
  • Automated the deployment process of REST API server using Ansible by implementing videoodyssee-api playbook.

Video review process:

  1. Admin logs into the Videoodyssee admin dashboard by using username and password.
  2. Reviews the video details and if clicks the video title he can open the video in another tab to watch it.
  3. If everything is fine he will approve the video.
  4. If some of the details are wrong he can change the details and then approves the video.
  5. if the admin thinks that the video is a spam video he can reject it.

Conclusion:

Finally the GSoC’22 has been pretty exciting and challenging for me .My mentor Andi Bräu really helped me a lot in the development process. I have learned a lot about open source development workflows, automation using Ansible and unit testing.

And finally , I would like to thank my mentor Andi Bräu for his incredible support and feedback.

If you got interested in my project and wanted to know more about it you can reach out to me on Linkedin.

Report Final GSOC 2022- Try LibreMesh without having a router

In the first blog, some problems that users were running into when trying LibreMesh were mentioned. In short, they consisted of:

  1. When you wanted to shut down a node that had been run previously, you left an interface up on the host.
  2. When you wanted to start a node to change internet access, this was not possible since the port required by the qemu_dev_start script was always occupied by the systemd_resolved process.
  3. When the node cloud was brought up, it had called ifconfig, and since this functionality is not installed in versions later than Ubuntu 18.04, the user could not run the node cloud.

To solve these problems, the strategy that was chosen was to modify the files already created, also creating detailed documentation to facilitate virtualization for anyone who wanted to try LibreMesh.

Milestones accoplished

1. Improvement of the environment:

All the problems mentioned meant obstacles when testing LibreMesh, thus causing users to decline in the knowledge of the tools that LibreMesh proposes. So, identifying and correcting them would offer any user a cleaner testing environment, free from any obstacles that depend on the LibreMesh code.

1.1 Close interface on the host

The solution to this was to modify the qemu_dev_stop script. The interface that was raised is called lime_br0, so the line was placed inside this file:

lime_br0 ip link

In this way, the interface on the host will stop the execution of the node.

1. 2. Collision in port

In this case, the solution was to change the port occupied by the DHCP service for the wan interface:

dnsmasq -F 172.99.0.100,172.99.0.100 –dhcp-option=3,172.99.0.1 -i “$WAN_IFC” –dhcp- authorized –log-dhcp –port=5353 –bind-dynamic

1. 3. Calls to ifconfig

In this case, the files that were modified were two:

– linux_bridge.py

– linux_bridge_port.py

which are located inside the lime-packages/tools/ansible/modules directory.

Where there were ifconfig calls, ip was used.

In linux_bridge.py the changes were the following:

– def brctl (self, cmd) :

– return self.module.run_command ([‘brctl’] + cmd)

– def ifconfig (self, cmd) :

– return self.module.run_command ([‘ifconfig’] + cmd)

+ def ip(self, cmd) :

+ return self.module.run_command ([‘ip’] + cmd)

– (rc, out, err) = self.brctl ([‘addbr’, self.bridge])

+ (rc, out, err) = self.ip ([‘link’, ‘add’, ‘name’, self.bridge, ‘type’, ‘bridge’])

– self.ifconfig ([self.bridge, ‘up’])

+ self.ip([‘link’,’set’,’up’, self.bridge])

– self.ifconfig ([self.bridge, ‘down’])

– (rc, out, err) = self.brctl ([‘delbr’, self.bridge])

+ self.ip([‘link’,’set’, ‘down’, self.bridge])

+ (rc, out, err) = self.ip ([‘link’, ‘del’, self.bridge])

In linux_bridge_port.py the changes were:

– def brctl (self, cmd) :

– return self.module.run_command ([‘brctl’] + cmd)

+ def ip(self, cmd) :

+ return self.module.run_command ([‘ip’] + cmd)

– (rc, out, err) = self.brctl ([‘addif’, self.bridge, self.port])

+ (rc, out, err) = self.ip ([‘link’, ‘set’, self.port,’master’,self.bridge])

– (rc, out, err) = self.brctl ([‘delif’, self.bridge, self.port])

+ (rc, out, err) = self.ip ([‘link’, ‘del’, self.port,’dev’,self.bridge])

qemu_cloud_start.yml was also modified:

+ enable_wan_param: “{{ (‘–enable-wan ‘ + enable_wan ) if enable_wan is defined else ” }}”

– shell: (../qemu_dev_start –node-id {{ node_id }} –eth0 {{ lm_ifname }}_0 –eth2 {{ lm_ifname }}_2 {{ rootfs }} {{ ramfs }} &)

+ shell: (../qemu_dev_start –node-id {{ node_id }} –eth0 {{ lm_ifname }}_0 –eth2 {{ lm_ifname }}_2 {{ enable_wan_param }} {{ rootfs }} {{ ramfs }} &)

– linklocal: “fe80::5000:ff:feab:c0{{ node_id }}%lm_cloud{{cloud}}”

+ linklocals: “{{ linklocals | default([]) | union([ hostvars[item].linklocal | default() ]) }}”

Conclusion:

The solutions that were given to the different problems and everything documented to be able to use LibreMesh greatly contribute to anyone who doesn’t even know what LibreMesh is and who wants to try it for the first time.

When a person is not knowledgeable about something and there are multiple problems they run into, and no help to clarify the picture, that person is more likely to stop spending effort on domain knowledge and look for other similar options, which you can understand and access more easily.

That is why the fact of not encountering the problems raised and of having documentation that guides this testing process means that users do not decline so easily and want to use mesh networks, which were the objectives pursued from the beginning.

The development of this whole project was very challenging for me. From the use of tools that I did not know like ansible-playbook, network management with which I was not so familiar, the use of virtual machines, the familiarization with the LibreMesh community to the management of a project itself, there were many things that I learned and what I take away from this project.

I am enormously grateful to my mentor Germán Ferrero, a great person, who has been able to guide me along the way I have traveled, always encouraging me to move forward and being there to answer any questions.

I also thank my advisor Tomas Assenza, other great person, who encouraged me to carry out this project from the beginning and who was faithfully present throughout its duration.

I am very happy about everything I learned in this beautiful experience.

Thank you very much for reading and following this tour!

Final Report on Minstrel TX Rate Control in User space – GSoC ’22

Hi everyone! With the GSoC 2022 session almost coming to an end, this is my final blog post to share all of the accomplishments, conclusions, and outlook for the Minstrel TX Rate Control in user space.

Goals of the Project

  • Adding a proper output to the existing user space Minstrel HT to aid in rate control analysis.
  • Extending user space Minstrel HT with missing functions present in its kernel counterpart.
  • Addition of new estimators/filters for research purposes.
  • Proper documentation, demo, and guide on working with the Minstrel HT package.

All of the mentioned goals have been completed. Furthermore, there have been additional changes/improvements to the user space Minstrel HT that isn’t listed above. With this, the next sections will cover all of my contributions to the project followed by future work and limitations.

(1) New Output

Before GSoC ’22, the user space Minstrel HT didn’t have a proper output which wasn’t even easily interpretable. Furthermore, it wasn’t possible to run a script to analyze the performance of the user space Minstrel HT. Hence, I’ve replaced the old output which was simply a printout of the rate statistics dictionary with its own dedicated file and folder for each access point. For each access point and the physical WiFi chip, it creates two files: rc_stats and rc_stats_csv. The rc_stats is a human-readable output of the RateTable object for user convenience whereas the rc_stats_csv is a file that stores all the RateTable data throughout the execution of user space Minstrel HT for offline analysis.

The output has been previously discussed in the previous blog post with the screenshot of the outputs which can be found here.

(2) Addition of new functions from kernel Minstrel HT

The user space Minstrel HT has been extended with additional functionalities from the kernel Minstrel HT which was missing prior to GSoC ’22. Consequently, the user space Minstrel HT now, in terms of functionality, is closer to its kernel counterpart than before.

get_avg_ampdu_len: Calculates and returns the average AMPDU length for a station connected to the access point.

calc_retransmit: Previously, the retry count in the multi-rate retry chain was static and set to a specific value (10) for all the rates. With this new function, the user space Minstrel HT now dynamically computes the retry count for each rate in the rate setting.

check_sudden_death: Checks if the two best throughput rates have sudden packet loss and, if the packet count is greater than 30 with 75% loss, then the affected rate is downgraded.

prob_rate_reduce_streams: Tries to find a more robust rate index that uses fewer streams than the current maximum probability rate.

downgrade_rate: Reduces either the best or the second best throughput rate to the maximum group throughput rate from a lower group that uses fewer streams.

For the addition of these functionalities, the RateMan package had to be extended to parse additional station information such as AMPDU length, AMPDU packet, and overhead for MCS and legacy rates.

(3) New Estimators

The user space Minstrel HT has been added with two new filters: Butterworth Filter and Exponentially Discounted Averaging and Variance. The kernel Minstrel HT no longer uses the Exponentially Weighted Moving Average (EWMA) and has switched to the Butterworth filter, however, prior to GSoC ’22, the user space Minstrel HT only had a simple implementation of the EWMA. The details on the Butterworth and the Discounted filter have been already discussed in my previous blog post which also includes the implemented mathematical formulas.

Regarding the EWMA, the previous implementation of the filter in user space Minstrel HT was also not identical to the kernel variant. Since the EWMA is used to calculate the average AMPDU length for a station, I’ve updated the implementation of the EWMA in the user space Minstrel HT and now consists of both EWMA level and division.

(4) Restructure

During the GSoC ’22 coding period, the RateMan package was restructured twice and, as the user space Minstrel HT is dependent on RateMan to perform kernel rate control functionalities, it was also modified to make the rate control algorithm compatible with the changes. Additionally, the user space Minstrel HT was also refactored to split the minstrel module into two modules: minstrel and sample module. Consequently, this enables each station to have an independent Sample object for probing which can be scaled and customized as needed in the future. Furthermore, the previous structure of user space Minstrel HT consisted of several independent loops which could have been avoided or merged to save computation time. Therefore, I’ve reduced the number of iterations in the minstrel module which has significantly reduced the computation time.

Previously, to obtain the best throughput rate and also the maximum probability rate, the user space Minstrel HT ran all the computations of finding the rates every time and was not stored to be accessed later. However, I’ve changed it such that, for an update interval, the best throughput rate and maximum probability rate are calculated only once and are available as an attribute of the RateTable object for direct access. Furthermore, I’ve also added a new attribute called “max_group_prob_rates” which stores the maximum probability for each group.

(5) Configuration

The user space Minstrel HT has been added with a configuration module to tweak filter parameters and rate control properties such as sample interval and update interval. The configuration file also allows the user to select the filter to be used for rate control. Currently, the user can choose from the following filters: Exponentially Weighted Moving Average (EWMA), Butterworth filter, and Exponentially Discounted Averaging and Variance.

The configuration parameters have been discussed in the previous blog post if this is in the interest of the reader. I have not included it here as this blog post is meant to only serve as an overview of all the changes and reiterating it here would only make the post longer.

(6) Rate-setting Experiment

Towards the end of GSoC ’22, I was also involved in conducting several experiments to validate the rate setting functionality in user space. For this, I created a separate GitHub repository with scripts to configure and run rate-setting experiments. The experiment script, for all supported rates, sets the rate between the client and access point, and after a rate-setting delay collects packet statistics for a specified time duration. The rate-setting delay and the time duration for rate statistics collection are completely configurable by the user. The output is stored in a file that shows, for each rate setting, the rates used, and their statistics.

For example, a WiFi connection with three supported rates could have the following experiment output:

Setting Rate Index 00
{’00’: {‘attempts’: 275927, ‘success’: 236525, ‘timestamp’: ’16d54c2156813120′}}
Setting Rate Index 01
{’01’: {‘attempts’: 366064, ‘success’: 318634, ‘timestamp’: ’16d54c3dfcf4af70′}, ’00’: {‘attempts’: 1199, ‘success’: 584, ‘timestamp’: ’16d54c3cb7d445a0′}}
Setting Rate Index 40
{’40’: {‘attempts’: 270312, ‘success’: 255716, ‘timestamp’: ’16d54deb84d2b0a0′}}

The middle result would denote that after setting the rate between the connection to index ’01’, we still see that rate ’00’ was used where we would generally only expect to see ’01’.

(7) Sample Rate

The last thing that I worked on during GSoC ’22 was another prominent but older rate control algorithm, called Sample Rate, in user space. The Sample Rate algorithm was developed by John C. Bicket in 2005. Unlike Minstrel HT, Sample Rate is not time interval based but real-time packet-based rate control. In the beginning, the Sample Rate algorithm sets the rate index to the highest bit rate. If a rate index fails, then it switches to the next highest bit rate until it finds a rate that works. Every 10th packet, the algorithm samples a random bit rate that does not have four successive packet failures, and the lossless transmission time is less than the average transmission time of the current best rate.

The algorithm uses only the latest results that are not older than 10 seconds to determine the average transmission time for each bitrate. The best rate is the data rate with the lowest average transmission time per packet. The transmission time is calculated as follows:

tx_time(rate) = airtime(rate) + backoff_time(failed_packetes)

The average backoff time, in microseconds, for up to 8 attempts of a unicast packet is:

Where to find my work?

GitHub Repositories

  • scnx-py-minstrel
  • scnx-snippets/rate_setting_validation
  • scnx-sample-rate

The repositories have not been public yet, but they will soon be released as open-source.

Future Work and Limitations

The user space Minstrel HT is running and fully functional but it has been only run on decent hardware i.e. laptop and not on embedded systems. It could be that the user space Minstrel HT requires heavy computation and may not be able to perform timely update intervals when running on an OpenWRT router itself. The output file, rc_stats_csv, file stores all the historical RateTable data which causes it to grow quite quickly and infeasible for experiment durations longer than several hours. Hence, a good addition to the user space Minstrel HT could be an in-built compression for the CSV file.

Furthermore, there are still some minor functionalities remaining to be implemented in the user space Minstrel HT. The kernel Minstrel HT uses a random sampling table to select probe rates which could be also implemented in the user space Minstrel HT as well instead of filtering from a list of potential probe rates. Additionally, now that the average AMPDU length can be calculated in user space Minstrel HT, the AMPDU length can be used to find the estimated throughput in an exact way as in the kernel variant.

The user space rate control API (minstrel-rcd) doesn’t have a way to set the retry count for RTS/CTS frames. Once this is implemented in the API, the calc_retransmit function in the user space Minstrel HT can be extended to compute and set the retry count for RTS/CTS frames as well. Similarly, the minstrel_ht_get_max_amsdu_len from the kernel Minstrel should also be implemented in the user space, in case it can be explicitly used which isn’t the case as of now.

Concluding Thoughts

The GSoC ’22 session has been a great opportunity to research and develop user space rate control algorithms. Even though, this is only the start of user space rate control algorithms, within the short time frame, a lot of work has been completed which has met and exceeded the initial goals. The Minstrel HT user space algorithm is now almost complete with all the functionalities from its kernel variant. Aditionally, another user space algorithm, Sample Rate, has also been implemented as a python package using RateMan.

I would like to thank my mentor, Prof. Thomas Hühn for his time and guidance throughout the project. This was a fruitful experience and would love to see the future of rate control in user space.