Tag: LEDE

PowQuty Live Log GSoC 2017 Final Update

Posted on by Stefan Venz

This is the last blog entry in the series of Google Summer of Code project updates. It will describe, what has been done and what is left to improve in the future in the PowQuty project.

Powquty

PowQuty is a power quality monitoring tool, which can be installed on a router running LEDE or OpenWrt. The router can be connected to an USB oscilloscope providing measurements which powqutyd will process and provide to the user in human readable form.
All this was tested on a x86 based LEDE router.

GSoC 2017

During this Google Summer of Code a live log functionality was added to PowQuty to provide information on power quality events. These events are:

  • voltage dip of 10% – 90% of the reference voltage on the measurement signal
  • voltage swell > 110% of the reference voltage on the measurement signal
  • voltage dip < 10% of the reference voltage on the measurement signal
  • > 5% of the measured values of one specific harmonic are over the defined threshold

    • On event occurrence important information like time, duration and event type will be written to a log file and presented in the extended luci app.

    As shown in the above picture, the interface provides a traffic light like color system behind these events, green indicates everything is within the EN50160 power quality norm. Yellow means, that 80% of the maximum time per week is already reached, red means, that the norm was violated during the last week.
    In addition to log writes, notifications are send out with Mosquitto. Mosquitto is a message broker using the MQTT protocol. It provides a publish/subscribe model, which allows a central server to subscribe to a topic and clients to send out messages to the server with a topic. Mosquitto was already in use in powquty but was extended for EN50160 event notifications. This will allow a central logging of bigger power supply networks, monitored by multiple devices.

    As another option Slack messages can be send by powquty now. Slack is a messaging program, using (as one option among many) webhooks for interaction. Everyone with the webhook can send messages to the team. Sending out messages allows a user to react quickly to changing situations, or get immediately informed on power event occurrence.

    Beginning with pull request 20 [https://github.com/thuehn/PowQuty/pull/20] I started to implement these features.
    First an option was developed to read measurements from a file, as most power supply networks are pretty stable and wont provide many opportunities to test event handling in powquty.
    Afterwards slack and MQTT notifications where added.
    During testing of mosquitto event messages, some seemed to be lost on intervals with many En50160 events in a short period(sometimes more than 35 events per seconds). The solution seems to buffer all events before sending.
    Something similar happened with Slack. Slack only allows one message per second(short bursts excluded) [rate-limits].
    Buffering events would resolve this problem as well. An option for live email notification was considered at first, but was dropped as spam protection would stop most of the messages and probably list users as spammers.
    The last step was to add the traffic light system to the luci app, to enable users without knowledge of the norm to get an idea of the power quality of their power supply network.
    In addition a slack library was written [libwebslack] to send slack messages from PowQuty.

    What can be improved
  • As mentioned before event buffering is needed and will be added after GSoC
  • Email notification in form of a weekly summary
  • More Error checking and handling
  • improving libwebslack to not use libcurl to reduce its size
  • provide libwebslack as OpenWrt/LEDE package, for easier future use

    • Finally I have to thank Dr. Thomas Huehn for being my mentor and Freifunk for their work they do and especially for being a mentoring organisation for Google Summer of Code.
    Last but not least I would like to thank Google for making this all possible.

    If you want to review some of my earlier posts:

  • Introduction
  • First Update
  • Second Update

    • Best regards
    Stefan

    GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – Final

    Posted on by Fejes Ferenc

    Brief summary

    In the first post (beginning of the GSoC 2017 project) I made a few checkpoints to complete at the end of the summer. Now I don’t copy them here, but the good news is all of them completed successfully. The main goal was a very simple transparent multipath Wi-Fi link bandwidth aggregation. The proof in the video above. And the details below.

    Continue reading “GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – Final” →

    GSoC 2017 – Attended Sysupgrade – Final evaluation update

    Posted on by Paul Spooren

    Hi Freifunk,

    This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.

    tl;dr:
    Direct links to the work I’ve done this GSoC:

    First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.

    Progress

    Server

    At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.

    The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.

    Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.

    To try the the server at https://betaupdate.libremesh.org have a look at the demos I prepared. Continue reading “GSoC 2017 – Attended Sysupgrade – Final evaluation update” →

    GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – July progress

    Posted on by Fejes Ferenc

    What we have so far

    After lots of experiments, I figured out the way which enables the transparent MPTCP operation for both the client and the server (with the regular kernel).
    I tested the configurations at my virtualized environment, check the video below. For configuration infos, please read the video description. (the post continues)

    Continue reading “GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – July progress” →

    GSoC 2017 – Milestone 2

    Posted on by ArneKappen

    Following the rather stark change of direction after the last milestone (see my last post), I have worked on the integration of my project with Johannes Wegener’s: OpenWifi.

    July Progress

    For my software-defined networking (SDN) agent, I have added support for configuration through a UCI config file as well as process management through procd. This ties the daemon in neatly with the LEDE/OpenWRT system. Now, parameters such as the SDN controller’s address are read from this file and the configuration can be reloaded at runtime without stopping the agent.
    Using the UCI system also exposes the configuration parameters to Johannes’ OpenWifi system. He and I discussed the bootstrapping process with our mentor and we have come up with the following idea: When a newly installed LEDE/OpenWRT access point first boots, it discovers the OpenWifi server via mDNS and fetches its configuration from it. With the address contained in the configuration, my SDN agent on the access point is able to connect to the SDN controller and thus integrate automatically with an existing centrally managed deployment. In the event of a configuration parameter change (e.g. a switch to a different SDN controller), OpenWifi can trigger a configuration reload to quickly update all access points in the network.

    On the controller side, I have implemented a REST client to interface with OpenWifi. Through it, the controller can register with the OpenWifi server. During the registration, it installs its address and OpenFlow listening port in the UCI configuration which later gets sent to access points.
    I have also begun writing a REST interface for my controller to offer more comfortable management of SDN applications and the network state. Right now, I can query the controller about its resources: access points, switches and clients. With the basics in place, expanding the interface to expose more SDN functionality should be pretty straightforward.

    Next Steps

    During the final part of this year’s GSoC, I want to focus on adding functionality, testing and documentation. I will spend the remaining weeks of the project like this:
    1) write an SDN application
    2) expose the app’s functionality via the controller’s command-line and REST interfaces
    3) test it on the university department’s testbed
    4) document its usage
    5) goto 1)

    Since the foundation for the SDN applications is in place and running, I am optimistic about getting a lot done during August. I will start with a client hearing map that keeps track of associated and unassociated clients in the vicinity of the deployment’s access points. Leveraging the hearing map, I want to implement a client load balancer that distributes associated clients evenly across available access points. I will also look into automated channel selection to avoid interference between neighbouring access points.
    Finally, I would like to wrap up the controller in a docker image for easy deployment.

    PowQuty Live Log Second Update

    Posted on by Stefan Venz

    It’s been nearly a month since my last update on the PowQuty Live Log project and i would like to tell you, what has been done
    so far and provide information on what will be done in the next month.

    PowQuty got updated, to support Slack and mqtt event notification and can already be used in the current PowQuty version.
    In addition to this, there have been some bug fixes during the last month and some new features were added.
    On event occurrence the event gets stored in a csv file and each entry is displayed in the luci-app. To increase the usability,
    a traffic light system will be added, which will show for each event type its occurrence time and show if the current values
    are in violation of EN50160.

  • Green: Everything is ok, no violation
  • Yellow: Close to a violation
  • Red: This event time is in violation of the norm

    • The event messages contain a times stamp, the duration of the event and updated event Type information, as well as event type related
    information and GPS data.
    .

    As receiving notifications or emails on every event occasion can get noisy, we decided to provide a weekly summary of the events in
    addition to the regular notifications.
    The user will be able to decide if he wants to receive this summary, every event, or both. We consider using the traffic light system
    here as well, to increase the readability and enable users to understand the quality of their power supply network, without a lot of
    knowledge of the EN50160 norm.
    We discussed individual intervals and keep it in mind as a possible later feature.

    Best regards
    Stefan

    GSoC 2017 – wlan slovenija – Report 1 – HMAC signing of Nodewatcher data

    Posted on by red_moo

    What’s been done

    The first blog post that describes the idea and goals can be read here.

    After getting a bit more familiar with the code base the first thing that had to be done was to set up the work environment. I chose to go the virtual machine route so an actual router did not need to be used, making the development easier. An instance of the nodewatcher is running on a virtual machine running Ubuntu Linux. It runs in a Docker development environment and is set up by using the provided Docker Compose configuration. To enable data collection from nodes the monitor system also needs to be running.

    To test the the nodewatcher’s HMAC signing capabilities I set up a dummy test node, turned on HMAC signing and used a python script to push some data on to the nodewatcher.

    The next step was to set up a development node. With a lot of help from my mentor Kostko I used firmware-core to create a LEDE virtual machine, compile nodewatcher-agent packages, transferred them to the LEDE machine, used SSH to connect to it, installed them and tested the basic nodewatcher-agent functions. The whole setup and development process was documented in detail in the nodewatcher-agent README. Then I set up the agent’s http.push module and pushed some data on to the nodewatcher instance.

    A local network like one in the above image was set up for development. Due to my lack of experience with networking I encountered some issues like not being able to ssh into the LEDE VM and not being able to send packets from LEDE VM to Nodewatcher VM, but with patience from my mentor it all works and the lesson was surely learned.

    What’s next

    So the hardest part is done, right? I am now working on adding a HMAC signing like it’s already done in the python script. It needs to be developed for both pushing the data from a node to the nodewatcher and pulling the data from a node.

    I am contributing using my github account.

    Onwards! Good luck!

    GSoC 2017 – Milestone 1

    Posted on by ArneKappen

    June Progress

    During the past weeks, I have mainly worked on LoxiGen. LoxiGen is a compiler which generates OpenFlow serialization and de-serialization libraries for C and Java. I have used the tool before and was able to build on my own work introducing the necessary message types for my GSoC project. Since I have chosen ONOS as my Software-Defined Networking (SDN) controller, I used ONOS’ fork of LoxiGen as a starting point. This required hard-coding some message types to make ONOS’ fork produce valid C libraries. I am planning to get in touch with LoxiGen’s maintainers soon to try and figure out if some of my work can be used towards a merge of the two diverted code bases.

    [Edit: Major change of plan]

    In an older version of this blog post, I wrote that I was going to implement a kernel module to enable the de-coupling of hostapd and the physical access point. Yesterday, I had a long discussion with my mentor and Johannes, another GSoC participant for Freifunk (read about his project here). We decided to drastically change the direction of my project. Instead of aiming for the (undoubtedly) cool and advanced feature of running hostapd as a Virtual Network Function, I am going to focus my efforts on a simpler use-case. The reasoning is to have a more stable and ready-to-use toolset for the management of larger OpenWRT/LEDE deployments in the end. In other words: Do less but do it well.

    To this end, Johannes and I are going to work on a set of interfaces to make our projects compatible. The vision is as follows: I am going to write an SDN application that registers the ONOS controller with Johannes’ OpenWifi server. Whenever a new access point joins the network, it discovers the OpenWifi server and fetches its configuration from it. Included in this configuration are the address and port of the ONOS controller. Thus, the new network node is able to connect to the SDN controller without the need for manual configuration by the network operator.

    Next Steps

    As mentioned above, Johannes and I will have to establish interfaces for our projects to interact. We also need to set up a combined testbed to test our implementations. On my end, I will start by adding the possibility to configure the OpenWRT/LEDE agent that connects to the ONOS controller through UCI. That way, the agent’s configuration can be queried and altered through the OpenWifi REST API. This should be rather simple. Next, I will write a proper REST API for my controller as well as the application that communicates with the OpenWifi server. Then, I will add features for network management such as the ability to move clients to a specific access point or to ban them from connecting altogether. Lastly, I will write SDN applications for automatic network management, for example, a load balancer that distributes clients evenly across available access points that is powered by the data stored in a client hearing map.

    GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – June progress

    Posted on by Fejes Ferenc

    Info form the project

    For more info whats the goal of the project, please visit my first blogpost which is here.

    What done yet

     

     

     

     

     

     

    I created this simple topology for trying out the technologies. I’m still in experimenting status but have some results as well. This virtualized test enviroment run on my machine. The paths between the MPTCP supported machines are shaped to 10 Mbps. With iperf3 and Wireshark I verified that the MPTCP protocol is working fine: clearly shows 20 Mbps between the two MPTCP supported machine (I use the latest 0.92 MPTCP kernel which is based on the 4.4 kernel version). Tha path between the MPTCP and Endhost machines are not shaped, so the throughput is much higher than 20 Mbps. I configured shadowsocks-libev on MPTCP1 (ss-server) and Endhost1 (ss-redir) machine. Then I tired an iperf3 download between Endhost1 and MPTCP2. This looks like:
    Endhost1 <——————]TCP[——————-> MPTCP1 <———————]MPTCP[——————–> MPTCP2
    I successfully get the 20 Mbps on Enhost1 which is the aggregated throughput of path1 and path2. (continue)

    Continue reading “GSoC 2017 – Add MPTCP support in LEDE/OpenWRT trunk – June progress” →

    PowQuty Live Log First Update

    Posted on by Stefan Venz

    As mentioned in my previous blog post, i am going to add a live log and notification system for
    certain events to the power monitoring tool PowQuty. The first steps have been done and the
    configuration has been extended.
    Three types of notifications have been added to the configuration options during the first month of coding.
    Namely email, slack and mqtt. Mqtt was in use before, but was extended to allow a second host and topic for
    the power quality events.
    The powquty configuration page was redesigned to use a separate tab for each notification option
    to increase overview.

    The old configuration page would have been very crowded with all the new options
    The new configuration view with mqtt tab open

    Power quality events, that cause a notification are:

    • Voltage dip between 10% and 90% of the reference voltage of 230V
    • Voltage swell above 110% of the reference voltage of 230V
    • Voltage dip < 10% of the reference voltage
    • More than 5% of the samples of one harmonic are above the threshold

    As the power supply network in Berlin was not willing to provide such events an option for test measurement
    input was needed. A file read flag for powqutyd was added and needs a little bit of clean up
    before a pull request on the upstream powqutyd.
    The library for the USB-oscilloscope provides the number of EN50160 events per measure cycle and
    the kind of each event. As of now some basic slack notifications are added, which provide the event
    type and the event start time from measurement start in milliseconds to the channel and team set
    in the luci web interface or under /etc/conf/powquty.

    Slack notifications with start time in milliseconds relative to measurement start, probably will be UTC or local in the future

    In the notification the type of event is provided to allow the network administrator to react directly
    to the changes, without to check the log any further.
    The other notification options will be added and tested soon.