PowQuty Live Log GSoC 2017 Final Update

This is the last blog entry in the series of Google Summer of Code project updates. It will describe, what has been done and what is left to improve in the future in the PowQuty project.

Powquty

PowQuty is a power quality monitoring tool, which can be installed on a router running LEDE or OpenWrt. The router can be connected to an USB oscilloscope providing measurements which powqutyd will process and provide to the user in human readable form.
All this was tested on a x86 based LEDE router.

GSoC 2017

During this Google Summer of Code a live log functionality was added to PowQuty to provide information on power quality events. These events are:

  • voltage dip of 10% – 90% of the reference voltage on the measurement signal
  • voltage swell > 110% of the reference voltage on the measurement signal
  • voltage dip < 10% of the reference voltage on the measurement signal
  • > 5% of the measured values of one specific harmonic are over the defined threshold
  • On event occurrence important information like time, duration and event type will be written to a log file and presented in the extended luci app.

    As shown in the above picture, the interface provides a traffic light like color system behind these events, green indicates everything is within the EN50160 power quality norm. Yellow means, that 80% of the maximum time per week is already reached, red means, that the norm was violated during the last week.
    In addition to log writes, notifications are send out with Mosquitto. Mosquitto is a message broker using the MQTT protocol. It provides a publish/subscribe model, which allows a central server to subscribe to a topic and clients to send out messages to the server with a topic. Mosquitto was already in use in powquty but was extended for EN50160 event notifications. This will allow a central logging of bigger power supply networks, monitored by multiple devices.

    As another option Slack messages can be send by powquty now. Slack is a messaging program, using (as one option among many) webhooks for interaction. Everyone with the webhook can send messages to the team. Sending out messages allows a user to react quickly to changing situations, or get immediately informed on power event occurrence.

    Beginning with pull request 20 [https://github.com/thuehn/PowQuty/pull/20] I started to implement these features.
    First an option was developed to read measurements from a file, as most power supply networks are pretty stable and wont provide many opportunities to test event handling in powquty.
    Afterwards slack and MQTT notifications where added.
    During testing of mosquitto event messages, some seemed to be lost on intervals with many En50160 events in a short period(sometimes more than 35 events per seconds). The solution seems to buffer all events before sending.
    Something similar happened with Slack. Slack only allows one message per second(short bursts excluded) [rate-limits].
    Buffering events would resolve this problem as well. An option for live email notification was considered at first, but was dropped as spam protection would stop most of the messages and probably list users as spammers.
    The last step was to add the traffic light system to the luci app, to enable users without knowledge of the norm to get an idea of the power quality of their power supply network.
    In addition a slack library was written [libwebslack] to send slack messages from PowQuty.

    What can be improved
  • As mentioned before event buffering is needed and will be added after GSoC
  • Email notification in form of a weekly summary
  • More Error checking and handling
  • improving libwebslack to not use libcurl to reduce its size
  • provide libwebslack as OpenWrt/LEDE package, for easier future use
  • Finally I have to thank Dr. Thomas Huehn for being my mentor and Freifunk for their work they do and especially for being a mentoring organisation for Google Summer of Code.
    Last but not least I would like to thank Google for making this all possible.

    If you want to review some of my earlier posts:

  • Introduction
  • First Update
  • Second Update
  • Best regards
    Stefan

    GSoC 2017 – Attended Sysupgrade – Final evaluation update

    Hi Freifunk,

    This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.

    tl;dr:
    Direct links to the work I’ve done this GSoC:

    First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.

    Progress

    Server

    At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.

    The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.

    Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.

    To try the the server at https://betaupdate.libremesh.org have a look at the demos I prepared. Continue reading “GSoC 2017 – Attended Sysupgrade – Final evaluation update”

    GSoC 2017 – Milestone 2

    Following the rather stark change of direction after the last milestone (see my last post), I have worked on the integration of my project with Johannes Wegener’s: OpenWifi.

    July Progress

    For my software-defined networking (SDN) agent, I have added support for configuration through a UCI config file as well as process management through procd. This ties the daemon in neatly with the LEDE/OpenWRT system. Now, parameters such as the SDN controller’s address are read from this file and the configuration can be reloaded at runtime without stopping the agent.
    Using the UCI system also exposes the configuration parameters to Johannes’ OpenWifi system. He and I discussed the bootstrapping process with our mentor and we have come up with the following idea: When a newly installed LEDE/OpenWRT access point first boots, it discovers the OpenWifi server via mDNS and fetches its configuration from it. With the address contained in the configuration, my SDN agent on the access point is able to connect to the SDN controller and thus integrate automatically with an existing centrally managed deployment. In the event of a configuration parameter change (e.g. a switch to a different SDN controller), OpenWifi can trigger a configuration reload to quickly update all access points in the network.

    On the controller side, I have implemented a REST client to interface with OpenWifi. Through it, the controller can register with the OpenWifi server. During the registration, it installs its address and OpenFlow listening port in the UCI configuration which later gets sent to access points.
    I have also begun writing a REST interface for my controller to offer more comfortable management of SDN applications and the network state. Right now, I can query the controller about its resources: access points, switches and clients. With the basics in place, expanding the interface to expose more SDN functionality should be pretty straightforward.

    Next Steps

    During the final part of this year’s GSoC, I want to focus on adding functionality, testing and documentation. I will spend the remaining weeks of the project like this:
    1) write an SDN application
    2) expose the app’s functionality via the controller’s command-line and REST interfaces
    3) test it on the university department’s testbed
    4) document its usage
    5) goto 1)

    Since the foundation for the SDN applications is in place and running, I am optimistic about getting a lot done during August. I will start with a client hearing map that keeps track of associated and unassociated clients in the vicinity of the deployment’s access points. Leveraging the hearing map, I want to implement a client load balancer that distributes associated clients evenly across available access points. I will also look into automated channel selection to avoid interference between neighbouring access points.
    Finally, I would like to wrap up the controller in a docker image for easy deployment.

    GSoC 2017 Project – OpenWifi: LEDE/OpenWRT configuration Mangement

    GSoC 2017 – OpenWifi

    Hi, my name is Johannes Wegener and I’ll be working on OpenWifi this Google Summer of Code. I’m 27 years old and study computer engineering at TU Berlin. In this blog post I’m going to explain to you what OpenWifi is and what should be done during this summer of code.

    What is OpenWifi?

    OpenWifi is a OpenWRT/LEDE configuration management system. It is intended to manage a bigger or smaller amount of OpenWRT/LEDE devices.

    If a new access point joins a network it is be able to auto detect the management Server and register to it. After the node has been registered its configuration (static configuration like what you’ll typically find in /etc/config/ on OpenWRT/LEDE devices) will be downloaded by the server and stored in a database.

    It is now possible to query and modify aspects of the configuration. It also possible to change values depending on other values. This means configuration changes can be applied to a lot of different routers. There is an older templating system which shall be replaced by a new graph-oriented one. It also manages SSH-Keys, has a rudimentary file upload functionality (for uploading new images for example) and extensible API. It is possible for example for a node to ask for an image it should install. (Currently this is not very dynamic – but it is intended that an image could be selected on various parameters)

    The Server also regularly pulls the health-state of the node and displays it in an overview. Furthermore the server acts as a luci2 proxy.

    It is also extensible via PlugIns and is able to serve as an entry point to other services like icinga or location services. (Both already have a proof of concept PlugIn available)

    How does it work? aka what makes it tick?

    The main software is written in python and uses the pyramid framework and sqlalchemy as ORM. It uses pyramid-rpc for json-rpc requests (which is used for nodes to register for example) and cornice for a REST-style API (which is intended for the user of the system). The Core-System can be found in this repository.

    Most of the tasks operating on nodes are done by a jobserver that uses celery – so that they don’t block the main thread.

    All Webviews have been moved to a different repository and are realised in fact as a PlugIn. (If you just want to manage your nodes on CLI/with scripts this will be possible in the near future!)

    The nodes use a notification script written as a shell script. It uses a fixed DNS entry (openwifi), a configuration file (/etc/config/openwifi) or mdns (using umdns or avahi) to detect a server and register to it. The packages can be found in the openwifi-feed repository. There is also a boot-flasher which is intended for flashing an alix2-style board from ramfs – since it is also possible to execute commands on the node I’ll integrate an update solution that uses sysupgrade.

    The communication between the server and the node is realized currently via rpcd. But one goal during this Google Summer of Code is to abstract that and realize it via a rpcd-communication-PlugIn – this would make it possible to also a NetJSON-communication-PlugIn for example.

    The PlugIns are realized with python entry points. There are some special named entry points which will extend the main application. You could have a look at the example plugin to get an idea how it works.

    How to try it?

    You can easily try the software with docker. Navigate to the Docker directory – there three files that will assist you with the setup. In conf.sh you setup if you want to use LDAP for authentication, avahi for mdns announcement and dnsmasq as dhcp server in the docker container.

    With build_image.sh you build the docker image. It tries to detect if you need to use sudo for docker or not. Last but not least you need run_image.sh to start the image.

    You can easily add PlugIns to the docker image if you just check them out or copy them inside the Plugins directory. To start off I would recommend to add the Web-Views Plugin and use avahi. Now you can navigate your browser to http://localhost:6543 to see the webviews (you need to restart the container after Plugin install – use docker stop OpenWifi and docker start OpenWifi).

    What is needs to be done?

    In the next section I’ll explain what should be done during Google Summer of Code. I need to prioritize these things because I’m not sure if it is possible to do everything. I think most important things are Testing, Authentication and the new graph-based database model.

    Testing-Infrastructure

    Currently just very few things have tests. But because it is possible to start a LEDE container in docker (there are some scripts helping to create a LEDE container in the Docker/LEDEImage directory) a lot of things are possible to test with docker orchestration. I want to provide a small LEDE image with the code to test it.

    Before adding big new things I like to transfer the development to true test-driven development and having test for at least 90% of the core component code. And after that continuing development by writing tests first.

    For that I would also have a look at coverage.py.

    Security and authentication

    There is a simplistic authorization for OpenWifi with a LDAP backend. I would like to add authorization based on user/password (without LDAP), API-Key and client side certificate (for the nodes to authenticate theirself against the API).

    I would like to have the authorization as granular as possible. Authorization should check whether the authorized identity is able to operate on this node and what kind of action is allowed.

    Security is provided by TLS. Last week I added the last bits to also use https for client communication – but right now it needs to be setup up by the user. I would like to add the appropriate hooks to the notification shell script.

    Expand new graph-based DB-Model

    The new DB-Configuration model is based on a graph – where configurations can be connected by links (the link can contain addition information – like what kind of link it is – currently the addition data is the option name (for example if you have a wifi-iface configuration the “device” links it to a wifi-device)).

    The model is implemented and there is a query API to get and set options. But for communication purposes the configuration is stored as uci-json-config. There is code to convert between these two (but the one from uci to graph-model needs quite some more intelligence to detect everything right). There are also some hooks that update each part. But the update process is not consistent – the graph-based config gets a new ID for example. It would be nice to have a consistent conversion.

    Furthermore I like nodes to share parts of a configuration. This should be easy to implement with some small DB changes. For this consistency is also very important. By default UCI generates unique strings for every configuration – this could be used for consistency.

    The DB-Model also currently lacks creating new configurations and reordering configurations.

    API

    The API to modify nodes should get expanded and everything should get documented. (see above) This should be done in code and a document generated by sphinx.

    I also like to implement a CLI program to use the API.

    Versioning

    The uci-json parser supports diffs between configurations. I would like to save a diff for each change which is applied – to have roll back configurations or delete a specific change.

    There is a simplistic revisioning implemented. But the diff should be updated to a class of its own with json import and export and operations to apply a diff to a UCI configuration. (upgrade and downgrade)

    Modularity and expandability

    OpenWifi is right now is already quite modular – but I also want to modularize configuration parsing and communication. It would also be nice to have some interoperability with OpenWISP.

    I use alembic for tracking database changes it would be nice to find a good way to use alembic for database changes needed by PlugIns as well.

    Scheduled Updates

    I want to schedule config updates – for example if you have a mesh you probably want to update the deepest nested nodes first and than the ones above etc. – this should be easy with celery (as long as the mesh is known and somehow represented in the database).

    DynaPoint Final

    Hi everyone,

    this is the final blog post about DynaPoint. Short recap: I created a daemon which regularily monitors the Internet connectivity and depending on that activates and deactivates the proper access points. That way the handling with APs would become easier as you already could tell the status by the AP’s SSID. I also created a LuCI component for it to make the configuration more easy.

    In the past weeks I was able to add some new features, fix bugs and complete the LuCI component. Especially the latter was really interesting and gave me some knowledge about how LuCI works.

    In the last post I mentioned that it’s better to verify Internet connectivity by using wget instead of just pinging an IP address.
    Consequently I switched from Pingcheck to wget. I also added an option to use curl instead of wget. With curl you also get the option to choose the interface for the connection.
    When you provide internet via VPN-interface you can explicitly check the connection of that interface now. The reason why I don’t use curl as default is because of curl’s rather large size. For some routers with only 4 MB of storage it might be too much.

    I also added an “offline threshold”, which will delay the switch to offline mode. So for example when you set the interval to 60 seconds and offline_threshold to 5, the switch to offline mode will be made after 5 cycles of checking (=300 seconds).

    So how does an example configuration look like?

    To use dynapoint just add dynapoint_rule ‘internet’ and dynapoint_rule ‘!internet’ in the desired sections in /etc/config/wireless:

    config wifi-iface
        option device ‘radio0’
        option network ‘lan2’
        option mode ‘ap’
        option encryption ‘none’
        option ssid ‘freifunk’
        option dynapoint_rule ‘internet’

    config wifi-iface
        option device ‘radio0’
        option network ‘lan2’
        option mode ‘ap’
        option encryption ‘none’
        option ssid ‘freifunk-maintenance’
        option dynapoint_rule ‘!internet’

    The configuration of dynapoint takes place in /etc/config/dynapoint:

    config rule ‘internet’
        list hosts ‘http://www.example.com’
        list hosts ‘http://www.google.com’
        option interval ’60’
        option timeout ‘5’
        option offline_threshold ‘3’
        option add_hostname_to_ssid ‘0’
        option use_curl ‘1’
        option curl_interface ‘eth0’

    All of that can also be configured via LuCI:

    If you want to try out DynaPoint for yourself please visit https://github.com/thuehn/dynapoint for more information.

    Future work

    Currently there is only support for one AP per state. In the next weeks I want to add support for multiple APs per state.
    Also I want to add support for more rules. At this time there is only support for one rule “internet”. I want to make this more generic and provide support for custom rules.

    Acknowledgements

    I want to thank my mentor Thomas Hühn for his excellent mentoring and great ideas during this project. 
    Also of course thanks to Freifunk for letting me realize this project and thanks to Google for organizing GSoC.

    GSoC: The ECE configuration system – summary

    The Google Summer of Code is almost over, so in this blog post I’ll give a overview over the targets I’ve met (and those I haven’t).

    Code repositories

    • https://gitlab.com/neoraider/ece/commits/gsoc2016 (daemon, client libraries, CLI client)
    • https://gitlab.com/neoraider/pkg-ece/commits/gsoc2016 (OpenWrt/LEDE package feed)
    • https://gitlab.com/neoraider/uci-ece/compare/gsoc2016-upstream…gsoc2016 (UCI ECE backend)

    All code in the first two repositories has been developed by me during the GSoC. The third link shows the work I’ve done to integrate a ECE backend into libuci.

    What is working

    As described in earlier posts, my GSoC project was a configuration storage system for OpenWrt/LEDE, trying to solve various issues of the UCI config system. The principal points of this new system are

      • ubus-based config daemon maintaining a central storage database file
      • JSON-based configuration data model
      • Validation based on simplified JSON-Schema

    The Wiki at https://gitlab.com/neoraider/ece/wikis/home gives a good overview of the design and the usage of ECE and describes many features in detail. The pkg-ece package feed can be used to build and install the different components of ECE on OpenWrt/LEDE easily.

    If you’ve worked with OpenWrt/LEDE, you probably know the UCI config system. A UCI config file looks like this:

    config system
            option hostname 'lede'
            option timezone 'UTC'
    

    This format is very simple: Each file (called “package”) has a number of sections (named or unnamed) of different types (this example from the “system” package has a single unnamed section of type “system”). These section contain options with single values or lists of values.

    Unnamed sections are usually accessed using indices, for example a command to set the hostname would look like this:

    uci set system.@system[0].hostname='betterhostname'
    

    With the simplicity of UCI, there come various issues and missing features; these are only a few of them:

        • The fixed data model (package/section/option) makes some kinds of configuration very awkward: In the example above, the index 0 must be given for the system section, but having a second section of this kind would not make sense. In other cases, deeper configuration trees must be flattened to be stored in UCI, making the configuration harder to understand
        • All values in UCI are strings, which often causes inconsistencies (booleans are usually stored as ‘0’/’1′, but several other pairs like ‘false’/’true’ and ‘off’/’on’ are supported as well; different users of UCI sometimes parse numbers differently)
        • UCI doesn’t have built-in validation. Frontends like LuCI usually validate the entered data, but as soon as the CLI client is used, no validation is done.
        • UCI always stored the whole configuration file and not only changes from the defaults, making the storage inefficient on overlay-based filesystem setups as they are common on OpenWrt/LEDE
        • In some situations, upgrades to default values should also affect the effective values; but only if the user didn’t change the values themselves. With UCI, this is not possible, as it doesn’t store the information if a value was changed by a user.
        • UCI allows comments in config files, but they are lost as soon as libuci or the CLI tool is used to modify it

    The configuration given above could be represented in ECE as this JSON document:

    {
      "system": {
        "hostname": "lede",
        "timezone": "UCI"
      }
    }
    

    Note that this is only the external representation of the configuration; internally, it is stored in a more efficient binary format.

    JSON gives us a lot of features for free: arbitrary configuration trees with proper data types. Existing standards and standard drafts like JSON Pointer and JSON Schema can be used to reference and validate configuration (the JSON Schema specification is simplified for ECE a bit though to allow more efficient validation on embedded systems).

    The command for changing the hostname would look like this in ECE:

    ece set /system/hostname '"betterhostname"'
    

    The quoting is currently necessary to make the string a valid JSON document; this may change in a future version.

    The whole configuration is saved in a single JSON document, but the specific format is not defined by a single schema; instead, each package can provide a schema, and the configuration tree is validated against a merged schema definition.

    The schemas also provide default values for the configuration. Adding documentation for the configuration options to the schemas is planned as a future addition and might be used to support the user in configuration utilities and automatically generate web-based or other interfaces.

    This gives only a small example for the usage of ECE, the abovementioned ECE Wiki contains much more information about the usage of ECE and the ideas behind it.

    In addition to the daemon and a simple CLI utility, I’ve developed libraries for C, Lua and Shell which allow to access the configuration. While there are still some features missing (some points for future work are given in https://gitlab.com/neoraider/ece/wikis/todo ), I think most of the missing pieces can be added in the near future.

    The UCI/ECE bridge

    When I proposed my project for the GSoC, I didn’t aim at making it a full replacement for the current UCI system, at least not in the near future. While the possibility to move some of UCI config files into the ECE config database had been my plan from the beginning, my ideas for backwards compatibility didn’t go further than a one-time import from UCI to ECE, and one-way generation of UCI config files from ECE.

    After talking to a few LEDE developers and package maintainers, it became clear to me and my mentors that many people are interested in replacing UCI with a better system in the not-too-far future. But for ECE to become this replacement, a real two-way binding between UCI and ECE would be necessary to allow gradual migration, so configuration utilities like LuCI (and many other utilities somehow interacting with UCI) don’t need to be adjusted in a flag-day change.

    An incomplete design draft for this UCI/ECE bridge has been outlined in https://gitlab.com/neoraider/ece/wikis/design/uci-bridge . The code found in the UCI ECE backend repository linked above implements a part of this bridge (it can load “static” and “named” bindings from ECE into UCI, and commit “static” bindings back to ECE) and has been implemented as an API- and ABI-compatible extension to libuci. The development of this bridge has taken a lot of time (much more time than I had originally scheduled for UCI compatibility features), as the data models of UCI and ECE are very different.

    Future work

    Of all points given in https://gitlab.com/neoraider/ece/wikis/todo , finalizing the database format is the most important, as any future change in the storage format will either break compatibility or involve some kind of conversion. When it is clear the format won’t be changed anymore, ECE should be added to the OpenWrt community package repository to make it easily accessible to all OpenWrt and LEDE users.

    After that, other points given in the TODO should be dealt with, but none of those seem too pressing to prevent actually using ECE for some software (but some of the points given in the first section of the TODO page would need to be addressed to properly support software that requires more complex configuration).

    Last, but no least, I’d like to express my gratitude to my mentors and all people in the OpenWrt, LEDE and Freifunk communities who have helped me develop ECE by giving guidance and lots of useful feedback, and to Google, who allowed me to focus on this project throughout this summer.

    DynaPoint update

    Hi everyone,

    here are some updates regarding DynaPoint. The idea was to create a daemon, which regularily checks the Internet connection and changes the used access point depending on the result. That way the handling with APs would become easier as you already could tell the status by the AP’s SSID.

    A daemon with basic functionality is already working. After installation, there is one configuration step necessary.
    You have to choose in /etc/config/wireless which AP should be used, if Internet connectivity is available and which one if the connectivity is lost. You can do that by adding “dynapoint 1” or “dynapoint 0” to the respecive wifi-iface section.

    You can also configure dynapoint via LuCI, although it’s not yet complete.
    I was struggeling a bit with it, because the documentation of LuCI is a bit minimalistic…
    Here is a screenshot of how it currently looks like:

    Next steps

    To verify Internet connectivity, it is probably better to make a small http download than just ping an IP address.
    Using “wget –spider” should be suitable for that.

    Also, I will see if I can get rid of the required configuration step in /etc/config/wireless in the next weeks and provide fully automatic configuration.

    If you want to test dynapoint for yourself, just go to https://github.com/thuehn/dynapoint.

    Google Summer of Code 2016: External netifd Device Handlers – Milestone 1

    OVERVIEW OF THE LAST WEEKS
    During the last 5 to 6 weeks I have implemented the possibility to include wireless interfaces in Open vSwitch bridges, rewritten a lot of the code for creating bridges with external device handlers and brought my development environment up to speed. I am now working with an up-to-date copy of the LEDE repository.
    I have also implemented the possibility for users of external device handlers to define what information and statistics of their devices look like and to query the external device handler for that data through netifd.

    CHANGES TO THE DEVELOPMENT ENVIRONMENT
    So far, I have been using quilt to create and manage patches for my alterations of the netifd code.
    One day they were completely broken. I still do not know what and how it happened but it was not the first time and this time recovery would have been way too tedious.
    This is why I switched to a git-only setup. I now have a clone of the netifd repo on my development machine that is symlinked into the LEDE source tree using the nice ‘enable package source tree override’ option in the main Makefile. I used the oppportunity to update both the LEDE source tree and the netifd repository to the most recent versions.
    Before, I was working on an OpenWRT Chaos Calmer tree, because of a bug causing the Open vSwitch package to segfault with more recent kernels.
    Now, everything is up-to-date: LEDE, netifd and Open vSwitch.

    MY PROGRESS IN DETAIL

    More Dynamic Device Creation
    In actual coding news, I have refined the callback mechanism for creating bridges with external device handlers and the way they are created and brought up.
    Previously, a bridge and its ports were created and activated immediately when /etc/config/network was parsed. Now, the ubus call is postponed until the first port on the bridge is brought up.
    Because of the added asynchronicity, I had to add a ‘timeout and retry’-mechanism to keep track of the state of the structures in netifd and the external device handler.

    A few questions have come up regarding the device handler interface. As I have explained in my first blog post, I am working on Open vSwitch integration into LEDE writing an external device handler called ovsd. Obviously, this is very useful for testing as well.
    I have come across the issue of wanting to disable an bridge without deleting it. This means bringing down the bridge and removing the L2 devices from it. The device handler interface that I mirror for my ubus methods doesn’t really have a method for this. The closest thing is ‘hotplug remove’, which using feels a bit like a dirty hack to me.
    I have reached out no netifd’s maintainer about this issue. For the meantime, I stick to (ab)using the hotplug mechanism.

    On the ovsd side I have added a pretty central feature: OpenFlow controllers. Obviously, someone who uses Open vSwitch bridges is likely to want to use its SDN capabilities.
    Controllers can be configured directly in /etc/config/network with the UCI option ‘ofcontrollers’:

    config interface ‘lan’
            option ifname ‘eth0’
            option type ‘Open vSwitch’
            option proto ‘static’
            option ipaddr ‘1.2.3.4’
            option gateway ‘1.2.3.1’
            option netmask ‘255.255.255.0’
            option ofcontrollers ‘tcp:1.2.3.4:5678’
            option controller_fail_mode ‘standalone’

    The format in which the controllers are defined is exactly the one the ovs-vsctl command line tool expects.
    The other new UCI option below ofcontrollers configures the bridge’s behavior in case the configured controller is unreachable. It is a direct mapping to the ovs-vsctl command ‘set-fail-mode’. The default behavior in case of controller absence is called ‘standalone’ which makes the Open vSwitch behave like a learning switch. ‘secure’ disables the adding of flows if no controller is present.

    Function Coverage: Information and Statistics
    Netifd device handlers have functions to dump information and statistics about devices in JSON format: ‘dump_info’ and ‘dump_stats’. Usually, these just collect data from the structures in netifd and the kernel but with my external device handlers, it is not as simple. I have to relay the query to an external device handler program and parse the response. Since the interface is generic, I cannot hard-code the fields and types in the response. This is why I relied once more on the JSON data type description mechanism that I have already used for dynamic creation of device configuration descriptions.
    In addition to the mandatory ‘config’ description, users can now optionally provide ‘info’ and/or ‘stats’ fields. Just like the configuration descriptions they are stored in the stub device handler structs within netifd where they are available to serve as blueprints for how information and statistics coming from external device handlers have to be parsed.

    For my Open vSwitch setup, it currently looks like this in /lib/netifd/ubusdev-config/ovsd.json:
    {
        “name” : “Open vSwitch”,
        “ubus_name” : “ovs”,
        “bridge” : “1”,
        “br-prefix” : “ovs”, 
        “config” : [
            [“name”, 3],
            [“ifname”, 1],
            [“empty”, 7],
            [“parent”, 3],
            [“ofcontrollers”, 1],
            [“controller_fail_mode”, 3],
            [“vlan”, 6]
        ],
        “info” : [
            [“ofcontrollers”, 1],
            [“fail_mode”, 3],
            [“ports”, 1]
        ]
    }

    This is how it looks when I query the Open vSwitch bridge ‘ovs-lan’:

    THE NEXT STEPS

    During the weeks to come I want to look into some issues which occurred sometimes when I disabled and re-enabled a bridge: Some protocol-realated configuration went missing. This could mean that sometimes the configured IP address was gone. Something which could help me overcome the problem is also in need of some work: reloading/reconfiguring devices.
    Along with this, I want to get started with the documentation to prepare for the publication of the source code.

    OpenWrt – poWquty (poWer quality): Computing and providing power quality information on OpenWrt routers.

    Dear Freifunkers,

    Please allow me to introduce the poWquty Project within Google Summer of Code 2016 at Freifunk.

    The big picture behind this project relates to the energy production and consumption. Sustainable energy production and consumption are crucial for a prospering life on earth. The importance of energy led many theorists to even define the level of civilization by its energy profile. With the renewable energies shift the energy production paradigm from centralized to decentralized energy production which poses one of the next big challenges, which will influence the energy production in the future years.

    From the big picture we move to the concrete case, increasingly faced when dealing with renewable energies: monitoring and control.
    The emerging smart grids include an inherent need for communication for monitoring and control purposes in a more and more dynamic environment. One of the major challenges is monitoring the power quality parameters in a decentralized manner. In such case, decentralized information need to be retrieved and transported with the lowest latency possible. One way to solve this challenge could be to install expensive infrastructure to each point. The better way is to use wireless mesh infrastructure that could also serve this purpose.

    Here where Freifunk comes in: The Freifunk mesh network is an outstanding example for a decentralized infrastructure that could be augmented with grid related functionalities to cope with future energy challenges. In order to use wireless mesh networks such as Freifunk for energy monitoring, we could use extra hardware that does the power measurements and use the wireless networks solely for transporting the information. The drawback of this is the need to install separate hardware. But, since all routers run on power, we could integrate the measurements into the router, which is the main goal of this project: to enable power quality measurements on OpenWrt.

    Here is the initial plan how to do this. First we need to retrieve voltage samples from the power grid. For the beginning we will rely on an oscilloscope device that delivers the real time samples over a USB interface. This way voltage samples from the electric socket are retrieved at the router. With these voltage samples we can go ahead and calculate the power quality parameters, using moving window algorithms, fourrier transform, and z-transform to get the phase angle, the effective power, the frequency, and the harmonics. This calculation should be time, and memory efficient since it has to run on the OpenWrt embedded devices. Once these values are calculated we need to think about how we want to make them available for retrieval over IP networks.

    Now we come to the Code: The goal of the project is to create an OpenWrt package which ensures three functionalities:
    1-    Retrieving sample data from the measurement device
    2-    Calculating power quality parameters form the retrieved samples
    3-    Provisioning of the calculated parameters for retrieval

    This project is intended to strengthen the role of open software in the uprising smart grids by providing some essential functionalities, communication devices need to have in the context of smart grids, especially in regard to the future role of the home routers in the future energy solutions.

    More updates on this will follow in the next weeks.

    Cheers,

    Neez

    GSoC: A new configuration system for OpenWrt/LEDE

    Hi,
    I’m Matthias (aka NeoRaider), and this year, I’ll participate in the Google Summer of Code for the Freifunk project.

    The goal of my project is to develop an alternative to the UCI configuration system, as UCI has a number of issues that make it cumbersome to use in some situations.

    One of the basic issues of UCI that affects many Freifunk (or generally community mesh) firmwares is the upgrade behaviour. Mesh firmwares usually contain elaborate default configurations, which set up network interfaces and other things to allow participating in a mesh without deep knowledge of the setup.

    But this setup needs to change from time to time, as the firmware is upgraded. In the Gluon firmware framework, we usually solve this by providing upgrade scripts which modify the configuration after flashing the firmware. Writing these script is often a tedious task, and the scripts easily break when the configuration differs too much from the expected one.

    But the ability to change the configuration is important for many Freifunk users: They want to change the role of ethernet ports, WLAN settings, and a lot more. But UCI doesn’t provide information how a setting was changed: if a script encounters an unexpected value, it can’t find out if it is an old default value, or was changed by the user. This often leads to a difficult choice for the script author: either to overwrite the value unconditionally, maybe disregarding voluntary configuration changes, or not to overwrite it, rendering communities unable to change certain configuration during upgrades.

    My project aims at solving this by saving the user configuration independently of the defaults provided by packages. This way, a package upgrade can change the default values, but explicit user configuration will not be affected.

    Another issue is that the upgrade scripts are usually part of the packages that bring the configuration. Removing a package will leave the configuration behind, which is usually a good thing (for users which know about this and may be interested in the the old config), but for mostly-automatic upgrades, old configuration may accumulate, which can quickly become problematic on devices with very limited flash.

    I plan to fix this by basing the new configuration system on schema definitions, which specify which configuration options and values are valid. The schema will probably be based on JSON, as there are already lots of existing systems for defining JSON schemas, which may be used for my project or at least serve as inspiration. This will also finally provide real datatypes for the configuration and make things more consistent (finally no more 1/true/on/yes/0/false/off/no booleans!). If we want too keep the package/section/option organization of UCI, or rather allow defining schemas for any JSON structure, is still a subject of debate.

    Instead of going into detail even more in this post, I’ll provide a link to my Gitlab project: https://gitlab.com/neoraider/ece

    Design documents, examples for configuration and schemas, and first code will start to appear there very soon Laughing