PowQuty Live Log GSoC 2017 Final Update

This is the last blog entry in the series of Google Summer of Code project updates. It will describe, what has been done and what is left to improve in the future in the PowQuty project.

Powquty

PowQuty is a power quality monitoring tool, which can be installed on a router running LEDE or OpenWrt. The router can be connected to an USB oscilloscope providing measurements which powqutyd will process and provide to the user in human readable form.
All this was tested on a x86 based LEDE router.

GSoC 2017

During this Google Summer of Code a live log functionality was added to PowQuty to provide information on power quality events. These events are:

  • voltage dip of 10% – 90% of the reference voltage on the measurement signal
  • voltage swell > 110% of the reference voltage on the measurement signal
  • voltage dip < 10% of the reference voltage on the measurement signal
  • > 5% of the measured values of one specific harmonic are over the defined threshold
  • On event occurrence important information like time, duration and event type will be written to a log file and presented in the extended luci app.

    As shown in the above picture, the interface provides a traffic light like color system behind these events, green indicates everything is within the EN50160 power quality norm. Yellow means, that 80% of the maximum time per week is already reached, red means, that the norm was violated during the last week.
    In addition to log writes, notifications are send out with Mosquitto. Mosquitto is a message broker using the MQTT protocol. It provides a publish/subscribe model, which allows a central server to subscribe to a topic and clients to send out messages to the server with a topic. Mosquitto was already in use in powquty but was extended for EN50160 event notifications. This will allow a central logging of bigger power supply networks, monitored by multiple devices.

    As another option Slack messages can be send by powquty now. Slack is a messaging program, using (as one option among many) webhooks for interaction. Everyone with the webhook can send messages to the team. Sending out messages allows a user to react quickly to changing situations, or get immediately informed on power event occurrence.

    Beginning with pull request 20 [https://github.com/thuehn/PowQuty/pull/20] I started to implement these features.
    First an option was developed to read measurements from a file, as most power supply networks are pretty stable and wont provide many opportunities to test event handling in powquty.
    Afterwards slack and MQTT notifications where added.
    During testing of mosquitto event messages, some seemed to be lost on intervals with many En50160 events in a short period(sometimes more than 35 events per seconds). The solution seems to buffer all events before sending.
    Something similar happened with Slack. Slack only allows one message per second(short bursts excluded) [rate-limits].
    Buffering events would resolve this problem as well. An option for live email notification was considered at first, but was dropped as spam protection would stop most of the messages and probably list users as spammers.
    The last step was to add the traffic light system to the luci app, to enable users without knowledge of the norm to get an idea of the power quality of their power supply network.
    In addition a slack library was written [libwebslack] to send slack messages from PowQuty.

    What can be improved
  • As mentioned before event buffering is needed and will be added after GSoC
  • Email notification in form of a weekly summary
  • More Error checking and handling
  • improving libwebslack to not use libcurl to reduce its size
  • provide libwebslack as OpenWrt/LEDE package, for easier future use
  • Finally I have to thank Dr. Thomas Huehn for being my mentor and Freifunk for their work they do and especially for being a mentoring organisation for Google Summer of Code.
    Last but not least I would like to thank Google for making this all possible.

    If you want to review some of my earlier posts:

  • Introduction
  • First Update
  • Second Update
  • Best regards
    Stefan

    PowQuty Live Log Second Update

    It’s been nearly a month since my last update on the PowQuty Live Log project and i would like to tell you, what has been done
    so far and provide information on what will be done in the next month.

    PowQuty got updated, to support Slack and mqtt event notification and can already be used in the current PowQuty version.
    In addition to this, there have been some bug fixes during the last month and some new features were added.
    On event occurrence the event gets stored in a csv file and each entry is displayed in the luci-app. To increase the usability,
    a traffic light system will be added, which will show for each event type its occurrence time and show if the current values
    are in violation of EN50160.

  • Green: Everything is ok, no violation
  • Yellow: Close to a violation
  • Red: This event time is in violation of the norm
  • The event messages contain a times stamp, the duration of the event and updated event Type information, as well as event type related
    information and GPS data.
    .

    As receiving notifications or emails on every event occasion can get noisy, we decided to provide a weekly summary of the events in
    addition to the regular notifications.
    The user will be able to decide if he wants to receive this summary, every event, or both. We consider using the traffic light system
    here as well, to increase the readability and enable users to understand the quality of their power supply network, without a lot of
    knowledge of the EN50160 norm.
    We discussed individual intervals and keep it in mind as a possible later feature.

    Best regards
    Stefan

    PowQuty Live Log First Update

    As mentioned in my previous blog post, i am going to add a live log and notification system for
    certain events to the power monitoring tool PowQuty. The first steps have been done and the
    configuration has been extended.
    Three types of notifications have been added to the configuration options during the first month of coding.
    Namely email, slack and mqtt. Mqtt was in use before, but was extended to allow a second host and topic for
    the power quality events.
    The powquty configuration page was redesigned to use a separate tab for each notification option
    to increase overview.

    The old configuration page would have been very crowded with all the new options
    The new configuration view with mqtt tab open

    Power quality events, that cause a notification are:

    • Voltage dip between 10% and 90% of the reference voltage of 230V
    • Voltage swell above 110% of the reference voltage of 230V
    • Voltage dip < 10% of the reference voltage
    • More than 5% of the samples of one harmonic are above the threshold

    As the power supply network in Berlin was not willing to provide such events an option for test measurement
    input was needed. A file read flag for powqutyd was added and needs a little bit of clean up
    before a pull request on the upstream powqutyd.
    The library for the USB-oscilloscope provides the number of EN50160 events per measure cycle and
    the kind of each event. As of now some basic slack notifications are added, which provide the event
    type and the event start time from measurement start in milliseconds to the channel and team set
    in the luci web interface or under /etc/conf/powquty.

    Slack notifications with start time in milliseconds relative to measurement start, probably will be UTC or local in the future

    In the notification the type of event is provided to allow the network administrator to react directly
    to the changes, without to check the log any further.
    The other notification options will be added and tested soon.

    PowQuty Live Log introduction

    Hi everyone,

    I’m Stefan and happy to present my GSoC 2017 project to you.
    That is extending the PowQuty daemon and luci web-app with an event log and notification system for events violating the EN-50160 norm.
    Probably most of you don’t know powquty. Therefore i will give a short introduction to it, before I present my actual project.
     

    PowQuty

    PowQuty[1] is an application for measuring power quality parameters of the power grid like voltage and frequency. It receives samples by a USB connected oscilloscope and writes the calculated values to a log file. This log can be read by the powquty luci web-app. The web interface will plot the measurements directly into several graphs. These are power over time, frequency over time and metrics over time.
    It was implemented during the last GSoC by Neez El Sayed [2] and its intention is “to strengthen the role of open software in the uprising smart grids by providing some essential functionalities”.
     

    PowQuty Live-log

    The goal of this project is to provide an event log of measurements that are outside defined parameters.
    Public power distribution networks have to meet certain criteria. These are formulated in the EN50160 norm.
    If there is a power drop, or a frequency spike PowQuty will write this to its log like every other value. The user can examine the log file or its graphical representation for critical values, but essential equipment could already be damaged.
    If such an event occurred, a user or power network administrator should be notified. This event log will get a graphical interpretation in the luci powquty app as well to enable a better tracking of events.
    This will allow users of PowQuty to react quickly to changing conditions in power supply networks.

    Therefore we will add a notification system to PowQuty as well as additional logging options.
    To collect events at a central server we will extend the mqtt usage to send an mqtt notification on a violating event.
    In addition we will add an email and slack notification option to allow quick responses.

     

    [1]: https://github.com/thuehn/PowQuty
    [2]: https://blog.freifunk.net/author/Neez-El-Sayed/