Tag: gsoc2017

netjsongraph.js – Google Summer of Code (GSoC) 2017 summary

Posted on by GeekPlux

Throughout the last three months, I was quite fortunate to work for Freifunk on netjsongraph.js under the guidance of my mentor Federico Capoano. Thanks for this invaluable experience that I learned a lot of knowledge and use them in a practical project. Here is a summary of the work I have done during the Google Summer of Code (GSoC) 2017.

Google Summer of Code project page

Goal

netjsongraph.js is a visualization library for NetJSON, a network topology data format. The main goal of netjsongraph.js may be concluded in below three lines (more details you can see in GSoC 2017-netjsongraph.js: visualization of NetJSON data):

  • Apply the modern front-end development tools and add tests workflow (#1, #45)
  • Rewrite it with WebGL (#11, #29, #39, #42, #47)
  • Improve the performance (#41, #44, #46)

Achievement

Github Repository : https://github.com/netjson/netjsongraph.js

Examples on GitHub pages: https://netjson.github.io/netjsongraph.js/

You can browse all examples on GitHub pages. Some screen shots of the application:
basic example
performance example
The force-directed layout is usually used to visualize network data. It offers insights on the relationships between nodes and links. The previous version of netjsongraph.js is implemented by d3 and it’s rendered using SVG. It would be very slow if there were thousands or ten of thousands nodes or links. So I have to embrace the WebGL speeded up by GPU to have a better performance.

I have recorded my work in the blog every milestone:

BTW, It’s a great management method to make members submit weekly reports and blog posts in Freifunk.

During the three months, there have been 116 commits from me. I created a big Pull Request include them:
netjsongraph.js #48
netjsongraph.js project panels
Almost all goals have achieved:

  • Published a minor version
  • Improved development workflow
  • Tests Added
  • Refactored visualization by Three.js and d3-force
  • Added more interaction like hover (show nodes tooltips), click (show nodes or links information panel), pan and zoom
  • Improved performance

Especially on performance aspect, it runs efficiently on Chrome reached 60FPS under 5k nodes and 10k links. And if you don’t wanna animation, you can choose the static rendering layout.

Challenges

I also encounter some challenges I never met before.

Event binding and handling

As you know, WebGL renders all objects in one canvas tag. How to bind events on every geometry? You should use the Ray casting. Raycasting is used for mouse picking (working out what objects in the 3d space the mouse is over) amongst other things. So you can know which geometry your mouse over and add some interaction effect.
There are thousands of objects and every object has several events you should handle, I had to develop an event controller to manage it.

Performance

The bottleneck in this visualizer is performance(#41). I tried many methods to improve it include:

Reuse geometry and material

However, the color of every node is different and the one link should highlight itself when it hovered, so the material should be independent and can not use in common.

Combine the mesh

Same problem with above. It’s not flexible to combine them to one mesh, different nodes and links should have different positions.

Static rendering

Make calculation before rendering, so there is no animation and repaint.

Using Web Worker

Web Workers is a simple means for web content to run scripts in background threads. The worker thread can perform tasks without interfering with the user interface. So put static layout calculation into it will be efficient.

Force-directed algorithm

There are different complexity and cost in the different force-directed algorithm. The Force-Atlas2 algorithm has some benefits over the force layout implemented in d3-force. So current version may be refactored by an advanced algorithm in the future.

What is left to be done

  • Add optional geographic map (#40)
  • Using Force-Atlas2 algorithm

More interactions and features should be added, and performance may be optimized by using new algorithm. I’d like to continue developing this project after GSoC.

In the end, thanks for the great patience and guidance from my mentors. Thanks for Google to provide me with this rare chance to contribute to an open source community together with awesome members from all over the world. I really appreciate this invaluable experience accumulated this summer and I believe it will have the profound impact on my career and life.

​

GSoC 2017 – Attended Sysupgrade – Final evaluation update

Posted on by Paul Spooren

Hi Freifunk,

This is my last post within this years GSoC. I’ll cover the progress, challenges and future of the project.

tl;dr:
Direct links to the work I’ve done this GSoC:

First a small reminder what the project is about: Enable end-users to update their routers to new releases or bulk update installed packages via a simple click in the Luci web interface. The magic lies in a server producing sysupgrade images with the same packages preinstalled as installed on the router. That happens on demand when triggered via the web interface.The image is downloaded securely via encrypted HTTP as the used browser has all certificates installed. The router does neither need certificates nor a correct running clock.

Progress

Server

At this point I’m very happy to announce a test (and usable) version of the attended sysupgrade setup. All created package recipes were accepted into the official repository and are compiled within the daily build cycle. The server runs fine obeying the described API at the Github page. Images are build within a few seconds if the request appears for the first time. The server stores previous requests and forwards to existing images instead of building it again. This reduces significantly the amount of build images due to the likely case of identical images being requested again. In addition some basic information are offered via the web servers status pages.

The server is implemented in Python3 using flask for request routing and template rendering. While that’s only the tip of the iceberg behind the server lays a rather complex PostgreSQL database validating requests, checking for package changes or transforming packages when updating to another main release. More on the transformations later.

Images created on the server. A click on manifest shows all installed packages with version. All snapshot builds are deleted midnight UTC.

To try the the server at https://betaupdate.libremesh.org have a look at the demos I prepared. Continue reading “GSoC 2017 – Attended Sysupgrade – Final evaluation update” →

GSoC 2017 – Spectrum Analyzer updates (2)

Posted on by nicopace@gmail.com

This is the second update of the spectrum analyzer project.

I’m glad to say that we have a visualization that is allowing us to view the output of the card in a meaningful way.

After polishing this interface I will create the waterfall and the cumulative views that are useful for this.

I will leave you a screenshot of how it looks right now (it will get better soon).

Also, we have started conversations with a University that has done an MVP of a PAWS database. The purpose of this is to gather information on the spectrum usage in the TVWS frequency.

With the help of the TVWS Frequency Shifter that Elektra is developing, we will be able of doing site surveys on those frequencies.

This will allow any OpenWRT device with an ath9k radio and a frequency shifter and a GPS to act as a surveyor of those frequencies.

Also had the chance to support the LibreMesh project by adding a Continuous Integration server that will be soon merged to the master branch. That will allow the project to keep on safely by having tests that ensure that the code does what it says it does.

Easy ubus Daemons with rpcd

Posted on by nicopace@gmail.com
Libremesh /etc/banner

In this article we will see how to create a simple ubus daemon with RPCd.

Ubus is an amazing piece of software that allow inter-process communication inside and in between nodes.

For inter-process communications it uses a common bus with synchronous and asynchronous calling, publish/subscribe and event paradigms.

For in-between nodes communication it provices a JSON-RPC 2.0 Server (and I hope will do WebSockets soon!).

For more information on how ubus works check out OpenWRT’s Wiki page.

If you want to expose simple functionality to ubus (or you are prototyping an interface before creating an efficient daemon) you can use rpcd for this.

This is a simplified example based on the one in the wiki

$ cat << EOF > /usr/libexec/rpcd/banner
#!/bin/sh

# The RPCd interfaces with commands via two methods: list and call.
case "$1" in
	list)
                # List method must return the list of methods and parameters that the daemon will accept. Only methods listed here will available to call.
		echo '{ "append": { "content": "str"}, "show": { } }'
	;;
	call)
                # The way rpcd calls the methods is by calling your script like this: <script-name> call <method-name> << <json-input>
                # So, in order to retrieve the called method, you need to read $2
		case "$2" in
			append)
				# And in order to retrieve the method parameter, you need to read stdin
				read input;
                                CONTENT=`echo $input | jsonfilter -e '@.content'` 
                                echo $CONTENT >> /etc/banner
				echo '{ "result": "ok" }'
			;;
			show)
				# return json object or an array
				echo -n '{ "content": "`; echo $(cat /etc/banner | text_to_json); echo '"}'
			;;
		esac
	;;
esac

function text_to_json() {
JSON_TOPIC_RAW=`echo $1`
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\\/\\\\} # \ 
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\//\\\/} # / 
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\'/\\\'} # ' (not strictly needed ?)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\"/\\\"} # " 
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//   /\\t} # \t (tab)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//
/\\\n} # \n (newline)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^M/\\\r} # \r (carriage return)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^L/\\\f} # \f (form feed)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^H/\\\b} # \b (backspace)
echo JSON_TOPIC_RAW
}
EOF
$ chmod +x /usr/libexec/rpcd/banner
$ service rpcd restart

And then we can try the new daemon:

$ ubus -v list banner
'banner' @xxxxxxx
    "append":{"content":"String"}
    "show":{}

$ ubus call banner show
LEDE blah blah
...

$ ubus call banner append 'Hello world!'
LEDE blah blah
...
Hello world!

Update:
@Rundfreifunk comment suggested that if you want to use this RPC commands via JSON-RPC, you need to add authentication parameters.
In this case, let’s suppose that we want to grant anonymous read access to the banner.show method.

We could do that like this:

$cat << EOF > /usr/share/rpcd/acl.d/banner.json
{
  "unauthenticated": {
    "description": "Alfred data types via batman-adv meshrouting",
    "read": {
      "ubus": {
        "banner": ["show"]
      }
    }
  }
}

The documentation related to ACLs can be found in this OpenWRT wiki page.