GSoC 2014 Final – OpenWrt: IEEE 802.1ad VLAN support

Hi all!

Because I have worked very hard in the first part of GSoC, the implementation was almost done already for mid term, in the second part I have been mostly testing the code, and taking advantage of it in a lot of setups 🙂

The GSoC experience have been very formative to me and I would like to repeat it next year either as student or mentor 🙂

Moreover I’ll suggests to apply to GSoC to a lot of friends!

Many thanks to Freifunk to chose my proposal I hope you will take advantage of 802.1ad too 🙂

Cheers!

GSoC 2014: Hardware Detection [ Final ]

 

Hi everyone!

I am very happy to have participated do GSoC 2014, this experience have permitted to me to learn a lot about opensorce comminity and programming, i have learned also Lua programming language, while it seemed a little ugly  to me in the firsts times I ended up loving it.

This GSoC project permits to people installing Libre-Mesh on their devices and have device specific quirks already solved by the hardware detection module, without user intervention, while it permits to developers to write little modules to easly supports new hardwares and solve their quirks.

In particular in the second part of my GSoC I have improved hardware detection, in particular I have created a module to autoconfigure TL-WDR3600 and a module that permits to Libre-Mesh to detect wan port of a lot of routers taking advantage of the OpenWRT infrastructure.

While creating those new modules I have also realized how to improve general Libre-Mesh infrastructure and committed various improvement to the hardware-detection branch, that is now in the official repository ready for merging in develop branch.

Futhermore, during the last phase of this project I have optimized the code including modules I have written in the first period like usbradio detection module. 

Obviously all this work have been possible thank to the help of the community that helped me in the whole coding period.

Best Regards 😉

GSoC: nodewatcher v3

Hello all!

Thanks to this year’s GSoC, a lot of work has been done on nodewatcher v3 platform. It now has a better, modular monitoring agent that can run on OpenWrt-supported devices, with a new JSON-based output format that can be easily reused by other projects as well. The platform has been ported to the latest stable version of Django (1.6) together with all migrations and dependencies. Development environment setup now uses Docker and fig in order to make it very easy to dive into the code without having to battle with various dependencies.

The API for access to node configuration and monitoring data (registry API) has been much improved, with better, more usable querying capabilities and performance. During development we have discovered a bug with cascade deletions and  polymorphic models in Django. Node configuration editor based on the registry API now supports references between form models that have not yet been saved — this functionality enables configuration of bridge interfaces which are now also supported by the firmware generator. I have implemented type support in the datastream library for long-term monitoring data storage with a new type for storing graphs as datapoints. This enables nodewatcher v3 to use datastream to store how the network topology evolves over time.

All the code is available on GitHub in several repositories:

GSoC: Retroshare social network plugin review and future

During Google Summer of Code i did these things:
– learn what users expect from a social network. Figure out who should receive which messages.
– map social network functions on the existing Retroshare General Exchange System
– build a backend with basic features
– build a frontend with basic features
– started a new programming interface on top of libretroshare: a JSON over http interface

The Secushare homepage says: “RetroShare should provide more social functionality” Indeed! Every country is spying on their citizens. Storage, computing power and bandwidth became cheap. These are good conditions to build a distributed social network. The first step was already started a while ago with the General Exchange System for Retroshare v0.6. This project is the second step in this direction. One thing is missing: a release for normal users.

The backend and frontend made during GSoC can display a wall with avatar image and profile text. It is possible to create posts and reference them on walls. Posts can be commented. Read more details in the previous blog post. Still it is not suitable for daily use. There was not enough time to implement a scalable user interface architecture. The web interface can’t handle more than 100 posts, because pagination is missing. Whats more the UI is filled iterative. This causes many updates to the html tree with high cpu load. It would be better to send all information to the browser in one piece. The browser would then only have to update the HTML tree once. The design and layout should be improved to highlight the content and to get the date label out the way. (See the this screenshot.)

Future

Wt is nice because it allows to make a web interface without touching HTML JavaScript and CSS. But this is only the half truth. There where issues:
– completely destroyed layout by setting the image size
– menu bar was horizontal instead of vertical, this required to manual set a style class from C++
– WTimer stops working. I had to build my own server side timer

These issues where solvable. A real pain with Wt is the layout and UI design testing. You first have to go through the complete compile and start cycle to see changes. What if you missed a closing tag in a HTML template? You have to recompile and restart. Now i saw that with real web technologies you can see a live preview while you type. I think when doing layout and design it is important to immediately see the result. This is not possible with Wt. Conclusion: you can build a web interface with C++. But hard coding CSS class names and embedding HTML snippets in C++ is a pain that should be avoided.

On the other hand there exist advanced tools and frameworks for web development. AngularJS is a very nice JavaScript library. It offers data binding from JavaScript to HTML: you update a JavaScript object, and Angular updates the HTML. JavaScript objects are more flexible than C++ objects. You can even make a JavaScript object out of a JSON file. Angular makes handling of button clicks easy. AngularJS makes  Bootstrap offers a set of widgets with HTML example code and style sheets. Now i want to use other web frameworks instead of Wt.

I did not know about these web technologies when starting the project. This is some sort of a chicken-egg problem: Retroshare does not have web developers, because it doesn’t have a web interface. And there will be no web interface until a web developer can show how it can be done. Gladly there is now a web developer who can teach me how to make a web interface.

The new idea: make a web interface using web technologies

The next goal is to make a clean and easy to use JSON over HTTP api for Retroshare. Then web developers are free to make a nice web interface using their favorite frameworks. This api is not only useful for web interfaces, but also for scripting. You could then send a chat message from a shell script using curl:

curl -X PUT -d ‘{“msg”:”hi all<br/>(send from bash)”}’ http://localhost:9090/api/chatlobbies/<id>

What i will do now
I will do more research for a JSON over HTTP interface for Retroshare and rssocialnet. If this stays a good idea i want to implements it. Maybe i will try to start a new web interface. But i hope someone else is faster than me. I prefer to work on the rssocialnet backend and libretroshare. Whats more a web interface is a perfect place for new contributors. If i would make a clearly documented JSON over HTTP api, then a web developer would not need C++ knowledge or experience with libretroshare.

A dream is to bring Retroshare to Android. This is possible, but we have to make a new touch-friendly user interface and we have to optimize Retroshare to make it more battery friendly. I currently see two ways to build a GUI for Android: with QML/QtQuick or HTML based with the Ionic Framework. There is already a QML prototype.

To get rssocialnet ready for daily use, i need your help. Unfortunately I’m again in a research/planning phase. This means I’m not sure how the JSON over HTTP interface will look at the end. It also means if you tell me “i want to help coding”, i have to disappoint you because i don’t know what we have to code and how to code it. Anyway, here are some things you can think about:

– which features are important for a social network? read some ideas
– can you make a better gui mockup than me?
– which frontend technologie should be used? QML/QtQuick or Bootstrap and AngularJS?
– what are the requirements for a first release?
– how can you use your skills to help?

Thank you Freifunk and other mesh network communities for donating one GSoC slot to Retroshare. This was a good decision, it made it possible to start a mesh friendly social network application.

GSoC: Features of the Retroshare social network plugin

Content is obviously the most important element of a social network. Currently only support for plain text is implemented. The content can have an author, but this is not required.

Future: It would be nice to have support for images. This is very easy on the backend side, but it needs a frontend which scales the image to fit on the screen. With Retroshares file transfer capabilities it would also be possible to publish large files like audio and video files. It would be nice to restrict access to content to a set of people. Retroshare is prepared for this and it only needs small changes in the social network backend. Of course this requires a user interface to sort people into circles and to select circles.

Content alone is useless without a place where it gets displayed. As explained in a previous blog post, every piece of content is stored for its own. To make content visible it has to get referenced on a wall. This happens automatically on the own wall when creating a new post. It also gets triggered by clicking the share button. A reference always has an author.

Future: maybe allow to reference content from other services. For example if Retroshare gets a Photo Share service, allow to reference a picture or photo album on the wall.

A Wall is a place where a profile text, an avatar image and references to content are stored. A user subscribes to another user to download all posts referenced on the wall. The wall owner and others can reference content on a wall.

All new posts are collected and displayed in the news feed. A news feed shows the new content, the comments and how others interact with the content. Who commented this post? Who shared this post? Currently the news feed displays posts in the order in which they where received.

Future: it is probably desired to have a more advanced logic to sort news feed entries. Imagine a user comes online after a week and gets bombed by hundred new posts. It would be possible to sort news in two dimensions: topics and rating. Example: have one tab for content from close friends, and one tab for other content. Then calculate a score to display more important content at the top. This requires a bit of backend work, but it is doable.

Users can interact with content in two different ways: they can comment it, and they can share it. Sharing creates a reference to the content on the own wall, and thus forwards the content to friends. Comments are stored with the content, so everyone who received the content will also receive the comments.

Future: one can think of other ways to interact with content. Examples are like, bookmark, vote and hide. In general these interactions are each a form of tagging. For the backend it does not make a difference if content is tagged with “GSoC14” or “like” or 3.1415. This is more a matter on the frontend side: which meaning does the tag have for the user? How does the frontend show different tags? (star, heart, thumbs up, plus sign, text, …) How can the user filter posts with specific tags?

There has to be an entry point to let the user see the people around him. If the user recognizes a known person he might want to subscribe to this person. For now there is a widget to display all identities with their name and avatar image. Of course later this list should get filtered to fit on the screen. Retroshare circles could be used to make friends lists accessible to friends. This would allow automatic circle intersection to search for people the user probably knows.

Below is screenshot of the Retroshare social network plugin.

[GSoC-2014] Final report of the GSoC project: “BGP/Bird integration with OpenWRT and QMP”

Here I present you a report of the finals state of my GSoC project. For further information feel free to  contact me using the channels described in the github and documentation.

“BGP/Bird integration with OpenWRT and QMP” [0] project’s main goals were to improve Bird4/6 Daemon [1] adding a better integration with OpenWRT bringing UCI configuration to it, to add an user-friendly interface to make it easier using the LuCI web-framework, to be able to port it to QMP mesh networks and, finally, to automate the route exchange and metric translation between Guifi.net (BGP) and QMP (BMX6) [2].

Current solution consists on two OpenWRT packets: bird4/6-uci and bird4/6-luci. While bird4/6-uci allows the user to modify Bird’s configuration and apply it using the init.d script, the bird4/6-luci package brings a web interface to make this UCI configuration even easier.

Regarding bird4/6-uci package, UCI configuration scheme was agreed with Bird main developers owing, not just to make a solution, but also to consensus its development and characteristics with their main developers. The package includes a DOCUMENTATION file with all the available options, its description and examples.

Regarding bird4/6-luci package, it brings all the necessary files to add LuCI web-based configuration interface and has bird4/6-uci as a dependency.

Finally, the solution used to automate the translation and exchange of routes between BGP and BMX6, uses Bird filters instead of an external developed tool:

First of all, as BGP routes are automatically exported and imported only using UCI configuration, the efforts were put into the reverse part. Second, initial experiments were done in the WiBed platform [3], owing to be able to repeat and test the solution without the possibility of “breaking anything”. Once the solution was stable enough, packages were installed in a QMP mesh with 5 nodes (2x WDR4300, 1x WDR3900, 1x WRTNode and 1x WR703N) and also connected with a Mikrotik RouterBoard 750G to check the routes exported. Moreover, some tests were made connecting the RouterBoard to Guifi.net’s UPC point, working with more than 500 routes.

Example of original Bird configuration:

log “/tmp/bird4.log” all;
debug protocols all;
 
#Router ID
router id 10.1.26.50;
 
#Secondary tables
table aux;

Example of the same configuration using UCI:

config global ‘global’
    option log_file ‘/tmp/bird4.log’
    option log ‘all’
    option debug ‘all’
    option router_id ‘10.1.26.50’

config table
    option name ‘aux’

An example of the LuCI configuration web page can be seen here:

Bird BGP LuCI configuration example

Example of BMX6 Routes and how are they filtered:

# ip r show
10.0.0.0/8 dev bmxOut_HW-Ermi  proto static  metric 1024
10.1.32.0/27 dev bmxOut_HW-Ermi  proto static  metric 1024

The pattern used in IPv4 filters is the device name “bmx*” and also the metric “1024” owing not to repeat or export internal routes.

In IPv6 the procedure used is to filter the 60 kernel table, as it contains all BMX6 iroutes:

# ip -6 r s table 60
fd66:66:66:8:de9f:dbff:fe35:17b6 via fe80::de9f:dbff:fe34:17b6 dev wlan0.12  proto static  metric 1024
fd66:66:66:a:de9f:dbff:fe34:17b6 via fe80::de9f:dbff:fe34:17b6 dev wlan0.12  proto static  metric 1024

Future work:

  • Continue adding the rest of BGP options to improve the solution.
  • Add OSPF (first of all) and the rest of the protocols to the UCI and LuCI solution.
  • Send the bird4/6-uci/luci package to OpenWRT willing to became an official package.
  • Continue giving support to package users and maintaining it.

Both package repositories are actually in my personal Github account [4] and [5].

Finally, I want to  thank Freifunk for the opportunity given to me with this GSoC project, to my mentors Roger Baig and Axel Neumann, to Pau Escrich for his support during the project and to Guifi.net and QMP project and their communities for the support received.

Eloi Carbó Solé.

[0] http://blog.freifunk.net/2014/gsoc-bgpbird-integration-openwrt-and-qmp-project-report

[1] https://github.com/openwrt-routing/packages/tree/master/bird

[2] http://qmp.cat/News/12_Google_Summer_of_Code_2014_and_QMP

[3] http://wiki.confine-project.eu/wibed:start

[4] https://github.com/eloicaso/bird4-openwrt

[5] https://github.com/eloicaso/bird6-openwrt

GSoC: Netengine project

Here it is my second blog post for the Mid Term Evaluation of GSOC2014.

As I wrote in the previous one, I’m working on Netengine, a Python module to abstract network devices and get information from them.

The work is going very well, I’m learning new things every day with the help of my mentor, Federico Capoano, and I’m very happy with the development.

In this first part of the work we completed as per goals, the SNMP back-end for AirOS and OpenWRT firmwares.

The most difficult part of this first part was to work with SNMP (Simple Network Management Protocol) because I had never worked with it, so I had to learn it’s basics and how it works, in particular it’s way of retrieving info from devices.

It uses different codes (MIB), everyone of it gives access to different information of the device (e.g device name, addresses, interfaces); so before starting to write I had to look for the correct MIBs to query.

Now we are focusing on the ssh.OpenWRT back-end ready to switch the next one on the list once completed.

I’m definitely very happy with how the work is going, with the communications I’m having with my mentor and with all coding practices I’m learning from him.

The program gave me not only the possibility to improve my skills but also to meet new people very experienced on the field.

The next step is to start coding on the new back-end, probably an HTTP back-end for AirOS, to complete the program in time.

For further questions on the project please visit https://github.com/ninuxorg/netengine or email us at ninux­dev@ml.ninux.org.  

GSoC “BGP/Bird integration with OpenWRT and QMP” project report

This entry is for updating the information regarding this GSoC Project focused on the automate of BGP-BMX6 metrics and routes exchange and “translation”. 
 
During the WBMv7 in Leipzig, the WiBed platform [0] was presentated as well as the GSoC project[1] where I am participating. WiBed was used to deploy the testbed network where many routing and mesh related experiments were executed.
I was participating in the deployment and development teams of the WBMv7, so I contributed in many bug fixes and platform improvements.
The WiBed project is important for the GSOC because it provides a testing platform very similar to the production environment where we will apply the results of the GSOC project.
 
Currently I am studying and understanding the basis system we need to accomplish the objectives of my project. I am working on Bird4 and Bird6 [2] configuration transition to UCI[3] and LUCI[4] willing to give to the OpenWRT project and community a more user-friendly Bird Daemon configuration. For those who do not know, Bird is a lightweight and flexible BGP daemon which may be used as an alternative to Quagga (which is actually very heavy).
 
Once we got an usable version, we wll upload the work done to public GIT repository following the standard OpenWRT feeds format (so it might be included in the official repositories). To test our advances and implementation we are using the WiBed platform network deployed in our laboratory (at UPC university in Barcelona).
The first production test will be made in the QMP [5] network we have deployed in Barcelona.
 
The most challenging feature in our project is the exchange of routes (and associatd metrics) between routing protocols (BGP and BMX6 [6] in our case). We (me, the workgroup and mentors) are still discussing about the different ways to implement it and how to use the Bird solution to Guifi.net [7] where the main protocol is BGP and the most common OS is the privative routerOS from Mikrotik. Including Bird in the open/libre firmware QMP will allow people in Guifi.net to use this solution instead of routerOS. However to make the interconnections between the QMP (Mesh) networks and the current BGP/Infrastructure we need the called frontier (or border) nodes (those who exchange the routes between both network clouds). 
 
To not overload the current mesh clouds (running with BMX6), we will install a BGP Bird instance only in these QMP border nodes. They will exchange routes and metrics in the entrance of the network and summarize the result by publishing the aggregated routes to each network.
 
Another idea we are considering is to create a very small and simple OpenWRT image with the BMX6 daemon ready to perform the routing. This image may be installed as a virtual machine in the RouterOS firmware (present in 50-60% of the Guifi.net nodes). So Mikrotik nodes will be able to route BMX6 packets thus the BGP instance will not be longer necessary (we believe mesh routing protocols are a much better option than BGP/OSPF for a WiFi network). This approach is compatible with the (previous) border nodes one. We will provide both options to Guifi.net users to let them decide.
 
Finally, to conclude this  mid-term report, say that we expect to finish the project in time and  just mention that in the coming days we will start testing the first solution in a real production network.
 

GSoC: Freifunk API Query Client – A short report from the Wireless Community Weekend 2014

Photo taken from https://twitter.com/christianheise/status/472746947569520641

Photo taken from https://twitter.com/christianheise/status/472746947569520641

From the 29th May to the 1st of June we were with Andi and Bernd from Weimarnetz at the wunderful c-base Raumstation. We visited the Wireless Community Weekend. It was my first experience of this kind of community event and I enjoyed it very much.

Beer and Bratwurst did harmonize quite well with technical talks about OpenWrt and the Freifunk Community. I was especially surprised how diverse and open the community is and how enthusiastic everyone involved was.

Andi and Monic talked about the progress on the Freifunk API and presented their work. At the end of their talk I had the chance to present my work on the query client for the API. Here are the slides.

Shortly after the talk JĂŒrgen Neumann from Freifunk Berlin came to me and introduced me to DeepaMehta. My original plan was to use something like NodeJS for the backend the storage of the API data but DeepaMehta looked promising and offers unique features I didn’t even thought off.

So after talking with Andi about it we decided to use DeepaMehta as the foundation and storage tool for the API data. A seperate blogpost for the GSoC midterm evaluation will outline my work in this direction.

Overall it was a very exciting weekend for me at #ffwcw 2014.

GSoC : Source-Sensitive Routing

The first weeks of my Google Summer of Code project were a little complicated, as I still had exams at university, and I was not really aware of what mesh networks were about.  I also needed a little time to get to know network and routing better, both pratically and theorotically.

At the beginning, in order to understand quagga and babeld, Iread a lot of documentation on the routing topic including theoretical papers and some RFCs, while also browsing the code of both babeld and quagga.

On the other hand, I have been able to experiment mesh routing with the mesh network available at university.  In order to use that network from home, and being able to test my programs at all time, I also established a VPN connection between university and my home computer. By doing so, I can connect to the university’s babel network at any time. I have also been able to understand the functionning of quagga and zebra and to install source-sensitive static routes on a mesh network.

After having spent much time reading the codes of babel, quagga, and babel in quagga, I achieved to use the zebra’s API in babeld and began to add support for source-sensitive routing in babeld. Currently, my code runs and segfaults proudly ! I hope to see the first results of source-sensitive routing with my version of babel, in the worst case, at the end of the week.

At first, my goals were not really clear, but now, I have precise objectives on the short, middle, and long term.  In brief, my short term objectives would be to get a source-sensitive routing Babel running by the end of the week.  After getting a working version of Babel source sensitive, I will implement the same work in RIPng. RIPng is a quite simple protocol and Juliusz and Matthieu told
me it would be a good idea to offer it source-sensitive routing.  And finally, after everything will be tested and running fine, I will be implementing the source-sensitive commands in Babel.  Then, I will complete the documentation about my work. And in the end, the ultimate goal would be to be included in the official repository of Quagga.

If you want more details about the work I did, you can read my blog here : http://ariane.wifi.pps.univ-paris-diderot.fr/~olden/. I posted an entry every week to keep you informed of the progress on the short term.