Tag: GSoC2014

GSoC: Work on Freifunk API Query Client will go on

Posted on by Martin Tippmann

This is the final blogpost for my GSoC project for the Freifunk-API Query Client.

Goals
 
We want a comfortable tool to query all the Freifunk API files as there are nearly 100 communities all over Germany providing their data. There are already several applications like our community map, a common calendar, our feed aggrator or the community podcast collector. But it’s still hard to find communities by properties like routing protocols or focus topics.
 
Challenges
 
When we began this project we only planned to query the generated JSON data for the community in a browser and additionally provide query results via a webservice. But then we talked to several people and we heard about DeepaMehta with features like connectors to OpenStreetmap. So we did something what you don’t do normally: We changed our project goals before the midterm evaluations.
 
DeepaMehta is not just another database product, it provides a different way to store and handle data. It uses a graph to store connections between items and allows to modex complex datatypes and associations between them. We had to change our mind and had to learn a new kind of thinking. The API data is constantly evolving and changing and there a lot of cross-references in the data e.g. links to various nodemaps. We think the switch to DeepaMehta is useful because we can query the graph and add new relations and data without problems.

 
It’s difficult to handle different spec versions if you want to query all API files, because some fields changed, other fields were added to the specs or got another meaning. In an ideal world all communities update their files as soon as possible. But we all know, it will never happen like that. As a workaround we first focused on less fields, available in all versions.
 
What we got
 
We’re able to import communities from the API directory as a base entity. We also tried some different ways to import and store the specs, but we need some improvements here. By using the summarized API file, the import of our payload can be done via the DeepaMehta REST API.

The switch to DeepaMehta brought a lot of complexity to the project and I’m personally not happy my results at this point because I had trouble to spend enough time for the project. Additionally some basic problems like dealing with changing schema and data import are not really solved well at this point. The data is in DeepaMehta and can be queries with the included client but it’s not in a state where it’s usable for the community.

Overall the GSoC was an interesting experience for me. Through I’ve failed to set aside enough time for the tasks. The timely overlap with university lectures does not make it easier. So I can only recommend to know beforehand that you’ll have enough time to accomplish your goals. But the support from the Freifunk community was always great and helpful! As the project is not a state that can be considered ‘ready’ I’m continuing working on it.
 
Future Plans

I definitly want to finish the work at least to point where it can be used by the wider Freifunk community.

The default DeepaMehta client isn’t designated to query a lot of fields like our API provides. Here we need a new web based client to provide users an interface to select fields and get a proper response.

Work will continue on integrating the API data and DeepaMehta.

Repository: https://github.com/freifunk/query.api.freifunk.net

Final Blog Post: Netengine

Posted on by Alessandro Bucciarelli

Hi everybody, this is going to be my last blog post as a participant of GSOC2014. I’m very sad about this, those have been very hard worked months but very formative.

I improved both my coding skills but above all I have earned a work methodology thanks to my mentor Federico, who said to me to be more reflexive and to be more precise in what I do.

I have learnt some aspect of versioning (Git) I ignored before and learned much more on Python than I did by myself.

This is the “change log” of the last time period of the program: we developed a new backend, the HTTP backend. It aims to retrieve informations from the web admin interface of AirOS devices, that’s why we called it HTTP.

We wrote documentation about all the project, describing all the things an user should do both to contribute or use Netengine, trying to be as much more clear and to make it simpler as we could.

Unfortunately this is my last year as a student, no PhD on my way (for now), so I will not be eligible at all for next editions coming.

Greetings and thanks: I would like to thank Ninux , the community network I’ve worked with. I’ve had the possibility of joining their meetings, to talk a lot with every member and to be supported every time I had some problem about what I was doing.
Obviously my mentor Federico Capoano, Mario Behling from Freifunk who supervised projects.

For further questions on the project please visit https://github.com/ninuxorg/netengine or email us at ninuxdev@ml.ninux.org or read the docs!

GSoC 2014 Final – OpenWrt: IEEE 802.1ad VLAN support

Posted on by Gioacchino Mazzurco

Hi all!

Because I have worked very hard in the first part of GSoC, the implementation was almost done already for mid term, in the second part I have been mostly testing the code, and taking advantage of it in a lot of setups 🙂

The GSoC experience have been very formative to me and I would like to repeat it next year either as student or mentor 🙂

Moreover I’ll suggests to apply to GSoC to a lot of friends!

Many thanks to Freifunk to chose my proposal I hope you will take advantage of 802.1ad too 🙂

Cheers!

GSoC 2014: Hardware Detection [ Final ]

Posted on by Francisco Jiménez González

 

Hi everyone!

I am very happy to have participated do GSoC 2014, this experience have permitted to me to learn a lot about opensorce comminity and programming, i have learned also Lua programming language, while it seemed a little ugly  to me in the firsts times I ended up loving it.

This GSoC project permits to people installing Libre-Mesh on their devices and have device specific quirks already solved by the hardware detection module, without user intervention, while it permits to developers to write little modules to easly supports new hardwares and solve their quirks.

In particular in the second part of my GSoC I have improved hardware detection, in particular I have created a module to autoconfigure TL-WDR3600 and a module that permits to Libre-Mesh to detect wan port of a lot of routers taking advantage of the OpenWRT infrastructure.

While creating those new modules I have also realized how to improve general Libre-Mesh infrastructure and committed various improvement to the hardware-detection branch, that is now in the official repository ready for merging in develop branch.

Futhermore, during the last phase of this project I have optimized the code including modules I have written in the first period like usbradio detection module. 

Obviously all this work have been possible thank to the help of the community that helped me in the whole coding period.

Best Regards 😉

GSoC: Retroshare social network plugin review and future

Posted on by Tillmann Gansky

During Google Summer of Code i did these things:
– learn what users expect from a social network. Figure out who should receive which messages.
– map social network functions on the existing Retroshare General Exchange System
– build a backend with basic features
– build a frontend with basic features
– started a new programming interface on top of libretroshare: a JSON over http interface

The Secushare homepage says: “RetroShare should provide more social functionality” Indeed! Every country is spying on their citizens. Storage, computing power and bandwidth became cheap. These are good conditions to build a distributed social network. The first step was already started a while ago with the General Exchange System for Retroshare v0.6. This project is the second step in this direction. One thing is missing: a release for normal users.

The backend and frontend made during GSoC can display a wall with avatar image and profile text. It is possible to create posts and reference them on walls. Posts can be commented. Read more details in the previous blog post. Still it is not suitable for daily use. There was not enough time to implement a scalable user interface architecture. The web interface can’t handle more than 100 posts, because pagination is missing. Whats more the UI is filled iterative. This causes many updates to the html tree with high cpu load. It would be better to send all information to the browser in one piece. The browser would then only have to update the HTML tree once. The design and layout should be improved to highlight the content and to get the date label out the way. (See the this screenshot.)

Future

Wt is nice because it allows to make a web interface without touching HTML JavaScript and CSS. But this is only the half truth. There where issues:
– completely destroyed layout by setting the image size
– menu bar was horizontal instead of vertical, this required to manual set a style class from C++
– WTimer stops working. I had to build my own server side timer

These issues where solvable. A real pain with Wt is the layout and UI design testing. You first have to go through the complete compile and start cycle to see changes. What if you missed a closing tag in a HTML template? You have to recompile and restart. Now i saw that with real web technologies you can see a live preview while you type. I think when doing layout and design it is important to immediately see the result. This is not possible with Wt. Conclusion: you can build a web interface with C++. But hard coding CSS class names and embedding HTML snippets in C++ is a pain that should be avoided.

On the other hand there exist advanced tools and frameworks for web development. AngularJS is a very nice JavaScript library. It offers data binding from JavaScript to HTML: you update a JavaScript object, and Angular updates the HTML. JavaScript objects are more flexible than C++ objects. You can even make a JavaScript object out of a JSON file. Angular makes handling of button clicks easy. AngularJS makes  Bootstrap offers a set of widgets with HTML example code and style sheets. Now i want to use other web frameworks instead of Wt.

I did not know about these web technologies when starting the project. This is some sort of a chicken-egg problem: Retroshare does not have web developers, because it doesn’t have a web interface. And there will be no web interface until a web developer can show how it can be done. Gladly there is now a web developer who can teach me how to make a web interface.

The new idea: make a web interface using web technologies

The next goal is to make a clean and easy to use JSON over HTTP api for Retroshare. Then web developers are free to make a nice web interface using their favorite frameworks. This api is not only useful for web interfaces, but also for scripting. You could then send a chat message from a shell script using curl:

curl -X PUT -d ‘{“msg”:”hi all<br/>(send from bash)”}’ http://localhost:9090/api/chatlobbies/<id>

What i will do now
I will do more research for a JSON over HTTP interface for Retroshare and rssocialnet. If this stays a good idea i want to implements it. Maybe i will try to start a new web interface. But i hope someone else is faster than me. I prefer to work on the rssocialnet backend and libretroshare. Whats more a web interface is a perfect place for new contributors. If i would make a clearly documented JSON over HTTP api, then a web developer would not need C++ knowledge or experience with libretroshare.

A dream is to bring Retroshare to Android. This is possible, but we have to make a new touch-friendly user interface and we have to optimize Retroshare to make it more battery friendly. I currently see two ways to build a GUI for Android: with QML/QtQuick or HTML based with the Ionic Framework. There is already a QML prototype.

To get rssocialnet ready for daily use, i need your help. Unfortunately I’m again in a research/planning phase. This means I’m not sure how the JSON over HTTP interface will look at the end. It also means if you tell me “i want to help coding”, i have to disappoint you because i don’t know what we have to code and how to code it. Anyway, here are some things you can think about:

– which features are important for a social network? read some ideas
– can you make a better gui mockup than me?
– which frontend technologie should be used? QML/QtQuick or Bootstrap and AngularJS?
– what are the requirements for a first release?
– how can you use your skills to help?

Thank you Freifunk and other mesh network communities for donating one GSoC slot to Retroshare. This was a good decision, it made it possible to start a mesh friendly social network application.

GSoC: Features of the Retroshare social network plugin

Posted on by Tillmann Gansky

Content is obviously the most important element of a social network. Currently only support for plain text is implemented. The content can have an author, but this is not required.

Future: It would be nice to have support for images. This is very easy on the backend side, but it needs a frontend which scales the image to fit on the screen. With Retroshares file transfer capabilities it would also be possible to publish large files like audio and video files. It would be nice to restrict access to content to a set of people. Retroshare is prepared for this and it only needs small changes in the social network backend. Of course this requires a user interface to sort people into circles and to select circles.

Content alone is useless without a place where it gets displayed. As explained in a previous blog post, every piece of content is stored for its own. To make content visible it has to get referenced on a wall. This happens automatically on the own wall when creating a new post. It also gets triggered by clicking the share button. A reference always has an author.

Future: maybe allow to reference content from other services. For example if Retroshare gets a Photo Share service, allow to reference a picture or photo album on the wall.

A Wall is a place where a profile text, an avatar image and references to content are stored. A user subscribes to another user to download all posts referenced on the wall. The wall owner and others can reference content on a wall.

All new posts are collected and displayed in the news feed. A news feed shows the new content, the comments and how others interact with the content. Who commented this post? Who shared this post? Currently the news feed displays posts in the order in which they where received.

Future: it is probably desired to have a more advanced logic to sort news feed entries. Imagine a user comes online after a week and gets bombed by hundred new posts. It would be possible to sort news in two dimensions: topics and rating. Example: have one tab for content from close friends, and one tab for other content. Then calculate a score to display more important content at the top. This requires a bit of backend work, but it is doable.

Users can interact with content in two different ways: they can comment it, and they can share it. Sharing creates a reference to the content on the own wall, and thus forwards the content to friends. Comments are stored with the content, so everyone who received the content will also receive the comments.

Future: one can think of other ways to interact with content. Examples are like, bookmark, vote and hide. In general these interactions are each a form of tagging. For the backend it does not make a difference if content is tagged with “GSoC14” or “like” or 3.1415. This is more a matter on the frontend side: which meaning does the tag have for the user? How does the frontend show different tags? (star, heart, thumbs up, plus sign, text, …) How can the user filter posts with specific tags?

There has to be an entry point to let the user see the people around him. If the user recognizes a known person he might want to subscribe to this person. For now there is a widget to display all identities with their name and avatar image. Of course later this list should get filtered to fit on the screen. Retroshare circles could be used to make friends lists accessible to friends. This would allow automatic circle intersection to search for people the user probably knows.

Below is screenshot of the Retroshare social network plugin.

GSoC: Netengine project

Posted on by Alessandro Bucciarelli

Here it is my second blog post for the Mid Term Evaluation of GSOC2014.

As I wrote in the previous one, I’m working on Netengine, a Python module to abstract network devices and get information from them.

The work is going very well, I’m learning new things every day with the help of my mentor, Federico Capoano, and I’m very happy with the development.

In this first part of the work we completed as per goals, the SNMP back-end for AirOS and OpenWRT firmwares.

The most difficult part of this first part was to work with SNMP (Simple Network Management Protocol) because I had never worked with it, so I had to learn it’s basics and how it works, in particular it’s way of retrieving info from devices.

It uses different codes (MIB), everyone of it gives access to different information of the device (e.g device name, addresses, interfaces); so before starting to write I had to look for the correct MIBs to query.

Now we are focusing on the ssh.OpenWRT back-end ready to switch the next one on the list once completed.

I’m definitely very happy with how the work is going, with the communications I’m having with my mentor and with all coding practices I’m learning from him.

The program gave me not only the possibility to improve my skills but also to meet new people very experienced on the field.

The next step is to start coding on the new back-end, probably an HTTP back-end for AirOS, to complete the program in time.

For further questions on the project please visit https://github.com/ninuxorg/netengine or email us at ninux­dev@ml.ninux.org.  

GSoC: Freifunk API Query Client meets DeepaMehta

Posted on by Martin Tippmann

This post will give an overview about the ongoing work on the API query client GSoC project. As I’ve wrote a few days ago I met Jürgen Neumann at the WCW 2014 and he introduced me to DeepaMehta. We decided to use this tool as a database for the API data. This approach is quite a leap from my original proposal and idea but after a few discussions we realized there are a lot of benefits to this approach. Here I want to give a short overview about this new approach.

What is DeepaMehta?

DeepaMehta represents information contexts as a network of relationships. This graphical representation exploits the cognitive benefits of mind maps and concept maps. Visual maps — in DeepaMehta called Topic Maps — support the user’s process of thinking, learning, remembering and generating ideas. We think that working with DeepaMehta stimulates creativity and increases productivity. – Welcome to DeepaMehta

This sounds interesting but one may ask where is the connection to community data in machine readable form? The answer is in the data model. Here is an example from the website:

The data is organised in a topic map. There are topics that can represent e.g an organisation or a person or an event. These topics are associated through a hypergraph relationship. This means that it is possible to model all kinds of possible relationships between topics. For example a person can be modelled as a topic, that is associated with an address and the adress consists of location data, email adresses and so on… this person can be part of several organisations and these organisations can be aggrated by several parent organisations and so on…

We have a powerful graph to represent all kinds of information and we know about the relationships of each information-bit to other bits…

We have a graph that we can traverse for queries. It’s straight forward to e.g. list all organisations, that a person is a member of.

To take an example from the API data: We want to know which communities use “olsr” as their routing protocol . This would be an instance of the topic type “routing protocol”. We now only need to follow the links to all instances of the type e.g. “freifunk-community” that are connected to the “oslr” instance of the “routing protocol” topic.

This would allow for flexible queries. Another example would be a map where all instances of location topics are displayed and their parent-topics are included as label for the points on the map. If e.g. node data is present for communities this would allow for a global node map that shows not only node locoations but also community event locations and meeting places. Of course there is a huge amount work do before this will be working but overall I hope this explains why there are a lot of benefits for using this representation.

Freifunk API data in DeepaMehta

So how will it work? I’ve tried to put it in a diagram:

More details:

Data

At the moment there is a specification – a JSON schema file – for the API data and an instance – a JSON API file.

Magic

Magic is probably the wrong word for that, but all the hard work is done by DeepaMehta and I only build a plugin on top of that – unaware of the implementation details – so I thought it is appropriate. To quote Arthur C. Clarke: Any sufficiently advanced technology is indistinguishable from magic

At first, we need to put the schema into the DeepaMehta platform. This is possible using a plugin that creates the topics in DeepaMehta for the entries in the JSON schema.

The next step is to feed the current data into DeepaMehta. This creates instances of the previously defined topic types. E.g. a topic for each community.

Once the data is in DeepaMetha it’s possible to query that data.

Presentation

We can now speak JSON over HTTP using REST with the platform and present the results in various ways. E.g. display communities in a map or provide an text interface to query the data. DeepaMehta already provides an REST API and a web-based interface for exploring and editing topic maps but while testing and playing with the interface we found it too complicated.

Current Progress

Feeding the data by hand into the plattform is not practical and I’m working on an import script for the schema and the API data. At the moment mapping the basic JSON types (string, integer, ..) into DeepaMehta is working but more works need to be done to get a better representation. Once the data is in the focus will shift on a doing actual queries.

Problems

Complexity. These are for the most part new concepts for me and I had little prior experience with semantic web technologies. DeepaMehta also covers quite a few other usecase and I need to learn more about the system.

Open Questions

Doing the actualy queries and traversing the graph is something I need to find a workable solution for. There are also different specifications of the API schema and different communities use different versions of the API. At the moment I’m ignoring that detail but here I need to find a solution. Another nice to have feature would be access to historic data. 

Lots of interesting problems.. unfortunatly I’ve been short on time in the past days and I’m quite behind the shedule but I’m optimistic that this approach is flexible enough to provide a solid ground all kinds of experiments with data. Once queries are possible things will hopefully move forward at a faster and more visible pace.

GSoC “BGP/Bird integration with OpenWRT and QMP” project report

Posted on by Eloi Carbó Solé
This entry is for updating the information regarding this GSoC Project focused on the automate of BGP-BMX6 metrics and routes exchange and “translation”. 
 
During the WBMv7 in Leipzig, the WiBed platform [0] was presentated as well as the GSoC project[1] where I am participating. WiBed was used to deploy the testbed network where many routing and mesh related experiments were executed.
I was participating in the deployment and development teams of the WBMv7, so I contributed in many bug fixes and platform improvements.
The WiBed project is important for the GSOC because it provides a testing platform very similar to the production environment where we will apply the results of the GSOC project.
 
Currently I am studying and understanding the basis system we need to accomplish the objectives of my project. I am working on Bird4 and Bird6 [2] configuration transition to UCI[3] and LUCI[4] willing to give to the OpenWRT project and community a more user-friendly Bird Daemon configuration. For those who do not know, Bird is a lightweight and flexible BGP daemon which may be used as an alternative to Quagga (which is actually very heavy).
 
Once we got an usable version, we wll upload the work done to public GIT repository following the standard OpenWRT feeds format (so it might be included in the official repositories). To test our advances and implementation we are using the WiBed platform network deployed in our laboratory (at UPC university in Barcelona).
The first production test will be made in the QMP [5] network we have deployed in Barcelona.
 
The most challenging feature in our project is the exchange of routes (and associatd metrics) between routing protocols (BGP and BMX6 [6] in our case). We (me, the workgroup and mentors) are still discussing about the different ways to implement it and how to use the Bird solution to Guifi.net [7] where the main protocol is BGP and the most common OS is the privative routerOS from Mikrotik. Including Bird in the open/libre firmware QMP will allow people in Guifi.net to use this solution instead of routerOS. However to make the interconnections between the QMP (Mesh) networks and the current BGP/Infrastructure we need the called frontier (or border) nodes (those who exchange the routes between both network clouds). 
 
To not overload the current mesh clouds (running with BMX6), we will install a BGP Bird instance only in these QMP border nodes. They will exchange routes and metrics in the entrance of the network and summarize the result by publishing the aggregated routes to each network.
 
Another idea we are considering is to create a very small and simple OpenWRT image with the BMX6 daemon ready to perform the routing. This image may be installed as a virtual machine in the RouterOS firmware (present in 50-60% of the Guifi.net nodes). So Mikrotik nodes will be able to route BMX6 packets thus the BGP instance will not be longer necessary (we believe mesh routing protocols are a much better option than BGP/OSPF for a WiFi network). This approach is compatible with the (previous) border nodes one. We will provide both options to Guifi.net users to let them decide.
 
Finally, to conclude this  mid-term report, say that we expect to finish the project in time and  just mention that in the coming days we will start testing the first solution in a real production network.
 

GSoC: Freifunk API Query Client – A short report from the Wireless Community Weekend 2014

Posted on by Martin Tippmann

Photo taken from https://twitter.com/christianheise/status/472746947569520641

Photo taken from https://twitter.com/christianheise/status/472746947569520641

From the 29th May to the 1st of June we were with Andi and Bernd from Weimarnetz at the wunderful c-base Raumstation. We visited the Wireless Community Weekend. It was my first experience of this kind of community event and I enjoyed it very much.

Beer and Bratwurst did harmonize quite well with technical talks about OpenWrt and the Freifunk Community. I was especially surprised how diverse and open the community is and how enthusiastic everyone involved was.

Andi and Monic talked about the progress on the Freifunk API and presented their work. At the end of their talk I had the chance to present my work on the query client for the API. Here are the slides.

Shortly after the talk Jürgen Neumann from Freifunk Berlin came to me and introduced me to DeepaMehta. My original plan was to use something like NodeJS for the backend the storage of the API data but DeepaMehta looked promising and offers unique features I didn’t even thought off.

So after talking with Andi about it we decided to use DeepaMehta as the foundation and storage tool for the API data. A seperate blogpost for the GSoC midterm evaluation will outline my work in this direction.

Overall it was a very exciting weekend for me at #ffwcw 2014.