Node Database and Map Server Yaffmap: Yet Another Freifunk Map

yaffmap is a project from Dennis Bartsch from Freifunk Berlin, which he started with a friend. It is called yaffmap. It got its name because at the beginning of the project we had many mapservers in Berlin and so it started as yet another approach for a Freifunk map. 
 
yaffmap – screenshot
 
The intention was to make a server that not just produces points and lines (nodes and their links) but to gather all information that might help to understand why a link is as bad as it is. This includes to gather wireless scan results, the effective rate chosen/calculated by the wireless driver to a specific neighbour and so on. Furthermore it had to be independent from the routing-protocol and its daemons (but needs it to gather useful info) and the IP version (or even no IP version for RPs like batman) and had to be able to upload and store data from multiple routing protocols on the same node. In order to sample so much information we went the route of scripting an agent for the map-server which runs on the nodes gathering the information and uploading it through a JSON interface to the server. For link-state-protocols like OLSR we even implemented the upload of the global topology to the server, which gave us some headache. From the beginning on the need for decentralized operation was stressed, so replication between the servers was implemented and any community which wants can have their own map data server. Moreover they cleary wanted the datacollection/storage to be independent from the frontend the map user is presented. In Berlin everytime a new map came along and an old one was gone we saw huge ammounts of data simply disappear. So the server only provides a SOAP interface for UIs and other services to use to get map data. The representation of a node in the database ist best seen on a graphic. Dennis uploaded it to imageshack, see [1]. A little bit of documentation is found under [2], but wasn’t thoroughly updated after we left concept stage.
 
We got to the point where the agent (bunch of shell/AWK) for the nodes runs on openWRT (best with madwifi and ath9k/ath5k) with olsrd (other routing daemons simply need scripts for data gathering, agent is held modular), uploades many useful information to the server (or another if the first does not respond; multiple can be configured) using JSON, where it gets stored correctly into a SQL database and replicated to other servers and provided through a SOAP interface. The agent even is already provided as a package in the respected PBerg Freifunk Firmware. There also exists a ‘proof-of-concept’ implementation for a map frontend, see [3]. The code is hosted on github as seen und [4]. Because of the frontend development got stuck the frontend SOAP interface is not very sophisticated nor much tested as is the frontend itself.
 
The result is a database which can answer more complex question to the mesh engineer or software developer. Maybe you want to know the average effective rate on all 11n interfaces or all 5GHz interfaces? Or you might be interested in the average ETX on different channels in a certain geographic area. Or you want to compare how different routing daemons evaluate the quality of a certain link. Maybe you even want to draw a noise map. Or wouldn’t it be interesting to see the correlation between the effective rate of a wifi link and the metrics which result in the routing daemons? Because the node data includes a ‘misc’ field through which any kind of node id or statistics could be stored/sent this is can even be integrated into existing community portals.
 
 

International Community Map Project and Node Database

One of the things every community would like to see are maps of their network. There are many initiatives and map projects around, but how much more progress could we make, if we work together? This is why a group of free networks contributors started a group to work together on a common project. We started a mailing list to coordinate efforts and welcome everyone who would like to collaborate: https://lists.funkfeuer.at/mailman/listinfo/interop-dev
 
The current goal is to define a basis for a common map. We are looking at different map solutions and try to find out how we can merge the best of each into one. The best approach seems to be a modular system. In the following three great projects of different communities.
 
Nodeshot comes from the community in Italy and is widely being used there (map.ninux.org). The main developer is Federico Capoano. The goal of the Django application is to have “a nice snapshot of your wireless community network”. Nodeshot is a web tool for wireless community network. It allows members to add their node and to share and manage information about their configurations like devices, ip addresses, wireless parameters etc. In this way, newcomers can easily contact/connect with them. Internal scripts will update the topology and retrieve nodes information via snmp, or parsing routing information given by olsr, batman or whatever. It is super-fast, nice and easy to use. It rises from the ashes of WNMap (sourceforge.net/projects/wnmap/), powered by django, released under GPLv3 and tested inside the Ninux wireless community network (wiki.ninux.org). How to install: A basic guide on how to setup nodeshot for your community is available here: wiki.ninux.org/InstallNodeshot
 
Nodewatcher, another Django based application was started in Slowenia by Mitar. Its main goal is the development of an open source network planning, deployment, monitoring and maintanance platform. The project is divided into multiple components:
* node telemetry provider: A simple shell script that is accessible via HTTP interface and is used for node status data acquisition that is performed by the data collection system. Its role is similar to the one of SNMP, however it uses less resources (CPU-wise and, mostly, when it comes to memory and flash space). Our firmware has this preinstalled, others can follow these instructions to install it.
* data collection system : A daemon written in Python that periodically collects data regarding the OLSR topology, active HNAs, node telemetry and performs active reachability tests for visible nodes. Using rrdtool it can then generate graphs that are used by the web interface.
* web interface: A web-based application, written in Python and using the Django framework. It is used by the users to monitor the status of the network and individual nodes and by the node owners to manage their nodes.
* firmware image generator: A daemon that handles per-node configuration and firmware image generation via the OpenWrt buildroot using our custom firmware. It receives requests from the web interface. Live version We use nodewatcher in wlan slovenija network (this is why we are developing it), so you can see a live version of nodewatcher for real deployed network here
 
yaffmap is a project from Dennis Bartsch from Freifunk Berlin. He is working on an implementation of a node database he started with a friend. It is called yaffmap. It got its name because at the beginning of the project we had many mapservers in Berlin and so it started as yet another approach for a Freifunk map. The intention was to make a server that not just produces points and lines (nodes and their links) but to gather all information that might help to understand why a link is as bad as it is. This includes to gather wireless scan results, the effective rate chosen/calculated by the wireless driver to a specific neighbour and so on. Furthermore it had to be independent from the routing-protocol and its daemons (but needs it to gather useful info) and the IP version (or even no IP version for RPs like batman) and had to be able to upload and store data from multiple routing protocols on the same node. In order to sample so much information we went the route of scripting an agent for the map-server which runs on the nodes gathering the information and uploading it through a JSON interface to the server. For link-state-protocols like OLSR we even implemented the upload of the global topology to the server, which gave us some headache. From the beginning on I stressed the need for decentralized operation, so replication between the servers was implemented and any community which wants can have their own map data server. 
 

Projekt Freikarte: Visualisierung der Standorte von Freifunkern auf freifunk.net – Mitstreiter gesucht!

Das Nutzerkonto auf freifunk.net, die Freikarte, hat in den
letzten Jahren viele Freifunk-Interessierte zusammen geführt. Mit der Freikarte
ist es möglich direkt direkt nach Freifunkern in der Nachbarschaft zu
suchen. Doch die Funktionen der Freikarte stoßen immer mehr an Grenzen. Bei der
großen Anzahl von Freifunkern in manchen Gegenden erscheinen bei der
Postleitzahlsuche manchmal Hunderte von Treffern – da wird es unter Umständen
schwierig den richtigen Ansprechpartner zu finden. Als Ziel hat das Kernteam
der Freifunk.net-Website daher anvisiert eine Karte in Verbindung mit einer
genaueren Suche einzuführen. Hierzu sollen in Zukunft auch Straße und
Hausnummer von Aktiven und Interessierten aufgenommen werden. In Verbindung mit
Google Maps oder in der Entstehung befindlichen Free und Open Source Maps
können lokale Mitwirkende und Usergroups so leichter gefunden werden und sich
vernetzen.

Ein Beispiel für die Implementierung von Karten bieten
bereits Communities in Frankreich (http://carte.wireless-fr.org),
Spanien (http://maps.guifi.net), in Terni
(Italien) (http://www.terniwireless.info/mappaterni),
Berlin (http://www.olsrexperiment.de/samples..)
und Leipzig (http://map.freifunk-leipzig.public-ip.org/polymap2.ffl/app).
Auch London hat bereits ein OpenMap-Projekt (http://uo.space.frot.org/freemap/).
Ein globales Map-Projekt, dass als Datenbasis für Freifunkkarten dienen kann,
wurde zudem auf http://www.openstreetmap.org
gestartet.

Fragen die für die Implementierung einer Karte auf
freifunk.net zu klären sind, werden bereits auf deutschen Mailinglisten
diskutiert. Andreas Hubel schrieb auf der deutschen Liste:

„Dann sollten wir uns auf ein einheitliches XML Format für
die Daten einigen und die WRT's dazu bringen die Daten in diesem Format
auszuspucken. Als nächstes sollte dann in jeder Stadt irgend ein Rechner
laufen, der diese XML Dateien via HTTP von den WRT's holt und in einer einzigen
Datei sammelt. Mit der können dann die eigenen Systeme gefüttert werden, oder
man schickt diese gesammt Datei eben an eine zentrale Stelle, an der dann eine
Freifunk weite XML-Datei generiert wird. Basierend auf diesen XML-Dateien
können wir ja dann die verschieden Kartensysteme die es so gibt ansprechen. Nen
Konverter für KML Dateien für Google Earth (und jetzt anscheind auch Google
Maps) wäre z.B. nicht schlecht, aber auch andere Formate wären möglich. Die
Leute vom Funkfeuer haben ja schon sowas in der Richtung gemacht, dort kann
lässt sich ne KML Datei runterladen, die dann in Google Earth in die 3D Karte
die Nodes einblendet. Wir bräuchten halt nur mal nen einheitlichen Standard für
die XML-Dateien, ich denke der Rest wäre das kleinere Problem.“ (http://permalink.gmane.org/gmane.org.freifunk.wlannews/1443)

Ein schönes Feature
für die Zukunft wäre auch, wenn Freifunker ihren lokalen Ausschnitt, wie bei
frappr.com in ihre eigen Website oder Blog einfügen könnten. Fernziel könnte
eine Free und Open Source Map sein, die Freifunk-Initiativen in Deutschland und
der ganzen Welt auf einer Karte anzeigt und die Verbindungen von Freifunkern
und zwischen den Communities sichtbar macht. Derartige Ideen sind im Moment
noch Zukunftsmusik. Für die Umsetzung braucht das Website-Team daher
Verstärkung. Also wer Ideen hat und mitmachen möchte, meldet euch bei uns!

Email: info@freifunk.net