OpenWLANMap App: Update 2

Hi,

In the last weeks I was working on  the storing process as described in the architecture in the last blog post [0].

Storage Handler:

Old app: the old app saves the data as byte in a file. A data entry is 28 bytes of MAC-address(12 bytes for 12 characters) and latitude(8 bytes for double) and longitude(8 bytes for double). An entry could be saved more than once in the file. There are 2 files, one for data which should be updated and one for data which should be deleted from backend.

New app: Firstly I wanted to adapt the structure from the old app. But since I saw some unreasonable points such as saving redundant data, flash workload, maintenance problem and unstructured storage, I decided for a standard database with more structure and easy to maintain: sqlite. Also I am using the new persistence lib, which provides an abstract layer for database: Room, newly released last year, as a part of android architecture components, with a lot of bug fixed since then. A lib with a lot of advantage when working with sqlite database: verify queries at compile time, reduce a lot of duplicate code in comparison with the last approach with DbHelper etc. In order to store the access point in the database, I implemented a seperate thread, which reads data from a blocking queue and saves it in the database, which works parallel with the scan thread and will be interrupted if there is nothing in the queue to store. Also to save energy and not force the store thread to run the whole time, a list of access points will be put into the blocking queue as an element. To pretend redundant data in storage, a data entry with BSSID will not be saved many times as in the old app but only once. The BSSID is used as primary key in the sqlite table. It will be updated the next time if the received signal strength is better than the last entry in the database. An explicit transaction is implemented to solve this case since the lib Room has only supported annotation for standard update/insert. To decide if a access point should be deleted or updated from backend, a flag is set.

Upload Handler:

The WifiUploader is in process. I did take a look at the uploading format in the old app and how it communicates with the current backend. Also the upload sequence is already defined, mean the scanning thread will be interrupted, all the rest of access point will be stored, the store thread will be interrupted to pretend conflict while 2 threads try to access same database at the same time before the uploading process is started. Also the WifiUploader will read maximum a number of data entries from the database and upload it, not the whole database like old app but one after another,  in order to pretend out-of-memory problem at device with small ram. (see more in the below diagram)

flowchart of uploading process

 

But since I am in the middle of my final exam period, there will be a small delay until this weekend for the WifiUploader to be published. Also from next week I will spend full time making the other features done includes implementing all saving resource features such as adaptive scanning, implementing all settings option. A clean and full documentation will be provided at the end as well.

Available issue: Permission request and handling

[0] https://blog.freifunk.net/2018/06/10/openwlanmap-app-update-1/

OpenWLANMap App: Update 1

Hi,

As I mentioned in the last blog post [0], the first step I did is defining the app’s functionalities[1] and designing the app architecture.

Basically the app contains 1 service, which runs in the background and communicates with UI thread per broadcast (public-subscriber pattern). Since as default, the service will run in the main thread, which is not wanted, I created a ScanThread to handle the scanning. It sends every 2s as default (and should be adapted with user’s speed etc. later) a scan request to the WifiLocator and gets a scan result back from it asynchronous. The WifiValidator then validates the scan result as well as the returned location, and puts the valid wifi access point in a WifiQueue. The WifiStorer will take anything from the WifiQueue and writes it to local disk (simple Consumer-Producer pattern). Based on the user’s upload mode setting, the WifiReader will be triggered if uploading is wanted, reads the local disk files in wanted format for uploading and passes it to the WifiUploader. It then uploads the data to any supported project api and as soon as the uploading is successful, the data will be deleted and ranking will be updated.

The next step I did is designing the new UI, got some feedback from mentor and changed it appropriately. Also in the process I defined all the user’s setting options. I spent a lot of time reading the android documentation for parallel processing and made decision for each functionalities, which is relevant for the next part. (WifiStorer, WifiReader: normal Thread, WifiUploader: AsynTaskLoader etc.) I write more about it in the next post.

Finally I jumped into implementing. I started with the demo mockup and then slowly implemented the logic part. I have finished the scan service and a part of the WifiValidator.The WifiLocator uses gps for defining location if available, otherwise it makes a request to openwifi.su with the surrounding wifis. I provided methods to do it with both new and old openwifi.su api in case we want to use any of them in the future. I ran into an android bug, where the wifi scan result is always 0 if the user disables GPS, even the location permission is granted.(Tested on Android 6). It is kind of weird because scanning wifi does not have anything to do with the gps and turning on gps the whole time will cost phone a lot of energy. Still it’s kind of wanted feature from Android to make users aware that their location information is being accessed when they use kind of app. Because the location of user’s phone could be defined based on the collected wifis. Since it is OS design, I pop up users a message with those information to ask them to turn on their GPS if they turn it off. I also implemented a part of WifiValidator, the WifiFilterer to check if an access point is openwifi, from freifunk or mobil hotspot or marked with _nomap (which should not be collected).

What’s now?

If you want to check the app, feel free to download install file .apk from [2].

If you as usual do not want to install an unknown source, I also provide a short demo video

 

What’s next?

In the next time, I will finish the WifiValidator, which should not only filter the access point but also validate the location to provide scan service a better scan period to save energy (in case the location is not changed for a long time, the scan service should be stopped etc.) and then other parts as shown in architecture image above.

Links

[0] https://blog.freifunk.net/2018/05/14/introduction-openwlanmap-app/

[1] https://github.com/openwifi-su/OpenWLANMap-App

[2] https://androidsmyadventure.wordpress.com/2018/06/03/openwlanmap/

 

Introduction: OpenWLANMap App

Hi,

My name is Lilli and I am studying technical computer science at the 6.th semester in Hamburg, Germany. In this summer, i will work on a new wardriving app for openwifi.su

OpenWifi.su is working on wifi positioning system. It uses an android app called OpenWLANMap App as the wardriving tool to collect wifi access points and sends it with the geolocation of the phone to the backend. The backend stores the data in database and also offers an API for non-GPS devices to request their positions based on the surrounding wifis.The backend currently uses triangulation technique on the numbers of access points it receives from the request device to calculate it’s geolocation.

A data entry from the wardriving app is currently BSSID + LATITUDE + LONGITUDE, which is stored temporarily effectively in local disk with 28 bytes (12 bytes for 12 characters of the MAC Address, 8 bytes for each latitude and longitude) before being uploaded to backend. The wardriver can do it manual or automatically. The app respects _nomap Wifi APs and does filter out some mobile hotspots on public transportation in europe. Unfortunately the app is hardly out of date. There are no developers working on the app and no updates for years. It does not run in new android devices. The OpenWifi.su has an amazing community of wardriver, over four thousands people. They have to keep very old phones to be able to run the app. But this community began to shrink in fact because peoples buy new phones which are not able to run the old app anymore.

I am so glad to get involved in the project and can spend my Google Summer of Code rewriting the app. Thank to freifunk community I was allowed to participate in the Wireless Community Weekend in Berlin in the community bonding period and had the chance to present my project, as well as talk to people about it and possible solutions for many problems. I spent the last weeks analysing the old app code and was talking a lot with my mentor about the old app performance and functionalities and ended up with many important decision about the design for new app. Here are some of them

  1. Wlocator is a service of getting GPS either from the device itself. In worst case where the GPS is undefined, it will send a request with the surrounding wifis to the Openwifi.su backend to ask for it’s geolocation. The service will run every n seconds
  2. Wifiscan is a service of scanning wifi access points. All the data of the AP such as BSSID, SSID, RSSI, frequency, channel , encrypted method, scan timestamp etc. will be scanned and display as user’s option. Necessary data will then be stored temporally local and later on uploaded to the database. The service will be stopped if the GPS of the device isn’t changed after n seconds in order to save device’s battery.
  3. Wififilter helps filter out: _nomap, mobile hotspot ( Call for help to collect mobile hotspot from different countries), ad_hoc network (I am working on it), collect open wifi to automatic connect and upload data if possible.
  4. Use trilateration or other techniques to define the location of the scanned AP better
  5. WifiUpload let users upload the data manual, automatic if internet available, automatic if only wifi available
  6. Extension: upload data to different APIs
  7. Saving resouces: stop scan service after n second of not changing GPS, put app in standby/doze mode if GPS does not change after n second, reduce brightness, kill app if battery critic
  8. Different languages available
  9. etc.

The plan for the next three months will be:

  1. Design new architecture for new app functionalities
  2. Design new UI
  3. Implement all logic functionalities
  4. JUNIT test + documentation

Stay tuned, I will update it soon.

L3I2