OpenWLANMap App Final Update

Hi,

This is my final update for GsoC.

In this blogpost I would like to summary all the work I have done in the last 3 months, as well as available problems and future plan.

An introduction, my progress and further information can be found under [0] [1] [2]

The new app is compatible to the old app on all basic functionalities [3]. Beside that the code is validated after google check style, contains a full java doc, clear interfaces and the app performance is partly improved.

Final architecture and app design:

Basic changes in comparison to old app

  • The old broken UI is replaced by a new designed UI.
  • Storing: The old app uses a non-standard database by writing multiple access points in bytes in file, which stores redundant data and difficult to maintain. In the new app, Sqlite is used for storing data in order to make it easier to maintain and extend. There is no redundant of multiple access points in database since BSSID is used as primary key and data is updated if RSSID is bigger. The storing process is done not by scan thread, but by a separate thread (WifiStorer), which reads from a blocking queue (WifiQueue) . A list of 50 APs (can be moved up if necessary) is put into the blocking queue as an item and storer thread is blocked on no item in the queue to pretend writing around storage the whole time.
  • Uploading: As old app, the uploading depends on user’s setting: manual, automatic on internet or on wireless connection. User can also set up the start number of APs to trigger auto upload from 5000 to 50000. User can only trigger uploading at at least 250 APs. The new app processes uploading with maximal 5000 APs once in order to pretend out-of-memory problem at device with small ram. An message contains upload process summary, new rank or error is given back to user. The WifiUploader uses UploadingQueryUtils for openwifi.su and can be changed quickly if backend changes.
  • Scanning: Scan period is dynamically set not only depending on speed but also night mode and I am working on movement detection base on sensors. Every 2s as default , the scanner thread sends Wifilocator a request on position and scanned wifis. Wifilocator uses GPS to define position, in case of no GPS defined, scanned wifis are used to define user’s position which is not working anymore at old app. The method for location can be displayed by overlay big number color as in setting.
  • Resource is managed and can be set as user’s options (kill app on low battery, on long time no gps etc.). While the old app triggers compare on noExistGps every time it scans, the new apps starts a new thread for checking resources only if user configures.
  • The old app exports only user’s own bssid and puts the export file/requires import file default in external storage. The new app allows user to import/export account with team and tag information as well as reset all settings back to default. User can browser the import file, as well as save the export file her/himself at any writable place in storage.
  • A map of all data from openwifi.su and a map from user’s contributed data is immigrated directly into the app (using osmdroid) , as well as a ranking list.
  • Min app api is moved to 19, which covers over 96% devices currently [4]. Permission is checked at runtime as required.

What I learned

I learned a lot about android developemnt.

  • Life cycles: The UI and service communicates per LocalBroadCast, the BroadCastReceiver has to be register and unregister on Resume/on Pause. Also the SettingPreferenceListener to have a proper lifecycle management of the activity.
  • Using LocalBroadCast instead of global BroadCast to keep data inside the app only.
  • From Api 23, the “dangerous permission” has to be asked at run time. Many system flags and parameters differ from api versions, which requires a lot of version control check in runtime
  • Working and managing service with a lot of parallel processes.
  • Osmdroid: open source lib for working with osm which is compatible to google map API
  • Database sqlite with lib Room
  • etc. 

    Also I learned how important architecture is since I was too fast jumping in coding where my mentor had to stop me and gave me some helpful advice. We re-designed the architecture with a component controller in center. All other components should only do its job and communicate with controller and not with others directly to make it easy to extend/replace any components.

    Difficulties I met

    It was hard to work on the app while I have no access on backend. I have to test all the APIs while analyzing the old app, which is also not nicely documented and implemented. Furthermore the backend is quite unstable and sometime unreachable. Another problem is testing. Since the app works with collected wifi access points, testing and debugging at home became very hard.

    Future plan

    There are still some points on the app performance I want to optimize further. I already started on working with the android sensors to detect movement, to scale the scan time more effectively to save resource since scanning wifi and gps are two of the services which cost bunch of phone battery.

    The app is currently only in development mode since I haven’t had a google play store account. But as soon as I do, I will release it. Until then, if you want to try it, an .apk is to download here [5]

    Acknowledgement

    Many thanks to freifunk community and my mentor Jan-Tarek Butt for this amazing opportunity. Even though there are still some small stuffs to do/fix, I am so glad that a new wardriving app is coming soon for openwifi.su. Many thank to Google summer of Code team for making this happen.

    [0] https://blog.freifunk.net/2018/05/14/introduction-openwlanmap-app/

    [1] https://blog.freifunk.net/2018/06/10/openwlanmap-app-update-1/

    [2] https://blog.freifunk.net/2018/07/09/openwlanmap-app-update-2/

    [3] https://github.com/openwifi-su/OpenWLANMap-App

    [4] https://developer.android.com/about/dashboards/

    [5] https://androidsmyadventure.wordpress.com/2018/06/03/openwlanmap/

 

OpenWLANMap App: Update 2

Hi,

In the last weeks I was working on  the storing process as described in the architecture in the last blog post [0].

Storage Handler:

Old app: the old app saves the data as byte in a file. A data entry is 28 bytes of MAC-address(12 bytes for 12 characters) and latitude(8 bytes for double) and longitude(8 bytes for double). An entry could be saved more than once in the file. There are 2 files, one for data which should be updated and one for data which should be deleted from backend.

New app: Firstly I wanted to adapt the structure from the old app. But since I saw some unreasonable points such as saving redundant data, flash workload, maintenance problem and unstructured storage, I decided for a standard database with more structure and easy to maintain: sqlite. Also I am using the new persistence lib, which provides an abstract layer for database: Room, newly released last year, as a part of android architecture components, with a lot of bug fixed since then. A lib with a lot of advantage when working with sqlite database: verify queries at compile time, reduce a lot of duplicate code in comparison with the last approach with DbHelper etc. In order to store the access point in the database, I implemented a seperate thread, which reads data from a blocking queue and saves it in the database, which works parallel with the scan thread and will be interrupted if there is nothing in the queue to store. Also to save energy and not force the store thread to run the whole time, a list of access points will be put into the blocking queue as an element. To pretend redundant data in storage, a data entry with BSSID will not be saved many times as in the old app but only once. The BSSID is used as primary key in the sqlite table. It will be updated the next time if the received signal strength is better than the last entry in the database. An explicit transaction is implemented to solve this case since the lib Room has only supported annotation for standard update/insert. To decide if a access point should be deleted or updated from backend, a flag is set.

Upload Handler:

The WifiUploader is in process. I did take a look at the uploading format in the old app and how it communicates with the current backend. Also the upload sequence is already defined, mean the scanning thread will be interrupted, all the rest of access point will be stored, the store thread will be interrupted to pretend conflict while 2 threads try to access same database at the same time before the uploading process is started. Also the WifiUploader will read maximum a number of data entries from the database and upload it, not the whole database like old app but one after another,  in order to pretend out-of-memory problem at device with small ram. (see more in the below diagram)

flowchart of uploading process

 

But since I am in the middle of my final exam period, there will be a small delay until this weekend for the WifiUploader to be published. Also from next week I will spend full time making the other features done includes implementing all saving resource features such as adaptive scanning, implementing all settings option. A clean and full documentation will be provided at the end as well.

Available issue: Permission request and handling

[0] https://blog.freifunk.net/2018/06/10/openwlanmap-app-update-1/

OpenWLANMap App: Update 1

Hi,

As I mentioned in the last blog post [0], the first step I did is defining the app’s functionalities[1] and designing the app architecture.

Basically the app contains 1 service, which runs in the background and communicates with UI thread per broadcast (public-subscriber pattern). Since as default, the service will run in the main thread, which is not wanted, I created a ScanThread to handle the scanning. It sends every 2s as default (and should be adapted with user’s speed etc. later) a scan request to the WifiLocator and gets a scan result back from it asynchronous. The WifiValidator then validates the scan result as well as the returned location, and puts the valid wifi access point in a WifiQueue. The WifiStorer will take anything from the WifiQueue and writes it to local disk (simple Consumer-Producer pattern). Based on the user’s upload mode setting, the WifiReader will be triggered if uploading is wanted, reads the local disk files in wanted format for uploading and passes it to the WifiUploader. It then uploads the data to any supported project api and as soon as the uploading is successful, the data will be deleted and ranking will be updated.

The next step I did is designing the new UI, got some feedback from mentor and changed it appropriately. Also in the process I defined all the user’s setting options. I spent a lot of time reading the android documentation for parallel processing and made decision for each functionalities, which is relevant for the next part. (WifiStorer, WifiReader: normal Thread, WifiUploader: AsynTaskLoader etc.) I write more about it in the next post.

Finally I jumped into implementing. I started with the demo mockup and then slowly implemented the logic part. I have finished the scan service and a part of the WifiValidator.The WifiLocator uses gps for defining location if available, otherwise it makes a request to openwifi.su with the surrounding wifis. I provided methods to do it with both new and old openwifi.su api in case we want to use any of them in the future. I ran into an android bug, where the wifi scan result is always 0 if the user disables GPS, even the location permission is granted.(Tested on Android 6). It is kind of weird because scanning wifi does not have anything to do with the gps and turning on gps the whole time will cost phone a lot of energy. Still it’s kind of wanted feature from Android to make users aware that their location information is being accessed when they use kind of app. Because the location of user’s phone could be defined based on the collected wifis. Since it is OS design, I pop up users a message with those information to ask them to turn on their GPS if they turn it off. I also implemented a part of WifiValidator, the WifiFilterer to check if an access point is openwifi, from freifunk or mobil hotspot or marked with _nomap (which should not be collected).

What’s now?

If you want to check the app, feel free to download install file .apk from [2].

If you as usual do not want to install an unknown source, I also provide a short demo video

 

What’s next?

In the next time, I will finish the WifiValidator, which should not only filter the access point but also validate the location to provide scan service a better scan period to save energy (in case the location is not changed for a long time, the scan service should be stopped etc.) and then other parts as shown in architecture image above.

Links

[0] https://blog.freifunk.net/2018/05/14/introduction-openwlanmap-app/

[1] https://github.com/openwifi-su/OpenWLANMap-App

[2] https://androidsmyadventure.wordpress.com/2018/06/03/openwlanmap/

 

Introduction: OpenWLANMap App

Hi,

My name is Lilli and I am studying technical computer science at the 6.th semester in Hamburg, Germany. In this summer, i will work on a new wardriving app for openwifi.su

OpenWifi.su is working on wifi positioning system. It uses an android app called OpenWLANMap App as the wardriving tool to collect wifi access points and sends it with the geolocation of the phone to the backend. The backend stores the data in database and also offers an API for non-GPS devices to request their positions based on the surrounding wifis.The backend currently uses triangulation technique on the numbers of access points it receives from the request device to calculate it’s geolocation.

A data entry from the wardriving app is currently BSSID + LATITUDE + LONGITUDE, which is stored temporarily effectively in local disk with 28 bytes (12 bytes for 12 characters of the MAC Address, 8 bytes for each latitude and longitude) before being uploaded to backend. The wardriver can do it manual or automatically. The app respects _nomap Wifi APs and does filter out some mobile hotspots on public transportation in europe. Unfortunately the app is hardly out of date. There are no developers working on the app and no updates for years. It does not run in new android devices. The OpenWifi.su has an amazing community of wardriver, over four thousands people. They have to keep very old phones to be able to run the app. But this community began to shrink in fact because peoples buy new phones which are not able to run the old app anymore.

I am so glad to get involved in the project and can spend my Google Summer of Code rewriting the app. Thank to freifunk community I was allowed to participate in the Wireless Community Weekend in Berlin in the community bonding period and had the chance to present my project, as well as talk to people about it and possible solutions for many problems. I spent the last weeks analysing the old app code and was talking a lot with my mentor about the old app performance and functionalities and ended up with many important decision about the design for new app. Here are some of them

  1. Wlocator is a service of getting GPS either from the device itself. In worst case where the GPS is undefined, it will send a request with the surrounding wifis to the Openwifi.su backend to ask for it’s geolocation. The service will run every n seconds
  2. Wifiscan is a service of scanning wifi access points. All the data of the AP such as BSSID, SSID, RSSI, frequency, channel , encrypted method, scan timestamp etc. will be scanned and display as user’s option. Necessary data will then be stored temporally local and later on uploaded to the database. The service will be stopped if the GPS of the device isn’t changed after n seconds in order to save device’s battery.
  3. Wififilter helps filter out: _nomap, mobile hotspot ( Call for help to collect mobile hotspot from different countries), ad_hoc network (I am working on it), collect open wifi to automatic connect and upload data if possible.
  4. Use trilateration or other techniques to define the location of the scanned AP better
  5. WifiUpload let users upload the data manual, automatic if internet available, automatic if only wifi available
  6. Extension: upload data to different APIs
  7. Saving resouces: stop scan service after n second of not changing GPS, put app in standby/doze mode if GPS does not change after n second, reduce brightness, kill app if battery critic
  8. Different languages available
  9. etc.

The plan for the next three months will be:

  1. Design new architecture for new app functionalities
  2. Design new UI
  3. Implement all logic functionalities
  4. JUNIT test + documentation

Stay tuned, I will update it soon.

L3I2