Author: Thi Huyen (Lilli) Cao

Hi,

This is my final update for GsoC.

In this blogpost I would like to summary all the work I have done in the last 3 months, as well as available problems and future plan.

An introduction, my progress and further information can be found under [0] [1] [2]

The new app is compatible to the old app on all basic functionalities [3]. Beside that the code is validated after google check style, contains a full java doc, clear interfaces and the app performance is partly improved.

Final architecture and app design:

Basic changes in comparison to old app

• The old broken UI is replaced by a new designed UI.
• Storing: The old app uses a non-standard database by writing multiple access points in bytes in file, which stores redundant data and difficult to maintain. In the new app, Sqlite is used for storing data in order to make it easier to maintain and extend. There is no redundant of multiple access points in database since BSSID is used as primary key and data is updated if RSSID is bigger. The storing process is done not by scan thread, but by a separate thread (WifiStorer), which reads from a blocking queue (WifiQueue) . A list of 50 APs (can be moved up if necessary) is put into the blocking queue as an item and storer thread is blocked on no item in the queue to pretend writing around storage the whole time.

• Uploading: As old app, the uploading depends on user’s setting: manual, automatic on internet or on wireless connection. User can also set up the start number of APs to trigger auto upload from 5000 to 50000. User can only trigger uploading at at least 250 APs. The new app processes uploading with maximal 5000 APs once in order to pretend out-of-memory problem at device with small ram. An message contains upload process summary, new rank or error is given back to user. The WifiUploader uses UploadingQueryUtils for openwifi.su and can be changed quickly if backend changes.
• Scanning: Scan period is dynamically set not only depending on speed but also night mode and I am working on movement detection base on sensors. Every 2s as default , the scanner thread sends Wifilocator a request on position and scanned wifis. Wifilocator uses GPS to define position, in case of no GPS defined, scanned wifis are used to define user’s position which is not working anymore at old app. The method for location can be displayed by overlay big number color as in setting.
• Resource is managed and can be set as user’s options (kill app on low battery, on long time no gps etc.). While the old app triggers compare on noExistGps every time it scans, the new apps starts a new thread for checking resources only if user configures.
• The old app exports only user’s own bssid and puts the export file/requires import file default in external storage. The new app allows user to import/export account with team and tag information as well as reset all settings back to default. User can browser the import file, as well as save the export file her/himself at any writable place in storage.
• A map of all data from openwifi.su and a map from user’s contributed data is immigrated directly into the app (using osmdroid) , as well as a ranking list.
• Min app api is moved to 19, which covers over 96% devices currently [4]. Permission is checked at runtime as required.

What I learned

I learned a lot about android developemnt.

• Life cycles: The UI and service communicates per LocalBroadCast, the BroadCastReceiver has to be register and unregister on Resume/on Pause. Also the SettingPreferenceListener to have a proper lifecycle management of the activity.
• Using LocalBroadCast instead of global BroadCast to keep data inside the app only.
• From Api 23, the “dangerous permission” has to be asked at run time. Many system flags and parameters differ from api versions, which requires a lot of version control check in runtime
• Working and managing service with a lot of parallel processes.
• Osmdroid: open source lib for working with osm which is compatible to google map API
• Database sqlite with lib Room
• etc. 

  Also I learned how important architecture is since I was too fast jumping in coding where my mentor had to stop me and gave me some helpful advice. We re-designed the architecture with a component controller in center. All other components should only do its job and communicate with controller and not with others directly to make it easy to extend/replace any components.

  Difficulties I met

  It was hard to work on the app while I have no access on backend. I have to test all the APIs while analyzing the old app, which is also not nicely documented and implemented. Furthermore the backend is quite unstable and sometime unreachable. Another problem is testing. Since the app works with collected wifi access points, testing and debugging at home became very hard.

  Future plan

  There are still some points on the app performance I want to optimize further. I already started on working with the android sensors to detect movement, to scale the scan time more effectively to save resource since scanning wifi and gps are two of the services which cost bunch of phone battery.

  The app is currently only in development mode since I haven’t had a google play store account. But as soon as I do, I will release it. Until then, if you want to try it, an .apk is to download here [5]

  Acknowledgement

  Many thanks to freifunk community and my mentor Jan-Tarek Butt for this amazing opportunity. Even though there are still some small stuffs to do/fix, I am so glad that a new wardriving app is coming soon for openwifi.su. Many thank to Google summer of Code team for making this happen.

  [0] https://blog.freifunk.net/2018/05/14/introduction-openwlanmap-app/

  [1] https://blog.freifunk.net/2018/06/10/openwlanmap-app-update-1/

  [2] https://blog.freifunk.net/2018/07/09/openwlanmap-app-update-2/

  [3] https://github.com/openwifi-su/OpenWLANMap-App

  [4] https://developer.android.com/about/dashboards/

  [5] https://androidsmyadventure.wordpress.com/2018/06/03/openwlanmap/

Hi,

In the last weeks I was working on  the storing process as described in the architecture in the last blog post [0].

Storage Handler:

Old app: the old app saves the data as byte in a file. A data entry is 28 bytes of MAC-address(12 bytes for 12 characters) and latitude(8 bytes for double) and longitude(8 bytes for double). An entry could be saved more than once in the file. There are 2 files, one for data which should be updated and one for data which should be deleted from backend.

New app: Firstly I wanted to adapt the structure from the old app. But since I saw some unreasonable points such as saving redundant data, flash workload, maintenance problem and unstructured storage, I decided for a standard database with more structure and easy to maintain: sqlite. Also I am using the new persistence lib, which provides an abstract layer for database: Room, newly released last year, as a part of android architecture components, with a lot of bug fixed since then. A lib with a lot of advantage when working with sqlite database: verify queries at compile time, reduce a lot of duplicate code in comparison with the last approach with DbHelper etc. In order to store the access point in the database, I implemented a seperate thread, which reads data from a blocking queue and saves it in the database, which works parallel with the scan thread and will be interrupted if there is nothing in the queue to store. Also to save energy and not force the store thread to run the whole time, a list of access points will be put into the blocking queue as an element. To pretend redundant data in storage, a data entry with BSSID will not be saved many times as in the old app but only once. The BSSID is used as primary key in the sqlite table. It will be updated the next time if the received signal strength is better than the last entry in the database. An explicit transaction is implemented to solve this case since the lib Room has only supported annotation for standard update/insert. To decide if a access point should be deleted or updated from backend, a flag is set.

Upload Handler:

The WifiUploader is in process. I did take a look at the uploading format in the old app and how it communicates with the current backend. Also the upload sequence is already defined, mean the scanning thread will be interrupted, all the rest of access point will be stored, the store thread will be interrupted to pretend conflict while 2 threads try to access same database at the same time before the uploading process is started. Also the WifiUploader will read maximum a number of data entries from the database and upload it, not the whole database like old app but one after another,  in order to pretend out-of-memory problem at device with small ram. (see more in the below diagram)

But since I am in the middle of my final exam period, there will be a small delay until this weekend for the WifiUploader to be published. Also from next week I will spend full time making the other features done includes implementing all saving resource features such as adaptive scanning, implementing all settings option. A clean and full documentation will be provided at the end as well.

Available issue: Permission request and handling

[0] https://blog.freifunk.net/2018/06/10/openwlanmap-app-update-1/