GSoC 2019 – Import public datasets to Retroshare network final evalutaion

Hi all!

On this final evaluation we focus the work on give to the automatic API wrapper generator support for openapi-swagger specification.

As Wikipedia said:

The OpenAPI Specification, is a specification for machine-readable interface files for describing, producing, consuming, and visualizing RESTful web services.

In addition, there are lots of tools that generate the front and backend code automatically from an OpenAPI specification. So, theoretically, an API interfacedefined with the OpenAPI specification, can easily be generated in an amount of languages using the OpenAPI Generator Project .

So with my mentor decided that this is more useful and priority for RetroShare project, because anybody could generate their own wrapper in a lots of languages in an automated way from the last commit of RetroShare master branch without write any code. 

So the workflow should be:

  1. Compile the RetroShare desired commit to generate the Doxygen documentation.
  2. Use the jsonapiwrapper-generator-openapi.py script that generates the OpenAPI YAML specification for RetroShare API.
  3. Generate the frontend code using  the OpenAPI Generator Project on to the desired language. For example, you can check the result for python on this example.

Following this steps, easily you can have an updated RetroShare API wrapper without coding anything!.

Some interesting tip about auto generated code is that the documentation are generated automatically with working examples.

Workaround

After we decided to give the support to OpenAPI specification we start to adapt the previous script https://gitlab.com/jpascualsana/retroshare-api-wrapper-generator/blob/master/jsonapiwrapper-generator.py to support the new specification. 

Trying to use the generated YAML against the generator validator shows us a lotsof warnings and crashes of our script. So we need to solve lots of bugs that we didn’t know that exists, bugs related of RsClasses generation, types etc… https://github.com/RetroShare/RetroShare/pull/1614

Also we deeply studied how to translate the RS API on to the OpenAPI specification, studying how to translate the types, types of functions, the authentication etc… 

Then, how to integrate the documentation on to the YAML to see the result on the generated code, following the specification. 

Finally we get in trouble with the asynchronous functions, that are not fully supported for the python generator and we are already looking for a solution.

So once we had the generated API wrapper we start to test it:

  • Tests for models: when the model objects are created using the OpenAPI Generator, you can pass an entire object as dictionary or instantiate it with the provided Class created by the generator. For example, on the link above we can see how to instantiate the object instead of pass it as dictionary.
groupMetada = RsGroupMetaData(m_group_name="Just another test channel2", m_group_flags=4, m_sign_flags=520)
channel = RsGxsChannelGroup(m_meta=groupMetada, m_description="Test it!")
req_rs_gxs_channels_create_channel = {"channel": channel}

Conclusion

So on this GSoC we started thinking that we  are going to code a way to import public datasets on the RetroShare network .

To do that we understood the necessity of create an automatically API wrapper, to do not code by hand each end-point of the API. 

Finally we found the necessity to provide a way to generate wrappers on any language needed on an automatic way, finding an already created solution on the OpenAPI Generator Project. 

Now is easier for developers to write applications over RetroShare network using the RetroShare JSON API, unifying the wrappers generation. 

On the back we let the project to import pulic datasets that I would like to finish on the following months, after GSoC, implementing the OpenAPI wrapper to the “import public datasets” repository. 

Repositories

Freifunk posts

  1. Arrival
  2. First evaluation
  3. Second evaluation
  4. Third evaluation

GSoC report

https://gitlab.com/jpascualsana/gsoc-2019-report

GSoC 2019 – Import public datasets to Retroshare network second evalutaion

Here again!

This evaluation I spend the work creating an automatically generated wrapper for the API. This wrapper is generated analyzing the Doxygen XML files generated when Retroshare is build. 

Creating the API wrapper

First of all, I modified the python script (made by @sehraf ) that generates the C++ API files, to create a python wrapper for the API. Analyzing the script and the XML files I get my script working generating a first version of the wrapper. Then I test the wrapper, giving support to async functions also. Some features of the wrapper are:

  • Document the code using DocString convention.
  • Parse also ‘manualwrappers’ like attempt login.
  • Requests with authentication and without.
  • Support basic authorization or token auth via ‘Authentication: basic bas64Token’ header.
  • Async methods support and callback implementation.

Here an example of the API wrapper generated: https://gitlab.com/snippets/1877207 . Some tests for the wrapper can be found here:

class TestMultiple(TestCase):
    def test_login(self):
        res = wrapper.RsLoginHelper.isLoggedIn()
        print(res)
        # Do login
        if not res['retval']:
            res = wrapper.RsLoginHelper.attemptLogin(ACCOUNT, PASSWORD)
            print(res)
            self.assertEqual(res['retval'], 0, "CANT LOG IN")
            return
        self.assertEqual(res['retval'], True, "is not loged in")

    def test_authorizedMethod(self):
        res = wrapper.RsGxsChannels.getChannelsSummaries()
        print(res)
        self.assertEqual(res['retval'], True, "Can't get channel summaries")

class TestAsyncMethods(TestCase):
    def cb(self,res):
        print("cb", res)
    def test_asyncMeth(self):
        wrapper.RsGxsChannels.turtleSearchRequest("XRCB", 300, wrCallback=self.cb, wrTimeout=4)

Creating Retroshare Classes wrapper

After that, the problem was that a lot of functions need to have Retroshare classes as parameters. For example, to create a Retroshare forum, are needed classes like RsGxsForumGroup that at the same time need other inner classes like RsGroupMetaData. With the first version of the wrapper all this classes was passed in JSON format that was really annoying to assemble. 
So the next step was parse also this Retroshare classes recursively from the XML files to a Retroshare classes wrapper. On this step it was difficult to parse it correctly: differentiating the different types of classes, and class attributes, translating the types to python, if they are enums, primitive types etc… Finally I created this second class wrapper so when you need to pass a RsGxsForumGroup to the API wrapper you can just instantiate it and the wrapper do all the necessary to transform it to python and call the API. Some features:

  • Parse “compound” classes (structs on C++) recursively.
  • Parse “enums” and get their values.
  • Parse “typedef” and “using” classes and translate it to the appropriate type on Python .
  • Document it using Docstring convention .

Here an example for the class wrapper: https://gitlab.com/snippets/1875153 . Some tests can be found here:

    def test_createChannel(self):
        channelMetadata = RsClass.RsGroupMetaData(mGroupName="TestChdddannelCreation2", mGroupFlags=4, mSignFlags=520)
        channel = RsClass.RsGxsChannelGroup(mMeta=channelMetadata, mDescription="Channel Test")
        res = wrapper.RsGxsChannels.createChannel(channel)
        print(res)
        self.assertEqual(res['retval'], True, "Can't create channel")

For “v2” methods I opened an issue because I can’t communicate with the API. It was resolved as my “retroshare-service” wasn’t updated at all.

Next steps

This script will be adaptable to generate the wrappers for the language needed, for example for an OpenAPI format, TypeScript… Making much easy to other developers to start developing over Retroshare network.

Also it will be very easy to update when new feature is added to the API because it can be generated each time Retroshare is built.

Now will be time to apply the wrapper to the scripts that will import the public datasets!

GSoC 2019 Import public datasets to Retroshare network – Update 1

After three weeks of code here we have the first evaluation! 
The first week I started to talk with my mentors of how to guide the project. I started a [repo](https://gitlab.com/jpascualsana/retroshare-python-bot) to code a “bot” that will wrap the Retroshare JSON API for better interaction. But I didn’t continue the job because we are looking a way to wrap the API using Doxygen generation (looking at [https://gist.github.com/sehraf/23cbc8ba076b63634fee0235d74cff4b](@Sehraf work)).


So I get a list of different projects, provided by my mentors, and I started to[write different scripts](https://gitlab.com/jpascualsana/public-datasets-import) to import the data on to Retroshare network. Some of this projects are:

  • Wikimedia based projects
  • WordPress blogs
  • Gutemberg project
  • ActivityPub
  • RSS
  • Radio onda Rossa
  • XRCB.cat
  • RadioTeca

So this scripts parse in different ways this sites, and get their information as previous step to publish it on to RetroShare network categorized as channels.This scripts are able to:

  • Parse the site/project getting all “pages” of interest with different strategies.
  • Get updates (the pages that have changed since last time the info has been retrived)
  • Command line executable with argument parse. See -h option to get supported options.

So the next step is to use this scripts to import this information on Retroshare network using a wrapper dynamically generated by Doxygen.

On the next screenshot we can see the help for the script that import from ActivityPub

Library to export/import public datasets to Retroshare network

Hi all!
My name is Joan Pascual and I’m going to develop Library to export/import public datasets to Retroshare network.
RetroShare is a distributed F2F network that is in my daily use to share content or chat with friends. I like a lot the project and after the creation of the Retroshare JSON API I start to develop the RetroShare Web Bridges (https://gitlab.com/r3sistance/retroshare-web-bridges). And now, with GSoC, I would like to participate a little more with RetroShare project and its community.
The idea is to import public datasets into RetroShare network such as Wikipedia, WordPress or other stuff in order to populate RetroShare network with this data, creating distributed repository of all this information. 
Following with this, I’ll create a series of scripts that will help to import and update this information, so anyone can publish at the same time on RetroShare network or over a centralized service, creating some kind of bridge.
I’m going to investigate Doxygen, a part of the framework that compiles the JSON API endpoints for RetroShare, and I’ll will make the needed pull requests to expose new parts of the API that actually are not supported and could be needed for the libraries. Also maybe is possible to use Doxygen to create the library in python that will interact with API endpoints.
The whole project will be written in Python except when a new API endpoint should be exposed that will be in C++ and Doxygen (languages used by libretroshare), but this will be mostly adding inline documentation with the special @jsonapi{development} annotation.
At the end, it will be a reference library to interact with RS JSON API that will engage people to create apps over RetriSshare network. Also, taking advantage of this library, some public datasets will be imported to RS network.
I’m very happy to have been selected this year as GSoC student for RetroShare, a program that is in my daily life. So thanks to GSoC team, Freifunk, RetroShare developers and of course to my Mentors! It will be an awesome experience! 


RetroShare is a distributed F2F network that is in my daily use to share content or chat with friends. I like a lot the project and after the creation of the Retroshare JSON API I start to develop the RetroShare Web Bridges (https://gitlab.com/r3sistance/retroshare-web-bridges). And now, with GSoC, I would like to participate a little more with RetroShare project and its community.
The idea is to import public datasets into RetroShare network such as Wikipedia, WordPress or other stuff in order to populate RetroShare network with this data, creating distributed repository of all this information. 
Following with this, I’ll create a series of scripts that will help to import and update this information, so anyone can publish at the same time on RetroShare network or over a centralized service, creating some kind of bridge.
I’m going to investigate Doxygen, a part of the framework that compiles the JSON API endpoints for RetroShare, and I’ll will make the needed pull requests to expose new parts of the API that actually are not supported and could be needed for the libraries. Also maybe is possible to use Doxygen to create the library in python that will interact with API endpoints.
The whole project will be written in Python except when a new API endpoint should be exposed that will be in C++ and Doxygen (languages used by libretroshare), but this will be mostly adding inline documentation with the special @jsonapi{development} annotation.
At the end, it will be a reference library to interact with RS JSON API that will engage people to create apps over RetriSshare network. Also, taking advantage of this library, some public datasets will be imported to RS network.
I’m very happy to have been selected this year as GSoC student for RetroShare, a program that is in my daily life. So thanks to GSoC team, Freifunk, RetroShare developers and of course to my Mentors! It will be an awesome experience!