[GSoC’21] API Generator and tools with Draft-7 JSON schema

Before you ingress

Hello everyone!, The main intention of this project is to update the existing API spec schema files to the latest version of JSON schema, and also few tools which are dependent on the spec files.

The current latest version of JSON schema is version 2012-12, but unfortunately there isn’t much support to update the dependent tools(to be specific, the generator). So my mentor Andibraeu and I have chosen to work with draft7 version, as it has upper hand with implementation support compared to other recent versions.

Our initial map out

  • Migrate all the spec schema file to draft 7 version.
  • Pick out a framework to update the generator.
  • Generate and test the forms.[1]
  • List and update all the remaining dependent tools.
  • Test the updated tools.
  • Fix bugs, if any.

Spec files

Migrating the spec files to draft 7 version is not a difficult task. At the beginning, I have only migrated the recent version of spec file, so that we can immediately start working with the tools. And after updating all the tools, I have migrated all the spec files to draft 7 version.

Some features to point out:

References:

  1. Initial Migration -pull request
  2. Patch -pull request
  3. Patch -pull request

Generator

This is one of the significant tool depending on spec schema files. The job of the tool is very simple, it takes JSON schema as input, generates HTML forms to render in the browsers, handles validation of the form data against the input JSON schema and finally, generates a JSON out file.

Generator Layout

Unfortunately, no framework seemed perfect at the beginning, so I have picked up several frameworks to try them out and weigh the pros and cons to finally pick one. I had mentioned all the pros and cons in a document(check the references section).

Frameworks that I have tried

  • UI Schema for React
  • React JSON schema forms (mozilla)
  • Restspace Schema forms
  • JSONForms (Eclipse Source)

By weighing all the pros and cons, we have chosen JSONForms (Eclipse Source) to proceed with the further development.

JSONFORMS (Eclipse Source)

And thereafter, I developed UI schema for custom layout for the form field in the webpage. Also, at this point, I have to develop a custom renderer to render a map for picking the latitude and longitude of the communities locations. So upon looking at the documentation I developed the custom renderer (renderer, control and tester) thereafter to adopt this renderer to schema we need to have single object which only embeds longitude and latitude fields. So I quickly discussed with my mentor and added a new spec file to the organization. In conclusion, we have our generator up and running a demo in GitHub Pages.

References:

  1. Evaluation of frameworks -document
  2. Implemented frameworks -repo
  3. Generator tool -repo
  4. live demo -GHPages

Dependent tools

The API viewer and the Travis job CI are completely dependent on the spec schema files to validate the communities API files.

API Viewer

This tool generates a static build of pages, which show the validation result of the communities API file data.

A Valid API file
An Invalid API file
Validation Errors

Improvements:

  • The prior tool existed in python 2, I have updated to python 3.
  • Update to Validate data against draft 7 schema and show validation.
  • Added datatables to list the communities.

References:

  1. Python 3 migration with Draft 7 data validation -pull request
  2. Patch -pull request

Travis Job

All the API files are collected in the directory repository. And this Travis job validates the data of the API file data, when they are updated or added to the directory.json

Travis job Build
Job console output

A build of the travis job can be found here.

Improvements:

  • Also, this tool (test) existed in python 2, I have updated it to python 3.
  • Updated to validate data against draft 7.

References:

  1. Python 3 migration with draft 7 validation -pull request

Common API(collector script)

If you recall as I have added new spec file by embedding latitude, longitude into an object to adopt with the jsonforms custom renderer for map picker using react-leaflet. This would affect a lot of other tools like Community finden, Kontakt, etc which are truly based on the lon, lat fields of the API files. But luckily all these tools use a summarized API file, And the collector script is used to collect all the communities files.

Improvements:

  • Deserialized geoCode object and appended the fields to the respective locations. So that the fields are set to their old locations.

References:

  1. Altering Location fields -pull request

References

Here are the previous blogs of the project at different stages:

  1. Initial Stage(Before coding period)
  2. Phase I evaluation

Wind-Up

I have started the project with minimal understanding of react, typescript and jsonschema. But it was very fun to understand and work on. I really liked this way to learn new things rather than reading or doing a course. Every issue that I have encountered had leaded me to understand the things briefly. I’m really thankful to freifunk for the opportunity. And a big shout out to my mentor Andreas Bräu for absolutely wonderful guidance and support.

~ Shiva Shankar aka sh15h4nk

[GSoC’21] JSON Schema WebUI Generator – II

Prologue

Welcome back!, This blog is for the first evaluation of the project.

Recap

In the prev section, we saw a viable way-out for the inconsistency of the generator tool for the latest version of the schema(2020-12) by mentioning some frameworks and also some targets.

Frameworks Review

I have tested the frameworks with the draft 7 version of the schema and I evaluated the framework by noting down the pros and cons of the results by each framework.

  • UI schema for react: This framework supports very advanced features of the schema, but has less UI integration. Also the framework supports the 2019-09 version of the schema but sadly UI schema cannot be segregated from our original JSON schema.
  • React JSON schema forms: This framework supports basic features of the schema and also has good UI customization, but has less integration of validation UI.
  • Restspace schema forms: This framework supports basic features of schema, but misses proper documentation.
  • JSONForms (eclipse source): This framework supports basic features of the schema and also has good UI customization with limited options.

A full document of the pros and cons of the frameworks can be found here.

Summary

By considering all the pros and cons of each framework, my mentor Andi and I have decided to work on with JSONForms (eclipse source).

Ongoing Status

I rewrote the schema for the API generator tool to the draft 7 version of JSON schema also I have added formats to the schema. And generated the forms from the schema. I have been using React library (JSONForms) to generate the forms. Also, I developed the UI schema which is required to generate the form.

I have rendered the form, and also the bounded data of the form, which is the current data of the in the form and will be updated on change of the form data. Also, I have rendered the validations errors to show all the validation errors at one place before and after submission of the form.

Validation and Submission.

The JSONForms only emits errors through an event. So I have added a state to track the errors emitted by the event and on the event emit I recorded the errors into the state.

And for the submission of form we have to consider there are no recorded errors and the form data should not be empty because I have recorded errors into state only if the forms data is not empty. Errors are emitted by the event even before starting to fill the form. By validating all this checks, I have generated the output JSON file.

Loading data

For the testing purposes, I have added a button to load the weimarnetz API file data into the forms. I have fetched the ffSummarizedDir.json file from api.feifunk.net which consisted of all the communities API file data and rendered all the communities into a select field. So then I can add a on change event to load the data into the form.

References

  • A full document of the pros and cons of the frameworks can be found here.
  • A live development site can be found here.
  • The GitHub repository link is here.

Accomplished Targets

  • Developing the Schema to the latest version (taking draft 7 as base version).
  • Choosing a Frameworks (Chosen JSONForms).
  • Integrating the updated schema with the framework to generate the forms.
  • Examining the forms (validating input data and functionality of the forms).

Final goals

  • Try creating a custom renderer for location (lon,lat) picker.
  • Upgrading all the dependent tools.(API viewer, Travis job validator)
  • Testing all the tools and fixing bugs if any found.

~ Shiva Shankar Genji aka sh15h4nk

[GSOC’21] JSON Schema WEBUI Generator

Prologue

Hello there, Major objective of this project is to update the JSON schema of the API generator tool and other tools to the latest version (2020-12 not perfectly sure) and integrate the schema with a WebUI generator framework to generate HTML forms.

Prior

Currently, JSON Forms is being used to generate the forms from a JSON Schema of version draft 3 for the API and other tools.

Here is the form(API Generator) generated by the JSON Forms, Which currently uses JSON Schema of version draft 3 to generate the forms.

This API generator simply generates a JSON file for the API.

This also has several other fields, which contains specified data about the community.

Discrepancy

As the JSON schema versions have been updated to 2020 – 12, but the current schema of the API and other tools is still at version draft 3. Updating the schema to the latest version is no such big deal. but there is an inconsistency with the JSON Forms as it only supports up to the draft version 3 of JSON Schema.

Viable Way-out

In order to update the schema to the latest version, I have to find a WEBUI generator framework or tool that generates HTML forms from the latest version of the JSON schema.

But Unfortunately, there is no tool or framework that generates forms from the latest schema version 2020-12.
As it is officially mentioned here. Then I had to limit myself to the previous versions, so my mentor Andi suggested choosing draft 7 as our base version for our tests and implementations.
Then I had to choose a framework that supports version draft 7 and above so that we can update the schema to higher versions in the future by considering the support and compatibility of the WEBUI generator.

Frameworks

I have chosen several frameworks based on the version support and compatibility of the frameworks to integrate the schema and test out the results.

UI Schema for React

The first framework I want to try out is UI Schema for React. because the official documentation of the JSON schema states that, this framework supports the most recent version of schema draft(2019-09) but with limitations.

Apart from this, it has this pretty decent documentation here.

React JSON Schema Forms (mozilla)

Next, I have this, while testing on the playground demo, This doesn’t support $ref. But yet I want to give it a try. Its documentation is here. It seems this has some limitations.

Restspace Schema Form (MIT)

This is something cool, This doesn’t depend on the versions of the schema but this depends on the features of the schema. All the supported features are here. As I see It supports everything, which might be in our schema, and I could not able to find good documentation for this framework yet.

JSONForms (jsonforms.io) (EclipseSource)

This supports version draft 7 of the JSON schema. It has documentation here. I see this is an extension to the current JSON Forms used by the API and other tools.

Targets

  • Developing the Schema to the latest version (taking draft 7 as base version).
  • Choosing a Frameworks (mostly UI Schema for React).
  • Integrating the updated schema with the framework to generate the forms.
  • Examining the forms (validating input data and functionality of the forms).
  • Try using a higher version of schema, and test out all the functionality.
  • Finding and resolving the bugs.
  • Updating all the tools (using the schema).
  • Final testing for all the tools using new generator and updated schema and resolving bugs if any.

~ Shiva Shankar Genji aka sh15h4nk