Author: jess3jane

GSOC 2019 is drawing to a close and with it the first part of the HTTP API. To be completely honest this is not where I’d hoped to be at this stage, but we’ve set up a solid foundation for future work to build the rest of the API.

A Framework

I wrote about choosing a web framework a while back. The choice I arrived at, Iron, was in retrospect not the best choice. I chose it because I liked its middleware model, but it has only recently been renewed and most of its ecosystem hasn’t been updated in three years. This meant writing a lot of new middleware for the API. Additionally the only existing Iron testing framework was difficult to use and relied on fragile string manipulation to generate Request objects. I wrote a new testing framework called Anneal which uses hyper to generate Request objects and follows a builder pattern to simplify testing.

A Service

The HTTP Api is designed to operate as an independent service within a Qaul instance. An instance may disable the API, or not ever include it, and things should still work. Currently the mechanism by which services communicate is still a work in progress but a big part of the api was trying to pull as much of the boilerplate code used for validating incoming requests into the http-api service as possible.

Authentication is handled by the api, parsing of JSON:API requests (using the json-api crate I talked about in a previous post) is handled by the API, scoping of requests is handled by the api. The benefits of this model will become more apparent as we start to give services HTTP APIs.

A Plan

While my work on Qaul under GSOC is coming to a close I fully intend to complete my work on the HTTP API. I have a branch for adding unit tests to the api waiting on my user creation merge request, I have been experimenting with implementing an inter-service messaging system.

I added an http-api service, built login and logout endpoints, built a hot pluggable mount middleware for use in mounting services, implemented middleware for dealing with cookies and authenticating with cookies. I have written extensive error messages and documentation for all of these components and hopefully the foundation they have created will allow future development of the API to proceed with ease.

So long, and thanks for all the fish.

The new qaul.net HTTP API speaks JSON, as do increasingly many things. It allows you to express complex types, it maps well to most programmers’ mental models, it’s self describing, and there’s a decent library for it in every language under the sun. In the Rust world JSON is primarily dealt with using the serde_json crate (https://crates.io/crates/serde_json) which allows the programmer to easily map strictly typed structures into JSON and back. Today we’re going to be talking about the difficulties we encountered building a type for JSON:API’s relationship data field (https://jsonapi.org/format/#document-resource-object-linkage).

The data field of a relationship can be any of the following things:

• non-existant
• null
• a single object
• []
• an array of objects

Each of these options semantically represents a distinct thing and so we should be able to tell them apart. We will use the following enumto represent our value:

#[derive(Serialize)]
#[serde(untagged)]
enum OptionalVec<T> {
  NotPresent,
  One(Option<T>),
  Many(Vec<T>),
}

Non-existant relationships will be represented by OptionalVec::NotPresent, to-one relationships (empty or otherwise) will be represented by OptionalVec::One, and to-many relationships will be represented by OptionalVec::Many.

Serializing

To allow our enum to serialize properly we just need to implement a method to tell if it’s supposed to be present or not:

impl<T> OptionalVec<T> {
  pub fn is_not_present(&self) -> bool {
    match self {
      OptionalVec::NotPresent => true,
      _ => false,
    }
  }
}

Now when we wish to use our enum we simply need to put the #[serde(skip_serializing_if = "OptionalVec::is_not_present")] attribute before the field.

Deserializing

To cover the OptionalVec::NotPresent case we will need to implement std::default::Default for OptionalVec as follows:

impl<T> Default for OptionalVec<T> {
  fn default() -> Self {
    OptionalVec::NotPresent
  }
}

Now whenever we use OptionalVec we need to add the #[serde(default)] attribute to the field. This tells serde to fill the field with a default value if the key isn’t present.

For the other options, we need to implement a custom deserializer. The technically proper way to do this is to build a Visitor, but we’re going to take the simpler route and deserialize it to serde_json::Value first. Our deserializer is as follows:

impl<'de, T> Deserialize<'de> for OptionalVec<T>
where T: DeserializeOwned {
  fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
  where D: Deserializer<'de> {
    let v = Value::deserialize(deserializer)?;
    match serde_json::from_value::<Option<T>>(v.clone()) {
      Ok(one) => Ok(OptionalVec::One(one)),
      Err(_) => match serde_json::from_value(v) {
        Ok(many) => Ok(OptionalVec::Many(many)),
        Err(_) => Err(D::Error::custom("Neither one nor many")),
      },
    }
  }
}

And that’s it! Effectively we try first to deserialize the singular case and if that fails we try to deserialize the multiple case. The first case will catch null as Option<T> will deserialize null as None.

Conclusion

This is just one of the many challenges we faced writing a framework for JSON:API parsing in Rust. The contents of this article in their proper context can be found here: https://github.com/qaul/json-api/blob/master/src/optional_vec.rs

These days, Rust has quite the collection of web frameworks to choose from. The Are we web yet? framework list lists some 17 frameworks and this number is growing constantly. One of my activities since the last update has been selecting one of these frameworks to base qaul.net’s http api on.

Requirements

For qaul.net we are doing our best to keep the binary size as low as possible. The qaul.net binary should be simple to move and the larger the program gets the harder that becomes. Aside size we also want to pick a framework that’s easy to use and that’s likely to be maintained for a while (it’d be a shame to have the framework be abandoned after we complete the project).

Results

From this initial list of 17 frameworks, 6 were cut because they were either abandoned or I couldn’t get any of their examples running.

We evaluated a series of test programs for various web frameworks and found that for the most part everything is within a couple hundred kilobytes in size. actix-web was the largest in our testing coming in at around 2.8 Mb for a fairly simple example program. This makes sense, actix pulls in the actix actor framework and that probably comes with a fair bit of code the other frameworks don’t bother with. rouille and Thruster were the smallest by a decent margin coming in at 1.2 Mb for a hello world. This makes sense as rouille is based on tiny-http and foregoes any sort of async code and Thruster by default uses it’s own backend. Most of the rest of the frameworks all came in between 1.7 and 2.3 Mb, and this too makes sense. Most of these frameworks were based on hyper and even using hyper directly gets you an executable around 1.6 Mb.

There were a few frameworks I was quite interested in that I decided against because of lack of documentation or non-functional examples.

Conclusion

In the end I decided to go with iron as I find it’s middleware model to be quite pleasant and extensible and it’s fairly well documented. Iron came in squarely in the middle of the pack in terms of binary size but I had strong maintainability and usability concerns for the very smallest frameworks making this choice a matter of a couple hundred kilobytes.

Next Steps

Presently I’m working on an authentication system for the api. Hopefully next month I’ll be able to talk about that and a bit more about the modular design of the api.

qaul.net allows spontaneous, ad-hoc networks to form between any wireless-enabled devices over whatever medium is available. Currently, the project is undergoing a Rust rewrite, which will provide enhanced security, modularity, and performance.

Project Overview

To make networking with qaul.net as easy as possible an HTTP API layer is being developed. This should enable applications to be written using qaul.net from any language with a decent HTTP implementation lowering the bar of entry substantially.

In the next couple of weeks a majority of my focus will be on making sure that we’re building from a strong base. As with all projects we want to make sure that we’re not going to end up with a hacky mess in a month or so and with qaul.net we want to keep binary size to a minimum. Rust is a language with many, many web frameworks so I’ll be spending a lot of time evaluating them looking mostly at ease of use and binary size.

Presently my top two contenders are `actix-web` and `gotham` but in the coming weeks I’ll be evaluating as many as I can so stay tuned for the results.

About Me

Hi, I’m jess, a Computer Engineering student in the Boston area.