qaul.net – Strictly Typed Code in a Stringly Typed World

The new qaul.net HTTP API speaks JSON, as do increasingly many things. It allows you to express complex types, it maps well to most programmers’ mental models, it’s self describing, and there’s a decent library for it in every language under the sun. In the Rust world JSON is primarily dealt with using the serde_json crate (https://crates.io/crates/serde_json) which allows the programmer to easily map strictly typed structures into JSON and back. Today we’re going to be talking about the difficulties we encountered building a type for JSON:API’s relationship data field (https://jsonapi.org/format/#document-resource-object-linkage).

The data field of a relationship can be any of the following things:

  • non-existant
  • null
  • a single object
  • []
  • an array of objects

Each of these options semantically represents a distinct thing and so we should be able to tell them apart. We will use the following enumto represent our value:

#[derive(Serialize)]
#[serde(untagged)]
enum OptionalVec<T> {
  NotPresent,
  One(Option<T>),
  Many(Vec<T>),
}

Non-existant relationships will be represented by OptionalVec::NotPresent, to-one relationships (empty or otherwise) will be represented by OptionalVec::One, and to-many relationships will be represented by OptionalVec::Many.

Serializing

To allow our enum to serialize properly we just need to implement a method to tell if it’s supposed to be present or not:

impl<T> OptionalVec<T> {
  pub fn is_not_present(&self) -> bool {
    match self {
      OptionalVec::NotPresent => true,
      _ => false,
    }
  }
}

Now when we wish to use our enum we simply need to put the #[serde(skip_serializing_if = "OptionalVec::is_not_present")] attribute before the field.

Deserializing

To cover the OptionalVec::NotPresent case we will need to implement std::default::Default for OptionalVec as follows:

impl<T> Default for OptionalVec<T> {
  fn default() -> Self {
    OptionalVec::NotPresent
  }
}

Now whenever we use OptionalVec we need to add the #[serde(default)] attribute to the field. This tells serde to fill the field with a default value if the key isn’t present.

For the other options, we need to implement a custom deserializer. The technically proper way to do this is to build a Visitor, but we’re going to take the simpler route and deserialize it to serde_json::Value first. Our deserializer is as follows:

impl<'de, T> Deserialize<'de> for OptionalVec<T>
where T: DeserializeOwned {
  fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
  where D: Deserializer<'de> {
    let v = Value::deserialize(deserializer)?;
    match serde_json::from_value::<Option<T>>(v.clone()) {
      Ok(one) => Ok(OptionalVec::One(one)),
      Err(_) => match serde_json::from_value(v) {
        Ok(many) => Ok(OptionalVec::Many(many)),
        Err(_) => Err(D::Error::custom("Neither one nor many")),
      },
    }
  }
}

And that’s it! Effectively we try first to deserialize the singular case and if that fails we try to deserialize the multiple case. The first case will catch null as Option<T> will deserialize null as None.

Conclusion

This is just one of the many challenges we faced writing a framework for JSON:API parsing in Rust. The contents of this article in their proper context can be found here: https://github.com/qaul/json-api/blob/master/src/optional_vec.rs

qaul.net – Choosing a Web Framework

These days, Rust has quite the collection of web frameworks to choose from. The Are we web yet? framework list lists some 17 frameworks and this number is growing constantly. One of my activities since the last update has been selecting one of these frameworks to base qaul.net’s http api on.

Requirements

For qaul.net we are doing our best to keep the binary size as low as possible. The qaul.net binary should be simple to move and the larger the program gets the harder that becomes. Aside size we also want to pick a framework that’s easy to use and that’s likely to be maintained for a while (it’d be a shame to have the framework be abandoned after we complete the project).

Results

From this initial list of 17 frameworks, 6 were cut because they were either abandoned or I couldn’t get any of their examples running.

We evaluated a series of test programs for various web frameworks and found that for the most part everything is within a couple hundred kilobytes in size. actix-web was the largest in our testing coming in at around 2.8 Mb for a fairly simple example program. This makes sense, actix pulls in the actix actor framework and that probably comes with a fair bit of code the other frameworks don’t bother with. rouille and Thruster were the smallest by a decent margin coming in at 1.2 Mb for a hello world. This makes sense as rouille is based on tiny-http and foregoes any sort of async code and Thruster by default uses it’s own backend. Most of the rest of the frameworks all came in between 1.7 and 2.3 Mb, and this too makes sense. Most of these frameworks were based on hyper and even using hyper directly gets you an executable around 1.6 Mb.

There were a few frameworks I was quite interested in that I decided against because of lack of documentation or non-functional examples.

Conclusion

In the end I decided to go with iron as I find it’s middleware model to be quite pleasant and extensible and it’s fairly well documented. Iron came in squarely in the middle of the pack in terms of binary size but I had strong maintainability and usability concerns for the very smallest frameworks making this choice a matter of a couple hundred kilobytes.

Next Steps

Presently I’m working on an authentication system for the api. Hopefully next month I’ll be able to talk about that and a bit more about the modular design of the api.

GSoC 2019 – qaul.net HTTP API

qaul.net allows spontaneous, ad-hoc networks to form between any wireless-enabled devices over whatever medium is available. Currently, the project is undergoing a Rust rewrite, which will provide enhanced security, modularity, and performance.

Project Overview

To make networking with qaul.net as easy as possible an HTTP API layer is being developed. This should enable applications to be written using qaul.net from any language with a decent HTTP implementation lowering the bar of entry substantially.

In the next couple of weeks a majority of my focus will be on making sure that we’re building from a strong base. As with all projects we want to make sure that we’re not going to end up with a hacky mess in a month or so and with qaul.net we want to keep binary size to a minimum. Rust is a language with many, many web frameworks so I’ll be spending a lot of time evaluating them looking mostly at ease of use and binary size.

Presently my top two contenders are `actix-web` and `gotham` but in the coming weeks I’ll be evaluating as many as I can so stay tuned for the results.

About Me

Hi, I’m jess, a Computer Engineering student in the Boston area.