Relish: A new serialization format

I’ve recently created and open sourced a new serialization format, named Relish1, and a Rust library implementing it. They can be found here. I don’t expect many people to adopt it – there are quite a few choices for serialization libraries available to people already – however, it has some nice design properties that I want to describe.

Relish is a binary serialization format; it does not have an IDL. I drew ideas primarily from ASN.1 DER and Protocol Buffers. I’ve maintained a Rust ASN.1 DER library for several years, and I’ve been known to tell people that most of the reasons they think ASN.1 is bad are wrong, and in fact there are a bunch of other things that are bad about ASN.1.2 As much as anything else, Relish was created as a tech demo for what it’d look like if we took some basic ideas about serialization formats and implemented them cleanly. It’s designed for things like RPC messages or database values that have modestly sized messages.

Relish is a Type-[Length]-Value format. Each type code is defined as either being a fixed-width type or a variable-width type. Fixed-width types do not encode a length, variable-width types do. For variable-width types, lengths are encoded using a varint encoding: lengths less than 128 are encoded in a single byte, and lengths up to 2**31 - 1 are encoded in four bytes with one signaling bit (larger messages are unsupported).3 Compared to more general varint schemes (like LEB128), this allows much simpler code for encoding/decoding, while keeping minimal overhead (though encoding the value “200” takes two extra bytes compared to LEB128, because we’re encoding a length we know that we have at least 200 bytes of data, meaning we’re still at only 1% overhead).

Relish supports a variety of types: null, bools, fixed-width signed and unsigned integers, UTF-8 strings, arrays, maps, structs, enums, and timestamps. You may have noticed that there’s no bytes type at all. That’s because it’s not necessary, due to clever choices around how arrays are encoded.

You can have an array of any other type. An array’s value begins by encoding the type code for its element’s type, followed by the encoding of the value for each element. As a concrete example, to encode an array of u8 containing b"\x01\x02\x03", the encoding looks like [array type code] [length (4)] [u8 type code] 0x01 0x02 0x03. This nicely generalizes: the encoding of an array of any fixed-width integer type can be a simple memcpy from an in-memory buffer. Arrays of variable-width types are still supported, each element is encoded with its length prefix.

Structs use explicit integer field IDs for each field, which allows for extremely clear backwards compatibility semantics. Similarly, enums use an integer field ID to indicate which variant is encoded. This is in contrast to ASN.1, which relies on different type tags to identify fields (with EXPLICIT/IMPLICIT tagging to disambiguate where required). One of the consequences of this choice is that serde cannot be used in Rust (because it doesn’t support integer field tags).

Overall, Relish can be (hopefully!) seen as a set of reasonable choices, executed well, with no unforced errors. I have not yet declared the spec final, meaning for a short period I’m still considering backwards-incompatible changes to it. If, despite the plethora of other options, you think this might be of use, please take a look and provide feedback (particularly feedback that can only be resolved with a backwards-incompatible change).

  1. So named because, Pickles are for Delis, not for Software ↩︎

  2. In particular, many people believe that ASN.1 is responsible for a huge number of security issues. In reality, the vast majority of “ASN.1 security problems” are actually memory unsafety issues. ASN.1’s biggest issues are that it has a Noah’s Ark of available types, including no fewer than 12 string types, 2 bytes types, and 7 date/time-related types. ↩︎

  3. I don’t think this approach is novel, however despite its utility I haven’t been able to find a name for it! ↩︎