Revisit of Flatbuffers vs. Protocol Buffers

Since this article (written by Kenton Varda in 2014), has anything changed about the use cases of using FlatBuffers vs. Protobuf? Alternatively, has something else come along that is now the preferred format/library for data exchange?

As best as I can tell, there seems to be no or minimal changes since the 2014 article which can be summarized by:

• Protobuf

  • Preferred for smaller messages (1MB or less)
  • More programmer friendly

• FlatBuffer

  • Used for larger messages
  • Optimized for efficient parsing
  • Better in-memory representation

There's been a lot of improvements to FlatBuffers since 9 years ago, but from that table alone these entries would change:

RPC system: FlatBuffers has out of the box gRPC support (for multiple languages).
Usable as mutable state: FlatBuffers now has an "Object API" that is similar to Protobuf, on top of the base API.
Other languages: lots!

It is perfectly fine for smaller messages, as long as you don't expect the same level of "compression" Protobuf's varints give.

The Object API is more programmer friendly, though is also slower, much like Protobuf.

As ever, with such questions the answer is "it depends"!

Personally speaking, I'd argue that that feature list is incomplete. I'd add:

• "Constraints" - the ability to define valid value ranges for fields, and valid lengths for lists
• "Multiple wire formats" - the ability to encode data in different ways, e.g. highly packed bit-optimised encodings for transmission over radio links (which tend to be bandwidth constrained) as well as more programmer-friendly formats like XML
• "Values Definition" - the ability not only to define data types, but to also define fixed instances of data types that show up in generated source code
• "Constraints Definitions in Terms of Values" - the ability to use defined values in the specification of constraints. This should include syntax to indicate "1 less than X" where X is a defined value
• "Strict Contract" - the wireformat is rigid in its representation of the schema

Constraints

Serialisation schemas such as ASN.1, XSD, and JSON schema all allow constraints on value and length. This is very useful, because it's fairly rare that unbounded values and lists are in fact valid within the application. Having constraints expressed in the schema and honoured in the generated code is comparatively rare - some paid-for XSD / XML tools do it, most ASN.1 tools do, and most JSON validators do.

The advantage is that one can have a more exact contract.

I'm not quite sure what is meant by "safe against malicious input" in that feature set. But so far as I know, in GPB the only way of conveying what is a valid value for a message field is to 1) put it in as a comment and hope the developer spots it, or, 2) use the (beta) third party extensions for GPB that mimic ASN.1's constraints.

Multiple Wire Formats

It's sometimes useful to be able to serialise data to, say, a packed binary format for wireless transmission, but also be able to serialise it to a readable format like XML / JSON. GPB does this kind of thing (JSON), but not to the extent ASN.1 does (numerous binary encodings, plus JSON and XML wireformats too).

ASN.1 uses the constraints to understand, for example, exactly how many bits an INTEGER field actually needs. If it's constrained to valid values between 1000 and 1015 inclusive, it'll use only 4 bits in its unaligned packed encoding rules.

The advantage is that you can get good efficiency on links where that matters, but also more programmer-friendly storage where size is less relevant.

Values Definition

If one is using a schema to define an interface between systems, it's quite likely there are system constants that they need to share. Putting them in a schema is quite useful.

Of them all, AFAIK only ASN.1 has this in its schema language.

Constraints Definitions in Terms of Values

This is just an extension of constraints. Rather than express constraints with literals, express them with values. For instance, in ASN.1 you can have:

listLen INTEGER ::= 10

List ::= SET
{
   list [0] SEQUENCE (SIZE(listLen)) OF REAL,
   defaultEntry [1] INTEGER (0..<listLen)
}

The first line defines a constant int of value 10. The next chunk defines a class that contains a list of floating point values listLen entries long, and an index into that list that is constrained to the values 0 to 9. The application logic would use listLen to iterate over the list, and the defaultEntry is guaranteed to be a valid value. If you ever need the list to be 11 long, just change line 1 and rebuild.

For communications systems this can be gold-dust; the entirety of the protocol message set and the protocol constants can be defined this way, and any tuning of it takes place solely within the ASN.1 schema; no application source code need be changed (you just tweak and rebuild).

Strict Contract

This is where valid wireformat strongly adheres to the schema. GPB is quite bad at this - for example it'll quite happily and silently parse multiple oneof fields in a message keeping only the last one, which I find surprising; I'd expect the detection of multiple fields to throw some sort of error!

Overall

Essentially, it boils down to "I quite like ASN.1", born out of experience building communications sytems. It's not surprising that the telephony industry is built on ASN.1.

It's quite intersting seeing teams that don't consider aspects such as the above launch off into project development and just pick the first thing they've heard of, and then get into a dreadful mess when it turns out that no one has really read the ICDs, code bases become to difficult to change, etc.

ASN.1 might have old origins, but it's been constantly updated over the decades and solves an awful lot of problems when it comes to getting systems communicating easily and reliably. Good tools for it cost money, but I'm quite happy to spend a bit of money to safe a ton of time, risk.