What is Object Serialization?

Objects organize data in memory according to their defined semantics within the programming language. Transferring that data to other environments–such as for persistence or transmission across a network– often requires conversion. Serialization is a general, well-understood term for this process. It is not the only term. Ruby calls it Marshalling, a common phrase in the 1980s for such operations also included in Robert Martin’s Clean Code--though used there for command line parameters. .NET also uses “serialization” and “marshalling”. Python calls it “pickling”. Yet common usage of “serialization” occurs in their documentation as well, and “serialization” is so common that it is even used in the documentation of “exotic languages” like Haskell and Julia.

Bytes? Can’t Objects be serialized to JSON/XML/CSV/YAML/...?

Many works also describe data formats such as the JSON, XML, and CSV as “serialization.” This arguably abuses terminology.

In the early days of computing, it was quite common for software companies to develop everything from scratch. This usually included both binary encoding for objects and their transmission protocols. As the desire for interoperability rose, software engineers started pushing to adopt more unified standards for object representation. This led to the rise of efforts like CORBA, RMI, and web services described in WSDL. As the World Wide Web exploded in popularity and network systems became the norm, using strings across the wire became another de facto standard. Hunt and Thomas' Pragmatic Programmer provided a watershed moment. They argued, quite convincingly, that human-readable string formats are superior for readability, debuggability, and extensibility. Nevertheless, it's worth considering the performance trade-off.

Consider the case of a birthdate. For simplicity, assume ISO8601. Figure 1 shows a basic Java class representing this concept–let’s ignore the lack of error checking for now, it’s not relevant for this example.

A birthdate in ISO8601 is represented as YYYY-MM-DD. Assuming a UTF-8 encoding, each glyph requires 2 bytes. Hence a string of 20 bytes in extended representation.

Integers are 4 bytes, so a birthdate transmitted as 3 integers requires 12 bytes. Clearly a savings. Can the size be reduced further?

Further optimization realizes that valid months (1-12) can be represented in only 4 unsigned bits, and valid dates (1-31) can be represented in 5 unsigned bits. Years are domain dependent. An 8 byte Java Long can go back to the Big Bang and forward to the projected Heat Death of the universe. Most domains dealing with birthdates, however, don’t need such length. If a particular domain requires birth dates within a bounded range, such as 1920+128 years, it is possible to use 7 bits to represent the year. In such a domain, one can represent a birthdate in 2 bytes.

Software engineers have long known the difference in efficiency between text and binary formats. Yet often such concerns are described abstractly in the literature, which can downplay the extent of the impact. Service Oriented Architectures with web services at a scale of thousands or more TPS often have significantly different performance profiles and resource needs when payloads are orders of magnitudes smaller. Such differences can really add up.

Investigating existing serialization libraries

As object serialization is a standard problem, many solutions have been developed for solving it. From a purely theoretical point of view, analyzing and comparing the entire population of solutions would be ideal. From a practical point of view, the amount of work necessary to discover, collate, and analyze dozens/hundreds/thousands of solutions is prohibitive. Consequently, I sample available solutions.

Sampling itself is a bit more art than science, with certain risks and biases attached. One risks omitting important libraries that many software engineers would likely want to try. Nevertheless, using intuition from field experience with thousands of Java projects, as well as my perceptions of library popularity by others, I’ve chosen a handful of libraries to review. I outline some rationale for excluding others.

Ignoring certain options

ProtocolBuffers/FlatBuffers/Thrift/Avro/MessagePack

Projects such as Google’s ProtocolBuffers and FlatBuffers, Apache’s Avro and Thrift, and MessagePack offer options for language-independent data serialization. Arguably they represent an advanced approach to serialization.

These projects apply a Schema First model. In Schema First, an object’s structure and data format are pushed outside of the code of a particular application. This allows for cross-platform sharing–such as serializing an object in Java and deserializing it in Ruby– and swapping a format such as XML to JSON to save bytes.

The downside of these options is that they generally force the use of separate extra-linguistic definitions of structures written according to their specification. A project that uses one of these options includes the library dependency and modifies its build process to generate code from these definitions. This extra setup is unnecessary in projects where Java is known to be used on both sides of serialization/deserialization.

They add some conceptual complexity–projects using them must learn an additional format and syntax outside of the code itself. Applications content to ignore schema and wire format as implementation details may find such complexity not strictly necessary. Tools such as the Java Serialization framework or Jackson function well for such applications.

They also invert the thinking of the application development process. Rather than writing Java classes and defining their serialization format later–so-called Code First– users should define the type in external files and rely on generated code–so-called Schema First. An ideal solution would enable Code First style development and be able to generate the platform-independent schema from the Code.

Java is particularly well-suited for this. Annotation Processing allows a build-system independent approach to code generation as the Java compiler itself generates the code—rather than an external tool or a plugin integration into a build system--and there are many in Java like Ant/Maven/Gradle/SBT/Bazel/etc. Once the rules for mapping a Java object to some platform independent schema are well known, generating the schema from an object can simply be integrated into an annotation processor.

Without commenting on the viability or effectiveness of such tools, I will omit them from analysis. I do not want to take on the additional overhead of writing extra files for objects that I plan to be canonically defined in Java classes, simultaneously learning different specifications such as Avro schemas or ProtocolBuffer messages and incurring the cognitive load of mapping between them. I believe the right Serialization API in Java enables usage of such tools seamlessly without disrupting a Code First iterative development process.

Random JSON/XML/YAML bespoke library

The wide selection of available serialization libraries complicates broad sampling. Results discovered can be greatly influenced by search methodology, including what the precise search query is, when the query is made, which search engines are used, geographical location of the query, and more. Some libraries discovered but not explored in-depth during my inspection include:

Why not include these libraries? Many I had never heard of before some searches on Google in late 2022. Many have specific implementation issues that I believe hinder an overall design, like a proliferation of annotations. I have yet not found enough evidence that these are advanced enough in adoption, performance, or architectural novelty, to warrant further investment. Based on community feedback and reception, I may include them at a later date

Selecting serialization libraries for analysis

• Speed
• Leanness and Code Quality Metrics
• Architecture and archaelogical analysis