Ordered Binary Representation of JS Primitives

Key-value stores are a powerful tool for fast storage of data. We use LMDB, which provides phenomenal performance, for caching data. However, storage systems like LMDB have a low-level interface, using binary data for keys (and values). In JavaScript it is much more convenient to use JS primitives like strings and numbers as identifiers or keys (null and booleans are also primitives). What is the best way to convert these primitive values to binary form, as a Buffer in Node.js, for use in getting and storing values in key-value store? We have created the ordered-binary package to provide a single simple tool for serializing primitives to binaries and back with a compact representation with consistent ordering. Let’s consider why we need this.

Node.js has functions for converting primitives to binary data in buffers. However, there are a number of shortcomings of using these for the purpose of keys for key-value stores. First, you must use a different method for each primitive type (at least to do so efficiently). Not only is there a different function for converting a string to a buffer, but making a compact representation of a number can potentially involve over a dozen different functions to choose from. Using a 64-bit representation for consistently small numbers is very inefficient. One can also use a string representation of numbers for flexible length, but it is inefficient for large numbers. And it is important that you know which function was used to serialize the primitives in order to convert it back from binary to primitive form.

Another consideration with binary keys is that key-value stores are typically ordered, with the ability to query or traverse over a range of keys. This is very important and valuable functionality. But ordered keys only work coherently, if the key binary data is ordered coherently. And if you mix different functions for different primitives (even adjusting for different number sizes), the binary keys will no longer be ordered consistently with actual number magnitudes.

The ordered-binary package solves these issues by providing a single simple function that can convert any primitive to a compact binary buffer, records the type and has deterministic, consistent ordering. This package provides distinct representation for strings and numbers (again, type preserving), and handles both small and large numbers, positive and negative, integer and floating point numbers, using a minimum number of bytes for each, while again preserving ordering even between floating numbers and integers.

Here are some examples of how we can use the serialization function:

import { toBufferKey } from 'ordered-binary'
let buffer1 = toBufferKey('and')
let buffer2 = toBufferKey('then')
buffer1.compare(buffer2) // -1 "then" comes after "and"
toBufferKey(1).compare(toBufferKey(2)) // -1, 2 comes after 1
toBufferKey(1.73).compare(toBufferKey(2)) // -1, 2 comes after 1.73
toBufferKey(-2.3).compare(toBufferKey(-1)) // -1, -1 comes after -2.3
toBufferKey(344).compare(toBufferKey('hi')) // -1, strings always comes after numbers

And we can easily parse ordered-binary buffers back to primitives, which would be important for traversing a range of keys where the database is returning keys as binary/buffer data that needs to be converted to JS primitives:

import { toBufferKey, fromBufferKey } from 'ordered-binary'
let buffer = toBufferKey('hi')
fromBufferKey(buffer) => 'hi'

Finally an example of how you would use this with a key-value store, like lmdb-store:

let result = store.get(toBufferKey('test')) // get the entry for "test"

The ordered-binary package is a very simply little library, that makes it very easy to use JS with key-value stores with efficient and consistently ordered keys.