zubin
Read and Write Structured Binary Data
Describe a fixed binary record once, as a layout, and read or write millions of them in one vectorised call: integers from 8 to 128 bits, the half, single and double precision floats of the IEEE 754 (Institute of Electrical and Electronics Engineers) standard, 'bfloat16' (brain floating point), complex numbers, booleans, fixed bytes and fixed strings, padding, arrays and 'C' struct alignment, in either byte order. Typed codecs convert whole vectors to and from bytes at every width and byte order, refusing rather than rounding a value that does not fit, and a growable byte builder with a hard cap appends without quadratic copying. The same buffer, cursor, typed reads and writes and layout kernels are a header-only 'C' library that other packages use through 'LinkingTo' alone.
zubin
Every binary format, from an image header to a market data feed to a C
struct dumped to disk, is specified the same way: a table of fields,
each with a name, a type, a width and an offset. Base R reads such data
one readBin() call at a time, loops over the records itself, has no
unsigned 32-bit integer and no half-precision float, and reads an 8-byte
integer silently wrong once it outgrows 32 bits.
zubin lets you write the table down once, as a layout, and then read or write any number of records in one call.
- Describe once, read millions. A layout is one string,
"<id:u32 ts:i64 price:f64 side:u8", andbin_unpack()turns a raw vector into a data frame of every record in it, one tight loop per field. - Every type a format uses. Integers from 8 to 128 bits, signed and
unsigned;
f16,bf16,f32andf64; complex numbers; booleans; fixed bytes and strings; arrays; padding; either byte order, per field; and C struct alignment. - Exact, or an error. Nothing is silently rounded, wrapped or truncated. A 64-bit integer that a double cannot hold, or a value too large for its field, is an error that names the field and the record.
- Writing too.
bin_pack()andbin_encode()are the inverses, and a byte builder appends without quadratic copying, up to a hard cap.bin_serialize()writes an R object into one without first making a raw vector, andbin_hash_object()fingerprints an object without making its serialisation at all. - A C library underneath. The same buffers, cursors and kernels are
a header-only C99 library that other packages use through
LinkingToalone.
Installation
zubin is not on CRAN yet. Once it is, install it with:
install.packages("zubin")
Until then, and for the development version, install it from GitHub:
# install.packages("pak")
pak::pak("pedrobtz/zubin")
Usage
library(zubin)
Read a real header. A PNG file opens with an 8-byte signature and
then an IHDR chunk, big-endian, as its specification lays it out. That
table, written down, reads the image’s size from the file R just wrote:
png_file <- tempfile(fileext = ".png")
png(png_file, width = 640, height = 480)
plot(pressure)
invisible(dev.off())
ihdr <- bin_layout(">x8 length:u32 type:s4 width:u32 height:u32 depth:u8 colour:u8")
bin_unpack(readBin(png_file, "raw", 64), ihdr, n = 1)
#> length type width height depth colour
#> 1 13 IHDR 640 480 8 6
The types base R lacks. readBin() has no unsigned 32-bit integers,
so it warns and reads four ff bytes as -1, and it has no half floats.
zubin decodes both, in either byte order:
x <- as.raw(c(0xff, 0xff, 0xff, 0xff))
readBin(x, "integer", size = 4, signed = FALSE)
#> Warning in readBin(x, "integer", size = 4, signed = FALSE): 'signed = FALSE' is
#> only valid for integers of sizes 1 and 2
#> [1] -1
bin_decode(x, "u32")
#> [1] 4294967295
# half-precision floats, as machine-learning weights are stored
bin_decode(as.raw(c(0x00, 0x3c, 0x00, 0xc0, 0x00, 0x7c)), "f16")
#> [1] 1 -2 Inf
Exact, or an error. readBin() reads the 8-byte integer 2^32 as 0,
without a word. zubin reads it exactly, and refuses a value above 2^53,
which has no exact double, naming the record rather than handing back a
neighbouring number (int64 = "integer64" returns the exact bits
instead):
counts <- bin_encode(c(2^32, 2^53 + 2), "u64")
readBin(counts, "integer", size = 8, n = 2)
#> [1] 0 2
bin_decode(counts[1:8], "u64")
#> [1] 4294967296
bin_decode(counts, "u64")
#> Error in `bin_decode()`:
#> ! The value of record 1 has no exact R value.
Take a number apart. A double is a sign bit, an 11-bit exponent and
a 52-bit fraction. Encoded and read back as two unsigned words, its
fields come out exactly, and 0.1 turns out to be
7205759403792794 / 2^56, a little more than a tenth:
w <- bin_unpack(bin_encode(c(1, 0.1, -2.5), "f64", endian = "big"), ">hi:u32 lo:u32")
bits <- function(v, from, width) (v %/% 2^from) %% 2^width
parts <- data.frame(sign = bits(w$hi, 31, 1), exponent = bits(w$hi, 20, 11) - 1023,
fraction = bits(w$hi, 0, 20) * 2^32 + w$lo)
format(parts, scientific = FALSE)
#> sign exponent fraction
#> 1 0 0 0
#> 2 0 -4 2702159776422298
#> 3 1 1 1125899906842624
(2^52 + parts$fraction[2]) / 2^56 == 0.1
#> [1] TRUE
A million records in one call. Packing a data frame and unpacking it again is one call each way, and the round trip is exact:
set.seed(1)
rec <- bin_layout("<id:i32 price:f64 qty:i16 x2")
trades <- data.frame(id = 1:1e6, price = round(runif(1e6, 90, 110), 2),
qty = sample(1:500, 1e6, replace = TRUE))
bytes <- bin_pack(rec, trades)
length(bytes)
#> [1] 16000000
identical(bin_unpack(bytes, rec), trades)
#> [1] TRUE
Mixed byte orders: a shapefile. The .shp part of a shapefile mixes
byte orders inside one record: its file code and length are big-endian,
everything after them little-endian. A .be suffix overrides the
layout’s order for one field, so each record is still a single layout.
Here the quakes data become a point shapefile, written with a builder,
as the ESRI specification lays it out:
shp_header <- bin_layout("<code:i32.be x20 words:i32.be version:i32 type:i32
xmin:f64 ymin:f64 xmax:f64 ymax:f64 x32")
point <- bin_layout("<id:i32.be words:i32.be type:i32 x:f64 y:f64")
n <- nrow(quakes)
b <- bin_builder()
bin_put(b, bin_pack(shp_header, code = 9994, words = (100 + 28 * n) / 2,
version = 1000, type = 1,
xmin = min(quakes$long), ymin = min(quakes$lat),
xmax = max(quakes$long), ymax = max(quakes$lat)))
bin_put(b, bin_pack(point, id = seq_len(n), words = 10, type = 1,
x = quakes$long, y = quakes$lat))
shp <- bin_take(b)
bin_hexdump(shp, n = 48)
#> 00000000: 0000 270a 0000 0000 0000 0000 0000 0000 ..'.............
#> 00000010: 0000 0000 0000 0000 0000 36e2 e803 0000 ..........6.....
#> 00000020: 0100 0000 3d0a d7a3 70b5 6440 ec51 b81e [email protected]..
The file code 9994 opens the file big-endian (0000 270a), and the
version 1000 at offset 28 is little-endian (e803 0000). Reading it
back is one call for the header and one for every point after it:
bin_unpack(shp, shp_header, n = 1)
#> code words version type xmin ymin xmax ymax
#> 1 9994 14050 1000 1 165.67 -38.59 188.13 -10.72
pts <- bin_unpack(shp, point, offset = 100)
head(pts, 3)
#> id words type x y
#> 1 1 10 1 181.62 -20.42
#> 2 2 10 1 181.03 -20.62
#> 3 3 10 1 184.10 -26.00
identical(pts$x, quakes$long)
#> [1] TRUE
Every function is prefixed bin_, and offsets are 0-based everywhere,
as in every format specification.
Learn more
- Structured binary data with zubin, the getting-started guide: layouts, a WAV file written and read back, C structs, arrays, 64-bit integers and errors.
- The C API, for
package authors: the
LinkingTorecipe, the cursor, the record kernels, and buffers owned by R. - Reading R’s own object header, a layout over bytes zubin did not write: the fixed-size C header behind every R object, in native order, with a bitfield word, pointers and a body at a known offset.
- The reference index,
with the full type table under
?bin_layoutand the error classes under?zubin-conditions.
Related work
zubin’s integer loads and stores, half-float conversions and UTF-8
validation come from zufast.
Python’s struct
module and the structured dtypes of NumPy are the nearest equivalents in
other languages.