zukomp

zukomp compresses and decompresses raw vectors through a single codec-neutral interface backed by a runtime registry, so the set of available codecs is discovered rather than fixed at compile time. The DEFLATE family (gzip, zlib and headerless deflate-raw), zstd, and LZ4 (lz4-frame and the bare lz4-block) are built in from vendored miniz, zstd and lz4 sources, so no system compression library is required.

Installation

install.packages("zukomp")

Or the development version from GitHub:

# install.packages("pak")
# pak::pak("pedrobtz/zukomp")
install.packages("zukomp", repos = c('https://pedrobtz.r-universe.dev', 'https://cloud.r-project.org'))

Usage

Bytes in, bytes out. komp_compress() names a codec; komp_decompress() detects it by default.

library(zukomp)

x <- charToRaw(strrep("the quick brown fox ", 500))
z <- komp_compress(x, "gzip")

length(x)                        # 10000
length(z)                        # 81
identical(komp_decompress(z), x) # TRUE

komp_codecs() is the registry. Codecs this build knows the name of but does not implement still get a row, so "not implemented here" is a possible answer:

komp_codecs()[, c("id", "available", "content_encoding", "source")]
#>              id available content_encoding source
#> 1      identity      TRUE         identity zukomp
#> 2   deflate-raw      TRUE             <NA> zukomp
#> 3          zlib      TRUE          deflate zukomp
#> 4          gzip      TRUE             gzip zukomp
#> 5        brotli     FALSE               br   <NA>
#> 6          zstd      TRUE             zstd zukomp
#> 7     lz4-frame      TRUE             <NA> zukomp
#> 8     lz4-block      TRUE             <NA> zukomp
#> ...

Decompression is safe to point at untrusted input: output is capped at 1 GiB by default, and the cap is enforced by the core stream driver, so no codec — including a third-party one — can bypass it. The getting started article covers the limits, detection, the structured error conditions, and the C ABI that lets another package drive these codecs or register one of its own.

Using zukomp from C

There are two ways for another package to reach the codecs, and they suit different consumers.

The registered function table is the one to prefer. Declare

Imports:    zukomp
LinkingTo:  zukomp

and include <zukomp-r.h>, which resolves zukomp_api() through R_GetCCallable(). The consumer drives whatever codecs the installed zukomp has, including ones registered after it was built, and links nothing.

A container format that records each body's decoded length (an Arrow IPC buffer's 8-byte prefix, a Parquet page header) decompresses one shot into a buffer of that length, and passes the length so zukomp holds the codec to it:

const zukomp_api_v1 *api = zukomp_api();       /* NULL: zukomp missing or too old */
zu_decoder_opts o;
memset(&o, 0, sizeof o);
o.struct_size   = sizeof o;
o.codec         = api->codec_lookup("zstd");   /* or "lz4-frame", "lz4-block" */
o.max_output    = limit;                        /* 0 = unlimited */
o.flags         = ZU_DEC_REJECT_TRAILING | ZU_DEC_EXPECTED_SIZE;
o.expected_size = n;
size_t written;
zu_status st = api->decompress_one(&o, src, src_len, dst, n, &written);
/* ZU_OK, or ZU_ERR_INVALID_DATA if the body is not exactly n bytes */

The static archive is for a consumer that needs the ZIP container layer rather than a codec: reading the members of a .zip, an .xlsx or any other ZIP-shaped format. That layer is trimmed out of zukomp.so on purpose and is not reachable through the table. An installed zukomp carries

zukomp/include/miniz.h
zukomp/lib/libzukomp.a          # lib/x64/ on Windows; see below
zukomp/licenses/miniz-LICENSE

where the archive is a second compilation of the same vendored miniz.c, with the ZIP reader, stdio and timestamps left in. LinkingTo: zukomp puts the header on the include path; the archive's location comes from system.file("lib", .Platform$r_arch, package = "zukomp"), which a configure script can resolve into src/Makevars without adding an Imports: dependency:

ZUKOMP_LIB=$("${R_HOME}/bin/Rscript" -e \
  'cat(system.file("lib", .Platform$r_arch, package = "zukomp"))')
sed "s|@ZUKOMP_LIB@|${ZUKOMP_LIB}|" src/Makevars.in > src/Makevars

PKG_CPPFLAGS = -DMINIZ_NO_ZLIB_COMPATIBLE_NAMES
PKG_LIBS = "@ZUKOMP_LIB@/libzukomp.a"

.Platform$r_arch is "" on every single-arch platform and "x64" on Windows, where the archive installs under that sub-directory beside the shared object's. Quote the expansion in PKG_LIBS: an R library path containing a space (C:/Program Files/R/...) otherwise arrives at the linker as two arguments. Windows needs the same two lines in configure.win, since R runs only that file there.

Four things to know before taking this route:

miniz is MIT-licensed third-party code. An installed zukomp carries the notice alongside it, at zukomp/licenses/miniz-LICENSE; inst/COPYRIGHTS records the provenance.