Heavily optimized library for DEFLATE/zlib/gzip compression and decompression
libdeflate is a library for fast, whole-buffer DEFLATE-based compression and
decompression.
The supported formats are:
libdeflate is heavily optimized. It is significantly faster than the zlib
library, both for compression and decompression, and especially on x86 and ARM
processors. In addition, libdeflate provides optional high compression modes
that provide a better compression ratio than the zlib’s “level 9”.
libdeflate itself is a library. The following command-line programs which use
this library are also included:
libdeflate-gzip, a program which can be a drop-in replacement for standard
gzip under some circumstances. Note that libdeflate-gzip has some
limitations; it is provided for convenience and is not meant to be the
main use case of libdeflate. It needs a lot of memory to process large files,
and it omits support for some infrequently-used options of GNU gzip.
benchmark, a test program that does round-trip compression and decompression
of the provided data, and measures the compression and decompression speed.
It can use libdeflate, zlib, or a combination of the two.
checksum, a test program that checksums the provided data with Adler-32 or
CRC-32, and optionally measures the speed. It can use libdeflate or zlib.
For the release notes, see the NEWS file.
libdeflate uses CMake. It can be built just like any
other CMake project, e.g. with:
cmake -B build && cmake --build build
By default the following targets are built:
libdeflate.a)libdeflate.so)libdeflate-gzip program, including its alias libdeflate-gunzipBesides the standard CMake build and installation options, there are some
libdeflate-specific build options. See CMakeLists.txt for the list of these
options. To set an option, add -DOPTION=VALUE to the cmake command.
Prebuilt Windows binaries can be downloaded from
https://github.com/ebiggers/libdeflate/releases.
Although the official build system is CMake, care has been taken to keep the
library source files compilable directly, without a prerequisite configuration
step. Therefore, it is also fine to just add the library source files directly
to your application, without using CMake.
You should compile both lib/*.c and lib/*/*.c. You don’t need to worry
about excluding irrelevant architecture-specific code, as this is already
handled in the source files themselves using #ifdefs.
If you are doing a freestanding build with -ffreestanding, you must add
-DFREESTANDING as well (matching what the CMakeLists.txt does).
The above are the minimums, but using a newer compiler allows more of the
architecture-optimized code to be built. libdeflate is most heavily optimized
for gcc and clang, but MSVC is supported fairly well now too.
The recommended optimization flag is -O2, and the CMakeLists.txt sets this
for release builds. -O3 is fine too, but often -O2 actually gives better
results. It’s unnecessary to add flags such as -mavx2 or /arch:AVX2, though
you can do so if you want to. Most of the relevant optimized functions are
built regardless of such flags, and appropriate ones are selected at runtime.
For the same reason, flags like -mno-avx2 do not cause all code using the
corresponding instruction set extension to be omitted from the binary; this is
working as intended due to the use of runtime CPU feature detection.
If using gcc, your gcc should always be paired with a binutils version that is
not much older than itself, to avoid problems where the compiler generates
instructions the assembler cannot assemble. Usually systems have their gcc and
binutils paired properly, but rarely a mismatch can arise in cases such as the
user installing a newer gcc version without a proper binutils alongside it.
Since libdeflate v1.22, the CMake-based build system will detect incompatible
binutils versions and disable some optimized code accordingly. In older
versions of libdeflate, or if CMake is not being used, a too-old binutils can
cause build errors like “no such instruction” from the assembler.
libdeflate has a simple API that is not zlib-compatible. You can create
compressors and decompressors and use them to compress or decompress buffers.
See libdeflate.h for details.
There is currently no support for streaming. This has been considered, but it
always significantly increases complexity and slows down fast paths.
Unfortunately, at this point it remains a future TODO. So: if your application
compresses data in “chunks”, say, less than 1 MB in size, then libdeflate is a
great choice for you; that’s what it’s designed to do. This is perfect for
certain use cases such as transparent filesystem compression. But if your
application compresses large files as a single compressed stream, similarly to
the gzip program, then libdeflate isn’t for you.
Note that with chunk-based compression, you generally should have the
uncompressed size of each chunk stored outside of the compressed data itself.
This enables you to allocate an output buffer of the correct size without
guessing. However, libdeflate’s decompression routines do optionally provide
the actual number of output bytes in case you need it.
Windows developers: note that the calling convention of libdeflate.dll is
“cdecl”. (libdeflate v1.4 through v1.12 used “stdcall” instead.)
The libdeflate project itself only provides a C library. If you need to use
libdeflate from a programming language other than C or C++, consider using the
following bindings:
Note: these are third-party projects which haven’t necessarily been vetted by
the authors of libdeflate. Please direct all questions, bugs, and improvements
for these bindings to their authors.
Also, unfortunately many of these bindings bundle or pin an old version of
libdeflate. To avoid known issues in old versions and to improve performance,
before using any of these bindings please ensure that the bundled or pinned
version of libdeflate has been upgraded to the latest release.
The DEFLATE format (rfc1951), the zlib
format (rfc1950), and the gzip format
(rfc1952) are commonly confused with
each other as well as with the zlib software library, which
actually supports all three formats. libdeflate (this library) also supports
all three formats.
Briefly, DEFLATE is a raw compressed stream, whereas zlib and gzip are different
wrappers for this stream. Both zlib and gzip include checksums, but gzip can
include extra information such as the original filename. Generally, you should
choose a format as follows:
gzipNote that gzip and zlib streams can be distinguished from each other based on
their starting bytes, but this is not necessarily true of raw DEFLATE streams.
An often-underappreciated fact of compression formats such as DEFLATE is that
there are an enormous number of different ways that a given input could be
compressed. Different algorithms and different amounts of computation time will
result in different compression ratios, while remaining equally compatible with
the decompressor.
For this reason, the commonly used zlib library provides nine compression
levels. Level 1 is the fastest but provides the worst compression; level 9
provides the best compression but is the slowest. It defaults to level 6.
libdeflate uses this same design but is designed to improve on both zlib’s
performance and compression ratio at every compression level. In addition,
libdeflate’s levels go up to 12 to make room for a
minimum-cost-path based algorithm (sometimes called “optimal parsing”) that can
significantly improve on zlib’s compression ratio.
If you are using DEFLATE (or zlib, or gzip) in your application, you should test
different levels to see which works best for your application.
Despite DEFLATE’s widespread use mainly through the zlib library, in the
compression community this format from the early 1990s is often considered
obsolete. And in a few significant ways, it is.
So why implement DEFLATE at all, instead of focusing entirely on
bzip2/LZMA/xz/LZ4/LZX/ZSTD/Brotli/LZHAM/LZFSE/[insert cool new format here]?
To do something better, you need to understand what came before. And it turns
out that most ideas from DEFLATE are still relevant. Many of the newer formats
share a similar structure as DEFLATE, with different tweaks. The effects of
trivial but very useful tweaks, such as increasing the sliding window size, are
often confused with the effects of nontrivial but less useful tweaks. And
actually, many of these formats are similar enough that common algorithms and
optimizations (e.g. those dealing with LZ77 matchfinding) can be reused.
In addition, comparing compressors fairly is difficult because the performance
of a compressor depends heavily on optimizations which are not intrinsic to the
compression format itself. In this respect, the zlib library sometimes compares
poorly to certain newer code because zlib is not well optimized for modern
processors. libdeflate addresses this by providing an optimized DEFLATE
implementation which can be used for benchmarking purposes. And, of course,
real applications can use it as well.
libdeflate is MIT-licensed.
I am not aware of any patents or patent applications relevant to libdeflate.