BitPacking

Documentation for BitPacking.

BitPacking packs values narrower than a byte, or of any width up to 128 bits, densely: NArray packs a static group of values into one storage value, and NarrowArray presents an array of such groups as an array of values. The bit model page states the rules they share.

Element protocol

BitPacking.bitwidth — Function
bitwidth(T)::Int
bitwidth(x)::Int

Return the number of value bits used by T or by the type of x.

The default for bitstypes is Core.bitsizeof(T) on Julia 1.14 and later, which is the declared width of a primitive type such as primitive type Float4 4 end, and 8 * sizeof(T) before that. Bool is specialized to one bit. Packages and users can overload this for narrow types declared with whole bytes and for packed containers.

source
BitPacking.raw_bits — Function
raw_bits(x)::Unsigned

Return the value of x as an unsigned integer: its bitwidth value bits in the low bits, and zeros above them. The integer is the narrowest of UInt8 to UInt128 that holds sizeof(x) bytes.

This is how BitPacking reads every element it packs, and it differs from reinterpret in two ways. It changes width where needed: a primitive type that declares fewer bits than it stores, such as primitive type Float4 4 end on Julia 1.14 and later, or a 3-byte type, is zero-extended, where reinterpret refuses it. And a type declared with whole bytes but a narrower bitwidth, such as Bool or a 4-bit float stored in a byte, has the bits above its width cleared, whatever they held.

Together with bitwidth and from_raw_bits this is the element protocol: any bitstype of at most 16 bytes takes part by default, and a type can specialize all three. An NArray does, returning its storage word, so that it can itself be an element.

source
BitPacking.from_raw_bits — Function
from_raw_bits(T, bits::Unsigned)::T

Return the T whose value bits are the low bitwidth(T) bits of bits; the bits above them are ignored, whatever the width of bits. The inverse of raw_bits: from_raw_bits(T, raw_bits(x)) === x for every x whose bits above its width are zero.

source

Narrow-element arrays

BitPacking.NArray — Type
NArray{T,N,S,D} <: StaticArray{S,T,N}
NArray{T,N,S}(values::Tuple)
NVector{T,L}
NMatrix{T,M,N}

Small statically sized array whose elements are packed into a single storage value.

NArray behaves like a StaticArrays.StaticArray, but stores its elements LSB-first in the data field: element 1 occupies the lowest bitwidth(T) bits, element 2 the next, and so on, in linear index order. The storage D holds the total bits rounded up to whole bytes: the unsigned integer of that size when there is one, and otherwise an NTuple of the widest unsigned integers that divide it, such as NTuple{3,UInt8} for 24 bits and NTuple{4,UInt64} for 256. Padding bits past the last element are always zero. Elements may be 1 to 128 bits wide.

Use NVector{T,L} and NMatrix{T,M,N} for the common one- and two-dimensional forms. Constructors always take values:

julia> v = NVector(true, false, true, false, true, false, true, false)
8-element NVector{Bool, 8, UInt8} with indices SOneTo(8):
 1
 0
 1
 0
 1
 0
 1
 0

reinterpret(T, x) follows the array rule, as for Base arrays: the bits are kept, and the length of dimension 1 is rescaled so that the total bit count stays the same, throwing when it does not divide. An NArray is itself an element: raw_bits returns its bits as one unsigned integer, and from_raw_bits builds one from them.

julia> reinterpret(UInt8, v)
1-element NVector{UInt8, 1, UInt8} with indices SOneTo(1):
 0x55

julia> BitPacking.raw_bits(v)
0x55

julia> BitPacking.from_raw_bits(NVector{Bool,8}, 0x0f) == NVector{Bool,8}(1, 1, 1, 1, 0, 0, 0, 0)
true

map keeps an NArray packed, while broadcasting treats it as the unpacked SArray of its values.

source
BitPacking.NarrowArray — Type
NarrowArray{T}(array::AbstractArray)
NarrowArray{T,N,L}(array::AbstractArray)
NarrowArray{T}(undef, dims...)

Array wrapper whose parent stores packed NArray chunks.

NarrowArray{T} presents an AbstractArray{T} interface while storing groups of logical values in the first dimension as NVector chunks. For an element type T, the default chunk length is pack_count(T) == 8 ÷ gcd(bitwidth(T), 8). Use NarrowArray{T,N,L} or NarrowVector{T,L} to choose a wider chunk length.

The constructor packs an existing logical array:

julia> x = NarrowArray{Bool}([true, false, true, false, true, false, true, false])
8-element NarrowVector{Bool, 8, Vector{NVector{Bool, 8, UInt8}}}:
 1
 0
 1
 0
 1
 0
 1
 0

NarrowArray{T}(undef, dims...) allocates an uninitialized one on the CPU, and similar(array, Narrow(T), dims) one on the backend of array.

The input's first dimension must contain whole chunks. For example, NarrowArray{Bool} requires the first dimension to be divisible by 8, and a 4-bit element type requires it to be divisible by 2.

NarrowArray{T}(x) converts values to T before packing when eltype(x) != T. parent(x) exposes the packed chunk array. copy(x) materializes the logical values, while reinterpret(T, x) reinterprets the whole logical bit buffer and rescales the leading dimension by bitwidth. Type broadcasts such as T.(x) convert packed chunks to dense logical T values. Display uses a host-adapted copy so GPU-backed parents do not print through scalar indexing.

source
BitPacking.Narrow — Type
Narrow(T)

Representation tag for the packed form of logical element type T. A Narrow instance substitutes for a Type in array operations whose result is a NarrowArray{T}:

operationto a NarrowArray{T}from a NarrowArray
reinterpretreinterpret(Narrow(T), data)reinterpret(T, narr)
broadcastNarrow(T).(array)T.(narr)
similarsimilar(array, Narrow(T), dims)similar(narr, T, dims)

For broadcast these are value conversions: Narrow(T).(array) packs values into a NarrowArray{T} and T.(narr) unpacks to dense T values. reinterpret never converts and never unpacks: reinterpret(Narrow(T), data) views the bytes or packed chunks of data as a NarrowArray{T}, and reinterpret(T, narr) views the packed bits of narr as T, which is a plain array when T fills its bytes (UInt8, Float32) and a NarrowArray{T} otherwise (Bool, a 4-bit float). Both keep the bits and rescale dimension 1. similar(array, Narrow(T), dims) allocates an uninitialized NarrowArray{T} of logical size dims whose chunk parent follows the backend of array, and similar(narr, T, dims) a dense array on the backend of narr.

Narrow(T).(array) makes the narrowing explicit where NarrowArray{T}(array) hides it; the equivalent in-place form is dest .= expr for a preallocated NarrowArray{T} destination. All forms use the default chunk length pack_count(T), so the leading dimension must be a whole number of chunks.

source