Holds a "chunked array" that allows a number of array fragments (represented by
Vector instances) to be treated logically as a single vector.
Vector instances can be concatenated into a
Chunked without any memory being copied.
Create a new contiguous typed array from a
Chunked instance (note that this creates a new typed array unless only one chunk)
const typedArray = chunked.toArray();
Chunked array supports iteration, random element access and mutation.
class Chunked extends Vector
Utility method that flattens a number of
Vector instances or Arrays of
Vector instances into a single Array of
Vector instances. If the incoming Vectors are instances of
Chunked, the child chunks are extracted and flattened into the resulting Array. Does not mutate or copy data from the Vector instances.
Returns an Array of
Concatenates a number of
Vector instances of the same type into a single
Chunked Vector. Returns a new
Note: This method extracts the inner chunks of any incoming
Chunked instances, and flattens them into the
chunks array of the returned
Chunked arrays are iterable, allowing you to use constructs like
for (const element of chunked) to iterate over elements. For in-order traversal, this is more performant than random-element access.
Returns the DataType instance which determines the type of elements this
Chunked instance contains. All vector chunks will have this type.
Returns the total number of elements in this
Chunked instance, representing the length of of all chunks.
Returns an array of the
Vector chunks that hold the elements in this
typeId enum value of the
Data instance of the first chunk in the list of inner Vectors.
Returns the constructor of the underlying typed array for the values buffer as determined by this Vector's DataType.
The number of logical Vector children for the Chunked Vector. Only applicable if the DataType of the Vector is one of the nested types (List, FixedSizeList, Struct, or Map).
The number of elements in the underlying data buffer that constitute a single logical value for the given type. The stride for all DataTypes is 1 unless noted here:
Decimaltypes, the stride is 4.
Datetypes, the stride is 1 if the
unitis DateUnit.DAY, else 2.
Timetypes, the stride is 1 if
bitWidth <= 32, else 2.
FixedSizeListtypes, the stride is the
listSizeproperty of the
FixedSizeBinarytypes, the stride is the
byteWidthproperty of the
Number of null values across all Vector chunks in this chunked array.
If this is a dictionary encoded column, returns a
Chunked instance of the indicies of all the inner chunks. Otherwise, returns
If this is a dictionary encoded column, returns the Dictionary.
Creates a new
Chunked array instance of the given
type and optionally initializes it with a list of
type- The DataType of the inner chunks
chunks= - Vectors must all be compatible with
offsets= - A Uint32Array of offsets where each inner chunk starts and ends. If not provided, offsets are automatically calculated from the list of chunks.
TBD - Confirm/provide some information on how
offsets can be used?
Returns a new
Chunked instance that is a clone of this instance. Does not copy the actual chunks, so the new
Chunked instance will reference the same chunks.
Concatenates a number of
Vector instances after the chunks. Returns a new
Vector chunks must be the same DataType as the
Returns a new chunked array representing the logical array containing the elements within the index range, potentially dropping some chunks at beginning and end.
0- The first logical index to be included as index 0 in the new array.
end- The first logical index to be included as index 0 in the new array. Defaults to the last element in the range.
Returns a zero-copy slice of this Vector. The begin and end arguments are handled the same way as JS'
Array.prototype.slice; they are clamped between 0 and
vector.length and wrap around when negative, e.g.
slice(-1, 5) or
Chunked Vector's DataType is one of the nested types (Map or Struct), returns a
Chunked Vector view over all the chunks for the child Vector at
index that is relative to the whole
Chunked Vector, binary search through the list of inner chunks using supplied "global"
index to find the chunk at that location. Returns the child index of the inner chunk and an element index that has been adjusted to the keyspace of the found inner chunk.
search() can be called with only an integer index, in which case a pair of
[chunkIndex, valueIndex] are returned as a two-element Array:
let chunked = [ Int32Vector.from([0, 1, 2, 3]), Int32Vector.from([4, 5, 6, 7, 8]) ].reduce((x, y) => x.concat(y)); let [chunkIndex, valueIndex] = chunked.search(6) assert(chunkIndex === 1) assert(valueIndex === 3)
search() is called with an integer index and a callback, the callback will be invoked with the
Chunked instance as the first argument, then the
valueIndex as the second and third arguments:
let getChildValue = (parent, childIndex, valueIndex) => chunked.chunks[childIndex].get(valueIndex); let childValue = chunked.search(6, (chunked, childIndex, valueIndex) => )
Checks if the element at
index in the logical array is valid.
Checks the null map (if present) to determine if the value in the logical
index is included.
Returns the element at
index in the logical array, or
null if no such element exists (e.e.g if
index is out of range).
Writes the given
value at the provided
index. If the value is null, the null bitmap is updated.
Returns the index of the first occurrence of
-1 if the value was not found.
offset- the index to start searching from.
Returns a single contiguous typed array containing data in all the chunks (effectively "flattening" the chunks.