Skip to main content

ParquetLoader

From-v3.1

 

BETA

Streaming loader for Apache Parquet encoded files.

LoaderCharacteristic
File FormatParquet
Data FormatClassic Table
File Extension.parquet,
MIME TypeN/A (application/octet-stream)
File TypeBinary
Supported APIsload, parse, parseInBatches

Please refer to the parquet format page for information on which Parquet format features are supported.

Usage​

Load a Parquet file as a table.

import {ParquetLoader} from '@loaders.gl/parquet';
import {load} from '@loaders.gl/core';

const data = await load(url, ParquetLoader, {parquet: options});

The ParquetLoader supports streaming parsing, in which case it will yield "batches" of rows.

import {ParquetLoader} from '@loaders.gl/parquet';
import {loadInBatches} from '@loaders.gl/core';

const batches = await loadInBatches('geo.parquet', ParquetLoader, {parquet: options}});

for await (const batch of batches) {
// batch.data will contain a number of rows
for (const feature of batch.data) {
switch (feature.geometry.type) {
case 'Polygon':
...
}
}
}

Compressions​

Some compressions are big and need to be imported explicitly by the application and passed to the ParquetLoader

import {ParquetLoader} from '@loaders.gl/parquet';
import {load} from '@loaders.gl/core';

import {ZstdCodec} from 'zstd-codec';
import lz4js from 'lz4js';

const data = await load(url, ParquetLoader, {modules: {
'zstd-codec': ZstdCodec,
'lz4js': lz4js,
// brotli - only needed for compression
});

Data Format​

For details see parquet documentation.

Options​

Supports table category options such as batchType and batchSize.

OptionFromTypeDefaultDescription