ParquetLoader 🆕 🚧

 

Streaming loader for Apache Parquet encoded files.

Loader	Characteristic
File Format	Parquet
Data Format	Classic Table
File Extension	.parquet,
MIME Type	N/A (application/octet-stream)
File Type	Binary
Supported APIs	load, parse, parseInBatches

Please refer to the parquet format page for information on which Parquet format features are supported.

Usage

Load a Parquet file as a table.

import {ParquetLoader} from '@loaders.gl/parquet';
import {load} from '@loaders.gl/core';

const data = await load(url, ParquetLoader, {parquet: options});

The ParquetLoader supports streaming parsing, in which case it will yield "batches" of rows.

import {ParquetLoader} from '@loaders.gl/parquet';
import {loadInBatches} from '@loaders.gl/core';

const batches = await loadInBatches('geo.parquet', ParquetLoader, {parquet: options}});

for await (const batch of batches) {
  // batch.data will contain a number of rows
  for (const feature of batch.data) {
    switch (feature.geometry.type) {
      case 'Polygon':
      ...
    }
  }
}

Compressions

Some compressions are big and need to be imported explicitly by the application and passed to the ParquetLoader

import {ParquetLoader} from '@loaders.gl/parquet';
import {load} from '@loaders.gl/core';

import {ZstdCodec} from 'zstd-codec';
import lz4js from 'lz4js';

const data = await load(url, ParquetLoader, {modules: {
  'zstd-codec': ZstdCodec,
  'lz4js': lz4js,
  // brotli - only needed for compression
});

Data Format

For details see parquet documentation.

Options

Supports table category options such as batchType and batchSize.

Option	From	Type	Default	Description