NDJSONLoader
Streaming loader for NDJSON encoded files and related formats (LDJSON and JSONL).
Loader | Characteristic |
---|---|
File Extension | .ndjson , .jsonl , .ldjson |
Media Type | application/x-ndjson , application/x-ldjson , application/json-seq |
File Type | Text |
File Format | NDJSON, [LDJSON][format_], [][format_] |
Data Format | Classic Table |
Supported APIs | load , parse , parseSync , parseInBatches |
Usage​
import {NDJSONLoader} from '@loaders.gl/json';
import {load} from '@loaders.gl/core';
const data = await load(url, NDJSONLoader, {ndjson: options});
The NDJSONLoader supports streaming NDJSON parsing, in which case it will yield "batches" of rows, where each row is a parsed line from the NDJSON stream.
import {NDJSONLoader} from '@loaders.gl/json';
import {loadInBatches} from '@loaders.gl/core';
const batches = await loadInBatches('ndjson.ndjson', NDJSONLoader);
for await (const batch of batches) {
// batch.data will contain a number of rows
for (const obj of batch.data) {
// Process obj
...
}
}
Data Format​
Parsed batches are of the format.
{
// standard batch payload
data: any[] | any;
bytesUsed: number;
batchCount: number;
}
Each element in the data
array corresponds to a line (Object) in the NDJSON data.
Options​
Supports the table category options such as batchSize
.