JSONLoader

Streaming loader for JSON encoded files.

Loader	Characteristic
File Extension	.json
Media Type	application/json
File Type	Text
File Format	JSON
Data Format	Classic Table
Supported APIs	load, parse, parseSync, parseInBatches

Usage

For simple usage, you can load and parse a JSON file atomically:

import {JSONLoader} from '@loaders.gl/json';
import {load} from '@loaders.gl/core';

const data = await load(url, JSONLoader, {json: options});

For larger files, JSONLoader supports streaming JSON parsing, in which case it will yield "batches" of rows from one array. To parse a stream of GeoJSON, the user can specify the options.json.jsonpaths to stream the features array.

import {JSONLoader} from '@loaders.gl/json';
import {loadInBatches} from '@loaders.gl/core';

const batches = await loadInBatches('geojson.json', JSONLoader, {json: {jsonpaths: ['$.features']}});

for await (const batch of batches) {
  // batch.data will contain a number of rows
  for (const feature of batch.data) {
    switch (feature.geometry.type) {
      case 'Polygon':
      ...
    }
  }
}

If no JSONPath is specified the loader will stream the first array it encounters in the JSON payload.

When batch parsing an embedded JSON array as a table, it is possible to get access to the containing object supplying the {metadata: true} option.

The loader will yield an initial and a final batch with batch.container providing the container object and batch.batchType set to partial-result and final-result respectively.

import {JSONLoader} from '@loaders.gl/json';
import {loadInBatches} from '@loaders.gl/core';

const batches = await loadInBatches('geojson.json', JSONLoader);

for await (const batch of batches) {
  switch (batch.batchType) {
    case 'partial-result': // contains fields seen so far
    case 'final-result': // contains all fields except the streamed array
      console.log(batch.container);
      break;
    case 'data':
      // batch.data will contain a number of rows
      for (const feature of batch.data) {
        switch (feature.geometry.type) {
          case 'Polygon':
          ...
        }
      }
  }
}

Data Format

Parsed batches are of the format

{
  batchType: 'metadata' | 'partial-result' | 'final-result' | undefined;
  jsonpath: string;

  // standard batch payload
  data: any[] | any;
  bytesUsed: number;
  batchCount: number;
}

Options

Supports table category options such as batchType and batchSize.

Option	From	Type	Default	Description
json.table	[]	boolean	false	Parses non-streaming JSON as table, i.e. return the first embedded array in the JSON. Always true during batched/streaming parsing.
json.jsonpaths	[]	string[]	[]	A list of JSON paths (see below) indicating the array that can be streamed.
metadata (top level)	[]	boolean	If true, yields an initial and final batch containing the partial and final result (i.e. the root object, excluding the array being streamed).

JSONPaths

A minimal subset of the JSONPath syntax is supported, to specify which array in a JSON object should be streamed as batchs.

$.component1.component2.component3

• No support for wildcards, brackets etc. Only paths starting with $ (JSON root) are supported.
• Regardless of the paths provided, only arrays will be streamed.

Attribution

This loader is based on a fork of dscape's clarinet under BSD 2-clause license.