Introduction
Why does loaders.gl provide an Arrow JS API Reference?
The idea is that this documentation should at some point be contributed back to the Apache Arrow project/repository, but so far this has not happened.
loaders.gl is designed to output parsed tables and meshes in binary columnar format (whenever the parsed data structure allows). Binary columnar tables are a compact and efficient representation that is easy to work with analytically in JavaScript and to seamlessly upload to GPUs (via e.g. WebGL or WebGPU) for ultra-performance rendering and computation.
While loaders.gl can load data into binary columnar tables, it only provides limited support for working with binary tables. The intention is that the application should be able to use complementary libraries like Apache Arrow JS.
While the Apache Arrow JS library itself is excellent, the reference documentation for the JavaScript bindings is unfortunately rather thin. It can therefore be challenging to get up to speed on the Arrow JS API, which is why this documentation is provided in loaders.gl.
About Apache Arrow JS
The Apache Arrow JavaScript API is designed to help applications work with binary columnar data in the Apache Arrow format. Arrow JS offers a core set of classes that supports use cases such as batched loading and writing, column and row access, schemas etc.
Getting Started
To install and start coding with Apache Arrow JS bindings, see the Getting Started.
About Apache Arrow
Apache Arrow is a performance-optimized binary columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. The Arrow spec is design to eliminate memory copies and aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.
Apache Arrow is emerging as the standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...). By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.
Resources
There are some excellent resources available that can help you quickly get a feel for what capabilities the Arrow JS API offers:
- Observable: Introduction to Apache Arrow
- Observable: Using Apache Arrow JS with Large Datasets
- Observable: Manipulating Flat Arrays, Arrow-Style
- Manipulating Flat Arrays General article on Columnar Data and Data Frames
Apache Arrow project links: