Why does loaders.gl provide an Arrow JS API Reference?
The idea is that this documentation should at some point be contributed back to the Apache Arrow project/repository, but so far this has not happened.
While loaders.gl can load data into binary columnar tables, it only provides limited support for working with binary tables. The intention is that the application should be able to use complementary libraries like Apache Arrow JS.
About Apache Arrow JS
To install and start coding with Apache Arrow JS bindings, see the Getting Started.
About Apache Arrow
Apache Arrow is a performance-optimized binary columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. The Arrow spec is design to eliminate memory copies and aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.
Apache Arrow is emerging as the standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...). By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.
There are some excellent resources available that can help you quickly get a feel for what capabilities the Arrow JS API offers:
- Observable: Introduction to Apache Arrow
- Observable: Using Apache Arrow JS with Large Datasets
- Observable: Manipulating Flat Arrays, Arrow-Style
- Manipulating Flat Arrays General article on Columnar Data and Data Frames
Apache Arrow project links: