Skip to main content

Working with Tables

References:

Loading Arrow Data​

Applications often start with loading some Arrow formatted data. The Arrow API provides several ways to do this, but in many cases, the simplest approach is to use Table.from().

import {Table} from 'apache-arrow';
const response = await fetch(dataUrl);
const arrayBuffer = await response.arrayBuffer();
const dataTable = arrow.Table.from(new Uint8Array(arrayBuffer));

Getting Records Count​

const count = table.count();

Getting Arrow Schema Metadata​

const fieldNames = table.schema.fields.map(f => f.name);
// Array(3) ["Latitude", "Longitude", "Date"]
const fieldTypes = tables.schema.fields.map(f => f.type)
// Array(3) [Float, Float, Timestamp]

const fieldTypeNames = ...;
// Array(3) ["Float64", "Float64", "Timestamp<MICROSECOND>"]

Accessing Arrow Table Row Data​

const firstRow = tables.get(0) // 1st row data
const lastRow = tables.get(rowCount-1)

Record toJSON and toArray​

It is easy to converting Rows to JSON/Arrays/Strings:

toJSON = Array(3) [41.890751259, -87.71617311899999, Int32Array(2)]
toArray = Array(3) [41.933659084, -87.72369064600001, Int32Array(2)]

Similar conversion methods are avaiable on many Arrow classes.

tables.get(0).toJSON()

Slicing Arrow Data​

every10KRow = Array(17) [Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3)]

Our custom arrow data range stepper for sampling data:

range = ƒ(start, end, step)

Iterating over Rows and Cells​

for (let row of dataFrame) {
  for (let cell of row) {
    if ( Array.isArray(cell) ) {
      td = '[' + cell.map((value) => value == null ? 'null' : value).join(', ') + ']';
    } else if (fields[k] === 'Date') {
      td = toDate(cell); // convert Apache arrow Timestamp to Date
    } else {
      td = cell.toString();
    }
    k++;
  }
}

Converting Dates​

Apache Arrow Timestamp is a 64-bit int of milliseconds since the epoch, represented as two 32-bit ints in JS to preserve precision. The fist number is the "low" int and the second number is the "high" int.

function toDate(timestamp) {
  return new Date((timestamp[1] * Math.pow(2, 32) + timestamp[0])/1000);
}

Column Data Vectors​

Apache Arrow stores columns in typed arrays and vectors:

Typed vectors have convinience methods to convert Int32 arrays data to JS values you can work with.

For example, to get timestamps in milliseconds:

timestamps = Array(10) [2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01, 2017-01-01]

Filtering Timestamped Data​

function filterByDate(startDate, endDate) {
  const dateFilter = arrow.predicate.custom(i => {
  	const arrowDate = table.getColumn('Date').get(i);
    const date = toDate(arrowDate);
    return date >= startDate && date <= endDate;
  }, b => 1);

  const getDate;
  const results = [];
  table.filter(dateFilter)
    .scan(
      index => {
        results.push({
          'date': toDate(getDate(index))
        });
      },
      batch => {
        getDate = arrow.predicate.col('Date').bind(batch);
      }
    );

  return results;
}

Our custom filter by date method uses custom arrow table predicate filter and scan methods to generate JS friendly data you can map or graph: