Skip to main content

AsyncIterators

Streaming functionality in loaders.gl is built on the ES2018 AsyncIterator concept. This page gives some background on AsyncIterator.

Availability​

AsyncIterator and the for await of iteration syntax are standard JavaScript ES2018 features and are supported by all recent evergreen browsers and Node.js versions as well as e.g. the babel transpiler.

Iterators and TypeScript​

There are multiple similar-sounding types supporting the type safe use of iterators which can be a source of confusion to users, so some information is provided here:

| Type | Async Type | Type Parameters | Description | | ------------------ | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Iterator | AsyncIterator | <...> | An iterator has next(), ... methods | | Iterable | AsyncIterable | <...> | An iterable is a class that has a [Symbol.iterator] or [Symbol.asyncIterator] property that returns an Iterator or AsyncIterator | | Generator | AsyncGenerator | <...> | A generator is a function that takes some parameters and when called returns an Iterator or AsyncIterator | | IterableIterator | AsyncIterableIterator | <...> | It is convenient to define Iterators that are also Iterable. Most built in container classes return this type. |

An IterableIterator can

  • be used in a for..of loop
  • be spread into an array
  • be spread into a parameter list
  • be used in APIs that accept iterables like Array.from(), new Set(), new Map()
interface Iterable {
    [Symbol.iterator]() : Iterator;
}
interface Iterator {
    next() : IteratorResult;
    return?(value? : any) : IteratorResult;
}
interface IteratorResult {
    value: any;
    done: boolean;
}

https://exploringjs.com/es6/ch_iteration.html#sec_implementing-iterables

Batched Parsing and Endcoding using AsyncIterators​

The input and output from streaming loaders and writers can both be expressed in terms of async iterators.

Using AsyncIterators​

Remember that an AsyncIterator or AsyncIterable can be consumed (iterated over) via the for-await construct:

for await (const x of asyncIterable) {
}

Using Streams as AsyncIterators​

With a little effort, streams in JavaScript can be treated as AsyncIterators. As the section about Javascript Streams explains, instead of registering callbacks on the stream, you can now work with streams in this way:

for await (const buf of fs.createReadStream('foo.txt')) {
  // do something
}

Creating AsyncIterators​

Remember that any object in JavaScript that implements the [Symbol.asyncIterator]() method is an AsyncIterable.

And the async generator syntax async function * can be used to generate new async iterators

async function * makeAsyncIterator() {
  yield new Promise(...)
}

for await (const x of makeAsyncIterator()) {} // Notice parens after 'makeAsyncIterator'