halley

sheetstream

Read and write million-row XLSX and CSV files in Node.js in about 80 MB of memory

Area
Libraries
Language
Rust
License
MIT
Latest
v0.3.0
Plate
No. 12 of 14
Fig. 12sheetstream
Move your pointer over it

What came back

  1. 82 MB

    peak memory writing 1M rows by 10 columns, where common JavaScript libraries need 749 MB to 3.3 GB.source

  2. 4x

    faster reading 1M rows: 2.4 s instead of 9.1 s.source

  3. 2x

    faster writing 1M rows: 5.5 s instead of 12.3 s.source

sheetstream reads and writes XLSX and CSV from Node.js without holding the rows in JavaScript. A Rust core does the parsing and zipping and rows cross in batches, so a million-row export fits in a small serverless function. It takes any iterable or async iterable as a source, so a database cursor or a generator works as is.

How it works

  1. 01

    Native Rust code (through napi-rs) does the XML, zip and CSV work, so the rows never become JavaScript objects in bulk.

  2. 02

    Rows go in and out in batches of 1000 by default, and a parser thread runs a few batches ahead of your loop.

  3. 03

    Writing in constant mode streams the sheet XML to a temp file and compresses it into the output, so the cost is disk, not RAM.

What it does

  • writeXlsx takes any Iterable or AsyncIterable of arrays or objects, and readXlsx is an async iterator of batches
  • Streaming CSV read and write, plus csvStream, a Node Readable with real backpressure for HTTP responses
  • XLSX formatting: header style, column widths, per-column number formats, frozen header row and header filter, at no extra memory
  • Multiple sheets, choose by name or index, with xlsxStream and toBuffer for HTTP and small outputs
  • Prebuilt binaries for macOS (arm64, x64), Linux (x64 glibc, x64 musl, arm64 glibc) and Windows (x64), nothing to compile on install