DuckDB + Parquet: give the engine and the files separate jobs

@Author: Garfield Zhu

English edition: AI-assisted translation from the Chinese original.

DuckDB and Parquet are not the same thing

They get along extremely well, but their jobs are different:

The one-line version: DuckDB queries and computes; Parquet packs and travels.

Parquet is a columnar file format for analytics. It is made for writing batches, compression, partitioning, and sharing; Spark, Polars, Pandas, Trino, and DuckDB can all read it. It is not a transactional database, and a single file is not meant to receive one tiny row update every morning.

DuckDB is an embedded SQL engine: no server, port, or connection pool. Load the library into a process and query. Its columnar execution and vectorized batches are a good match for filtering, aggregating, and joining lots of rows.

So the value of DuckDB + Parquet is not “two databases stacked together”. It is a decoupled engine and portable storage: query with DuckDB today, switch to Spark, Polars, or a cloud warehouse tomorrow, and the files still work.

How should .duckdb and Parquet share the work?

The application defines this boundary; DuckDB does not silently decide which table belongs where. A useful hot/cold split looks like this:

• frequently changed settings, metadata, and recent records: keep them in .duckdb;
• old, immutable history: export it to Parquet by day or month;
• hide both behind a view that becomes the logical table your app sees.

For example, an archive job can own the rule:

COPY (
  SELECT *
  FROM events
  WHERE event_time < current_date - INTERVAL '90 days'
) TO 'archive/events_2026_05.parquet'
(FORMAT parquet, COMPRESSION zstd);

DELETE FROM events
WHERE event_time < current_date - INTERVAL '90 days';

Then hide hot and cold data behind one query surface:

CREATE VIEW all_events AS
SELECT * FROM recent_events
UNION ALL
SELECT * FROM read_parquet('archive/events/*.parquet');

Business code queries all_events and does not need to care where the bytes live. DuckDB gives you COPY, read_parquet, and views; “archive after 90 days” is your product policy.

One small rule matters: do not treat one Parquet file like an updatable database table. Append new batch files, partition them by date, and query the glob. Parquet is excellent at “write a batch, read a lot”; it is not a row-level transaction log.

Why bring DuckDB-Wasm into the browser?

Browsers already have IndexedDB, OPFS, and localStorage, but those solve storage problems, not the same analytical workload:

DuckDB-Wasm compiles the full DuckDB engine to WebAssembly, usually in a Web Worker. A web page can query local CSV, JSON, Parquet, or a file the user just dropped in; the result can travel as a columnar structure such as Arrow into tables and charts.

That unlocks a few genuinely useful scenarios:

Keep the data on the user’s machine

Medical, financial, log, and personal data can be analyzed locally. You do not upload raw data just to draw a chart. Privacy is not “one more API layer”; it is simply not sending the data.

Turn backend analytics endpoints into static data

The server can periodically produce Parquet from a business database such as PostgreSQL and publish it to object storage or a CDN. DuckDB-Wasm queries it in the browser, so the user’s machine does the computation. More users mean fewer server-side aggregations; bandwidth and storage still cost money, so the free lunch has a small asterisk.

Parquet’s columns, row groups, and statistics also pair nicely with HTTP Range requests: a query fetches the bytes it needs instead of downloading a 1 GB file end to end. Configure CORS for cross-origin files, and make sure the object store or CDN supports Range.

Offline and local-first

Ship Parquet with a PWA or cache it on first open, and analysis keeps working offline. On a plane, behind a firewall, or in a subway tunnel, the chart can keep doing its job.

One mental model for Web, Tauri, and Electron

A desktop build can use native DuckDB + Parquet; the web build can use DuckDB-Wasm + Parquet. Keep the frontend components and SQL as similar as possible. Tauri/Electron handles files, windows, and system integration; DuckDB handles analysis; Parquet carries portable history. You get a desktop-grade local tool without maintaining two unrelated query models.

Wasm is not magic, of course:

• browser memory and single-thread limits still exist; do not cram a petabyte warehouse into a tab;
• extensions have to be compiled into the Wasm bundle;
• DuckDB-Wasm is a great read/analyze layer, while frequent disk writes belong to the application storage layer.

Run it once and watch the division become a performance gap

The demo below generates one deterministic trade dataset and lets DuckDB-Wasm, SQLite-Wasm, and IndexedDB + JS start together on the same semantic aggregation:

SELECT region, product, SUM(amount) AS total, COUNT(*) AS cnt
FROM trades
GROUP BY region, product
ORDER BY total DESC;

Loading and writing are included in each engine’s elapsed time; the one-time Wasm startup is prewarmed first and kept out of the race. The bars share one live scale, and each bar freezes at its own measured duration as soon as that engine finishes.

// DuckDB query: columnar storage + vectorized execution
// Turn the data into an Arrow table, then let SQL do the aggregation.
const arrow = await import('apache-arrow')
const table = arrow.tableFromJSON(rows)

await conn.query('DROP TABLE IF EXISTS trades')
await conn.insertArrowTable(table, { name: 'trades' })

const result = await conn.query(
  "SELECT region, product, SUM(amount) AS total, COUNT(*) AS cnt" +
   " FROM trades GROUP BY region, product ORDER BY total DESC"
)

// The engine reads the columns it needs and processes them in batches.

This is not a claim that IndexedDB is bad. It is an excellent browser-native transactional store. The trouble starts when it is asked to analyze millions of rows: first read every object back, then hand-write the grouping in JavaScript. SQLite gives you SQL, but it is row-oriented; DuckDB is designed for this columnar aggregation workload.

How I would choose

For small, frequently changing objects and row-level CRUD, pick IndexedDB or SQLite. For lots of locally generated records with filtering, grouping, and window calculations, consider DuckDB first. If history also needs to move across tools, languages, and machines, let Parquet be the common language.

The comfortable composition is usually:

Business database / .duckdb for mutable hot data → periodic Parquet exports → DuckDB-Wasm or desktop DuckDB for hot+cold analytics → Arrow → tables and charts.

Four traps I hit while putting this in a blog

1. CSP blocked a CDN worker. Self-host the Wasm and worker files under public/vendor/ so they stay same-origin.
2. SSR has no Worker. Load DuckDB-Wasm dynamically inside the browser callback, never at module top level.
3. Arrow wants RecordBatch, not any random object array. Use tableFromJSON to make the table shape explicit.
4. SQLite-Wasm statements use finalize(), not free(). One word, half an hour of life.

Takeaway

DuckDB is the analytics engine, Parquet is portable analytical data, and DuckDB-Wasm is how this combination moves into the browser. It shines when data already lives locally, is not tiny, and users want to explore it themselves: local tools, privacy-friendly analytics, public data portals, offline dashboards, and products that have both web and desktop shapes.

Ship the engine with the app, keep the data near the user, and let the server publish files. It sounds like the backend secretly got lighter; really, we just woke up the CPU that was supposed to do the work.