DuckDB-Wasm: bring an analytical engine to the browser

@Author: Garfield Zhu

English edition: AI-assisted translation from the Chinese original.

First, what is DuckDB responsible for?

If data analysis is cooking, the data file is the pantry and DuckDB is the chef who is very good at prep: it reads columns, filters, joins, aggregates, sorts, and serves the result to a table or chart. It is not a database service you deploy on a port. It is an analytical engine you can embed in an application, without a connection pool or a “please start the server first” ritual.

A note from my own workbench: most of the production databases I deal with are PostgreSQL. Postgres is absurdly capable—transactions, MVCC, concurrency, complex SQL, JSON/JSONB, indexes, replication, permissions, and extensions cover almost anything a business system asks for; its feature overview and MVCC documentation are good evidence. It is a great fit for shared, mutable business truth that must stay consistent under concurrent writes. But DuckDB and PostgreSQL are not even running the same race: DuckDB is an embeddable, read-heavy OLAP engine, while Postgres is a general-purpose server database. This is not a “who replaces whom” contest. Postgres keeps the facts consistent; DuckDB slices those facts quickly. Need transactions and concurrent writes? Let Postgres sit at the main table. Need to scan and aggregate a snapshot, Parquet files, or local data? Invite DuckDB.

They can still shake hands: DuckDB’s PostgreSQL extension can read a running Postgres database as a query source and export analytical results or Parquet. My practical model is simple: Postgres produces the truth; DuckDB turns that truth into slices and charts.

The design focus is clear: analytics (OLAP) first, row-by-row transactions (OLTP) second. DuckDB uses columnar execution and vectorized batches, then lets its SQL optimizer choose how filters, aggregations, and joins should flow. The official vectorized execution notes put it plainly: data moves through operators in vectors, rather than shuffling one row at a time.

That creates a useful boundary. Application code decides where the data can be read from; DuckDB decides how to query it efficiently. A local file, object storage, an in-memory table, or a file someone just dropped onto the page can all be query inputs. The analytical engine and the place where bytes live do not have to be welded together.

Parquet is part of the story, not another database

Parquet fits nicely here, but it does a different job:

The short version: DuckDB queries and computes; Parquet packs and travels.

Parquet is built for “write a batch, read a lot”. A backend can export date-partitioned history from its business database, and the same files remain readable from Python, Spark, Polars, or DuckDB. It is not especially interested in receiving one tiny transactional update every morning; that belongs in a business database or a .duckdb file.

This is the useful frontend/backend contract: the backend publishes Parquet, and the browser uses DuckDB-Wasm to query the same data. A desktop build can reuse the SQL too. The file format becomes a common language, the engine is a replaceable execution layer, and every chart does not need its own “return exactly these four numbers” endpoint.

DuckDB-Wasm: a small analytical machine in the browser

DuckDB-Wasm compiles DuckDB to WebAssembly and usually runs it in a Web Worker. A browser can query CSV, JSON, Parquet, or a file the user just dropped in, then send the result to tables, charts, and interactive filters.

The point is not to put a database in a webpage for sport. It is to shorten the path from data to exploration:

• Keep data on the user’s machine. Medical, financial, log, and personal data does not need to be uploaded just to draw a pie chart. Privacy is not another API wrapper; it is simply not sending the raw data.
• Let the backend publish data while the frontend explores it. The server can produce Parquet from PostgreSQL or another business store, while the browser handles ad-hoc filters, groups, and sorts. The server repeats fewer aggregations, and the user gets more immediate feedback. Bandwidth and storage still cost money; the free lunch comes in a smaller portion.
• Do not necessarily download the whole file. Parquet columns, row groups, and statistics pair with DuckDB’s HTTP Range reads, so a query can fetch the bytes it needs. Configure CORS and Range support on the object store, or the optimization becomes “why is it still downloading?”.
• Keep working offline. A PWA can cache a batch of Parquet and continue analyzing it without a network. An internal data portal does not need a backend round trip for every filter either.

Desktop is just an extension of the same idea: Tauri or Electron handles windows, files, and system integration; DuckDB (native or Wasm) handles analysis; Parquet carries historical data. The interesting part is sharing a data model and query habits between Web and desktop, not arguing about which shell is holier.

Wasm is not magic, of course. Browser memory is finite, so huge datasets still need partitions, sampling, or progressive loading. Frequent row-level writes also belong to a storage layer, not to an analytical engine pretending to be a transaction log.

A small experiment: measure analysis, not who can insert rows

The demo below generates one deterministic trade dataset and asks DuckDB-Wasm, SQLite-Wasm, and IndexedDB + JavaScript to run the same semantic full-year Top-N query:

WITH full_year AS (
  SELECT SUBSTR(t.date, 1, 7) AS month,
         t.region, t.product, t.channel, t.device,
         c.segment, c.weight, t.campaign_id, t.amount, t.units, t.discount, t.latency_ms
  FROM trades AS t
  JOIN campaigns AS c ON c.id = t.campaign_id
  WHERE t.date >= '2024-01-01'
    AND t.date < '2025-01-01'
), monthly AS (
  SELECT month, region, product, channel, device, segment,
         SUM(amount * (1 - discount) * weight) AS total,
         SUM(units) AS units,
         AVG(latency_ms) AS avg_latency_ms,
         QUANTILE_DISC(latency_ms, 0.95) AS p95_latency_ms,
         COUNT(*) AS cnt,
         COUNT(DISTINCT campaign_id) AS campaign_count
  FROM full_year
  GROUP BY month, region, product, channel, device, segment
), ranked AS (
  SELECT *, ROW_NUMBER() OVER (
    PARTITION BY region, product ORDER BY total DESC
  ) AS product_rank,
  RANK() OVER (PARTITION BY segment ORDER BY total DESC) AS segment_rank,
  SUM(total) OVER (PARTITION BY month, region) AS region_month_total
  FROM monthly
)
SELECT month, region, product, channel, device, segment, total, units, avg_latency_ms, p95_latency_ms, cnt, campaign_count
FROM ranked
WHERE product_rank <= 3
ORDER BY total DESC
LIMIT 20;

This is deliberately more than a tiny two-column group-by. It filters a wide fact table with channel, device, a raw payload, and several numeric metrics for a full year (the payload is intentionally not selected, leaving column pruning something to do), joins a campaign dimension table, groups by month, region, product, channel, device, and campaign segment, computes weighted net revenue, units, average latency, P95 latency, and the number of distinct campaigns (COUNT(DISTINCT)), uses several window functions to keep the top three rows per product and calculate partition totals, and then takes a global Top-N. That looks like a sales dashboard, game telemetry, or event-replay summary—and is much closer to the star-shaped analytical workloads DuckDB is built for.

Dataset generation shows overall progress; once it is ready, three empty bars appear immediately. DuckDB, SQLite, and IndexedDB then prepare and query in order. The active engine’s timer starts before its preparation work, and the engine is closed before the next one begins. Only one implementation owns the CPU at a time, so the page does not freeze while three Wasm/storage paths compete; there is no fake animation that jumps from 0 to 80% after a long pause. Bars show the full end-to-end wait honestly, while each completed row also shows query-only time; the conclusion compares query-only time, so fixed setup cost does not hide the analytical advantage on small samples.

// DuckDB query: columnar storage + vectorized execution
// Turn the data into an Arrow table, then let SQL do the aggregation.
const arrow = await import('apache-arrow')
const table = arrow.tableFromArrays({ id, date, region, product, channel, device, payload, campaign_id, amount, units, discount, latency_ms })

await conn.query('DROP TABLE IF EXISTS trades')
await conn.insertArrowTable(table, { name: 'trades' })

const result = await conn.query(
  "WITH full_year AS (SELECT SUBSTR(t.date, 1, 7) AS month, t.region, t.product, t.channel, t.device, c.segment, c.weight, t.campaign_id," +
   " t.amount, t.units, t.discount, t.latency_ms FROM trades t JOIN campaigns c" +
   " ON c.id = t.campaign_id WHERE t.date >= '2024-01-01' AND t.date < '2025-01-01'), monthly AS (" +
   "SELECT month, region, product, channel, device, segment, SUM(amount * (1 - discount) * weight) AS total," +
   " SUM(units) AS units, AVG(latency_ms) AS avg_latency_ms, QUANTILE_DISC(latency_ms, 0.95) AS p95_latency_ms," +
   " COUNT(*) AS cnt, COUNT(DISTINCT campaign_id) AS campaign_count FROM full_year" +
   " GROUP BY month, region, product, channel, device, segment), ranked AS (SELECT *, ROW_NUMBER() OVER" +
   " (PARTITION BY region, product ORDER BY total DESC) AS product_rank, RANK() OVER" +
   " (PARTITION BY segment ORDER BY total DESC) AS segment_rank, SUM(total) OVER" +
   " (PARTITION BY month, region) AS region_month_total FROM monthly)" +
   " SELECT month, region, product, channel, device, segment, total, units, avg_latency_ms, p95_latency_ms, cnt, campaign_count FROM ranked" +
   " WHERE product_rank <= 3 ORDER BY total DESC LIMIT 20"
)

// The engine reads the columns it needs and processes them in batches.

This is not a declaration that IndexedDB is bad. It is an excellent browser-native transactional store. The trouble starts when it is asked to analyze a million rows: read every object back into JavaScript, then hand-write the filtering, grouping, and sorting. SQLite gives you SQL, but it is more row-oriented; this experiment also adds P95 latency and distinct campaign counts, so SQLite has to emulate the discrete percentile with window functions while DuckDB has a native analytical aggregate. DuckDB’s columnar, vectorized execution is closer to this “read a few columns from many rows, then aggregate and sort” workload. The exact numbers depend on the browser, CPU, and caches. The experiment demonstrates a workload shape, not a universal law of physics.

The recent direction: analytical engines are moving closer to apps

This is not a one-person hunch. DuckDB’s recent benchmark and ecosystem update keeps emphasizing local analytics, columnar files, and interoperability; querying Iceberg in the browser is another sign that browser analytics is moving beyond “upload a CSV for five minutes”.

The interesting trend, to me, is that query capability is becoming an application capability. A user can open a page and get a small analytical workbench, instead of waiting for the backend to send four pre-baked charts. The frontend is not only a renderer anymore; it owns part of the data product.

My take: out-of-the-box analysis has room to grow

For small, frequently changing objects and row-level CRUD, IndexedDB, SQLite, or a business database is the better fit. For lots of locally generated records with filtering, grouping, sorting, and window calculations, DuckDB deserves a serious look. If the history needs to move across tools, languages, and machines, let Parquet be the common format.

The composition I have in mind is:

Business database or .duckdb for mutable hot data → periodic Parquet exports → DuckDB-Wasm for hot+cold Web analysis → Arrow/JSON for charts.

It could become a Web/desktop BI tool, a download-and-go personal data analyzer, or a game utility with live replays and leaderboards. The server publishes data, the app lets users explore it, and the CPU that was supposed to do the work finally gets woken up—without every filter button knocking on the backend’s door first.

A few implementation sticky notes for future me:

1. If CSP dislikes a CDN Worker, self-host the Wasm and Worker files under a same-origin path.
2. SSR has no Worker, so load DuckDB-Wasm dynamically in the browser.
3. Give Arrow an explicit column shape before handing it to DuckDB; an arbitrary object array is not a magic passport.
4. SQLite-Wasm statements use finalize(), not free(). One word, half an hour of life.