Exporting Your Data as CSV, JSON, SQL, HTML, XLSX, and Parquet

Exporting data from a database client seems trivial. In practice, it's an engineering problem that touches on streaming, memory management, output format, and user experience. In QoreDB, export relies on a unified asynchronous pipeline that feeds six distinct writers: CSV, JSON, SQL INSERT, HTML, XLSX, and Parquet.

The goal wasn't to pile up formats to tick boxes. Each format meets a real need, identified in the context of everyday development.

A streaming pipeline, not an in-memory dump

The starting point is that the export doesn't load the entire result into memory before writing. The pipeline uses the drivers' native streaming (PostgreSQL cursors, MySQL iterative fetch, etc.) to consume rows as they come. On the Rust side, a tokio::spawn task runs the query via execute_stream_in_namespace, which pushes each row into an mpsc channel with a capacity of 100. The writer consumes this channel and writes directly to disk via an asynchronous BufWriter.

This design makes it possible to export tables of several million rows without memory consumption exceeding a few megabytes. The disk flush happens every 1,000 records by default (configurable via batch_size). The user can follow the progress in real time thanks to the Tauri events emitted every 250 ms, with the number of rows exported, the bytes written, and the throughput in rows per second.

The export is also cancelable at any moment. A tokio CancellationToken makes it possible to cleanly interrupt the streaming and release the driver-side resources, including canceling the in-flight query on the database server.

The ExportWriter trait: one interface per format

All formats implement the same Rust trait: ExportWriter. This trait defines four methods: write_header for the columns, write_row for each row, flush to force the write to disk, and finish to finalize the file (write an HTML footer, close an XLSX archive, etc.).

This abstraction lets the pipeline stay identical regardless of the format. The create_writer function instantiates the right writer according to the ExportFormat enum, and the pipeline consumes the stream in the same way. Adding a format amounts to implementing the trait and adding a variant in the match.

Six formats, six use cases

CSV is the universal format. It works everywhere, imports into any spreadsheet or script, and stays lightweight. The writer handles escaping of quotes and line breaks within values. The include_headers option controls whether the header row is present.

JSON produces an array of objects, one per row. It's the natural format for feeding a script, an API, or a processing pipeline. Types are preserved: integers stay numbers, booleans stay booleans, null values are explicit.

SQL INSERT generates INSERT INTO statements ready to be replayed on a database of the same engine. The writer uses the source driver's SqlDialect to produce syntactically correct SQL (escaping quotes, literal formatting). This format requires a table name, validated before the export is launched.

HTML is the most unusual format. QoreDB generates a self-contained HTML file that embeds the CSS and JavaScript needed to display the data with per-column sorting, text filtering, pagination, and type coloring. The file opens in any browser, with no server or external dependency. It's the ideal format for sharing a dataset with a colleague who doesn't have QoreDB.

XLSX and Parquet are professional formats reserved for QoreDB Pro. XLSX allows direct import into Excel with column types preserved. Parquet is a compressed columnar format, designed for analytics and ingestion into tools like DuckDB, Spark, or BigQuery. These two writers manage their own I/O (no shared BufWriter) because their binary formats require it.

In practice: exporting from the interface

The export is triggered from the Data Grid or after running a query. The user chooses a format, a destination path, and launches the operation. A progress bar appears with the real-time throughput. They can cancel at any moment without corrupting the output file, because the writer only finalizes at the end (the HTML footer is only written at finish, the XLSX archive is only closed after the last row).

The export goes through the Tauri command start_export, which validates the output path (it must be absolute, with no parent-directory traversal, in an existing folder), checks that the driver supports streaming, then launches the asynchronous task. Each export receives a unique identifier (UUID) that lets you track its progress or cancel it via cancel_export.

A limit parameter lets you cap the number of rows exported. This is useful for extracting a data sample without waiting for the full export of a large table. When the limit is reached, the pipeline sends a cancellation signal to the driver and cleanly finalizes the file.

Why separate Core and Pro on the formats

CSV, JSON, SQL INSERT, and HTML cover the vast majority of a developer's daily needs. These four formats are available in the Core (open source) version of QoreDB. XLSX and Parquet, on the other hand, target more specialized workflows: Excel reporting, analytics ingestion, columnar archiving. They pull in additional Rust dependencies and are compiled conditionally via the "pro" feature flag. This separation is consistent with QoreDB's open-core model: the technical foundation stays free, and the professional formats are part of the Pro offering.

Export in QoreDB is a good example of the project's philosophy: a clean technical mechanism (streaming pipeline, unified trait, real-time progress) in service of a simple need (getting your data out in the right format). No magic, no implicit conversion between engines. The file produced contains exactly what the query returned, in the chosen format.