The Background Job Manager: Long Exports and Asynchronous Tasks

Exporting five million rows to CSV, launching a pg_dump backup, aggregating a MongoDB stream through a federation. These operations take anywhere from a few seconds to several minutes. In a desktop application, they can't run on the main thread without freezing the interface. QoreDB relies on a Background Job Manager built around tokio to offload these tasks to the background, track their progress, and allow them to be cancelled without corrupting state.

The principle is simple to state but demands rigor in the implementation: each job is a tokio::spawn coroutine that has its own identity, its own progress channel, and its own cancellation signal. The Tauri code exposed to the frontend simply launches the job, returns an identifier immediately, and everything else happens through an event stream.

Why a Job Manager in a desktop client

A short interactive query (a SELECT on a table with LIMIT) resolves in a few milliseconds. It can be awaited directly in the Tauri handler. But as soon as you're talking about a full table export, generating a multi-gigabyte Parquet file, or a complete pg_dump, you enter another regime. The operation must be able to be launched, tracked, and cancelled at any time without blocking the user's next click.

The Job Manager plays this role in three concrete cases: exports to CSV, JSON, SQL INSERT, HTML, XLSX, and Parquet; backups and restores that drive an external binary such as pg_dump or mysqldump; and federated executions that orchestrate several drivers at once. They all share the same skeleton: a job has a job_id, a state (Pending, Running, Completed, Cancelled, Failed), a progress event channel, and a cooperative cancellation mechanism.

Isolation with tokio::spawn

At the heart of the system is a recurring pattern. The Tauri handler receives the request, validates the parameters, allocates an identifier, registers the job in a shared table, then calls tokio::spawn to launch the background coroutine. The handler returns immediately with the job_id. The frontend can then update the list of running jobs, display a progress bar, and listen to the events emitted by that task.

The choice of tokio::spawn over spawn_blocking is deliberate: these tasks are IO-bound. They spend their time waiting for data from the database engine or writing to disk, not burning CPU. Tokio multiplexes them onto its runtime pool without monopolizing OS threads. When it comes to write-intensive work like Parquet, the writer exposes its interface in async and delegates the CPU-bound blocks to the blocking pool internally.

Cooperative cancellation with CancellationToken

Killing a Tokio task abruptly would leave resources in an undefined state: a writer half-flushed to disk, an open transaction on the database side, a locked file. The export pipeline therefore uses tokio_util::sync::CancellationToken to signal cancellation, and the job's inner loop checks this token via a tokio::select! on every iteration.

Concretely, the loop waits in parallel for two things: a row event coming from the driver through an mpsc channel, or the token's signal. If the token is triggered, the job calls driver.cancel(session_id, Some(query_id)) to release the query on the server side, transitions to the Cancelled state, then runs the writer's flush and finalization phase. The partial file stays on disk with a known row count, which lets the user resume where they left off if they wish.

For backups launched via an external binary, the mechanism is different but the spirit is the same. A oneshot::Sender is registered in an ActiveBackups structure indexed by job_id. When the user clicks Cancel, the sender sends a signal to the wrapper task, which calls child.kill() on the child process. This path is necessary because we don't control the code of pg_dump; we can only terminate its process cleanly.

Real-time progress feedback

An export that runs for three minutes with no sign of life is unacceptable from a UX standpoint. Each job therefore emits a stream of progress events through Tauri's Window::emit system. The channel is named export_progress:<job_id>, which lets the frontend subscribe precisely to the job it cares about.

The ExportProgress structure contains the number of exported rows, the bytes written, the elapsed time in milliseconds, the throughput in rows per second, and an optional error message. To avoid flooding the frontend with one event per row, the loop emits at most once every 250 milliseconds. This throttling is computed by looking at last_emit.elapsed() rather than by counting rows, which keeps a predictable cadence even when throughput varies.

For external backups, it's the child process's stdout and stderr that are relayed line by line to the frontend as BackupEvent::Log events. The user therefore sees pg_dump's native output scroll by as if they had launched it in a terminal.

The registry of active jobs

Each family of jobs has a small registry structure. For exports, it's ExportPipeline with a RwLock<HashMap<String, ExportJob>> that stores the CancellationToken of each running job. For backups, it's ActiveBackups with a Mutex<HashMap<String, oneshot::Sender<()>>>. The registry is placed in the Tauri state and accessible from any command.

Registration happens just before the spawn, deregistration just after the writer is finalized or the child process terminates. The cycle is short and keeps the table clean. A dedicated Tauri command enumerates the active jobs to feed the status bar, and another targets a specific job_id to cancel it. The frontend doesn't need to aggregate this information itself.

Error recovery and lifecycle

A job has only three possible terminal states: Completed, Cancelled, Failed. Inside the job, every potentially fallible operation (writer write, flush, finalization, stream read) is intercepted. On failure, the job switches to Failed, the message is captured in the error field, and the loop is interrupted cleanly. A final flush and finish on the writer are attempted so as not to leave a corrupted file behind.

The driver task itself gets special handling. When the job ends on cancel or failure, the associated driver task is granted two seconds to stop via timeout, after which it is aborted. This delay prevents a long-running query from keeping the interface from returning to a clean state, while still giving a well-behaved driver the chance to stop gracefully.

The user experience

On the frontend side, the work is reduced to the essentials: call the Tauri command that launches the job, receive the job_id, listen to the corresponding event channel, and update a progress component. The user can keep navigating in QoreDB, open another tab, run another query. A four-gigabyte Parquet export runs in the background, and the next click stays responsive.

Cancellation is instantaneous from the user's point of view, even though it actually triggers an ordered cascade: token triggered, driver.cancel sent to the engine, writer flush and finalization, deregistration from the registry, emission of a final Cancelled event. The partial file remains usable. If an error occurs along the way, the user receives a clear message with the number of rows processed before the failure.

A design suited to the desktop

This Job Manager is not a distributed queue system. There is no persistence of jobs to disk, no recovery after a process crash, no cron scheduling. It's a design meant for single-user desktop use: the jobs are numerous but short to medium in length, they live in the process, and their lifecycle is tied to that of the application. This simplicity is what lets the code stay readable, cancellation be reliable, and feedback be instantaneous.

The result is a system that fades into the background when everything goes well and responds cleanly when the user wants to take back control. Launching an export feels like pressing play; cancelling it feels like pressing stop. All the coordination, signaling, and cleanup work is encapsulated in a handful of Rust structures that share the same skeleton: an identifier, a channel, a token, a registry.