qumirdb

KO | EN

gitlite — search

#svg #rag #mysql #list #rust #macos #security #chatgpt #monitoring #cross-platform

(★ 9)

QumirDB is an analytical query engine designed to run parquet-backed workloads

File Explorer

ZIP Download

- - c-cpp.yml
  - deb.yml
- - run_clickbench.sh
- - run_tpcds.sh
- - run_tpch.sh
- cli.cpp
- CMakeLists.txt
- plan_export.cpp
- plan_export.h
- QdbPackaging.cmake
- - aggregation.md
  - architecture.tex
  - benchmark_results.md
  - building.md
  - decorrelation.md
  - dispatch.md
  - join.md
  - join_reorder.md
  - kernel-compilation-phases.md
  - late_materialization.md
  - logical_plan_build.md
  - optimizations.md
  - predicate_pushdown.md
  - README.md
  - rust_toolchain.md
  - scheduler_runtime.md
  - sql_pseudo_calls.md
  - string_values.md
- CMakeLists.txt
- coroio
- qumir
- - external_module.cpp
  - external_module.h
- - aggregate_exec.cpp
  - aggregate_exec.h
  - filter_exec.cpp
  - filter_exec.h
  - join_exec.cpp
  - join_exec.h
  - kernel_rowset.h
  - late_materialize_exec.cpp
  - late_materialize_exec.h
  - plan.h
  - plan_builder.cpp
  - plan_builder.h
  - planner_helpers.cpp
  - planner_helpers.h
  - project_exec.cpp
  - project_exec.h
  - runtime_context.h
  - sort_exec.cpp
  - sort_exec.h
  - sort_key_ops.cpp
  - sort_key_ops.h
  - stage.h
  - unary_process.h
  - window_exec.cpp
  - window_exec.h
- - io.h
  - physical_row_id.h
  - schema.h
- - aggregate_key.cpp
  - aggregate_key.h
  - annotate_type.cpp
  - annotate_type.h
  - builder.cpp
  - builder.h
  - column_value.cpp
  - column_value.h
  - compiler.cpp
  - compiler.h
  - finalize.cpp
  - finalize.h
  - finalize_fused.cpp
  - finalize_fused.h
  - gen.cpp
  - gen.h
  - generated.h
  - join_gen.cpp
  - join_gen.h
  - join_key.cpp
  - join_key.h
  - lib.cpp
  - lib.h
  - regex.cpp
  - regex.h
  - spec.cpp
  - spec.h
  - vm_function.cpp
  - vm_function.h
- - qumirdb.cpp
  - qumirdb.h
  - qumirdb.oz
  - qumirdb_runtime.cpp
  - qumirdb_runtime.h
  - qumirdb_types.h
  - row_group_interval.oz
- - build.cpp
  - build.h
  - clone_expr.cpp
  - clone_expr.h
  - clone_operator.cpp
  - clone_operator.h
  - pipeline.cpp
  - pipeline.h
  - plan_print.cpp
  - plan_print.h
  - stats_codec.h
- - connection.cpp
  - connection.h
  - executor.cpp
  - executor.h
  - graph.cpp
  - graph.h
  - mpmc.h
  - plan_lowerer.cpp
  - plan_lowerer.h
  - runtime_adapter.cpp
  - runtime_adapter.h
  - scan_split.cpp
  - scan_split.h
  - settings.h
  - single_threaded_scheduler.cpp
  - single_threaded_scheduler.h
  - spsc.h
  - state.h
  - threaded_scheduler.cpp
  - threaded_scheduler.h
- - parser.cpp
  - parser.h
  - printer.cpp
  - printer.h
- - ast.cpp
  - ast.h
  - lexer.cpp
  - lexer.h
  - parser.cpp
  - parser.h
  - printer.cpp
  - printer.h
- - module_path.cpp
  - module_path.h
  - regex.cpp
  - regex.h
  - rust_std_loader.cpp
  - rust_std_loader.h
  - sha1.cpp
  - sha1.h
  - union_find.h
- CMakeLists.txt
- stub.cpp
- - api.js
  - app.js
  - browser_parquet.js
  - browser_runtime.js
  - browser_storage.js
  - browser_worker.js
  - external_queries.js
  - graph.js
  - index.html
  - oz_type.js
  - package.json
  - styles.css
  - tpcds_queries.js
  - tpch_queries.js
  - wasm_allocator.js
  - wasm_capabilities.js
- CMakeLists.txt
- postinst
- qumirdb-service.service.in
- server.cpp
- - run.mjs
- CMakeLists.txt
- mock_source.h
- plan_export_native_runner.cpp
- plan_runner.h
- qumirdb_source_module.h
- test_aggregate.cpp
- test_aggregate_f64.cpp
- test_cbo.cpp
- test_const_fold.cpp
- test_cte_materialize.cpp
- test_cte_reuse.cpp
- test_equijoin.cpp
- test_exec_plan_builder.cpp
- test_external_module.cpp
- test_filter.cpp
- test_finalize_fused.cpp
- test_heap.cpp
- test_io.cpp
- test_join.cpp
- test_join_asymmetric.cpp
- test_join_exec.cpp
- test_join_kernel.cpp
- test_join_key.cpp
- test_join_node.cpp
- test_join_pipeline.cpp
- test_join_planner.cpp
- test_kernel_spec.cpp
- test_late_materialization.cpp
- test_lexer.cpp
- test_owned_blocks.cpp
- test_parquet_row_reader.cpp
- test_project_e2e.cpp
- test_project_kernel.cpp
- test_project_type.cpp
- test_reg.cpp
- test_row_group_predicate.cpp
- test_row_group_pushdown_e2e.cpp
- test_scheduler.cpp
- test_scheduler_connection.cpp
- test_scheduler_core.cpp
- test_scheduler_graph.cpp
- test_scheduler_queues.cpp
- test_scheduler_runtime_adapter.cpp
- test_scheduler_scan_split.cpp
- test_scheduler_threaded.cpp
- test_sexp.cpp
- test_sort.cpp
- test_sort_plan.cpp
- test_sort_row_id.cpp
- test_stats.cpp
- test_string_key_descriptor.cpp
- test_string_ops.cpp
- test_string_types.cpp
- test_swiss_group.cpp
- test_window.cpp
- test_window_exec.cpp
- test_window_plan.cpp
.gitignore
.gitmodules
CMakeLists.txt
LICENSE
README.md
todo.txt

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**The same compiled query runs natively and in your browser.**

QumirDB is an analytical query engine for Parquet. It handles scans,
filters, joins, aggregates, and sort. A planner uses predicate pushdown,
join reorder, and row-group pruning to plan each query.

Every query compiles through LLVM into a generated kernel. It does not run
through an interpreter. The same codegen path builds host machine code and
also builds WebAssembly. The same fused kernel runs on the server, or in the
browser against OPFS-backed datasets. There is no separate browser engine to
keep in sync with the native one.

Compiled queries have one more benefit. User code can join the same kernel.
You register a Rust module with `CREATE MODULE`. Its bitcode joins the
query's LLVM module before optimization runs. Then LLVM inlines the Rust
code, propagates constants through it, and vectorizes it together with the
SQL. See [UDFs that disappear into the
query](#udfs-that-disappear-into-the-query) below.

Try it live at **[db.qumir.dev](https://db.qumir.dev)**.

## SQL, plainly

```sql
SELECT
    l_returnflag,
    l_linestatus,
    SUM(l_quantity)                       AS sum_qty,
    SUM(l_extendedprice)                  AS sum_base_price,
    SUM(l_extendedprice * (1 - l_discount)) AS sum_disc_price,
    AVG(l_discount)                       AS avg_disc,
    COUNT(*)                              AS count_order
FROM lineitem
WHERE l_shipdate <= DATE '1998-09-02'
GROUP BY l_returnflag, l_linestatus
ORDER BY l_returnflag, l_linestatus;
```

The planner applies many rewrites: join reorder, predicate pushdown, column
pruning, and Parquet row-group pruning. Each rewrite has a doc with a
minimal query and the plan before and after it, in
[docs/arch/optimizations.md](docs/arch/optimizations.md).

## UDFs that disappear into the query

```sql
CREATE MODULE orbital LANGUAGE rust AS $$
#[no_mangle]
pub extern "C" fn orbit_is_neo(a: f64, e: f64) -> bool {
    a * (1.0 - e) < 1.3
}
$$;

CREATE FUNCTION orbit_is_neo(a DOUBLE, e DOUBLE) RETURNS BOOL
SET MODULE TO orbital;
```

Call it like any built-in function:

```sql
SELECT designation FROM sbdb_objects
WHERE orbit_is_neo(a, e) AND e < 0.3;
```

`orbit_is_neo` does not run through a function pointer call. Its Rust
bitcode joins the same LLVM module as the query kernel before optimization
starts. The compiler treats it just like SQL generated code. Without this,
a scalar call would stay inside the row loop, like this:

```llvm
%is_neo = call i1 @orbit_is_neo(double %a, double %e)
%low_e  = fcmp olt double %e, 0.3
```

After optimization, the call is gone. The code turns into a wide vector
loop instead. A test used a 32 million row, two column Parquet scan, on
one thread, on an Apple M5 Pro chip, with 20 paired runs. The kernel time
dropped from 292 ms to 108 ms. That is 2.72 times faster. Both numbers
include the Parquet scan cost.

A Rust struct becomes multiple SQL columns:

```sql
CREATE FUNCTION orbit_position(
    a DOUBLE, e DOUBLE, i DOUBLE, w DOUBLE,
    node DOUBLE, m DOUBLE, epoch DOUBLE, ts DOUBLE
) RETURNS (DOUBLE, DOUBLE, DOUBLE)
SET MODULE TO orbital;

SELECT orbit_position(a, e, i, w, node, m, epoch, ts) AS (x, y, z)
FROM unnumbered_asteroids;
```

The engine links Rust code before it runs optimization. Because of this,
the query can specialize the Rust code it calls. A mode argument can be a
constant value at query time. Then constant propagation can delete the
branch the query never uses:

```sql
CREATE FUNCTION hazard_score(diameter DOUBLE, velocity DOUBLE, mode INT)
RETURNS DOUBLE
SET MODULE TO orbital;

SELECT designation FROM sbdb_objects
WHERE hazard_score(diameter, velocity, 2) > 10;  -- 2 is a constant here
```

```rust
fn hazard_score(d: f64, v: f64, mode: i32) -> f64 {
    if mode == 2 { precise_model(d, v) } else { fast_model(d, v) }
}
```

The query around it stays a normal relational query. Joins, dates, grouping,
and aggregation stay in SQL. The domain model stays in Rust:

```sql
SELECT
    o.orbit_class,
    danger,
    COUNT(DISTINCT c.designation) AS asteroids,
    COUNT(*)                      AS approaches,
    MIN(c.dist_au)                AS closest_au,
    MAX(hazard_score(diameter, velocity, 2)) AS max_score
FROM close_approaches c
JOIN sbdb_objects o USING (designation)
LEFT JOIN sentry_virtual_impactors s USING (designation)
WHERE c.approach_date BETWEEN DATE '2026-08-13' AND DATE '2126-08-13'
  AND c.dist_au < 0.1
GROUP BY o.orbit_class, danger
ORDER BY danger DESC, o.orbit_class;
```

Parquet statistics prune row groups that do not matter, before the Rust
model runs at all. Scan tasks on separate threads feed normal columnar
batches into the fused kernels. The result is a small SQL table. It is not
a custom Rust output format.

## Why it exists

- **One optimizer, two languages.** SQL owns scans, predicate pushdown, join
  order, and grouping. Rust owns the domain logic that does not fit a query
  language: numeric models, parsers, state machines. LLVM optimizes both
  together, not one around the other.
- **Same query, two targets.** The native path uses a JIT to build host
  machine code. The browser path compiles the same fused kernel to
  WebAssembly. It runs against OPFS-backed datasets, with no server round
  trip per row.
- **Not tied to Rust.** Any language with an LLVM backend and a C ABI can use
  the same path. Rust just has no GC to fight and a small, clear boundary.
  See [docs/arch/rust_toolchain.md](docs/arch/rust_toolchain.md).

## Try it

Live playground: **[db.qumir.dev](https://db.qumir.dev)**. Write SQL,
inspect the logical and physical plans, and run the query natively on the
server or in your own browser.

To build and run locally, see
[docs/arch/building.md](docs/arch/building.md).

## Architecture

Design notes live in [docs/arch/](docs/arch/README.md).

- [Optimizations](docs/arch/optimizations.md): every rewrite the engine does,
  with a minimal query and the plan before and after.
- [Logical plan build](docs/arch/logical_plan_build.md): SQL AST to operator
  tree, aggregate rewrites, CTEs.
- [Predicate pushdown](docs/arch/predicate_pushdown.md) and
  [join reorder](docs/arch/join_reorder.md): the two passes that decide how
  much data the scans return.
- [Aggregation](docs/arch/aggregation.md) and [join](docs/arch/join.md):
  generated kernels, hash tables, runtime ABI.
- [Scheduler runtime](docs/arch/scheduler_runtime.md) and
  [dispatch](docs/arch/dispatch.md): the physical graph, tasks and
  connections.
- [Late materialization](docs/arch/late_materialization.md): a narrow Parquet
  scan and late column fetches after a small `LIMIT`.
- [Kernel compilation phases](docs/arch/kernel-compilation-phases.md): how a
  plan becomes Qumir code, LLVM IR, and finally machine code or wasm.

## Repository layout

- `qdb/`: engine library. SQL/parser, logical plan, passes, kernel
  generation, execution, scheduler, parquet IO.
- `bin/`: the `qdb` command line tool.
- `service/`: web server and static browser workbench.
- `test/`: C++ unit and regression tests.
- `benchmark/`: TPC-H, TPC-DS, and ClickBench query packs and runners.
- `docs/arch/`: design notes for planner, scheduler, joins, aggregation, and
  kernel dispatch, plus build and deployment instructions.

QumirDB is built on [Qumir](https://qumir.dev)
([GitHub](https://github.com/resetius/qumir)), the compiler and runtime
layer for generated query kernels, and on
[Coroio](https://coroio.dev) ([GitHub](https://github.com/resetius/coroio)),
the coroutine-based web service runtime.

qumirdb

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

qumirdb

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README