ginzburg-neural-denoiser

KO | EN

gitlite — search

#gpt #nlp #html #mysql #llm #chatgpt #redis #devops #agent #data-science

ginzburg-neural-denoiser

(★ 9)

ML research on temporally stable, detail-preserving denoising for rendered sequences and video. Automatic noise calibration and a standalone C++/CUDA inference runtime.

File Explorer

ZIP Download

- - ci.yml
  - gpu-tests.yml
- main.cc
- FindCUDNN.cmake
- - architecture.svg
  - flicker-face.png
  - flicker-hair.png
  - flicker-header.png
  - flicker-neck.png
  - noise-to-clean-comparison.jpg
  - ster-pipeline.svg
- architecture.md
- benchmarks.md
- build.md
- mc_noise_workflow.md
- usage.md
- weight_format.md
- - tensor.h
  - weight_loader.h
- - mc_noise_presets_demo.json
- export_mc_noise_presets.py
- exr_analyzer.py
- mc_noise_presets.nk
- MCNoise.blink
- MCNoise.nk
- generate_mc_noise_pairs.sh
- infer.sh
- tensorboard.sh
- train.sh
- - exr_io.cc
  - image_io.cc
  - video_io.cc
- - bn_inference.cuh
  - concat.cuh
  - deform_im2col.cuh
  - fp16_utils.cuh
  - model_kernels.cu
  - nafnet_kernels.cu
- - batchnorm2d.cu
  - conv2d.cu
  - deform_conv2d.cu
  - maxpool2d.cu
  - relu.cu
  - upsample.cu
- - nafnet.cu
  - nafnet_temporal.cu
- tensor.cu
- weight_loader.cc
- CMakeLists.txt
- conftest.py
- gen_fixtures.py
- gen_sample_images.py
- noise_preview.py
- test_batchnorm2d.cc
- test_conv2d.cc
- test_integration.py
- test_output_path.cc
- test_tensor.cc
- test_unet_forward.cc
- test_weight_loader.cc
- visualise_noise.py
- googletest
- nlohmann_json
- stb
- tinyexr
- - __init__.py
  - conftest.py
  - test_dataset.py
  - test_export.py
  - test_image_io.py
  - test_infer.py
  - test_metrics.py
  - test_models.py
  - test_noise_generators.py
  - test_noise_profiler.py
  - test_noise_visual.py
- assess_motion.py
- convert_renders_to_dataset.py
- dataset.py
- eval_benchmark.py
- export.py
- generate_mc_noise_pairs.py
- gpu_benchmark.py
- image_io.py
- infer.py
- losses.py
- metrics.py
- models.py
- motion_compensation.py
- noise_calibrate.py
- noise_generators.py
- noise_profiler.py
- preview_mc_noise.py
- preview_noise.py
- pyproject.toml
- run_benchmarks.py
- smoke_benchmark.py
- temporal_dataset_creator.py
- train.py
- video_extract.py
- .gitkeep
.clang-format
.gitignore
.gitmodules
CHANGELOG.md
CMakeLists.txt
LICENSE
Makefile
pyproject.toml
README.md
THIRD_PARTY_LICENSES.md
uv.lock

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

# Project Badges

CI Visit
License: MIT Visit

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Overview

**Ginzburg Neural Denoiser** is my research project on temporal denoising for Monte Carlo rendered
sequences, with tooling for calibrated camera noise. It spans the full stack: dataset generation,
temporal training and evaluation, model export, and a standalone C++/CUDA inference runtime (cuDNN,
cuBLAS, and custom kernels).

![Low-sample render and denoised result](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/noise-to-clean-comparison.jpg)

*Low-sample render and denoised result. Image by Joanna Kobierska.*

There are strong denoisers already (Intel OIDN, NVIDIA OptiX, Neat Video), and on a single frame
they look great. The trouble is motion. When a denoiser reconstructs each frame on its own, the
residual noise and reconstruction errors change from frame to frame, so a clip that looks clean when
paused can **boil and flicker** in playback. Some tools handle this with temporal modes; others leave
it to an artist to tune per shot. STeR targets that directly: offline, high-quality denoising that
stays stable across the whole sequence, with no per-shot tuning at inference.

So two things are the focus:

- **Temporal stability:** no flicker, boiling, ghosting, or trailing; the whole sequence holds
  together, frame after frame.
- **No per-shot tuning:** data-driven at inference; the noise level is estimated from the input, so
  an artist sets no denoising strength or temporal thresholds per shot.

![Noisy input, Intel OIDN, and STeR](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/flicker-header.png)

*Noisy input, Intel OIDN, and STeR (ours). STeR leaves the least flicker (lower LF-p95).*

---

## Evolution

- **2016–2020, [Ginzburg Denoiser](https://github.com/ginzburg-dev/ginzburg-denoiser).** My first
  denoiser: classical, deterministic, and built for temporal stability. Used in production on
  *The Fixies*, then on animated features and series.
- **2024–present, Ginzburg Neural Denoiser (this project).** I combine that classical work with an
  ML model I designed and trained on my own rendered dataset. The current method is **STeR**,
  Stable Temporal Restoration.
- **2025–present, [Denoise-Machine-X](https://github.com/ginzburg-dev/Denoise-Machine-X).** In
  development: a production plugin that brings STeR and other classical and neural filters into DCC
  applications.

---

## STeR: Stable Temporal Restoration

STeR (Stable Temporal Restoration) is my own temporal denoising method for rendered sequences. It
removes render noise while keeping the image stable across the whole clip, suppressing the flicker,
boiling, ghosting, and edge smearing that appear when a single-frame denoiser is applied to moving
footage. It needs no per-shot tuning: the noise level is estimated from the input, and the model
adapts to it along the sequence.

I designed STeR and trained it on the dataset described below; it is marked *Ours* in the comparison.
Its architecture, training recipe, and weights are mine and stay private, while the baselines,
tooling, and runtime around it are all in this repository.

Inside, STeR runs a data-driven pipeline:

![STeR pipeline: noisy window, then noise estimation, motion compensation and feature extraction, then STeR temporal fusion and refinement, to a clean stable frame](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/ster-pipeline.svg)

The pipeline is fully data-driven: the noise level and features are estimated from the input, so
there are no per-shot strength or threshold knobs to set at inference.

This is a flicker test. Each strip plays nine consecutive frames of one shot, cropped on a hard
region: hair, the shadow-side face, the neck and collar. The crop follows the character but the
frames are not aligned, so the movement you see is what the denoiser produced. Watch the flat areas:
Intel OIDN and NVIDIA OptiX boil and shimmer, STeR stays still. STeR uses only the noisy colour pass,
while the others also get the albedo and normal passes.

![Flicker on hair: Noisy, Intel OIDN, NVIDIA OptiX, STeR (ours), Reference](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/flicker-hair.png)

![Flicker on the shadow-side face, close crop: Noisy, Intel OIDN, NVIDIA OptiX, STeR (ours), Reference](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/flicker-face.png)

![Flicker on the neck and collar: Noisy, Intel OIDN, NVIDIA OptiX, STeR (ours), Reference](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/flicker-neck.png)

*Temporal stability comparison. Intel OIDN and NVIDIA OptiX flicker and boil heavily across the
sequence; STeR stays stable.*

The scores below are from a single 36-frame shot, measured against its clean render. On this
sequence STeR reads only the noisy colour pass, while Intel OIDN and NVIDIA OptiX also receive the
albedo and normal passes. It is one shot, not a multi-sequence average.

**Temporal stability** (higher PSNR is better, lower is better for the rest):

| Method | Temporal PSNR ↑ | Temporal MAE % ↓ | Temporal chroma % ↓ | Flicker LF-p95 % ↓ |
|---|---:|---:|---:|---:|
| **STeR (Ours)** | **43.78** | **0.56** | **0.39** | **0.045** |
| NVIDIA OptiX | 43.75 | 0.62 | 0.42 | 0.060 |
| NAFNet Temporal + AOV | 42.43 | 0.66 | 0.53 | 0.053 |
| Intel OIDN | 42.33 | 0.75 | 0.54 | 0.054 |
| Noisy input | 36.95 | 1.42 | 1.07 | 0.054 |

- **Temporal PSNR**: closeness to the clean reference over the clip. Higher is cleaner.
- **Temporal MAE**: frame-to-frame jitter. Lower is steadier.
- **Temporal chroma**: colour jitter between frames. Lower keeps colour steady.
- **Flicker LF-p95**: large-area brightness pulsing, near worst case. Lower is less flicker.

On this sequence STeR leads the temporal metrics; on PSNR it is effectively tied with OptiX
(43.78 vs 43.75), with clearly lower temporal MAE, chroma variation, and flicker, using only the
noisy colour pass.

**Spatial quality** (single-frame fidelity):

| Method | PSNR (dB) ↑ | SSIM ↑ |
|---|---:|---:|
| Intel OIDN | **41.74** | **0.975** |
| NVIDIA OptiX | 40.69 | 0.974 |
| **STeR (Ours)** | 40.02 | 0.970 |
| NAFNet Temporal + AOV | 38.73 | 0.966 |
| Noisy input | 36.82 | 0.900 |

On a single frame, Intel OIDN gives the strongest fidelity here; STeR trades a little per-frame
sharpness for temporal stability. More metrics and baselines are in
[docs/benchmarks.md](docs/benchmarks.md).

---

## Architecture

Path tracing estimates illumination from randomly sampled light paths; a smaller sampling budget
renders faster but leaves stochastic error in the image, and in an animation that error shifts from
frame to frame and reads as flicker. It resists naive filtering
because sampling noise and fine scene detail overlap in the same high-frequency range, so filtering
too hard takes the detail with it. A spatial denoiser only makes the temporal problem worse:
it processes each frame on its own and cannot enforce consistency across the sequence.

STeR takes the opposite route. It reconstructs the centre frame from a short temporal window,
deciding per pixel how much to trust each neighbouring frame and fusing them accordingly. It can
optionally take auxiliary render features (albedo, normals, depth) as guidance when they are
available, but the benchmark below used only the noisy colour pass. The objective is to remove the
temporally uncorrelated noise while preserving detail and frame-to-frame coherence.

The public NAFNet variants are reproducible baselines that exercise the same training, export, and
inference pipeline.

![NAFNet Temporal baseline architecture, with STeR shown as a black box](https://raw.githubusercontent.com/ginzburg-dev/ginzburg-neural-denoiser/main/docs/img/architecture.svg)

*Public NAFNet Temporal baseline: an encoder-decoder of NAFBlocks with per-level temporal mixing
(pink) on the skips. STeR reuses this training and inference pipeline; its internal blocks stay
private.*

---

## Training

### Dataset

I built the paired STeR dataset from real production scenes. For each scene I set up the render
and produced one clean reference plus seven progressively noisier variants on a render farm.

| Scenes | Frames per scene | Noisy variants | Total rendered frames |
|---:|---:|---:|---:|
| ≈75 | ≈60 | 7 | ≈36,000 |

That works out to roughly 75 × 60 × 8 renders: seven noise levels and one clean reference.
Sequence lengths vary. The samples are temporally contiguous rather than shuffled stills, so the
dataset carries real motion, occlusion, disocclusion, and changes in illumination and scene content.
The source frames themselves are not distributed.

### Augmentation

Noisy inputs and clean targets stay aligned through every crop and transform. Crop, flip, and
rotation parameters are sampled once per temporal window, so augmentation never introduces
artificial frame-to-frame inconsistency. Training can also apply constant-velocity shifts across
neighbouring frames to add controlled motion without breaking the input-target correspondence.

### Noise generation and profiling

Beyond the paired dataset, the pipeline can synthesise noise during training: Gaussian,
Poisson-Gaussian, ISO-parameterised camera noise, residual patches sampled from real footage, and
Monte Carlo render noise.

For camera footage, `noise_profiler.py` estimates signal-dependent shot noise and
signal-independent read noise from dark and flat-field clips, or extracts pools of residual patches
from real footage. The selected profile is held constant within each temporal window, so
augmentation does not fake a frame-to-frame change in the camera noise model.

### Export

The public models are trained in PyTorch. `training/export.py` writes the model description to
`model.json` and each tensor to a binary weight file in FP16 or FP32. The exporter can reload the
files and compare them with the PyTorch state dictionary before they are passed to the standalone
C++/CUDA runtime. Python/C++ parity tests then compare the output of both implementations.

---

## Models & weights

| Model / backend | Role | Params | Weights | Source |
|---|---|:--:|---|---|
| **`STeR` (ours)** | Internal stable temporal restoration model | not published | not published | Internal research |
| NAFNet (`spatial`) | public spatial baseline | configuration-dependent | bring your own checkpoint | Public |
| `mc-temporal` | minimal motion-compensated temporal baseline | n/a | algorithmic | Public |
| NAFNet Temporal (`nafnet-temporal-experimental`) | public learned temporal baseline | configuration-dependent | bring your own checkpoint | Public |

No trained weights are bundled. Export a compatible checkpoint with `uv run python training/export.py --checkpoint <ckpt> --output weights/spatial`.

---

## Inference engine

The deployment runtime is written directly in C++/CUDA rather than wrapping the PyTorch model in
TorchScript or ONNX; parity tests hold it to the Python reference.

```
   PyTorch checkpoint (.pth)
            |
        export.py
            |
   model.json + FP16/FP32 weight files
            |
   C++/CUDA runtime (cuDNN, cuBLAS, custom kernels)
            |
   denoised image, EXR, or video
```

- Standalone binary (`denoise`) plus a library, with no Python at runtime. CMake, GoogleTest,
  GitHub Actions CI.
- FP16 CUDA kernels, cuDNN/cuBLAS operators, memory-mapped weight files with lazy H2D upload, RAII
  CUDA tensors, and deformable convolution via im2col + cuBLAS GEMM.
- I/O: PNG/JPEG (stb), EXR/HDR (tinyexr), and video through an `ffmpeg` subprocess. The engine does
  not link against `libav*`.

---

## What's included

This repository covers the data pipeline, reproducible baselines, evaluation, model export, and the
C++/CUDA runtime. The production dataset and the STeR architecture, training recipe, calibration,
and weights stay private.

- **Standalone C++/CUDA inference engine.** Runs exported weights with no Python, ONNX, or LibTorch
  at runtime. GPU execution uses cuDNN, cuBLAS, and the CUDA runtime.
- **Motion compensation.** Hierarchical block matching (coarse-to-fine pyramid, sub-pixel parabolic
  refinement), with DIS or RAFT optical flow as optional public backends.
- **Public baselines.** NAFNet spatial and temporal models plus a motion-compensated baseline; see
  [Models & weights](#models--weights).
- **Python/C++ parity harness.** The parity test rejects a maximum per-pixel difference above 0.005.

---

## Quick start

**Build from source:**

```bash
git clone https://github.com/ginzburg-dev/ginzburg-neural-denoiser && cd ginzburg-neural-denoiser
make install                 # uv-managed Python env
make build CUDA_ARCH=89      # C++/CUDA engine, RTX 40 series
make fixtures && make test   # parity fixtures + full suite (Python, C++, parity)
```

**Export a checkpoint and denoise a frame:**

```bash
uv run python training/export.py --checkpoint /path/to/model.pth --output weights/spatial
./build/denoise --model weights/spatial/model.json --input frame.exr --output frame_denoised.exr
```

`make help` lists every command. `STeR` itself isn't distributed; the commands above run the
public baselines and the engine.

---

## Reproducible smoke benchmark

The only measured result currently committed to the repository is a CPU sanity check, not a
STeR evaluation. It takes 60 copies of `docs/img/noise-to-clean-comparison.jpg` (1000×500) and
adds independent additive white Gaussian noise (AWGN) with σ=25/255. Here Temporal MAE is simply the
mean absolute difference between adjacent frames; lower is better.

| Method | PSNR | SSIM | Temporal MAE | End-to-end ms/frame |
|---|---:|---:|---:|---:|
| Noisy input | 20.445 | 0.5595 | 0.106345 | n/a |
| ffmpeg `hqdn3d` | 20.523 | 0.5620 | 0.104407 | 16.16 |

Measured on macOS 12.4, Apple Silicon, ffmpeg 8.1.2. Timing is the median of five runs after
warm-up and includes PNG decode, filtering, and PNG encode, so it is not comparable to GPU
kernel-only timing. Re-run it with:

```bash
uv run --no-project --with numpy --with Pillow python training/smoke_benchmark.py
```

---

## Limitations

- **Fast non-rigid motion** such as cloth or splashing water remains difficult. Occlusion and poor alignment reduce the useful temporal information.
- **Noise models:** built for Monte Carlo render noise and calibrated camera noise. Heavily compressed footage (block/mosquito artifacts) is out of distribution.
- **Latency vs. throughput:** measure on the target GPU and resolution; no GPU timing is published yet.

---

## Documentation

| Doc | What it covers |
|---|---|
| [docs/build.md](docs/build.md) | Prerequisites and build options (CUDA, cuDNN, flags) |
| [docs/usage.md](docs/usage.md) | CLI, training, and Python inference/evaluation |
| [docs/benchmarks.md](docs/benchmarks.md) | Metrics and evaluation methodology (incl. temporal stability) |
| [docs/mc_noise_workflow.md](docs/mc_noise_workflow.md) | Monte Carlo noise: Nuke/Python preset workflow |
| [docs/architecture.md](docs/architecture.md) | Engine internals, layers, and model architecture |
| [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) | Bundled dependencies and their licenses |

---

## Citation

```bibtex
@software{ginzburg_neural_denoiser,
  author = {Ginzburg, Dmitri},
  title  = {STeR: Stable Temporal Restoration for Monte Carlo Rendered Sequences},
  year   = {2026},
  url    = {https://github.com/ginzburg-dev/ginzburg-neural-denoiser}
}
```

If you use the NAFNet building blocks, please also cite Chen et al., *Simple Baselines for Image Restoration*, ECCV 2022.

## Acknowledgements

The public baselines draw on **NAFNet** (Chen et al., ECCV 2022), **Deformable ConvNets v2**
(Zhu et al., CVPR 2019), **RAFT** (Teed & Deng, ECCV 2020), and **DIS optical flow** (Kroeger et al.,
ECCV 2016). The C++ dependencies are listed in
[THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md).

## License

---

**Dmitri Ginzburg** · [LinkedIn](https://www.linkedin.com/in/ginzburg-cg)

ginzburg-neural-denoiser

File Explorer

# Use via CDN

# Project Badges

Was this content helpful?

Recent Feedback

Download README