gsplat-mlx

KO | EN

gitlite — search

#editor #mcp #rust #gemini #golang #csharp #claude-code #containers #command-line #kubernetes

(★ 9)

Apple Silicon 3D Gaussian Splatting core with MLX primitives, Metal kernels, and gsplat CUDA parity tests.

파일 탐색기

ZIP 다운로드

- - binding.cpp
- - dummy.h
  - gsplat_intersect.h
  - gsplat_projection.h
  - gsplat_quat_scale_to_covar.h
  - gsplat_rasterize.h
  - gsplat_spherical_harmonics.h
  - helper.h
- - gsplat_intersect.metal
  - gsplat_projection.metal
  - gsplat_quat_scale_to_covar.metal
  - gsplat_rasterize.metal
  - gsplat_spherical_harmonics.metal
- - test.cpp
- dummy.cpp
- gsplat_intersect.cpp
- gsplat_projection.cpp
- gsplat_quat_scale_to_covar.cpp
- gsplat_rasterize.cpp
- gsplat_spherical_harmonics.cpp
- helper.cpp
- gsplat_3dgs_forward_sh_path.md
- gsplat_backward_api_and_fixture_schema.md
- gsplat_backward_source_map.md
- gsplat_cuda_source_map.md
- mlx_nanobind_version.md
- mlx_training_gradient_proxy.md
- mlx_vs_mlx2_scanner_training.md
- projection_backward_gpu_path.md
- - __init__.py
- - b075x65r3x_dataset.py
  - dodecahedron_dataset.py
  - image_fitting_dataset.py
  - training_dataset.py
- - export_forward_3dgs_chain.py
  - export_intersect_tile_forward.py
  - export_projection_ewa_3dgs_fused_backward.py
  - export_projection_ewa_3dgs_fused_edge_cases.py
  - export_projection_ewa_3dgs_fused_forward.py
  - export_quat_scale_to_covar_preci_backward.py
  - export_quat_scale_to_covar_preci_backward_edge_cases.py
  - export_quat_scale_to_covar_preci_edge_cases.py
  - export_quat_scale_to_covar_preci_forward.py
  - export_rasterize_to_pixels_3dgs_backward.py
  - export_rasterize_to_pixels_3dgs_forward.py
  - export_rasterize_to_pixels_3dgs_masks.py
  - export_spherical_harmonics_backward.py
  - export_spherical_harmonics_backward_degrees.py
  - export_spherical_harmonics_degree4_masks.py
  - export_spherical_harmonics_forward.py
  - export_training_reference_config.py
  - export_utils.py
  - README.md
- - absgrad_refine_debug.py
  - autograd_vjp_smoke.py
  - colmap_360_dataset.py
  - compare_exported_npz.py
  - compare_training_summary.py
  - dataset_geometry_diagnostics.py
  - dummy.py
  - forward_3dgs_chain.py
  - image_loss_debug.py
  - intersect_tile_forward.py
  - optimizer_lr_schedule_debug.py
  - parity_forward_3dgs_chain.py
  - parity_intersect_tile_forward.py
  - parity_projection_ewa_3dgs_fused_forward.py
  - parity_quat_scale_to_covar_preci_forward.py
  - parity_rasterize_to_pixels_3dgs_forward.py
  - parity_spherical_harmonics_forward.py
  - parity_utils.py
  - projection_ewa_3dgs_fused_forward.py
  - projection_vjp_guardrails.py
  - quat_scale_to_covar_preci_forward.py
  - rasterize_to_pixels_3dgs_forward.py
  - refine_state_reset_debug.py
  - refine_strategy_branches_debug.py
  - render_random_3dgs_png.py
  - scanner_dataloader_debug.py
  - scanner_dataset_utils.py
  - scanner_initialization_debug.py
  - scanner_points_training_utils.py
  - spherical_harmonics_forward.py
  - spz_convention_debug.py
  - train_360_points_multiview_3dgs_mlx.py
  - train_dodecahedron_points_multiview_3dgs_mlx.py
  - train_image_fitting_3dgs_mlx.py
  - train_scanapp_depth_multiview_3dgs_mlx.py
  - train_scanner_points_multiview_3dgs_mlx2.py
  - train_sofa_points_multiview_3dgs_mlx.py
  - training_dense_3dgs_loop_smoke.py
  - training_projection_viewspace_proxy_smoke.py
  - training_viewspace_proxy_smoke.py
.gitignore
.gitmodules
CMakeLists.txt
LICENSE
Makefile
pyproject.toml
README.md
setup.py
Task.md
Task_360.md
Task_Fixed_Points.md
Task_Metal.md

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## gsplat_core

`gsplat_core` is a CMake-based mlx custom extension project for Apple Silicon.
It implements 3D Gaussian Splatting operators with mlx c++ primitives, custom
metal kernels, nanobind python bindings, and an Xcode-friendly build/test
workflow.

The project is also useful as a template for writing mlx c++ custom extensions.

Lidar scan App source code is availiable in [ScanApp](https://github.com/a1091150/ScanApp/tree/main)
## Features

- mlx c++ custom extension with `mlx::core::Primitive`
- Support python mlx through nanobind.
- XCode project build and test for metal code.
- `Cmake` and `Makefile` for easy training setup.
- mlx python training scripts for image fitting, scanner captures, COLMAP / Mip-NeRF 360-style scenes, and SPZ export.

## Environment Setup, Build, Tests and Validation

### Environment Setup
1. Install Xcode from the App Store.
2. In Xcode, open Settings -> Components -> Other Components and make sure the metal Toolchain is installed.
3. Install CMake.
4. Create Conda environment `gsplat_core`. By default, `Makefile` assumes `gsplat_core` conda environment.
5. Install required python packages
6. Install `gsplat_core`
7. Install `spz`

```bash
# Install metal tool chain via terminal
xcodebuild -downloadComponent metalToolchain
# Install Cmake
brew instal cmake
# Create and activeate environment
conda create -n gsplat_core python=3.11
conda activate gsplat_core
# Install required package
pip install mlx==0.30.0 nanobind==2.4.0 cmake opencv-python plyfile pillow scipy pycolmap tyro
# Install gsplat_core
pip install . --no-build-isolation

# Install spz from submodule, spz does not have pip install.
git submodule update --init --recursive
cd submodules/spz
pip install .
```

Notes:
- mlx releases may require matching nanobind versions. mlx 0.30.0 uses nanobind 2.4.0.
- mlx >= 0.31.0 changes some code like `get_encoder`. The code would not compile on lastest mlx version.
- Some `spz` versions have export issues. This project uses `spz` on commit id `ef094fd1a96ca6ff414d72d7904ee4f4f6d97be9`.

### Build and Test
- The project uses `compare_exported_npz.py` to test with npz files from `gsplat`. See [scripts/export_ref/README.md](scripts/export_ref/README.md) for CUDA-side fixture export instructions.
- Only XCode can provide debug information and gpu trace on metal shading code. `make xcode-build` create a `xcode_build` directory and `test.cpp` to test.

`Makefile` provide some useful commands to use:
```bash
# Check the local environment
make env-check

# Build the python extension with Xcode.
make xcode-build

# Install the package
make pip-install

# For editable development install
make pip-develop

# Run the c++ / metal smoke test:
make codex-xcode-test

# Run a dense 3DGS training smoke test:
make codex-dense-training-smoke

# Render a random 3DGS PNG for manual inspection:
make codex-random-png

# Compare exported CUDA reference fixtures against the mlx / metal
python scripts/test/compare_exported_npz.py
```

## Datasets
The project supports 6 sources to train:
- Mip-NeRF 360
- B075X65R3X(Brown Sofa)
- 3D Scanner App Captures
- Open-sourced scanner app (See https://github.com/a1091150/ScanApp/tree/main)
- Dodecahedron (code-generated dataset)
- Image fittings (code-generated dataset)

Training scripts expect datasets under `datasets/` by default.

### Mip-NeRF 360 / COLMAP

The 360 training target uses `datasets/360_v2/<scene>`. Use gsplat's
[`download_dataset.py`](https://github.com/nerfstudio-project/gsplat/blob/main/examples/datasets/download_dataset.py) to download datasets. it download 360_v2.zip by default.

### B075X65R3X(Brown Sofa)

The `B075X65R3X` dataset is available from
[hbb1/torch-splatting](https://github.com/hbb1/torch-splatting). Download `B075X65R3X.zip`, unzip it, and place the extracted folder at `datasets/B075X65R3X`

### 3D Scanner App Captures

Scanner training currently targets datasets exported from the
[3D Scanner iPhone App](https://3dscannerapp.com/).

1. Use an iPhone with LiDAR support.
2. Install the 3D Scanner App.
3. Capture in landscape orientation, with the front camera on the left side.
4. Choose the Point Cloud option for scanning.
5. Export using the "All Data" option and transfer it to your Mac with AirDrop.

### Scan App
An app is 100% generated by Codex.
[ScanApp](https://github.com/a1091150/ScanApp/tree/main)

1. Use an iPhone with LiDAR support.
2. Download and build the code.
3. Capture in landscape orientation, with the front camera on the left side.
4. In preview, click share button to export directory transfer it to your Mac with AirDrop.

## Training

gsplat use 30000 training steps but it is taking too long on Macbook pro M5 pro 48GB. This projects use 4000 training steps, which takes 15 minutes.

The output directory is under `outputs/`
```bash
# Train Mip-NeRF 360 / COLMAP. The default scene is `garden`
make codex-360-points-train-spz

# Train using other dataset
make codex-360-points-train-spz \
  COLMAP_360_ROOT=datasets/360_v2 \
  COLMAP_360_SCENE=bonsai

# Train with 3D scanner app's dataset
make codex-scanner-points-train-spz2 \
  SCANNER_DATASET=/path/to/3d-scanner-app-export

# Train scan app's dataset.
make codex-scanapp-depth-train-spz

# Train B075X65R3X(Brown Sofa)
make codex-sofa-train-spz

# An example to use a single image to train.
# https://docs.gsplat.studio/main/examples/image.html
make codex-image-fitting-train

# Train using dodecahedron.
make codex-dodecahedron-train-spz
```

## Some Useful Hints and Notes

### Hints and Experience
I started to build 3dgs on mlx extension a year ago(2025/05). At the time:
- No mlx extension or opensource repo to start with, only `axpby` on mlx source repo.
- 3dgs is mainly on cuda and pytoch.
- ChatGPT (3 ~ 5.4) was not smart enough to generate mlx python and mlx c++ code.
- Only XCode can debug and profile metal code.
- pytorch has `retain_grad()` to get means2d's gradient. mlx does not have one.
- Learning rate, optimizer setting and loss function almost affect gaussian's quality.

The idea from `FastGS` to solve `retain_grad()` by using extra trainable argument `viewspace_points` to get means2d's gradient. In backward path return the means2d's gradient.

### Codex
This project's code 100% generated by Codex 5.5 Medium with fast mode. It took me a long time to find a solution, build the project, debug metal code, write mlx primitive, translate pytorch to mlx. Nowadays just tell Codex to do everything, save me months of time.

It is easy to write forward path of 3dgs, but it is hard to write and debug backward path. Codex debugs backward path by generate some npz files from gsplat cude then compare values in mlx.

Codex uses harness engineering, which is build tasks -> implement tasks -> validate and test. `Task.md` demonstrates some plan on cuda code to metal Code. I just tell Codex to read gsplat source code to map cuda to metal, split forward path to serveral mlx primitives.

### Current guardrails and known gaps

- Packed sparse `gsplat` paths are not currently implemented.
- Segmented sort paths are not currently implemented. This project use `mlx::core::argsort` and `mlx::core::take` to gather gaussians.
- Projection backward currently supports the pinhole path; ortho and fisheye
  backward VJPs remain guarded.
- Intersect tile currently uses the dense radius AABB path; CUDA AccuTile /
  SNUGBOX ellipse intersection with `conics` and `opacities` is not ported.
- 2DGS, LiDAR, rolling shutter, and world-ray rasterization paths are out of
  scope for now.
- Full high-level `gsplat` python API compatibility is not the current goal.

CUDA parity fixtures and training smoke tests are included as validation tools, but they are secondary to the repo's role as a practical mlx c++ / metal custom extension template.

gsplat-mlx

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README