bergamot-ncnn-vulkan - GitLite

KO | EN

gitlite — search

#hacktoberfest #shell #containers #security #mysql #proxy #terminal #cpp #awesome #redis

bergamot-ncnn-vulkan

(★ 9)

mobile-friendly mechine translator

File Explorer

ZIP Download

- ARCHITECTURE.md
- BUILDING.md
- MODEL_FORMAT.md
- README.md
- TRANSLATION_EXAMPLES.md
- USAGE.md
- __init__.py
- batch_export.py
- check_status.py
- compute_missing.py
- converter.py
- download_missing.py
- download_models.py
- export.py
- export_tokenizer.py
- int8_loader.py
- model_loader.py
- ncnn_format.py
- README.md
- validate_all.py
- validate_strict.py
- README.md
- bergamot_translator.cpp
- bergamot_translator.h
- CMakeLists.txt
- deps_ncnn.cmake
- main.cpp
- ncnn
- ssplit-cpp
- test_tokenizer.cpp
- tokenizer.cpp
- tokenizer.h
.gitignore
.gitmodules
LICENSE
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">

Local Marian/Bergamot translation with a C++17 runtime, ncnn, and optional
Vulkan execution.

![C++17](https://img.shields.io/badge/C%2B%2B-17-00599C?style=flat-square&logo=cplusplus)
![ncnn](https://img.shields.io/badge/runtime-ncnn-5B8FF9?style=flat-square)
![Vulkan](https://img.shields.io/badge/backend-CPU%20%7C%20Vulkan-AC162C?style=flat-square&logo=vulkan)
![Models](https://img.shields.io/badge/models-fp32%20%7C%20int8-2EA44F?style=flat-square)

[Build](docs/BUILDING.md) | [Run](docs/USAGE.md) | [Models](https://ai.atomgit.com/fdph/bergamot-ncnn-model) | [Examples](docs/TRANSLATION_EXAMPLES.md) | [Export models](export/README.md) | [Architecture](docs/ARCHITECTURE.md)

</div>

The executable reads exported ncnn translation models and supports four input
modes: raw text, tokenized sentences, pre-wrapped segments, and a single token
sequence. Raw-text mode includes ssplit-cpp sentence splitting and a small
runtime tokenizer for the exported text vocabulary format.

> [!IMPORTANT]
> The runtime tokenizer is not a complete SentencePiece implementation. It does
> not load the original SentencePiece normalization rules. See
> [Tokenizer compatibility](docs/USAGE.md#tokenizer-compatibility) before using
> raw-text mode for compatibility-sensitive applications.

## Quick start

### 1. Initialize dependencies

```powershell
git submodule update --init --recursive
```

### 2. Build

```powershell
cmake -S src -B build-vulkan -DCMAKE_BUILD_TYPE=Release -DBERGAMOT_ENABLE_VULKAN=ON
cmake --build build-vulkan --config Release --target bergamot-ncnn-vulkan
```

Use `-DBERGAMOT_ENABLE_VULKAN=OFF` and another build directory for a CPU-only
build. Platform and ncnn selection details are in [Building](docs/BUILDING.md).

### 3. Provide an exported model

Model artifacts are ignored by Git. The recommended runtime layout is:

```text
models/
|-- fp32/
|   `-- en-de/
`-- int8/
    `-- en-de/
```

Download ready-to-run FP32 and INT8 models from the
[Bergamot ncnn model repository on AtomGit AI](https://ai.atomgit.com/fdph/bergamot-ncnn-model).
Place the desired precision/pair directory at `models/<precision>/<pair>` while
preserving its filenames. To build models from the upstream Mozilla artifacts
instead, use the included [model exporter](export/README.md).

### 4. Translate UTF-8 text

```powershell
build-vulkan\Release\bergamot-ncnn-vulkan.exe `
  -m models\fp32\en-de `
  -i input.txt `
  -o output.txt `
  --input-mode text `
  --vulkan
```

For Ninja and Make generators, the executable is normally
`build-vulkan/bergamot-ncnn-vulkan`.

The selected backend is written to standard error:

```text
backend=vulkan:<device name>
```

This line reports device selection only. It does not claim that every graph
layer ran on Vulkan; ncnn may execute unsupported layers on CPU.

## Vulkan scope

For `text`, `sentences`, and `segments`, the Vulkan path keeps embeddings,
positional encodings, encoder output, decoder state, K/V cache, logit projection,
and argmax in VkMat storage when the involved ncnn layers support Vulkan. One
token ID is downloaded per decode step for EOS control.

The `tokens` input mode uses the generic Mat-based encode/decode functions and
does not use this custom VkMat chain. int8 MHA is not supported by the current
ncnn Vulkan layer, so int8 models incur Vulkan/CPU transfers around those layers.

## Documentation

| Document | Source-backed scope |
| --- | --- |
| [Documentation index](docs/README.md) | Navigation and conventions. |
| [Building](docs/BUILDING.md) | CMake options, submodules, CPU, and Vulkan builds. |
| [Runtime usage](docs/USAGE.md) | Input formats, option effects, discovery rules, and fallback scope. |
| [Translation examples](docs/TRANSLATION_EXAMPLES.md) | Actual FP32 and INT8 output for every available language direction. |
| [Ready-to-run models](https://ai.atomgit.com/fdph/bergamot-ncnn-model) | Download the exported FP32 and INT8 model collection. |
| [Architecture](docs/ARCHITECTURE.md) | Actual CPU/Vulkan paths and graph boundaries. |
| [Model format](docs/MODEL_FORMAT.md) | Exported filenames, blob names, shapes, and optional files. |
| [Model exporter](export/README.md) | Downloader and exporter behavior, including overwrite rules. |

## Repository layout

```text
.
|-- docs/               Documentation
|-- export/             Python conversion tools
|-- models/             Local artifacts; only README.md is tracked
`-- src/
    |-- ncnn/           Git submodule
    |-- ssplit-cpp/     Git submodule
    |-- main.cpp        CLI and text-mode orchestration
    |-- tokenizer.cpp   Custom text-vocabulary tokenizer
    `-- bergamot_translator.cpp
```

## Known limitations

- Greedy decoding is the only search strategy.
- Graph execution is sequential per segment; there is no graph-level batch.
- `--mini-batch-words` is parsed, but currently does not change the effective
  translation execution because `translate_segments()` creates its own default
  grouping and still executes every segment sequentially.
- For non-empty raw-text input, the current implementation prints first-sentence
  token diagnostics to standard error.
- The custom tokenizer only approximates the preprocessing needed by the
  original SentencePiece models.
- int8 MHA falls back to CPU when a Vulkan backend is selected.