LlamaKit - GitLite

KO | EN

gitlite — search

#svg #containers #developer-tools #json #ai #http #automation #tailwindcss #claude-code #golang

LlamaKit

(★ 17)

Delphi wrappers and compiled Windows DLLs for local LLM inference and embeddings via llama.cpp

File Explorer

ZIP Download

- ggml-base.dll
- ggml-cpu-alderlake.dll
- ggml-cpu-cannonlake.dll
- ggml-cpu-cascadelake.dll
- ggml-cpu-cooperlake.dll
- ggml-cpu-haswell.dll
- ggml-cpu-icelake.dll
- ggml-cpu-ivybridge.dll
- ggml-cpu-piledriver.dll
- ggml-cpu-sandybridge.dll
- ggml-cpu-sapphirerapids.dll
- ggml-cpu-skylakex.dll
- ggml-cpu-sse42.dll
- ggml-cpu-x64.dll
- ggml-cpu-zen4.dll
- ggml.dll
- llama.dll
- makerai.embedder.dll
- makerai.gen.dll
- - uMakerAi.Embeddings.Llamacpp.pas
- uMakerAi.Config.pas
- uMakerAi.Embedder.Import.pas
- uMakerAi.ErrorCodes.pas
- uMakerAi.Gen.Import.pas
- - embeddings_example.dpr
  - embeddings_example.dproj
- - generation_example.dpr
  - generation_example.dproj
- .gitignore
- llamakit.ini
- README.md
.gitignore
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

Command Glossary

Commands referenced in this DOCs, explained below.

🔍

`Write`

View Details ▼

Write data to memory.

write memory

Write current configuration to memory:

write {{username}} {{terminal_id}}

Send a message to a given user on a given terminal ID:

write {{username}} {{terminal_id}}

Send a message to a given user on a given terminal ID:

🔍

`ollama`

View Details ▼

A large language model runner.
For a list of available models, see <https://ollama.com/library>.

ollama serve

Start the daemon required to run other commands:

ollama run {{model}}

Run a model and chat with it (will automatically download the model if it's not downloaded):

ollama run {{model}} --think=false "{{prompt}}"

Run a model with a single prompt and thinking turned off:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Run local AI models in your Delphi applications — no server, no Python, no internet connection required.

LlamaKit is a set of compiled Windows DLLs and Delphi wrapper units that let you load **GGUF language models** directly into any Delphi application and use them for:

- **Text embeddings** — convert text to vectors for semantic search, RAG pipelines, similarity comparison
- **Text generation** — stream responses from any instruction-tuned or chat LLM locally

Built on top of [llama.cpp](https://github.com/ggml-org/llama.cpp), one of the fastest open-source LLM inference engines available.

> **Ready-to-run examples** are in the [`examples/`](examples/) folder — compile and run with a single model path argument.

---

## Why LlamaKit

Most Delphi AI integrations rely on HTTP calls to external APIs (OpenAI, Ollama, Cohere, etc.). That works well but has limits: it requires internet access, incurs per-token costs, and sends your data to third-party servers.

LlamaKit runs the model **inside your process**. You call a function, you get a result. No network, no latency from HTTP round-trips, no API keys, and full privacy — the data never leaves the machine.

```
Your Delphi app
  └── makerai.embedder.dll  ←→  llama.dll  ←→  GGUF model file
  └── makerai.gen.dll       ←→  llama.dll  ←→  GGUF model file
```

---

## Requirements

- Windows x64
- Delphi XE8 or newer (uses generics and anonymous methods)
- A GGUF model file (see [Models](#models) below)
- For GPU acceleration: an NVIDIA GPU with CUDA 12 support

---

## Getting started

### 1. Get the DLLs

**CPU-only (no GPU needed)**

Clone this repository and copy everything from the [`bin/`](bin/) folder alongside your application's `.exe`.

**GPU-accelerated (NVIDIA CUDA 12)**

Download [`LlamaKit-cuda12-win64.zip`](../../releases/latest) from the Releases page and extract it alongside your `.exe`. This includes the same CPU files plus the CUDA backend.

> The CUDA runtime (`cublas64_12.dll`, `cublasLt64_12.dll`, `cudart64_12.dll`) must be present on the target machine. They are redistributable from NVIDIA and included in the zip.

### 2. Get a model

Download any GGUF model from [Hugging Face](https://huggingface.co/models?library=gguf).

For embeddings, a good starting point is [mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1-gguf) — accurate, 1024-dimensional vectors, fast even on CPU.

For generation, [Llama 3.2 3B Instruct](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) is a solid general-purpose model that fits in 4 GB of VRAM at Q4 quantization.

> **Ollama users:** if you already have models pulled via `ollama pull`, you can point directly to the blob files at  
> `%LOCALAPPDATA%\Ollama\models\blobs\sha256-<hash>` — no `.gguf` extension needed.

### 3. Add the Delphi wrappers to your project

Add the [`delphi/`](delphi/) folder to your project's library search path. Three units cover everything:

| Unit | Contents |
|------|----------|
| `uMakerAi.ErrorCodes` | `TLlamaErrorCode` enum — returned by all operations |
| `uMakerAi.Embedder.Import` | `TEmbedder` — load a model and compute embeddings |
| `uMakerAi.Gen.Import` | `TLlamaGenerator` — load a model and generate text |

---

## Embeddings

### Basic usage

```pascal
uses uMakerAi.Embedder.Import;

var
  Emb: TEmbedder;
  VecA, VecB: TArray<Single>;
  Similarity: Single;
begin
  Emb := TEmbedder.Create; // loads makerai.embedder.dll from the exe folder
  try
    Emb.Load('C:\models\mxbai-embed-large-v1-q4_k_m.gguf', 99);
    // 99 = offload all layers to GPU; use 0 for CPU-only

VecA := Emb.Embed('The cat sat on the mat');
    VecB := Emb.Embed('A feline resting on a rug');

Similarity := TEmbedder.CosineSimilarity(VecA, VecB);
    Writeln(Format('Similarity: %.4f', [Similarity]));
    // → typically 0.85+ for semantically similar sentences
  finally
    Emb.Free;
  end;
end;
```

### Batch embedding

```pascal
var
  Texts: TArray<string>;
  Vecs:  TArray<TArray<Single>>;
begin
  Texts := ['first sentence', 'second sentence', 'third sentence'];
  Vecs  := Emb.EmbedBatch(Texts);
  // Vecs[0], Vecs[1], Vecs[2] — one vector per input
end;
```

### Validate a model before loading

```pascal
var Code: TLlamaErrorCode;
begin
  Code := Emb.ValidateModelFile('C:\models\my-model.gguf');
  if Code = llamaErrNone then
    Emb.Load('C:\models\my-model.gguf')
  else
    Writeln('Invalid model: ', Ord(Code));
end;
```

### Recommended embedding models

| Model | Dimensions | Best for |
|-------|:----------:|----------|
| [mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1-gguf) | 1024 | General-purpose, top accuracy |
| [snowflake-arctic-embed-l](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-gguf) | 1024 | Retrieval / RAG |
| [nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF) | 768 | Long documents |
| [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | 384 | Lightweight, fast |

---

## Text generation

### Simple generation

```pascal
uses uMakerAi.Gen.Import;

var
  Gen:    TLlamaGenerator;
  Output: string;
begin
  Gen := TLlamaGenerator.Create; // loads makerai.gen.dll
  try
    Gen.Load('C:\models\llama3.2-3b-instruct-q4_k_m.gguf', 99);
    // parameters: model path, GPU layers (99=all), context size (0=default 4096)

Output := Gen.Generate('Explain quantum entanglement in one paragraph:', 256);
    Writeln(Output);
  finally
    Gen.Free;
  end;
end;
```

### Streaming (token by token)

```pascal
Gen.GenerateStream(
  'Write a short poem about the Andes mountains:',
  128,   // max tokens
  0.8,   // temperature
  0.95,  // top-p
  procedure(const Piece: string; IsLast: Boolean)
  begin
    if not IsLast then
      Write(Piece)     // print each token as it arrives
    else
      Writeln;         // newline at the end
  end);
```

### Advanced samplers

The generator supports multiple sampling strategies via `GenerateStream` overload:

```pascal
uses uMakerAi.Gen.Import;

// Greedy decoding (deterministic, best for benchmarks)
Gen.GenerateStream(Prompt, 128, 0, 0, smGreedy, 0, 0, OnToken);

// Mirostat v2 (self-regulating quality, good for creative text)
Gen.GenerateStream(Prompt, 256, 0, 0, smMirostatV2,
  5.0,  // tau  — target entropy (higher = more creative)
  0.1,  // eta  — learning rate
  OnToken);

// Min-P sampling (better coherence at high temperatures)
Gen.GenerateStream(Prompt, 256, 0.9, 0, smMinP,
  0.05, // min_p
  1,    // min_keep
  OnToken);
```

| Mode | Constant | Description |
|------|----------|-------------|
| Greedy | `smGreedy` | Always picks the most probable token. Deterministic. |
| Temp + Top-P | `smTempTopP` | Standard sampling. Default. |
| Min-P | `smMinP` | Filters tokens below a minimum probability. |
| Mirostat v2 | `smMirostatV2` | Adaptive sampler targeting a perplexity budget. |
| Typical-P | `smTypical` | Samples from the most "typical" part of the distribution. |

### Multi-turn conversations

Call `Reset` between turns to clear the KV cache, then prepend the conversation history yourself:

```pascal
var History: string;
begin
  History := '';
  Gen.Load('C:\models\llama3.2-3b-instruct-q4_k_m.gguf');

// Turn 1
  Gen.Reset;
  History := History + '<|user|>What is the capital of France?<|end|><|assistant|>';
  History := History + Gen.Generate(History, 64);

// Turn 2
  Gen.Reset;
  History := History + '<|end|><|user|>And what is its population?<|end|><|assistant|>';
  History := History + Gen.Generate(History, 64);
end;
```

### Recommended generation models

Any GGUF instruction-tuned or chat model works. Popular options:

| Model | Size | Notes |
|-------|------|-------|
| Llama 3.2 3B Instruct | 3B | Fast, good quality, fits in 4 GB VRAM at Q4 |
| Gemma 3 1B / 4B | 1B–4B | Excellent instruction following |
| Phi-3 Mini | 3.8B | Strong reasoning for its size |
| DeepSeek-R1 1.5B | 1.5B | Built-in chain-of-thought (`<think>` blocks) |
| Mistral 7B Instruct | 7B | High quality, requires more VRAM |

---

## MakerAI integration

If you use the [MakerAI Suite](https://github.com/gustavoeenriquez/MakerAI), LlamaKit ships a drop-in embedding component that is fully API-compatible with the existing `TAiEmbeddings` subclasses (OpenAI, Ollama, Cohere, etc.).

Add [`delphi/makerAI/uMakerAi.Embeddings.Llamacpp.pas`](delphi/makerAI/uMakerAi.Embeddings.Llamacpp.pas) to your MakerAI project (no extra search paths needed — it depends only on the other files in `delphi/`).

### Using the component in code

```pascal
uses uMakerAi.Embeddings.Llamacpp;

var
  Emb: TAiLlamacppEmbeddings;
  Vec: TAiEmbeddingData;  // TArray<Double>
begin
  Emb := TAiLlamacppEmbeddings.Create(nil);
  try
    Emb.ModelPath  := 'C:\models\mxbai-embed-large-v1-q4_k_m.gguf';
    Emb.NGPULayers := 99;    // 0 for CPU-only
    Vec := Emb.CreateEmbedding('The quick brown fox jumps over the lazy dog', '');
    Writeln(Format('%d-dimensional vector, first value: %.6f', [Length(Vec), Vec[0]]));
  finally
    Emb.Free;
  end;
end;
```

### Using the component at design time

Install `uMakerAi.Embeddings.Llamacpp.pas` into a Delphi package — the `Register` procedure adds `TAiLlamacppEmbeddings` to the **MakerAI** tab in the component palette.

Drop it on a form and set the `ModelPath` property in the Object Inspector. It behaves identically to `TAiOllamaEmbeddings` or `TAiOpenAIEmbeddings` — the same `CreateEmbedding` call, the same `OnGetEmbedding` event, the same factory registration.

### Swapping between local and remote at runtime

```pascal
// Use local model during development / testing
Embedder := TAiLlamacppEmbeddings.Create(nil);
TAiLlamacppEmbeddings(Embedder).ModelPath := 'C:\models\mxbai-embed-large-v1-q4.gguf';

// Switch to OpenAI in production — zero code change at the call site
// Embedder := TAiEmbeddingFactory.Instance.CreateDriver('OpenAI', nil);

Vec := Embedder.CreateEmbedding(MyText, '');
```

---

## DLL reference

### makerai.embedder.dll

| Export | Description |
|--------|-------------|
| `EmbedderCreate` | Create a new embedder instance |
| `EmbedderLoad` | Load a GGUF model (default GPU layers) |
| `EmbedderLoadEx` | Load a GGUF model (explicit GPU layers) |
| `EmbedderGetDimensions` | Returns the vector dimension of the loaded model |
| `EmbedderEmbed` | Compute a single embedding vector |
| `EmbedderEmbedBatch` | Compute multiple embedding vectors |
| `EmbedderIsGPUEnabled` | True if ggml-cuda.dll is loaded |
| `EmbedderGetCUDAInfo` | Returns a GPU/CPU status string |
| `EmbedderValidateModelFile` | Check a model file before loading |
| `EmbedderGetLastError` | Get the last error message |
| `EmbedderGetLastErrorCode` | Get the last error code |
| `EmbedderFree` | Free an embedder instance |

### makerai.gen.dll

| Export | Description |
|--------|-------------|
| `LlamaGenCreate` | Create a new generator instance |
| `LlamaGenLoad` | Load a GGUF model |
| `LlamaGenerate` | Generate text with Temperature + Top-P sampler |
| `LlamaGenerateEx` | Generate text with explicit sampler selection |
| `LlamaGenReset` | Clear the KV cache |
| `LlamaGenIsGPUEnabled` | True if GPU backend is active |
| `LlamaGenValidateModelFile` | Check a model file before loading |
| `LlamaGenGetLastError` | Get the last error message |
| `LlamaGenGetLastErrorCode` | Get the last error code |
| `LlamaGenFree` | Free a generator instance |

---

## Error codes

`TLlamaErrorCode` is defined in `uMakerAi.ErrorCodes.pas` and returned by all operations:

| Constant | Value | Meaning |
|----------|:-----:|---------|
| `llamaErrNone` | 0 | Success |
| `llamaErrUnknown` | 1 | Unclassified exception inside the DLL |
| `llamaErrFileNotFound` | 2 | Model file does not exist |
| `llamaErrIO` | 3 | File unreadable or permission denied |
| `llamaErrInvalidArg` | 4 | Bad argument (empty path, wrong dims, etc.) |
| `llamaErrModelLoad` | 5 | llama.cpp rejected the model file |
| `llamaErrContextCreate` | 6 | Could not create an inference context |
| `llamaErrTokenize` | 7 | Tokenization failed or prompt too long |
| `llamaErrDecode` | 8 | Inference decode returned a non-zero error |
| `llamaErrNoEmbeddings` | 9 | Model loaded but returned no embedding data |
| `llamaErrDllLoad` | 11 | `LoadLibrary` failed (DLL not found or missing dependency) |
| `llamaErrEntryPoint` | 12 | `GetProcAddress` failed (wrong DLL version) |

---

## Examples

See the [`examples/`](examples/) folder for two complete, ready-to-compile Delphi console applications:

| Example | What it demonstrates |
|---------|----------------------|
| [`01-embeddings/`](examples/01-embeddings/) | Model validation, loading, cosine similarity, batch embedding, GPU vs CPU |
| [`02-generation/`](examples/02-generation/) | Simple generation, streaming, Greedy / Temp+TopP / Mirostat v2 / Min-P samplers, KV cache reset, GPU vs CPU |

Copy [`examples/llamakit.ini`](examples/llamakit.ini) alongside the compiled `.exe` and fill in your model paths — or pass `--model <path>` on the command line.

---

## Repository layout

```
bin/                          CPU DLLs — copy these alongside your .exe
  makerai.embedder.dll
  makerai.gen.dll
  llama.dll
  ggml.dll
  ggml-base.dll
  ggml-cpu.dll
  ggml-cpu-alderlake.dll      ← CPU dispatch variants, auto-selected at runtime
  ggml-cpu-haswell.dll
  ggml-cpu-zen4.dll
  ...

delphi/                       Delphi wrapper units — add to library search path
  uMakerAi.ErrorCodes.pas     Shared error code enum
  uMakerAi.Embedder.Import.pas  TEmbedder wrapper
  uMakerAi.Gen.Import.pas     TLlamaGenerator wrapper
  uMakerAi.Config.pas         INI-based model path helper (optional)

makerAI/                    MakerAI component integration
    uMakerAi.Embeddings.Llamacpp.pas   TAiLlamacppEmbeddings
```

GPU build (CUDA 12) is available as a release asset — see [Releases](../../releases).

---

## License

LlamaKit is released under the **MIT License**.

This project uses [llama.cpp](https://github.com/ggml-org/llama.cpp) which is also MIT licensed.  
CUDA runtime DLLs are redistributable under the [NVIDIA CUDA Toolkit EULA](https://docs.nvidia.com/cuda/eula/index.html).

---

*Part of the [MakerAI Suite](https://github.com/gustavoeenriquez/) — AI components for Delphi.*