gated_linear_attention - GitLite

KO | EN

gitlite — search

#javascript #ai #llm #go #machine-learning #rust #linux #docker #nodejs #java #windows #kubernetes #security #openai

gated_linear_attention

(★ 107)

No description available.

File Explorer

Download Latest Version (.zip)

.gitmodules
gla.py
gla_model.py
LICENSE
README.md

# Installation Guide

1. Get the code

git clone https://github.com/berlino/gated_linear_attention

Downloads the entire project code from GitHub to your computer.

cd gated_linear_attention

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended

Prerequisites

Python 3 Python is required to use pip.

pip install -U git+https://github.com/sustcsonglin/flash-linear-attention

Installs the package published on PyPI directly — no need to clone the source.

✅ After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Gated Linear Attention Layer

Standalone module of Gated Linear Attention (GLA) from [Gated Linear Attention Transformers with
Hardware-Efficient Training](https://arxiv.org/pdf/2312.06635.pdf).

```
pip install -U git+https://github.com/sustcsonglin/flash-linear-attention
```

Warning: ```fused_chunk``` mode needs Triton2.2 + CUDA12 (See [issue](https://github.com/berlino/gated_linear_attention/issues/8 )). You can use [test](https://github.com/sustcsonglin/flash-linear-attention/blob/main/tests/test_fused_chunk.py) to quickly see if you can use ```fused_chunk``` mode. If cannot, please refer to [link](https://github.com/sustcsonglin/flash-linear-attention/blob/main/fla/layers/gla.py#L44C1-L45C1) and use ```chunk``` mode instead.

## Usage

Load the checkpoint from huggingface.

```python
from gla_model import GLAForCausalLM
model = GLAForCausalLM.from_pretrained("bailin28/gla-1B-100B")
vocab_size = model.config.vocab_size
bsz, seq_len = 32, 2048
x = torch.randint(high=vocab_size, size=(bsz, seq_len))
model_output = model(x)
loss = model_output.loss
logits = model_output.logits
```