recurrentgemma - GitLite

KO | EN

gitlite — search

#ai #android #machine-learning #linux #macos #java #awesome #kubernetes #awesome-list #vue #pytorch #security #ios #openai

recurrentgemma

(★ 683)

Open weights language model from Google DeepMind, based on Griffin.

파일 탐색기

최종 버전 다운로드 (.zip)

- - pytest.yml
- fine_tuning_tutorial_jax.ipynb
- gsm8k_eval.ipynb
- sampling_tutorial_jax.ipynb
- sampling_tutorial_pytorch.ipynb
- sampling_jax.py
- sampling_pytorch.py
- simple_run_jax.py
- simple_run_pytorch.py
- - __init__.py
  - array_typing.py
  - complex_lib.py
  - complex_lib_test.py
  - griffin.py
  - griffin_test.py
  - layers.py
  - layers_test.py
  - modules.py
  - modules_test.py
  - pallas.py
  - sampler.py
  - sampler_test.py
  - scan.py
  - utils.py
- - __init__.py
  - array_typing.py
  - griffin.py
  - griffin_test.py
  - layers.py
  - layers_test.py
  - modules.py
  - modules_test.py
  - sampler.py
  - sampler_test.py
  - test_utils.py
- __init__.py
- common.py
- conversion.py
- conversion_test.py
.gitignore
CONTRIBUTING.md
LICENSE
pyproject.toml
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/google-deepmind/recurrentgemma

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd recurrentgemma

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install .

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

jupyter notebook

브라우저에서 노트북(.ipynb) 파일들을 열람하고 실행할 수 있는 Jupyter 화면을 켭니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# RecurrentGemma

RecurrentGemma is a family of open-weights Language Models by [Google DeepMind](https://deepmind.google/), based on the novel [Griffin architecture](https://arxiv.org/abs/2402.19427). This architecture achieves fast inference when generating long sequences by replacing global attention with a mixture of local attention and linear recurrences.

This repository contains the model implementation and examples for sampling and fine-tuning. We recommend most users adopt the [Flax](https://github.com/google/flax) implementation, which is highly optimized. We also provide an un-optimized [PyTorch](https://github.com/pytorch/pytorch) implementation for reference.

### Learn more about RecurrentGemma

-   The [RecurrentGemma technical report](https://storage.googleapis.com/deepmind-media/gemma/recurrentgemma-report.pdf) gives specific details on the training and evaluation of RecurrentGemma.
-   The [Griffin paper](https://arxiv.org/abs/2402.19427) describes the underlying model architecture.

## Quick start

### Installation

#### Using Poetry
RecurrentGemma uses [Poetry](https://python-poetry.org/docs/) for dependency
management.

To install dependencies for the full project:
* Checkout the code.
* `poetry install -E full` to create a virtual environment with all dependencies.
* `poetry shell` to activate the created virtual environment.

If you only need to install a subset of dependencies use one of the alternative
library-specific commands below.

#### Using pip
If you want to use `pip` instead of Poetry, 
then create a virtual environment (run `python -m venv recurrentgemma-demo` and `. recurrentgemma-demo/bin/activate`) and:

* Checkout the code.
* `pip install .[full]`

#### Installing library-specific packages

##### JAX
To install dependencies only for the JAX pathway use:
`poetry install -E jax` or (`pip install .[jax]`).

##### PyTorch
To install dependencies only for the PyTorch pathway use:
`poetry install -E torch` (or `pip install .[torch]`).

##### Tests
To install dependencies required for running unit tests use:
`poetry install -E test` (or `pip install .[test]`)

### Downloading the models

The model checkpoints are available through Kaggle at
http://kaggle.com/models/google/recurrentgemma.
Select either the **Flax** or **PyTorch** model variations, click the ⤓ button
to download the model archive, then extract the contents to a local directory.

In both cases, the archive contains both the model weights and
the tokenizer.

### Running the unit tests

To run the tests, install the optional `[test]` dependencies (e.g. using `pip install .[test]`) from the root of the source tree, then:

```
pytest .
```

## Examples

To run the example sampling script, pass the paths to the weights directory and tokenizer:

```
python examples/sampling_jax.py \
  --path_checkpoint=/path/to/archive/contents/2b/ \
  --path_tokenizer=/path/to/archive/contents/tokenizer.model
```

### Colab notebook tutorials

-   [`colabs/sampling_tutorial_jax.ipynb`](https://colab.sandbox.google.com/github/google-deepmind/recurrentgemma/blob/main/colabs/sampling_tutorial_jax.ipynb)
    contains a [Colab](http://colab.google) notebook with a sampling example using JAX.

-   [`colabs/sampling_tutorial_pytorch.ipynb`](https://colab.sandbox.google.com/github/google-deepmind/recurrentgemma/blob/main/colabs/sampling_tutorial_pytorch.ipynb)
    contains a [Colab](http://colab.google) notebook with a sampling example using PyTorch.

-   [`colabs/fine_tuning_tutorial_jax.ipynb`](https://colab.sandbox.google.com/github/google-deepmind/recurrentgemma/blob/main/colabs/fine_tuning_tutorial_jax.ipynb)
    contains a [Colab](http://colab.google) with a basic tutorial on how to
    fine-tune RecurrentGemma for a task, such as English to French translation, using JAX.

To run these notebooks you will need to have a Kaggle account and first read and accept
the Gemma license terms and conditions from the [RecurrentGemma page](http://kaggle.com/models/google/recurrentgemma).
After this you can run the notebooks, which will automatically download the weights and tokenizer from there.

Currently different notebooks are supported under the following hardware:

| Hardware            | T4  | P100 | V100 | A100 | TPUv2 | TPUv3+ |
|---------------------|:---:|:----:|:----:|:----:|:-----:|:------:|
| Sampling in Jax     | ✅  | ✅   | ✅   | ✅   | ✅    | ✅    |
| Sampling in PyTorch | ✅  | ✅   | ✅   | ✅   | ✅    | ✅    |
| Finetuning in Jax   | ✅  | ✅   | ✅   | ✅   | ❌    | ✅    |

## System Requirements

RecurrentGemma code can run on CPU, GPU or TPU.
The code has been optimized for running on TPU using the Flax implementation,
which contains a low level [Pallas](https://jax.readthedocs.io/en/latest/pallas/index.html) kernel to perform the linear scan in the recurrent layers.

## Contributing

We are open to bug reports and issues. Please see
[CONTRIBUTING.md](CONTRIBUTING.md) for details on PRs.

## License

Copyright 2024 DeepMind Technologies Limited

This code is licensed under the Apache License, Version 2.0 (the \"License\");
you may not use this file except in compliance with the License. You may obtain
a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

## Disclaimer

This is not an official Google product.