transformers-openai-api

KO | EN

gitlite — search

#javascript #python #ai #typescript #llm #android #machine-learning #linux #macos #java #cli #kubernetes #awesome-list #ios

transformers-openai-api

(★ 64)

An OpenAI Completions API compatible server for NLP transformers models

파일 탐색기

최종 버전 다운로드 (.zip)

- __init__.py
- __main__.py
- app.py
- metrics.py
- models.py
- serve.py
.gitignore
config.example.json
LICENSE
README.md
requirements.txt
setup.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# transformers-openai-api

`transformers-openai-api` is a server for hosting locally running NLP [transformers](https://github.com/huggingface/transformers/) models via the [OpenAI Completions API](https://beta.openai.com/docs/api-reference/completions). In short, you can run `transformers` models and offer them through an API compatible with existing OpenAI tooling such as the [OpenAI Python Client](https://github.com/openai/openai-python) itself or any package that uses it (e.g. [LangChain](https://github.com/hwchase17/langchain)).

## Quickstart

### From pip

```sh
pip install transformers-openai-api
wget https://raw.githubusercontent.com/jquesnelle/transformers-openai-api/master/config.example.json
mv config.example.json config.json
transformers-openai-api
```

### From source

```sh
git clone https://github.com/jquesnelle/transformers-openai-api
cd transformers-openai-api
cp config.example.json config.json
pip install -r requirements.txt
python transformers_openai_api/
```

## Using with OpenAI Python Client

Simply set the environment variable `OPENAI_API_BASE` to `http://HOST:PORT/v1` before importing the `openai` package. For example, to access a local instance of `transformers-openai-api`, set `OPENAI_API_BASE` to `http://127.0.0.1:5000/v1`. Alternatively, you can set the `api_base` property on the `openai` object:

```python
import openai
openai.api_base = 'http://HOST:PORT/v1'
```

## Configuration

All configuration is managed through `config.json`. By default `transformers-openai-api` looks for this file the in the current working directory, however a different path can be passed as the command-line argument to the program.  See [config.example.json](config.example.json).

### Hosting

By default the API server listens on `127.0.0.1:5000` to change this, add a `HOST` and/or `PORT` entries to the configuration file. For example to serve publicly:
```json
{
    "HOST": "0.0.0.0",
    "PORT": 80
}
```

### Models

#### Using accelerate

To use [accelerate](https://github.com/huggingface/accelerate), set `device_map` on the `MODEL_CONFIG` to `auto` and explicitly set `MODEL_DEVICE` to `null`. The default `text-davinci-003` model in [config.example.json](config.example.json) is an example of this.

#### Using CPU

To switch to CPU inference, set `MODEL_DEVICE` and `TOKENIZER_DEVICE` to `cpu`.

#### Using FP16

To use a model at half-precision, set `torch_dtype` on the `MODEL_CONFIG` to `torch_dtype`. The disabled `text-curie-001` model in [config.example.json](config.example.json) is an example of this.

### Authorization

To limit access to the API (i.e. enforcing `OPENAI_API_KEY`), fill in the `BEARER_TOKENS` object with a list of authorized tokens (e.g. your OpenAI key). If the `BEARER_TOKENS` list does not exist, no authorization will be enforced.
```json
{
    "BEARER_TOKENS": ["sk-..."]
}
```

transformers-openai-api

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

transformers-openai-api

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README