LitServe
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
File Explorer
- bug_report.md
- config.yml
- documentation.md
- feature_request.md
- ci-checks.yml
- ci-minimal-dependency-check.yml
- ci-parity.yml
- ci-testing.yml
- release-pypi.yml
- CODEOWNERS
- dependabot.yml
- PULL_REQUEST_TEMPLATE.md
- stale.yml
- tests.yml
- perf.txt
- __init__.py
- metric_callback.py
- __init__.py
- base.py
- __init__.py
- base.py
- continuous_batching_loop.py
- loops.py
- simple_loops.py
- streaming_loops.py
- __init__.py
- image.py
- __init__.py
- base.py
- openai.py
- openai_embedding.py
- __init__.py
- openai_embedding_spec_example.py
- openai_spec_example.py
- simple_example.py
- __init__.py
- base.py
- factory.py
- process_transport.py
- zmq_queue.py
- zmq_transport.py
- __about__.py
- __init__.py
- __main__.py
- api.py
- cli.py
- connector.py
- constants.py
- docker_builder.py
- loggers.py
- mcp.py
- middlewares.py
- python_client.py
- server.py
- utils.py
- default_api.py
- default_async_streaming.py
- default_batched_streaming.py
- default_batching.py
- default_mcp.py
- default_openai_embedding_spec.py
- default_openai_with_batching.py
- default_openaispec.py
- default_openaispec_response_format.py
- default_openaispec_tools.py
- default_single_streaming.py
- default_spec.py
- openai_embedding_with_batching.py
- test_e2e.py
- test_async.py
- benchmark.py
- fastapi-server.py
- ls-server.py
- main.py
- benchmark.py
- data.py
- run_test.sh
- server.py
- utils.py
- benchmark.py
- server.py
- run_test.sh
- test_auth.py
- test_batch.py
- test_callbacks.py
- test_cli.py
- test_compression.py
- test_connector.py
- test_docker_builder.py
- test_examples.py
- test_failed_workers.py
- test_form.py
- test_lit_server.py
- test_litapi.py
- test_logger.py
- test_logging.py
- test_loops.py
- test_mcp.py
- test_middlewares.py
- test_multiple_endpoints.py
- test_openai_embedding.py
- test_pydantic.py
- test_readme.py
- test_request_handlers.py
- test_schema.py
- test_simple.py
- test_specs.py
- test_torch.py
- test_transport.py
- test_utils.py
- test_zmq_queue.py
- __init__.py
- conftest.py
- minimal_run.py
- simple_server.py
- simple_server_diff_port.py
- .codecov.yml
- .gitignore
- .pre-commit-config.yaml
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- GOVERNANCE.md
- LICENSE
- MANIFEST.in
- pyproject.toml
- pytest.ini
- README.md
# Installation Guide
pythonSetup Steps
pip install litserve
Install the LitServe library.
python server.py
Run the defined LitServe server script.
curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"input": 4.0}'
Send a test request to the server prediction endpoint.
Key Commands
pip install litserve
Install LitServe using the Python package manager.
lightning deploy server.py
Deploy or run the server using the Lightning CLI.
python server.py
Directly execute the server script with the Python interpreter.
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
pip install
View Details ▼
pip install
Install Python packages.
pip install {{package1 package2 ...}}
Install one or more packages:
pip install {{package1 package2 ...}} {{[-U|--upgrade]}}
Upgrade all specified packages to the latest version, installing any that are not already present:
pip install {{package}}=={{version}}
Install a specific version of a package: