conformal-risk - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #go #android #rust #deep-learning #linux #nodejs #java #cli #security #ios #mcp #openai

conformal-risk

(★ 85)

Conformal prediction for controlling monotonic risk functions. Simple accompanying PyTorch code for conformal risk control in computer vision and natural language processing.

File Explorer

Download Latest Version (.zip)

- - infer.py
  - LICENSE
  - MODEL_ZOO.md
  - README.md
  - requirements.txt
- - coco_grid_fig.pdf
- - grid_fig.py
  - risk_histogram.py
  - utils.py
- .gitignore
- __init__.py
- get_lhat.py
- - hierarchical_grid_fig.pdf
- .gitignore
- __init__.py
- grid_hierarchical.py
- map_clsloc.txt
- ntree.py
- risk_histogram.py
- utils.py
- wordnet_hierarchy.json
- - .gitignore
  - __init__.py
  - process_all_datasets.py
  - README.md
- .gitignore
- multipolyp_grid_fig.py
- polyp_utils.py
- risk_histogram.py
- convert_predictions.py
- inference.sh
- print_examples.py
- README.md
- requirements.txt
- risk_histogram.py
- train_extractive_reader.py
.gitignore
environment.yml
LICENSE
README.md

# Installation Guide

1. Get the code

git clone https://github.com/aangelopoulos/conformal-risk

Downloads the entire project code from GitHub to your computer.

cd conformal-risk

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r coco/asl/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Conformal Risk Control
This is the official repository of [Conformal Risk Control](http://arxiv.org/abs/2208.02814) by Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster.

<p align="center">
    <a style="text-decoration:none !important;" href="http://arxiv.org/abs/2208.02814" alt="arXiv"> <img src="https://img.shields.io/badge/paper-arXiv-red" /> </a>
    <a style="text-decoration:none !important;" href="https://people.eecs.berkeley.edu/%7Eangelopoulos/" alt="website"> <img src="https://img.shields.io/badge/website-Berkeley-yellow" /> </a>
    <a style="text-decoration:none !important;" href="https://docs.conda.io/en/latest/miniconda.html" alt="package management"> <img src="https://img.shields.io/badge/conda-env-green" /> </a>
    <a style="text-decoration:none !important;" href="https://opensource.org/licenses/MIT" alt="License"> <img src="https://img.shields.io/badge/license-MIT-blue.svg" /> </a>

</p>

## Technical background
In the risk control problem, we are given some loss function $L_i(\lambda) = \ell(X_i,Y_i,\lambda)$.
For example, in multi-label classification, you can think of the loss function as the false negative proportion $L_i(\lambda) = 1 - \frac{|Y_{i} \cap C_{\lambda}(X_{i})|}{|Y_i|}$, where $C_{\lambda}(X_{i})$ is the set-valued output of a machine learning model. 
As $\lambda$ grows, so does the set $C_{\lambda}(X_{i})$, which shrinks the false negative proportion.
We seek to choose $\hat{\lambda}$ based on the first $n$ data points to control the expected value of its loss <i>on a new test point</i> at some user-specified risk level $\alpha$, $$\mathbb{E}\big[L_{n+1}(\hat{\lambda})\big] \leq \alpha.$$

The conformal risk control algorithm is in `core/get_lhat.py`. It is 5 lines long, including the function header.

## Examples
Each of the `{polyps, coco, hierarchical-imagenet, qa}` folders contains a worked example of conformal risk control with a different risk function.
`polyps` does gut polyp segmentation with false negative rate control. `coco` does multi-label classification with false negative rate control. `hierarchical-imagenet` does hierarchical classification and chooses the resolution of its prediction by bounding the graph distance to an ancestor of the true label. Finally, `qa` controls the F1-score in open-world question answering.

### Setup
For the computer vision experiments, run
```
  conda env create -f environment.yml
  conda activate conformal-risk
```
This will install all dependencies for the vision experiments.

For the question-answering task, follow the instructions in `qa/README.md`.

### Reproducing the experiments
After setting up the environment, enter the example folder and run the appropriate `risk_histogram.py` file.
To produce the grids of images in the paper, run the python file containing the word `grid` in each folder.

### Citation

```
@article{angelopoulos2022conformal,
  title={Conformal Risk Control},
  author={Angelopoulos, Anastasios N and Bates, Stephen and Fisch, Adam and Lei, Lihua and Schuster, Tal},
  journal={arXiv preprint arXiv:2208.02814},
  year={2022}
}
```