RFQI

KO | EN

gitlite — search

#hacktoberfest #react #typescript #llm #golang #machine-learning #deep-learning #macos #java #awesome #cli #vue #pytorch #open-source

(★ 25)

Implementation of Robust Reinforcement Learning using Offline Data [NeurIPS'22]

RFQI Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Robust Reinforcement Learning using Offline Data
Implementation of the algorithm Robust Fitted Q-Iteration (RFQI). RFQI is introduced in our paper [Robust Reinforcement Learning using Offline Data](https://arxiv.org/abs/2208.05129) (NeurIPS'22). This implementation of RFQI is based on the implementation of [BCQ](https://github.com/sfujim/BCQ) and the implementation of [PQL](https://github.com/yaoliucs/PQL).

Our method is tested in OpenAI gym discrete control task, [CartPole](https://www.gymlibrary.ml/environments/classic_control/cart_pole/), and two [MuJoCo](http://www.mujoco.org/) continuous control tasks, Hopper and HalfCheetah, using the [D4RL](https://github.com/rail-berkeley/d4rl) benchmark. **Thus it is required that MuJoCo and D4RL are both installed prior to using this repo**.

## Setup
Install requirements:
```
pip install -r requirements.txt
```
Next, you need to properly register the perturbed Gym environments which are placed under the folder perturbed_env. A recommended way to do this: first, place cartpole_perturbed.py under gym/envs/classic_control, hopper_perturbed.py and half_cheetah_perturbed.py under gym/envs/mujoco. Then add the following to \__init__.py under gym/envs:
```
register(
    id="CartPolePerturbed-v0",
    entry_point="gym.envs.classic_control.cartpole_perturbed:CartPolePerturbedEnv",
    max_episode_steps=200,
    reward_threshold=195.0,
)
register(
    id="HopperPerturbed-v3",
    entry_point="gym.envs.mujoco.hopper_perturbed:HopperPerturbedEnv",
    max_episode_steps=1000,
    reward_threshold=3800.0,
)
register(
    id="HalfCheetahPerturbed-v3",
    entry_point="gym.envs.mujoco.half_cheetah_perturbed:HalfCheetahPerturbedEnv",
    max_episode_steps=1000,
    reward_threshold=4800.0,
)
```
You can test this by running:
```
import gym

gym.make('HopperPerturbed-v3')
```
After installing MuJoCo and D4RL, you can run the following script to download D4RL offline data and make it conform to our format, or you can directly go to TL;DR section below:
```
python load_d4rl_data.py
```

# Training and Testing of MuJoCo Infographics: Robust MuJoCo Gym
![Robust MuJoCo Gym](https://github.com/user-attachments/assets/f867b467-ea43-4342-98ed-5ac25c498693)

The nominal stochastic probability transition model on which we train our policies is the vanilla MuJoCo setup inducing transition stochasticity using traditional action randomizations from [BCQ](https://github.com/sfujim/BCQ) and [PQL](https://github.com/yaoliucs/PQL) implementations. Finally, we evaluate the trained policies deployed on physics-informed perturbed MuJoCo environments.

## TL;DR
Here you can find shell scripts that take you directly from offline data generation to evaluation results.

To get all data, run
```
sh scripts/gen_all_data.sh
```
To get all results, run
```
sh scripts/run_cartpole.sh
sh scripts/run_hopper.sh
sh scripts/run_half_cheetah.sh
```
To evaluate all pre-trained models, run
```
sh scripts/eval_all.sh
```
## Detailed instructions 
To generate the epsilon-greedy dataset for `CartPole-v0` with `epsilon=0.3`, run the following:
```
python generate_offline_data.py --env=CartPole-v0 --gendata_pol=ppo --eps=0.3
```

To generate the mixed dataset specified in Appendix E.1, run the following:
```
python generate_offline_data.py --env=Hopper-v3 --gendata_pol=sac --eps=0.3 --mixed=True
```
To train an RFQI policy on `Hopper-v3` with `d4rl-hopper-medium-v0` and uncertainty hyperparameter `rho=0.5`, please run:
```
python train_rfqi.py --env=Hopper-v3 --d4rl=True --rho=0.5
```
You can also train an RFQI policy on `Hopper-v3` with mixed dataset and uncertainty hyperparameter `rho=0.5` by running
```
python train_rfqi.py --env=Hopper-v3 --data_eps=0.3 --gendata_pol=sac --mixed=True --rho0.5
```
## Miscellaneous
If you are using a remote machine to run this repo, please remember to assign a display/virtual display for the evaluation suite to properly generate gifs.

## Citation
Please consider citing our repository and paper if you find it useful in your research directions.
```
@article{panaganti-rfqi-2022,
  title={Robust Reinforcement Learning using Offline Data},
  author= {Panaganti, Kishan and Xu, Zaiyan and Kalathil, Dileep and Ghavamzadeh, Mohammad},
  journal={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2022}
}
```

RFQI

RFQI Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

RFQI

RFQI Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README