a2c - GitLite

KO | EN

gitlite — search

#openai #html #kotlin #nodejs #rag #svg #security #kubernetes #web #mcp

a2c

(★ 10)

No description available.

파일 탐색기

ZIP 다운로드

- pong.png
- space_invaders.png
.gitignore
a2c.py
batch_env.py
condv.py
distribution.py
env.py
eval.py
experience.py
main.py
network.py
noisy_net.py
README.md
train.py
utils.py

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

명령어 용어집

이 문서에서 사용된 명령어를 모아봤습니다. 낯선 명령어가 있다면 펼쳐서 확인해보세요.

🔍

`python3`

설명 보기 ▼

이 명령은 `python`의 별칭입니다.

tldr python

자세한 내용은 원본 명령을 참고하세요:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

This repo implements the A2C algorithm for reinforcement learning with
emphasis on speed and modularity. A2C is the synchronized version of
[Asynchronous advantage actor-critic(A3C)](https://arxiv.org/pdf/1602.01783.pdf).
A2C has been implemented and studied by many AI researchers and notablely been part of the
[OpenAI baseline](https://blog.openai.com/baselines-acktr-a2c/). However, the
detailed description of A2C is hard to find online. In this project,
we implement our own version with reference to the
[OpenAI baseline](https://github.com/openai/baselines/tree/master/baselines/a2c)
and write down some key design choices to help understand the codebase.

## Design

In the original A3C, N instances of the enviroment, e.g. an Atari
game, run asynchronously and each instance has its own actor to
produce trajactories (sequences of game play timesteps) used for
asynchronous parameter updates. In A2C, a
batch of environments are sychronized at every step, meaning that they
produce a batch of states, the agent consume that batch and produce a
batch of actions, and then the environments perform corresponding
actions to produce the next batch of states. This method can increase
the utilization of GPU and thus increase speed.

In A3C, each enviroment collect a trajectory of maximum length T and
the trajectory may be shorter if end state is encountered. In A2C, we
naturally repeat the batched step T times to collect N trajectories of
length T. However, end states may appear in one or more
trajactories. In those cases, the trajectory is simply a concatenation
of two shorter trajectories. For example, if T = 5, it may happen that
the first 2 nodes in the trajectory record the information of the last
two frames of one episode and the last 3 nodes in the trajactory
record the begining of a new episode. In this way, we can easily
convert N trajectories if length T into N * T pairs of (input, target)
tuples to maximize the training speed.

## Performance

This implementation is very fast. It runs at 2300 frames/s on my 4
core CPU + GTX1080 desktop to train on Pong with 16
environments/processes, which is more than 20% faster than some
[existing implementation](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr)
running on the same machine. Benchmarking on machines with more CPU
cores will be added.

<img src="figs/pong.png" width="425"/> <img src="figs/space_invaders.png" width="425"/>

## Usage
```
python3 main.py --env_name SpaceInvadersNoFrameskip-v4 --num_envs 16 --exp_name run1
```
See main.py for a complete list of optional command line arguments.

## Future Work
1. [NoisyNet for exploration](https://arxiv.org/abs/1706.10295)
2. [ACKTR](https://arxiv.org/abs/1708.05144)