transframer-pytorch - GitLite

KO | EN

gitlite — search

#python #react #typescript #machine-learning #rust #deep-learning #docker #macos #nodejs #java #kubernetes #ios #mcp #open-source

transframer-pytorch

(★ 73)

Implementation of Transframer, Deepmind's U-net + Transformer architecture for up to 30 seconds video generation, in Pytorch

파일 탐색기

최종 버전 다운로드 (.zip)

- - python-publish.yml
- __init__.py
- transframer_pytorch.py
.gitignore
LICENSE
README.md
setup.py
transframer.png

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/lucidrains/transframer-pytorch

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd transframer-pytorch

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install .

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<img src="./transframer.png" width="400px"></img>

## Transframer - Pytorch (wip)

Implementation of <a href="https://arxiv.org/abs/2203.09494">Transframer</a>, Deepmind's U-net + Transformer architecture for up to 30 seconds video generation, in Pytorch

The gist of the paper is the usage of a Unet as a multi-frame encoder, along with a regular transformer decoder cross attending and predicting the rest of the frames. The author builds upon his <a href="https://arxiv.org/abs/2103.03841">prior work</a> where images are encoded as sparse discrete cosine transform (DCT) sequences.

I will deviate from the implementation in this paper, using a <a href="https://github.com/lucidrains/RQ-Transformer/blob/main/rq_transformer/hierarchical_causal_transformer.py">hierarchical autoregressive transformer</a>, and just a regular resnet block in place of the NF-net block (this design choice is just Deepmind reusing their own code, as <a href="https://arxiv.org/abs/2102.06171">NF-net</a> was developed at Deepmind by Brock et al).

Update: On further meditation, there is nothing new in this paper except for generative modeling on DCT representations

## Appreciation

- This work would not be possible without the generous sponsorship from <a href="https://stability.ai/">Stability AI</a>, as well as my other sponsors

## Todo

- [ ] figure out if dct can be directly extracted from images in jpeg format

## Citations

```bibtex
@article{Nash2022TransframerAF,
    title   = {Transframer: Arbitrary Frame Prediction with Generative Models},
    author  = {Charlie Nash and Jo{\~a}o Carreira and Jacob Walker and Iain Barr and Andrew Jaegle and Mateusz Malinowski and Peter W. Battaglia},
    journal = {ArXiv},
    year    = {2022},
    volume  = {abs/2203.09494}
}
```