Transformer_Knowlegde

KO | EN

gitlite — search

#pdf #go #json #github #deep-learning #codex #dotnet #pytorch #cli #gemini

Transformer_Knowlegde

(★ 27)

从底层机理了解Transformer

File Explorer

ZIP Download

README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

> 从底层机理了解Transformer
>
> 知乎专栏：https://www.zhihu.com/column/c_1380571607480651776

# 1. 线性self-attention

## 1.1 稀疏Attention（sparse attention)

> 讲解：https://zhuanlan.zhihu.com/p/469853664

---

- **《Generating Long Sequences with Sparse Transformers》** 19年4月 OpenAI

> https://arxiv.org/abs/1904.10509

![img](https://gitee.com/wanghui88888888/picture/raw/master/img/v2-9433571376feed59813b182b1251afea_720w.jpg)

---

- **《Sparse Transformer: Concentrated Attention Through Explicit Selection》**  19年12月

> https://arxiv.org/pdf/1912.11637.pdf

![img](https://gitee.com/wanghui88888888/picture/raw/master/img/v2-0c50dc60aec5c346aa15ed1b2e17bf4e_720w.jpg)

---

- **《Longformer: The Long-Document Transforme》** 20年4月

> https://arxiv.org/abs/2004.05150

![image-20220304093109970](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304093109970.png)

---

- 《Reformer: The Efficient Transformer》 20年1月

> https://arxiv.org/abs/2001.04451

![image-20220304093336368](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304093336368.png)

---

- [Linformer: Self-Attention with Linear Complexity](https://link.zhihu.com/?target=https%3A//arxiv.org/abs/2006.04768)

> https://arxiv.org/abs/2006.04768

![image-20220304093639972](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304093639972.png)

---

- **Rethinking Attention with Performers ** **2020年9月**

> https://arxiv.org/abs/2009.14794

![image-20220304093832052](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304093832052.png)

---

- **Luna: Linear Unified Nested Attention**  2021年6月

> https://arxiv.org/abs/2106.01540

![image-20220304093922043](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304093922043.png)

## 1.2 softmax线性化 （linear softmax）

> 讲解：https://zhuanlan.zhihu.com/p/471291695

---

- **Efficient Attention: Attention with Linear Complexities** 18年12月

> https://arxiv.org/abs/1812.01243

![image-20220304094225376](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304094225376.png)

---

- **Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention** 20年6月

> https://arxiv.org/abs/2006.16236

![image-20220304094354606](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304094354606.png)

---

- **COSFORMER : RETHINKING SOFTMAX IN ATTENTION** 22年2月

> https://arxiv.org/pdf/2202.08791.pdf

![image-20220304094450162](https://gitee.com/wanghui88888888/picture/raw/master/img/image-20220304094450162.png)

# 2. PostNorm/PreNorm

**Content is constantly updated... ...**

Transformer_Knowlegde

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Transformer_Knowlegde

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README