avatar
Articles
18
Tags
5
Categories
0
Home
Archives
Tags
Categories
List
  • Music
  • Movie
Link
About
scmmm's blogFlashAttention 原理 Back to Home
Home
Archives
Tags
Categories
List
  • Music
  • Movie
Link
About

FlashAttention 原理

Created2026-07-20|Updated2026-07-20
|Post Views:
Author: scmmm
Link: http://example.com/2026/07/20/FlashAttention-%E5%8E%9F%E7%90%86/
Copyright Notice: All articles on this blog are licensed under CC BY-NC-SA 4.0 unless otherwise stated.
cover of next post
Next
面试积累 - FFN 层激活函数
在 FFN 层中，有几个常见的激活函数，这次我们尝试梳理它们。 GLU 变体公式 \text{激活} \quad \left\{ \begin{aligned} \mathrm{FFN}_{\mathrm{GLU}}(x, W, V, W_2) &= (\sigma(xW) \otimes xV)\,W_2 \\ \mathrm{FFN}_{\mathrm{Bilinear}}(x, W, V, W_2) &= (xW \otimes xV)\,W_2 \\ \mathrm{FFN}_{\mathrm{ReGLU}}(x, W, V, W_2) &= (\max(0, xW) \otimes xV)\,W_2 \\ \mathrm{FFN}_{\mathrm{GEGLU}}(x, W, V, W_2) &= (\mathrm{GELU}(xW) \otimes xV)\,W_2 \\ \mathrm{FFN}_{\mathrm{SwiGLU}}(x, W, V, W_2) &= (\mathrm{Swish}_{1}(xW) \otimes xV)\,W_2 \end{align...
avatar
scmmm
B.S. in Biology & Computer Science, Shandong University,2022-2026
M.S. in Computer Science, ShanghaiTech University(Research conducted at ICT,CAS),2026-
Research Focus: AI for EDA
Articles
18
Tags
5
Categories
0
Follow Me
Recent Posts
FlashAttention 原理
FlashAttention 原理2026-07-20
面试积累 - FFN 层激活函数
面试积累 - FFN 层激活函数2026-07-20
LLM学习计划
LLM学习计划2026-07-20
Hello World
Hello World2026-07-20
Digital VLSI Design-lecture 7
Digital VLSI Design-lecture 72025-04-10
© 2020 - 2026 By scmmm