state-space models

Mamba: Linear-Time Sequence Modeling with Selective State Spaces arXiv (Gu & Dao, 2024)

state-spaces/mamba, state-spaces/s4

Idea: to use state space representations in control theory to for sequences modeling.

Mamba

Mamba: Linear-Time Sequence Modeling with Selective State Spaces arXiv (Gu & Dao, 2024) Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality arXiv (Dao & Gu, 2024) Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality arXiv

State-space duality (SSD): SSM + attentions layers (SMA, or structured masked attention)

Bibliographie

Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. arXiv preprint arXiv:2405.21060 [arXiv]
Gu, A., & Dao, T. (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752 [arXiv]