Mamba - a replacement for Transformers? - YouTube
Excerpt
Mamba is a new neural network architecture proposed by Albert Gu and Tri Dao.Timestamps:00:00 - Mamba - a replacement for Transformers?00:19 - The Long RangeâŠ
This is my first time watching your channel. Impressive walkthrough. When I first heard of Q* my imagination started to build a very similar architecture⊠I donât follow too much of the technical, but I saw how the sandwiched gates, shown in the video, could be used almost in an analogue fashion. This is brilliant! Watching this made me grin like crazy⊠This might not be zero memory, but dang if it isnât a huge step in that direction. Using local memory is genius. And that token interpretation length, yes⊠So⊠physically, I guess, in my mind the next step is to localize the memory to the operation even more, but it looks like in that architecture itâs as local as itâs going to get⊠What about something like⊠âSample-and-hold,â from actual analogue circuits? That might be something to think about.
Read more