Anatomy of Trillion-Parameter Switchboards: Understanding Feedforward Blocks
Exploring the hidden layers of trillion-parameter switchboards: Feedforward Neural Networks and Activation Functions.
Deep-Learning
The Geometry of Meaning: Sine, ALiBi, RoPE, and HoPE
From Sinusoidal to RoPE and HoPE: How Transformers learn to process word order and sequence length.