Anatomy of Trillion-Parameter Switchboards: Understanding Feedforward Blocks
Exploring the hidden layers of trillion-parameter switchboards: Feedforward Neural Networks and Activation Functions.
Posts
Attention Is All You Need, but exactly which one?: MHA, GQA and MLA
We are about to touch the holy grail of modern AI. From the original 2017 paper to DeepSeek’s MLA, how has the definition of ‘Attention’ transformed?
The Geometry of Meaning: Sine, ALiBi, RoPE, and HoPE
From Sinusoidal to RoPE and HoPE: How Transformers learn to process word order and sequence length.
The Global Accountant and the Subword Surgeon: Decoding GloVe and FastText
How subword units and global co-occurrence matrices allow GloVe and FastText to capture nuances that Word2Vec missed.
Semantic Alchemy: Cracking Word2Vec with CBOW and Skip-Gram
Understanding the mathematics behind Word2Vec, CBOW, and Skip-Gram and how they map language to vector space.
The DNA of Language: A Deep Dive into LLM Tokenization concepts
A comprehensive guide to tokenization strategies: BPE, WordPiece, Unigram, and SentencePiece.
My First Post
A brief introduction to Vectors & Verbs and formatting verification.