Writing archive
Notes on language models, optimization, and the systems behind modern machine learning.
All neural networks use a form of gradient descent for updating their parameters. The fundamental intuition to all neural net’s parameter optimization seems obvious to us, i.e., to move opposite to the gradient. However, …
What is Positional Encoding and why it matters? When training any large language model based on Transformers architecture, our input token sequences tend to form a $\text{seq\_len} \times \text{seq\_len}$ dimension …
Recurrent Neural Networks (RNN) have existed for long at this point, and RNNs without attention mechanism (plain-simple RNN architecture) are no longer the hottest thing either. Still, RNN represents one of the first …