Mamba, Transformers, and Training

Glossary

Backpropagation

The standard algorithm for training deep neural networks by calculating the gradient (rate of change) of the loss function with respect to the network weights. It uses the chain rule to efficiently compute these gradients layer by layer, starting from the output and moving backward (the backward pass). (See Section 5)

Backward Pass

The part of the backpropagation algorithm where the error signal is propagated backward through the network to compute the gradients for each weight. Computationally intensive, often ~2x the cost of the forward pass. (See Section 5.3)

Context Resolution

The idea of reducing the level of detail or granularity stored or processed in a model's representation of context. Lower resolution can lead to efficiency gains, relating to model reduction. (See Section 6)

Context Synchronization

The concept of using multiple models whose understanding of the context is kept aligned or synchronized, potentially allowing several smaller models to replace one large one, relating to model reduction. (See Section 6)

Context Window

The maximum number of preceding tokens that a model (especially a Transformer) can look at when processing the current token. (See Section 2)

Evolutionary Algorithms (EAs)

Optimization algorithms inspired by biological evolution (selection, mutation, crossover). Used in Neuroevolution to train neural networks without backpropagation by evolving weights or architectures. (See Section 5.4)

Efficient Attention

Modifications to the standard self-attention mechanism in Transformers designed to reduce its quadratic computational cost, making it feasible to process longer sequences. Examples include sparse attention and linear attention. (See Section 4)

Extreme Learning Machines (ELMs)

Feedforward neural networks where input-to-hidden weights are random and fixed; only the hidden-to-output weights are trained (often analytically), avoiding backpropagation. (See Section 5.4)

Exploding Gradients

A problem during training where gradients become excessively large during backpropagation, leading to unstable updates and poor learning. Analogous to the "jackknife" in the trailer analogy. (See Section 5.1)

FlashAttention

A highly optimized, hardware-aware implementation of the standard self-attention mechanism. It doesn't change the math but significantly speeds up computation and reduces memory usage on GPUs by improving memory access patterns. A key part of "Transformers++". (See Section 4)

Forward Pass

The process of feeding input data through the neural network, layer by layer, to compute the output prediction. Precedes the backward pass during training. (See Section 5.3)

Hardware-Aware Implementation

Designing algorithms and code to take specific advantage of the underlying hardware (like GPUs) for maximum efficiency. Mamba's parallel scan and FlashAttention are examples. (See Section 1.3)

Hugging Face Hub

An online platform hosting thousands of pre-trained models (including Mamba and Transformers), datasets, and demos. Integrates with the Hugging Face transformers library. (See Section 3)

Hugging Face transformers Library

A popular open-source Python library providing a unified API for downloading, training, and using a wide range of pre-trained models (including Mamba and Transformers) for various tasks. (See Section 3)

Hidden State

Internal memory or representation within a neural network (especially recurrent models or SSMs like Mamba) that summarizes relevant information processed so far in a sequence. (See Section 1.3)

Inference

The process of using a trained model to make predictions on new, unseen data. Contrasts with training, where the model learns from data. (See Section 1.2)

Linear Scaling (O(N))

When the computational cost or memory requirement of an algorithm grows proportionally to the size of the input (N, e.g., sequence length). Mamba aims for this, contrasting with the quadratic scaling of standard Transformers. (See Section 1.2)

Lightning AI lit-gpt

An open-source library providing optimized implementations of various LLMs, including Mamba, focused on clarity, performance, and research flexibility. (See Section 3)

Lost in the Middle

The phenomenon where models (sometimes Transformers with very long context windows) struggle to effectively utilize information located in the middle of the input sequence compared to information at the beginning or end. (See Section 2)

Mamba

A novel neural network architecture for sequence modeling based on Selective State Space Models (SSMs). Designed as an efficient (linear scaling) alternative to Transformers, especially for long sequences. (See Section 1)

Model Reduction

Techniques aimed at reducing the computational or memory requirements of a machine learning model (making it smaller, faster, or less resource-intensive) while preserving performance as much as possible. Examples include pruning, quantization, knowledge distillation, and potentially approaches like context resolution or context synchronization. (See Section 6)

Mixture of Experts (MoE)

An architectural technique where parts of the network (often feed-forward layers) consist of multiple "expert" sub-networks. For each input token, only a few experts are selected and activated, increasing model capacity efficiently. (See Section 4)

Quadratic Complexity (O(N²))

When the computational cost or memory requirement of an algorithm grows with the square of the input size (N). The standard self-attention mechanism in Transformers has this complexity with respect to sequence length, making very long sequences expensive. (See Section 1.2)

Retrieval-Augmented Generation (RAG)

A technique enhancing generative models by first retrieving relevant documents or information from an external knowledge base and then using that information as context to generate a more accurate and informed response. (See Section 4)

Reservoir Computing (ESNs, LSMs)

A framework for computation using fixed, random recurrent neural networks ("reservoirs"). Only a simple readout layer is trained to interpret the reservoir's dynamics, avoiding backpropagation through the recurrent part. (See Section 5.4)

Self-Attention

The core mechanism in Transformers that allows each token in a sequence to weigh the importance of all other tokens (within the context window) when computing its own representation. Powerful but computationally expensive (O(N²)). (See Section 1.2)

Sequence Modeling

The task of understanding, processing, or generating sequences of data, where order matters. Common in natural language processing (text), audio processing, time series analysis, and genomics. (See Section 1)

State Space Models (SSMs)

A class of models, originating from control theory, used for modeling sequences. They maintain a latent hidden state that evolves over time based on inputs. Mamba uses a *Selective* SSM where the state dynamics are input-dependent. (See Section 1.3)

Transformer

A highly successful neural network architecture, primarily based on the self-attention mechanism. Dominant in NLP (e.g., GPT, BERT) but suffers from quadratic scaling with sequence length. (See Section 1)

Transformers++

A conceptual term representing the ongoing evolution and improvement in large-scale sequence modeling, including alternatives to Transformers (like Mamba), optimizations within the Transformer framework (like Efficient Attention, MoE), and augmenting techniques (like RAG). (See Section 4)

Vanishing Gradients

A problem during training deep networks where gradients become extremely small during backpropagation, making it difficult for earlier layers to learn effectively (the error signal "vanishes"). (See Section 5.1)