[ REFERENCE 02 ]

Component reference

All available components in Otter, organized by category. Each block is extracted from a production model and validated for dimension compatibility.

[ EMB ]

Embedding

Embedding

[B, S] → [B, S, hidden_dim]
[ from Standard ]

Maps token IDs to dense vectors. The entry point for most architectures.

[ PARAMETERS ]
vocab_size(int)— Vocabulary size (e.g., 32000)
hidden_dim(int)— Embedding dimension (e.g., 4096)
[ NORM ]

Normalization

RMSNorm

[B, S, hidden_dim] → [B, S, hidden_dim]
[ from Llama ]

Root Mean Square Layer Normalization. Faster than LayerNorm with similar results.

[ PARAMETERS ]
hidden_dim(int)— Hidden dimension to normalize
eps(float)— Epsilon for numerical stability (default: 1e-6)
[ POS ]

Positional

RoPE

[B, H, S, head_dim] → [B, H, S, head_dim]
[ from Llama ]

Rotary Position Embeddings. Applied to Q/K in attention for relative positions.

[ PARAMETERS ]
head_dim(int)— Dimension per head
max_seq_len(int)— Maximum sequence length
base(float)— Base frequency (default: 10000)
[ ATTN ]

Attention

GroupedQueryAttention

[B, S, hidden_dim] → [B, S, hidden_dim]
[ from Llama 2 ]

Multi-head attention with grouped key-value heads. Balances quality and efficiency.

[ PARAMETERS ]
hidden_dim(int)— Total hidden dimension
num_query_heads(int)— Number of query heads (e.g., 32)
num_kv_heads(int)— Number of KV heads (e.g., 8)
head_dim(int)— Dimension per head
[ FFN ]

FFN

SwiGLU

[B, S, hidden_dim] → [B, S, hidden_dim]
[ from PaLM ]

Feed-forward network with SiLU activation and gating. Standard in modern LLMs.

[ PARAMETERS ]
hidden_dim(int)— Input/output dimension
intermediate_dim(int)— Intermediate dimension (e.g., 11008)
[ MOE ]

Routing

TopKRouter

[B, S, hidden_dim] → [B, S, hidden_dim]
[ from Mixtral ]

Routes tokens to top-k experts. Core of Mixture-of-Experts architectures.

[ PARAMETERS ]
hidden_dim(int)— Input dimension
num_experts(int)— Total number of experts (e.g., 8)
top_k(int)— Experts per token (e.g., 2)
[ OUT ]

Output

OutputHead

[B, S, hidden_dim] → [B, S, vocab_size]
[ from Standard ]

Projects hidden states to vocabulary logits. Typically with tied embeddings.

[ PARAMETERS ]
hidden_dim(int)— Input hidden dimension
vocab_size(int)— Output vocabulary size
→ More components are added regularly. Request one via the feedback widget in the canvas.