MODEL FORGE

emergent architecture parameter calculator — agentic design space

Presets

Precision

Batch Size1

164

Model Dim (d_model)4,096

6416,384

Attention Heads32

1128

KV Heads (GQA)8

132

Layers32

1128

Context Length8.2K

512131.1K

Vocab Size128.3K

1.0K256.0K

FFN Multiplier3.5x

1.0x8.0x

7.50B

total parameters

15.01 GB

Model Memory

1.07 GB

KV Cache

4.29 GB

Activations

20.38 GB

Total VRAM

GPU Fit Estimate

RTX 4090✓ fits

A100 40GB✓ fits

A100 80GB✓ fits

H100 80GB✓ fits

Parameter Breakdown

Embedding

525.34M

Attention

1.34B

FFN

5.64B

LayerNorm

262.1K

Memory Breakdown

Model Weights

15.01B

KV Cache

1.07B

Activations

4.29B

we're so early — emergent architecture intuition is AI-native now · ← neural · tokens →