| Day / Status | Track A: Systems DSA (1.5h) | Track B: DL & Math (1.5h) | Track C: Domain & Build (2.0h) | Daily Deliverable | Notes / Benchmark Log |
|---|
| Done | Domain | Primary Book / Text | Companion Course / Codebase | Chapters to Prioritize (4-Week Scope) | Bookmark / Page Notes |
|---|
| Read | Paper & Authors | Whiteboard Derivation & Architecture Focus | Your Derivation / Key Equation Notes |
|---|
| Week / Status | Checkpoint Focus | Closed-Book Verification Criteria | Self-Grading Notes & Gaps to Review |
|---|
01. Roofline Model & GPU Arithmetic Intensity
Week 1 · GPU Perf
Arithmetic Intensity & Ridge Point
Operations per byte transferred from HBM global memory.
Arithmetic_Intensity (AI) = Total_FLOPs / Total_Bytes_Transferred
Ridge_Point = Peak_Compute_FLOPS / Peak_Memory_Bandwidth
• If AI < Ridge_Point → Memory-Bandwidth Bound (optimize coalescing, fusion, shared mem)
• If AI > Ridge_Point → Compute Bound (optimize Tensor Cores, warp scheduling, FP16/FP8)
GEMM Arithmetic Intensity (C[M,N] = A[M,K] × B[K,N] in FP16)
Each element multiply-accumulate counts as 2 FLOPs; FP16/BF16 uses 2 bytes per element.
FLOPs = 2 * M * N * K
Bytes = 2 * (M*K + K*N + M*N)
AI = (M * N * K) / (M*K + K*N + M*N)
• NVIDIA A100 (80GB SXM): 312 TFLOPS (FP16) | 2.0 TB/s HBM2e → Ridge ≈ 156 FLOPs/Byte
• NVIDIA H100 (80GB SXM): 989 TFLOPS (BF16) | 3.35 TB/s HBM3 → Ridge ≈ 295 FLOPs/Byte
Shared Memory Bank Conflict Avoidance
32 banks, 4-byte (32-bit) width per bank. Threads in a warp accessing different addresses in the same bank serialize.
Bank_ID = (Byte_Address / 4) % 32
// Fix column-stride bank conflicts in shared memory tiles by padding +1:
__shared__ float tile[BLOCK_SIZE][BLOCK_SIZE + 1];
02. LLM Memory Footprint, ZeRO & KV Cache Math
Week 3 · Scaling Math
Mixed-Precision Training State (16Ψ Rule)
For a model with Ψ parameters trained with BF16/FP16 weights and FP32 AdamW states:
Weights (BF16): 2 * Ψ bytes
Gradients (BF16): 2 * Ψ bytes
AdamW FP32 Master Copy: 4 * Ψ bytes
AdamW FP32 Momentum (m): 4 * Ψ bytes
AdamW FP32 Variance (v): 4 * Ψ bytes
------------------------------------
Total Static Memory = 16 * Ψ bytes (e.g., 7B model = 112 GB; 70B = 1,120 GB)
ZeRO-1 / ZeRO-2 / ZeRO-3 Sharding Across N_d Data-Parallel GPUs
Per-GPU static memory footprint as optimizer states, gradients, and weights are partitioned:
ZeRO-1 (Optimizer Sharded): (2Ψ + 2Ψ) + (12Ψ / N_d) = 4Ψ + 12Ψ/N_d
ZeRO-2 (Opt + Grad Sharded): (2Ψ) + (14Ψ / N_d) = 2Ψ + 14Ψ/N_d
ZeRO-3 (All Sharded): 16Ψ / N_d (requires 1.5x comm volume for All-Gather/Reduce-Scatter)
Inference KV-Cache Memory Footprint (GQA / MQA)
For Batch B, Sequence Length S, Layers L, KV Heads n_kv, Head Dim d_h in FP16 (2 bytes):
KV_Cache_Bytes = 2 (K & V) * 2 (FP16 bytes) * L * n_kv * d_h * B * S
// Example: Llama-3-70B (L=80, n_kv=8, d_h=128) in FP16:
Per-Token KV = 4 * 80 * 8 * 128 = 327,680 bytes (~320 KB / token)
At S=8192, B=16 → 327,680 B * 8192 * 16 ≈ 42.9 GB (= 40 GiB) KV Cache
03. Online Softmax & RLHF / GRPO / DPO Equations
Week 2 · Post-Training
FlashAttention Online Softmax Recurrence
Computes exact softmax in SRAM tiles without materializing the N×N attention matrix in HBM:
m_new = max(m_prev, rowmax(S_block))
l_new = l_prev * exp(m_prev - m_new) + rowsum(exp(S_block - m_new))
O_new = diag(l_new)^(-1) * (l_prev * exp(m_prev - m_new) * O_prev + exp(S_block - m_new) * V_block)
PPO Clipped Surrogate Objective & GAE
Probability ratio r_t(θ) = π_θ(y_t | x, y_
δ_t = R_t + γ * V(s_{t+1}) - V(s_t) // R_t = reward at step t (not the ratio r_t(θ))
A_t^GAE = Σ_{k=0}^{T-t-1} (γ * λ)^k * δ_{t+k}
L_PPO = E_t [ min( r_t(θ) * A_t, clip(r_t(θ), 1 - ε, 1 + ε) * A_t ) ] - β * D_KL(π_θ || π_ref)
GRPO (Critic-Free Group Relative Policy Optimization) & DPO Loss
GRPO normalizes rewards across G sampled completions {o_1...o_G} per prompt, eliminating the value network:
// GRPO Group Advantage & Unbiased KL Estimator:
A_i = (r_i - mean(r_1..r_G)) / (std(r_1..r_G) + 1e-8)
D_KL = (π_ref(o_i|q) / π_θ(o_i|q)) - log(π_ref(o_i|q) / π_θ(o_i|q)) - 1
// Direct Preference Optimization (DPO) on (x, y_win, y_lose):
L_DPO = -E [ log σ( β * log(π_θ(y_w|x)/π_ref(y_w|x)) - β * log(π_θ(y_l|x)/π_ref(y_l|x)) ) ]
04. AI Sandboxing & RLVR Code Execution Hardening
Week 3 · Security & Infra
1. Namespace & Filesystem Isolation (unshare / pivot_root)
Isolate PID tree, network stack, mount table, and UID mappings before executing untrusted LLM code:
# Launch a process in fresh user/PID/net/mount/IPC/UTS namespaces (no external network interfaces).
# For a read-only rootfs, pivot_root into a prepared root and remount it read-only before exec:
unshare --user --map-root-user --pid --net --mount --ipc --uts --fork \
sh -c "mount -t proc proc /proc && exec python3 -I /sandbox/eval_runner.py"
2. cgroups v2 Memory, Swap & Fork-Bomb Limits
Enforce hard kernel caps so generated code cannot exhaust host RAM or spawn runaway processes:
mkdir -p /sys/fs/cgroup/rlvr_worker_1
echo "+memory +pids +cpu" > /sys/fs/cgroup/cgroup.subtree_control # enable controllers for child cgroups
echo "536870912" > /sys/fs/cgroup/rlvr_worker_1/memory.max # 512 MB hard RAM cap
echo "0" > /sys/fs/cgroup/rlvr_worker_1/memory.swap.max # Disable swap
echo "32" > /sys/fs/cgroup/rlvr_worker_1/pids.max # Block fork bombs
echo "50000 100000" > /sys/fs/cgroup/rlvr_worker_1/cpu.max # 50% of 1 CPU core
3. seccomp-bpf Syscall Filter & MicroVM Defense-in-Depth
Block high-risk kernel syscalls and socket creation inside RLVR grading workers:
// High-priority syscalls to deny via seccomp-bpf (SCMP_ACT_ERRNO(EPERM)):
Deny: socket(AF_INET/AF_INET6), connect, ptrace, mount, umount2, pivot_root,
unshare, setns, kexec_load, bpf, perf_event_open, userfaultfd, keyctl
// Production layer: Wrap worker inside gVisor (runsc) or Firecracker MicroVM.
Saved