Project
CS336 Assignment 1: Implementation Notes and Takeaways
This page records implementation-level observations from building core language-model components and validating them against expected behavior.
Scope
The assignment centers on implementing foundational pieces of a language-model stack. The useful systems angle is how each local choice affects memory use, numerical behavior, and runtime measurement.
Implementation Notes
- Shape conventions should be explicit at module boundaries.
- Numerical tests need small deterministic inputs before larger training runs.
- Tokenizer, model, and optimizer code each create distinct failure modes.
- Runtime measurements are only meaningful when batch size, sequence length, and device placement are fixed.
Takeaways
The assignment is a reminder that AI-system work depends on both algorithmic clarity and low-level discipline. Correctness checks, tensor layout, memory use, and measurement design are part of the same implementation story.