AI & Machine Learning

Distributed LLM Pretraining Torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Publisher / source

Orchestra-Research/AI-Research-SKILLs (found on skills.sh, an awesome list)

Upstream record

Orchestra-Research/AI-Research-SKILLs

Version

Version 2

Published

2026-11-02

Publication review

Approved for publication

Content fingerprint

530b85ddadfc69e2cfd3bbdd7cdb03c89f7ddd00ba0e84fed8ac43e98a25a4d8