Software Development
Training Llms Megatron
Trains large language models (2B-462B parameters) with NVIDIA Megatron-Core using tensor, pipeline, sequence, context, and expert parallelism, plus FP8 on H100 and MoE configuration for Mixtral-style models. Covers choosing TP/PP/DP/CP sizes, launching distributed training, tuning micro-batch size, and fixing low MFU, out-of-memory errors, and diverging loss. Use when training models above 10B parameters on NVIDIA A100/H100 GPUs, when setting up 3D parallelism for a LLaMA-style model, when configuring expert parallelism for MoE training, or when trying to raise MFU toward 40-47%. Use PyTorch FSDP, DeepSpeed, or HuggingFace Accelerate instead for models under 70B or simpler setups.
- Publisher / source
- KalarisLabs/research-agent-skills (found on the CSV list)
- Upstream record
- KalarisLabs/research-agent-skills
- Version
- Version 2
- Published
- Publication review
- Approved for publication
- Content fingerprint
5130a2944e77482b145767289fc6a3ff3454419d853c06610f5d864792333d8e