🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Slicing a PyTorch Tensor Into Parallel Shards

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)

Tensor Parallelism with JAX + Shard Map

Fast and easy distributed model training examples.

Democratizing huggingface model training with InternEvo

This repository outlines a comprehensive guide for training a distributed deep learning model.