A collection of AWESOME things about mixture-of-experts

Reaching LLaMA2 Performance with 0.1M Dollars

Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method ; GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

Repo-Level Code generation papers

An incremental parsing system for programming tools

API Documentation Browser

An Extensible Deep Learning Library

Mixture-of-Experts (MoE) Language Model

Notebooks for the O'Reilly book "Learning Ray"

Agentic components of the Llama Stack APIs

A repository for research on medium sized language models.

Faker is a Python package that generates fake data for you.

LLM101n: Let's build a Storyteller

Scalable toolkit for efficient model alignment

DataComp: In search of the next generation of multimodal datasets

DataComp for Language Models

Provides end-to-end model development pipelines for LLMs and Multimodal models that can be launched on-prem or cloud-native.

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

Easily embed, cluster and semantically label text datasets

A simple, performant and scalable Jax LLM!

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

SimPO: Simple Preference Optimization with a Reference-Free Reward

Code examples and resources for DBRX, a large language model developed by Databricks

Official inference library for Mistral models

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Manage scalable open LLM inference endpoints in Slurm clusters

A massively parallel, high-level programming language

Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supportin…

Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.