pipeline-parallelism

#计算机科学#一个整合高效并行技术的AI大模型训练系统。

#计算机科学#DeepSpeed Chat: 一键式RLHF训练，让你的类ChatGPT千亿大模型提速省钱15倍

#自然语言处理#🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

#计算机科学#A GPipe implementation in PyTorch

翻译 - PyTorch中的GPipe实施

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train...

#自然语言处理#LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

#计算机科学#Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

#计算机科学#A curated list of awesome projects and papers for distributed training or inference

Serving Inside Pytorch

#计算机科学#Decentralized LLMs fine-tuning and inference with offloading

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

An Efficient Pipelined Data Parallel Approach for Training Large Model

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines.

#自然语言处理#FTPipe and related pipeline model parallelism research.

Implementation of autoregressive language model using improved Transformer and DeepSpeed pipeline parallelism.

#计算机科学#Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks

Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)

Docs for torchpipe: https://github.com/torchpipe/torchpipe