nccl

NumPy & SciPy for GPU

翻译 - CUDA加速了类似NumPy的API

Safe rust wrapper around CUDA toolkit

#自然语言处理#An open collection of methodologies to help with successful training of large language models.

#自然语言处理#An open collection of implementation tips, tricks and resources for training large language models

Best practices & guides on how to write distributed pytorch training code

#计算机科学#Distributed and decentralized training framework for PyTorch over graph

翻译 - PyTorch over graph 的分布式和去中心化训练框架

Efficient Distributed GPU Programming for Exascale, an SC/ISC Tutorial

#计算机科学#Federated Learning Utilities and Tools for Experimentation

#计算机科学#NCCL Fast Socket is a transport layer plugin to improve NCCL collective communication performance on Google Cloud.

Sample examples of how to call collective operation functions on multi-GPU environments. A simple example of using broadcast, reduce, allGather, reduceScatter and sendRecv operations.

A Julia wrapper for the NVIDIA Collective Communications Library.

#计算机科学#Python Distributed Non Negative Matrix Factorization with custom clustering

N-Ways to Multi-GPU Programming

#计算机科学#NCCL Examples from Official NVIDIA NCCL Developer Guide.

High performance NCCL plugin for Bagua.

use ncclSend ncclRecv realize ncclSendrecv ncclGather ncclScatter ncclAlltoall

Blink+: Increase GPU group bandwidth by utilizing across tenant NVLink.

Nvidia NCCL2 Python bindings using ctypes and numba.

Experiments with low level communication patterns that are useful for distributed training.

Summary of call graphs and data structures of NVIDIA Collective Communication Library (NCCL)