llm-serving · GitHub Topics

#大语言模型#A high-throughput and memory-efficient inference and serving engine for LLMs

gpt 大语言模型 PyTorch llmops mlops model-serving transformer llm-serving inference llama amd rocm CUDA inferentia trainium tpu xpu hpu deepseek qwen

Python 41.24 k

25 分钟前

ray-project / ray

#计算机科学#Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

翻译 - 一个快速简单的框架，用于构建和运行分布式应用程序。 Ray与RLlib（可扩展的强化学习库）和Tune（可扩展的超参数调整库）打包在一起。

Python 35.92 k

19 小时前

liguodongiot / llm-action

#大语言模型#本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

大语言模型 llm-inference llm-serving llm-training llmops

HTML 15.13 k

11 天前

sgl-project / sglang

#大语言模型#SGLang is a fast serving framework for large language models and vision language models.

CUDA inference llama llava 大语言模型 llm-serving moe PyTorch transformer vlm llama3 llama3-1 deepseek deepseek-llm deepseek-v3 deepseek-r1 deepseek-r1-zero

Python 11.8 k

1 小时前

bentoml / OpenLLM

#大语言模型#Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

大语言模型 llmops model-inference fine-tuning llm-serving llama vicuna bentoml llama2 llm-inference llm-ops mistral mlops llama3-1

Python 10.94 k

2 天前

skypilot-org / skypilot

#计算机科学#SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 14+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

Python 7.48 k

2 天前

bentoml / BentoML

#大语言模型#The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

翻译 - 轻松进行模型服务

model-serving mlops llmops generative-ai llm-inference 深度学习 llm-serving 机器学习 Python multimodal ml-engineering 大语言模型

Python 7.47 k

1 天前

superduper-io / superduper

#向量搜索引擎#Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

人工智能 mlops torch transformers MongoDB Python PyTorch 机器学习数据库 data inference llm-inference pretrained-models 聊天机器人 semantic-search llm-serving llmops vector-search rag

Jupyter Notebook 5 k

2 天前

predibase / lorax

#大语言模型#Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

fine-tuning gpt llama 大语言模型 llm-inference llm-serving llmops lora model-serving PyTorch transformers

Python 2.39 k

6 天前

microsoft / aici

#大语言模型#AICI: Prompts as (Wasm) Programs

人工智能 Rust WebAssembly wasmtime inference language-model 大语言模型 llm-framework llm-inference llm-serving llmops model-serving transformer

Rust 2 k

2 个月前

MoonshotAI / MoBA

#大语言模型#MoBA: Mixture of Block Attention for Long-Context LLMs

flash-attention 大语言模型 llm-serving llm-training moe PyTorch transformer

Python 1.64 k

6 天前

ray-project / ray-llm

#大语言模型#RayLLM - LLMs on Ray

distributed-systems large-language-models ray serving transformers 大语言模型 llm-inference llm-serving llmops

Python 1.26 k

9 个月前

zhihu / ZhiLight

#大语言模型#A highly optimized LLM inference acceleration engine for Llama and its variants.

inference-engine 大语言模型 CUDA gpt llama llm-serving PyTorch llm-inference model-serving deepseek-r1

C++ 872

10 天前

mosecorg / mosec

#大语言模型#A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

model-serving 深度学习机器学习 nerual-network mlops Hacktoberfest gpu Python PyTorch Tensorflow 大语言模型 jax llm-serving Rust cv mxnet tts

Python 827

3 天前

efeslab / Nanoflow

#大语言模型#A throughput-oriented high-performance serving framework for LLMs

CUDA inference llama2 大语言模型 llm-serving model-serving

Cuda 749

6 个月前

alibaba / rtp-llm

#大语言模型#RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

gpt inference llama 大语言模型 llm-serving llmops model-serving

C++ 640

2 个月前

rohan-paul / LLM-FineTuning-Large-Language-Models

#大语言模型#LLM (Large Language Model) FineTuning

gpt-3 gpt3-turbo large-language-models llama2 大语言模型 llm-inference llm-serving llm-training mistral-7b PyTorch

Jupyter Notebook 512

10 个月前

hpcaitech / SwiftInfer

#计算机科学#Efficient AI Inference & Serving

人工智能深度学习 gpt inference llama llama2 llm-inference llm-serving

Python 467

1 年前

helixml / helix

#大语言模型#🧬 Helix is a private GenAI stack for building AI applications with declarative pipelines, knowledge (RAG), API bindings, and first-class testing.

Go llama 大语言模型 mistral openai 自托管 mixtral sdxl stable-diffusion API finetuning function-calling llm-agent llm-serving OpenAPI Specification rag swagger helm Kubernetes

Go 435

7 天前

ray-project / ray-educational-materials

#大语言模型#This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.

深度学习 ray generative-ai 大语言模型 llm-inference llm-serving

Jupyter Notebook 378

1 年前