llm-inference · GitHub Topics

nomic-ai / gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

llm-inference ai-chat

C++ 72.76 k

16 天前

gitleaks / gitleaks

#大语言模型#Gitleaks 是一个开源SAST（静态应用安全测试）命令行工具，用于检测Git 仓库以防止把密码、API 密钥和访问令牌等机密信息硬编码到代码中

安全 Git Go secret gitleaks devsecops Hacktoberfest CI/CD 命令行界面 data-loss-prevention dlp Open Source ai-powered 大语言模型 llm-inference llm-training

Go 19.16 k

1 小时前

liguodongiot / llm-action

#大语言模型#本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

大语言模型 llm-inference llm-serving llm-training llmops

HTML 15.13 k

11 天前

Lightning-AI / litgpt

#大语言模型#20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

人工智能深度学习 large-language-models 大语言模型 llm-inference llms

Python 11.74 k

15 天前

bentoml / OpenLLM

#大语言模型#Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

大语言模型 llmops model-inference fine-tuning llm-serving llama vicuna bentoml llama2 llm-inference llm-ops mistral mlops llama3-1

Python 10.94 k

2 天前

mistralai / mistral-inference

#大语言模型#Official inference library for Mistral models

大语言模型 llm-inference mistralai

Jupyter Notebook 10.06 k

4 个月前

SJTU-IPADS / PowerInfer

#大语言模型#PowerInfer 是一个快速的、可运行在消费级GPU、个人电脑上的大模型服务

large-language-models llama 大语言模型 llm-inference local-inference

C++ 8.15 k

22 天前

openvinotoolkit / openvino

#自然语言处理#OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

翻译 - OpenVINO™工具包存储库

inference 深度学习 openvino 人工智能机器视觉 diffusion-models generative-ai llm-inference 自然语言处理 performance-boost speech-recognition stable-diffusion deploy-ai optimize-ai transformers yolo recommendation-system good-first-issue

C++ 7.96 k

1 小时前

bentoml / BentoML

#大语言模型#The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

翻译 - 轻松进行模型服务

model-serving mlops llmops generative-ai llm-inference 深度学习 llm-serving 机器学习 Python multimodal ml-engineering 大语言模型

Python 7.47 k

21 小时前

InternLM / lmdeploy

#大语言模型#LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

cuda-kernels deepspeed fastertransformer llm-inference turbomind internlm llama 大语言模型 codellama llama2 llama3

Python 5.82 k

19 小时前

superduper-io / superduper

#向量搜索引擎#Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

人工智能 mlops torch transformers MongoDB Python PyTorch 机器学习数据库 data inference llm-inference pretrained-models 聊天机器人 semantic-search llm-serving llmops vector-search rag

Jupyter Notebook 5 k

2 天前

kserve / kserve

#计算机科学#Standardized Serverless ML Inference Platform on Kubernetes

knative 机器学习 model-interpretability model-serving istio kubeflow 人工智能 Tensorflow PyTorch scikit-learn xgboost Kubernetes service-mesh Hacktoberfest mlops genai llm-inference

Python 3.97 k

4 天前

DefTruth / Awesome-LLM-Inference

📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, Flash-Attention, Paged-Attention, Parallelism, etc. 🎉🎉

flash-attention tensorrt-llm vllm llm-inference deepseek deepseek-v3 deepseek-r1

3.62 k

9 天前

neuralmagic / deepsparse

#自然语言处理#Sparsity-aware deep learning inference runtime for CPUs

机器学习 onnx inference 机器视觉 object-detection pruning quantization pretrained-models 自然语言处理 cpus sparsification llm-inference performance

Python 3.11 k

8 个月前

NVIDIA / GenerativeAIExamples

#大语言模型#Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

gpu-acceleration large-language-models 大语言模型 llm-inference 微服务 nemo rag retrieval-augmented-generation tensorrt triton-inference-server

Python 2.89 k

8 天前

FellouAI / eko

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

agent agentic-ai agentic-framework agentic-workflow natural-language-inference workflow rag agents chain-of-thought genai llm-inference llmapi prompt-engineering llm-agents ai-agents browser-automation

TypeScript 2.75 k

2 天前

databricks / dbrx

#大语言模型#Code examples and resources for DBRX, a large language model developed by Databricks

databricks gen-ai generative-ai 大语言模型 llm-inference llm-training mosaic-ai

Python 2.54 k

10 个月前

FasterDecoding / Medusa

#大语言模型#Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

大语言模型 llm-inference

Jupyter Notebook 2.45 k

9 个月前

predibase / lorax

#大语言模型#Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

fine-tuning gpt llama 大语言模型 llm-inference llm-serving llmops lora model-serving PyTorch transformers

Python 2.39 k

6 天前

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving

flash-attention gpu CUDA PyTorch llm-inference jit

Cuda 2.31 k

5 天前