llm-as-a-judge · GitHub Topics

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.

llm-tools prompt-engineering prompt-management llm-evaluation llm-framework rag-evaluation llm-observability llm-as-a-judge llm-monitoring llm-platform llm-playground llmops-platform

Python 2.19 k

1 天前

prometheus-eval / prometheus-eval

#大语言模型#Evaluate your LLM's response with Prometheus and GPT4 💯

evaluation 大语言模型 llmops Python vllm gpt4 llm-as-a-judge

Python 877

2 个月前

metauto-ai / agent-as-a-judge

🤠 Agent-as-a-Judge and DevAI dataset

llm-as-a-judge llms

Python 330

2 个月前

IAAR-Shanghai / xFinder

#大语言模型#[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

evaluation gpt 大语言模型 large-language-models Regular expression reliability benchmark dataset chatglm phi qwen llm-as-a-judge

Python 153

15 天前

martin-wey / CodeUltraFeedback

CodeUltraFeedback: aligning large language models to coding preferences

alignment code-generation dpo large-language-models llm-as-a-judge

Python 69

9 个月前

KID-22 / LLM-IR-Bias-Fairness-Survey

#大语言模型#This is the repo for the survey of Bias and Fairness in IR with LLMs.

bias fairness information-retrieval large-language-models recommender-systems ChatGPT 大语言模型 llm-as-a-judge

14 天前

MJ-Bench / MJ-Bench

Official implementation for "MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?"

llm-as-a-judge

Jupyter Notebook 43

17 天前

zhaochen0110 / Timo

Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)

llm-as-a-judge llms rlhf

Python 20

5 个月前

minnesotanlp / cobbler

#自然语言处理#Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

bias evaluation 大语言模型自然语言处理 bias-detection llm-as-a-judge llm-evaluation llms

Jupyter Notebook 18

1 年前

PKU-ONELab / Themis

The official repository for our EMNLP 2024 paper, Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability.

evaluation llm-as-a-judge nlg

Python 18

18 天前

OussamaSghaier / CuREV

Harnessing Large Language Models for Curated Code Reviews

代码审查 large-language-models llm-as-a-judge

Python 11

22 天前

root-signals / rs-python-sdk

#大语言模型#Root Signals Python SDK

evaluation 大语言模型 llm-as-a-judge observability evals

Python 8

15 天前

UMass-Meta-LLM-Eval / llm_eval

A comprehensive study of the LLM-as-a-judge paradigm in a controlled setup that reveals new results about its strengths and weaknesses.

large-language-models llm-as-a-judge 自然语言处理

Python 8

5 个月前

PKU-ONELab / LLM-evaluator-reliability

The official repository for our ACL 2024 paper: Are LLM-based Evaluators Confusing NLG Quality Criteria?

evaluation llm-as-a-judge nlg

Python 7

18 天前

aws-samples / genai-system-evaluation

A set of examples demonstrating how to evaluate Generative AI augmented systems using traditional information retrieval and LLM-As-A-Judge validation techniques

genai generative-ai information-retrieval llm-as-a-judge llm-evaluation

Jupyter Notebook 7

6 个月前

HillPhelmuth / LlmAsJudgeEvalPlugins

LLM-as-judge evals as Semantic Kernel Plugins

llm-as-a-judge llm-evaluation semantickernel

C# 6

2 个月前

aws-samples / model-as-a-judge-eval

#大语言模型#Notebooks for evaluating LLM based applications using the Model (LLM) as a judge pattern.

evaluation generative-ai 大语言模型 llm-as-a-judge

Jupyter Notebook 3

9 个月前

emory-irlab / conqret-rag

Controversial Questions for Argumentation and Retrieval

hallucination llm-as-a-judge retrieval retrieval-augmented-generation

Python 3

3 个月前

djokester / groqeval

#大语言模型#Use groq for evaluations

generative-ai llama3 大语言模型 llm-as-a-judge mixtral groq

Python 2

8 个月前

romaingrx / llm-as-a-jailbreak-judge

Explore techniques to use small models as jailbreaking judges

jailbreak llm-as-a-judge

Python 2

6 个月前