Назад к репозиториям
Pythonai-dynamo

Каталог репозиториев

ai-dynamo/aiperf

AIPerf — это всесторонний инструмент для бенчмаркинга, который измеряет производительность генеративных моделей ИИ, обслуживаемых вашим предпочтительным решением для вывода.

Открыть GitHub
780
Звёзды GitHub
Python
Язык
ai-dynamo
Автор
авг. 2026 г.
Добавлено

Описание

AIPerf — это всесторонний инструмент для бенчмаркинга, который измеряет производительность генеративных моделей ИИ, обслуживаемых вашим предпочтительным решением для вывода.

README

AIPerf

PyPI version License Codecov Discord Ask DeepWiki

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution. It provides detailed metrics using a command line display as well as extensive benchmark performance reports.

AIPerf UI Dashboard

Quick Start

This quick start guide leverages Ollama via Docker Desktop.

Setting up a Local Server

In order to set up an Ollama server, run granite4:350m using the following commands:

docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama-data:/root/.ollama \
  ollama/ollama:latest
docker exec -it ollama ollama pull granite4:350m

Basic Usage

Create a virtual environment and install AIPerf:

python3 -m venv venv
source venv/bin/activate
pip install aiperf

[!NOTE] The optional crick dependency has no prebuilt wheel for some platforms, including Linux aarch64 (arm64), and falls back to an sdist that needs a C compiler at install time. crick backs the non-default tdigest list-metric backend, so a plain pip install aiperf does not pull it and needs no toolchain. If you want that backend (AIPERF_METRICS_LIST_BACKEND=tdigest) on a platform that builds it from source, install the system build tools first: sudo apt install build-essential (Debian/Ubuntu), sudo yum groupinstall "Development Tools" (RHEL/CentOS), or equivalent. Everywhere a wheel exists, pip install "aiperf[tdigest]" is all you need.

crick 0.0.8 publishes wheels for Linux x86_64 and i686, macOS arm64, and Windows win32 and win_amd64. Everything else builds from source, which includes Linux aarch64, Intel macOS, and Windows on Arm.

Optional integrations:

  • pip install "aiperf[tdigest]" enables the bounded-memory t-digest list-metric backend
  • pip install "aiperf[mlflow]" enables MLflow uploads and live telemetry streaming
  • pip install "aiperf[otel]" enables OpenTelemetry metric streaming
  • pip install "aiperf[wandb]" enables Weights & Biases result uploads
  • pip install "aiperf[mlflow,otel,wandb]" installs all telemetry extras

To run a simple benchmark against your Ollama server:

aiperf profile \
  --model "granite4:350m" \
  --streaming \
  --endpoint-type chat \
  --tokenizer ibm-granite/granite-4.0-micro \
  --url http://localhost:11434 \
  --request-count 10

Example with Custom Configuration

aiperf profile \
  --model "granite4:350m" \
  --streaming \
  --endpoint-type chat \
  --tokenizer ibm-granite/granite-4.0-micro \
  --url http://localhost:11434 \
  --concurrency 5 \
  --request-count 10

Example output:

NOTE: The example performance is reflective of a CPU-only run and does not represent an official benchmark.

                                               NVIDIA AIPerf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃                               Metric ┃       avg ┃      min ┃       max ┃       p99 ┃       p90 ┃       p50 ┃      std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│             Time to First Token (ms) │  7,463.28 │ 7,125.81 │  9,484.24 │  9,295.48 │  7,596.62 │  7,240.23 │   677.23 │
│            Time to Second Token (ms) │     68.73 │    32.01 │    102.86 │    102.55 │     99.80 │     67.37 │    24.95 │
│      Time to First Output Token (ms) │  7,463.28 │ 7,125.81 │  9,484.24 │  9,295.48 │  7,596.62 │  7,240.23 │   677.23 │
│                 Request Latency (ms) │ 13,829.40 │ 9,029.36 │ 27,905.46 │ 27,237.77 │ 21,228.48 │ 11,338.31 │ 5,614.32 │
│             Inter Token Latency (ms) │     65.31 │    53.06 │     81.31 │     81.24 │     80.64 │     63.79 │     9.09 │
│     Output Token Throughput Per User │     15.60 │    12.30 │     18.85 │     18.77 │     18.08 │     15.68 │     2.05 │
│                    (tokens/sec/user) │           │          │           │           │           │           │          │
│      Output Sequence Length (tokens) │     95.20 │    29.00 │    295.00 │    283.12 │    176.20 │     63.00 │    77.08 │
│       Input Sequence Length (tokens) │    550.00 │   550.00 │    550.00 │    550.00 │    550.00 │    550.00 │     0.00 │
│ Output Token Throughput (tokens/sec) │      6.85 │      N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
│    Request Throughput (requests/sec) │      0.07 │      N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
│             Request Count (requests) │     10.00 │      N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
└──────────────────────────────────────┴───────────┴──────────┴───────────┴───────────┴───────────┴───────────┴──────────┘

CLI Command: aiperf profile --model 'granite4:350m' --streaming --endpoint-type 'chat' --tokenizer 'ibm-granite/granite-4.0-micro' --url 'http://localhost:11434'
Benchmark Duration: 138.89 sec
CSV Export: /home/user/aiperf/artifacts/granite4:350m-openai-chat-concurrency1/profile_export_aiperf.csv
JSON Export: /home/user/Code/aiperf/artifacts/granite4:350m-openai-chat-concurrency1/profile_export_aiperf.json
Log File: /home/user/Code/aiperf/artifacts/granite4:350m-openai-chat-concurrency1/logs/aiperf.log

Features

  • Scalable multiprocess architecture with 10 services communicating via ZMQ
  • 3 UI modes: dashboard (real-time TUI), simple (progress bars), none (headless)
  • Multiple benchmarking modes: concurrency, request-rate, request-rate with max concurrency, trace replay
  • Extensible plugin system for endpoints, datasets, transports, and metrics
  • Public dataset support including ShareGPT and custom formats

Supported APIs

  • OpenAI chat completions, completions, embeddings, audio, images
  • NIM embeddings, rankings

Tutorials and Feature Guides

Getting Started

Load Control and Timing

Workloads and Data

Endpoint Types

Kubernetes

Analysis and Monitoring

Documentation

DocumentPurpose
ArchitectureThree-plane architecture, core components, credit system, data flow
CLI OptionsComplete command and option reference
Metrics ReferenceAll metric definitions, formulas, and requirements
Environment VariablesAll AIPERF_* configuration variables
Plugin SystemPlugin architecture, 25+ categories, creation guide
Creating PluginsStep-by-step plugin tutorial
Accuracy BenchmarksAccuracy evaluation against MMLU, AIME, and other benchmarks
Benchmark ModesTrace replay and timing modes
Server MetricsPrometheus-compatible server metrics collection
Tokenizer Auto-DetectionPre-flight tokenizer detection
Conversation Context ModeHow conversation history accumulates in multi-turn
Dataset Synthesis APISynthesis module API reference
Code PatternsCode examples for services, models, messages, plugins
Migrating from Genai-PerfMigration guide and feature comparison
Design ProposalsEnhancement proposals and discussions

Contributing

See CONTRIBUTING.md for development setup, coding conventions, and contribution guidelines.

License

AIPerf is licensed under the Apache License 2.0. Because helm package only includes files beneath the chart directory, the aiperf-operator Helm chart contains a symlink to the repository's top-level license solely so the license is included in the packaged chart archive. Helm dereferences the symlink at packaging time, so the archive contains a regular LICENSE file with the top-level license contents.

Known Issues

  • Output sequence length constraints (--output-tokens-mean) cannot be guaranteed unless you pass ignore_eos and/or min_tokens via --extra-inputs to an inference server that supports them.
  • Very high concurrency settings (typically >15,000) may lead to port exhaustion on some systems. Adjust system limits or reduce concurrency if connection failures occur.
  • Startup errors caused by invalid configuration settings can cause AIPerf to hang indefinitely. Terminate the process and check configuration settings.
  • Copying selected text may not work reliably in the dashboard UI. Use the c key to copy all logs.