特点

相比于 llama.cpp，vllm 在兼顾性能的同时并发调度上优化更好

安装

Terminal

$vim ~/.bashrc

# 在配置文件最下面添加下列代码 12.8 对应下载的版本号

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.8/lib64

export PATH=$PATH:/usr/local/cuda-12.8/bin

export CUDA_HOME=$CUDA_HOME:/usr/local/cuda-12.8

export PATH=/usr/local/cuda/bin:$PATH

$source ~/.bashrc

$nvcc -V# 检测配置是否成功

nvcc: NVIDIA (R) Cuda compiler driver

Built on Thu_Mar_19_11:12:51_PM_PDT_2026

Cuda compilation tools, release 12.8, V12.8.78

Build cuda_12.8.r12.8/compiler.37668154_0

python 版本最好 3.12 兼容性好

Terminal

$pip isntall uv

$uv venv --seed --python 3.12# 在目标目录生成 vllm 的安装环境

$uv pip install vllm --torch-backend=auto

Terminal

$vim ~/.bashrc

export HF_HOME=/data/my_huggingface_cache

export HF_ENDPOINT=https://hf-mirror.com

$source ~/.bashrc