如何直接使用模型参数启动模型呢

[coyoteXujie]

以 qwen2.5 为例，观察到咱参数都是通过 huggingface 直接拉取的方式来实现的，能否给定模型参数的地址运行此模型
qwen2.5-7b-instruct-l4:
enabled: false
url: "hf://Qwen/Qwen2.5-7B-Instruct"
features: [TextGeneration]
env:
VLLM_ATTENTION_BACKEND: "FLASHINFER"
# VLLM_USE_V1: "1"
args:
- --max-model-len=8192
- --max-num-batched-token=8192
- --max-num-seqs=256
- --gpu-memory-utilization=0.95
- --kv-cache-dtype=fp8
- --enable-prefix-caching
# - --enforce-eager
engine: VLLM
resourceProfile: 'nvidia-gpu-l4:1'

[samos123]

Take a look here: https://www.kubeai.org/how-to/configure-text-generation-models/#insecure-model-pulling

You can configure a custom registry endpoint:

spec:
  url: ollama://my-local-registry:5000/my-model

In case your registry is insecure:

spec:
  url: ollama://my-local-registry:5000/my-model?insecure=true

@coyoteXujie Can you please give it a try and report back if that solved your issue?

[coyoteXujie]

非常感谢您的帮助。但我说的本地参数，指的是从 huggingface 上拉取下来的模型参数，是通过 git clone https://huggingface.co/xxx 得到的

[PeterYang12]

非常感谢您的帮助。但我说的本地参数，指的是从 huggingface 上拉取下来的模型参数，是通过 git clone https://huggingface.co/xxx 得到的

Hi, you can take a look at our community, https://github.com/opea-project/GenAIInfra/blob/main/kubeai/models/qwen2.5-7b-instruct-gaudi.yaml
We provide the Qwen/Qwen2.5-7B-Instruct model profile on Intel Gaudi platform.