Instructions to use Qwen/QwQ-32B-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/QwQ-32B-Preview with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/QwQ-32B-Preview")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/QwQ-32B-Preview")
model = AutoModelForCausalLM.from_pretrained("Qwen/QwQ-32B-Preview", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/QwQ-32B-Preview with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/QwQ-32B-Preview"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/QwQ-32B-Preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/QwQ-32B-Preview

SGLang

How to use Qwen/QwQ-32B-Preview with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/QwQ-32B-Preview" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/QwQ-32B-Preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/QwQ-32B-Preview" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/QwQ-32B-Preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/QwQ-32B-Preview with Docker Model Runner:
```
docker model run hf.co/Qwen/QwQ-32B-Preview
```

Remove CoT prompt from chat template

#58

by SumanthRH - opened Jan 22, 2025

base: refs/heads/main

←

from: refs/pr/58

Discussion Files changed

-1

Upload folder using huggingface_hubad2b7ed1

SumanthRH

Jan 22, 2025

•

edited Jan 22, 2025

This PR removes the CoT prompt from the QwQ default system message

The current default system prompt is:

You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step.

While QwQ has been specifically fine-tuned on reasoning tasks that require step-by-step thinking, the default system prompt should not include this because the language model can be used in other settings as well. For example, for basic QA tasks like ARC, MMLU where the model is expected to simply provide the final answer, the "step-by-step" system prompt will be at odds with user instruction.

This PR changes to default to

You are a helpful and harmless assistant. You are Qwen developed by Alibaba.

SumanthRH changed pull request title from Upload folder using huggingface_hub to Remove CoT prompt from chat template Jan 22, 2025

Upload folder using huggingface_hub6f5aea9a

rawgreaze

Feb 4, 2025

"You should think step-by-step unless specified otherwise" seems more flexible imo

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment