Instructions to use Qwen/QwQ-32B-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/QwQ-32B-Preview with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/QwQ-32B-Preview")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/QwQ-32B-Preview")
model = AutoModelForCausalLM.from_pretrained("Qwen/QwQ-32B-Preview", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/QwQ-32B-Preview with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/QwQ-32B-Preview"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/QwQ-32B-Preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/QwQ-32B-Preview

SGLang

How to use Qwen/QwQ-32B-Preview with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/QwQ-32B-Preview" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/QwQ-32B-Preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/QwQ-32B-Preview" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/QwQ-32B-Preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/QwQ-32B-Preview with Docker Model Runner:
```
docker model run hf.co/Qwen/QwQ-32B-Preview
```

Adds Chinese characters to responses

#16

by maxbenk - opened Nov 28, 2024

Discussion

maxbenk

Nov 28, 2024

Adds Chinese characters to responses

Вот один интересный факт: знаете ли вы, что в Японии существует особый вид искусства, называемый «кинуть кимоно»? Это древнее искусство складывания и ношения кимоно, которые являются традиционными японскими нарядами. Кимоно имеют сложную систему складок и узлов, и чтобы правильно их носить, требуется много практики и мастерства. Интересно, не правда ли?桷

here is an example, I added this symbol 桷

How can I make the message appear strictly in Russian?

jukofyork

Nov 28, 2024

Try setting min_p = 0.01 - this used to work well for the qwen-1.5 models and better than setting temperature = 0 as that causes looping.

maxbenk

Nov 28, 2024

unfortunately it didn't help.
this is the answer I received:

Привет! Рад видеть тебя здесь. Как我可以帮助你？如果有任何问题或需要 assistance，请随时告诉我。

imoc

Nov 29, 2024

This is a model problem. The dataset is too mixed up. I'm also getting random single character inside reply, albeit it is same meaning as English.

BenevolenceMessiah

Dec 3, 2024

This is a model problem. The dataset is too mixed up. I'm also getting random single character inside reply, albeit it is same meaning as English.

Thank you, this is actually very helpful to know.

BenevolenceMessiah

Dec 3, 2024

Adds Chinese characters to responses

Вот один интересный факт: знаете ли вы, что в Японии существует особый вид искусства, называемый «кинуть кимоно»? Это древнее искусство складывания и ношения кимоно, которые являются традиционными японскими нарядами. Кимоно имеют сложную систему складок и узлов, и чтобы правильно их носить, требуется много практики и мастерства. Интересно, не правда ли?桷

here is an example, I added this symbol 桷

How can I make the message appear strictly in Russian?

Have you tweaked the system prompt to something like: only output answers in the Russian language, do not output any responses in any other language unless the conversation context requires it. ?

maxbenk

Dec 3, 2024

Yes, I changed it, but it didn't work.
This is how I get responses like this:

Конечно! Вот один забавный анекдот:

Почему программисты喜欢住在高楼里?

Потому что у них есть elevate к offices!

Надеюсь, вам понравилось! 😄如果你想要更多的笑话,随时告诉我。

maxbenk

Dec 3, 2024

In this chat, this model works excellently and understands Russian perfectly.
https://huggingface.co/chat/models/Qwen/QwQ-32B-Preview

alexblack2202

Mar 15, 2025

In this chat, this model works excellently and understands Russian perfectly.
https://huggingface.co/chat/models/Qwen/QwQ-32B-Preview

i have same question, i write in vietnamese and model return some china character, but it not appear in hugging face chat , do you know how to fix this

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Adds Chinese characters to responses

Adds Chinese characters to responses

unfortunately it didn't help.this is the answer I received:

Привет! Рад видеть тебя здесь. Как我可以帮助你？如果有任何问题或需要 assistance，请随时告诉我。

Adds Chinese characters to responses

Yes, I changed it, but it didn't work.This is how I get responses like this:

Надеюсь, вам понравилось! 😄如果你想要更多的笑话,随时告诉我。

unfortunately it didn't help.
this is the answer I received:

Yes, I changed it, but it didn't work.
This is how I get responses like this: