Hindi code-switching fix + evaluation data (Gemma 4 12B, 25 prompts)
Sharing results in case they help others using Gemma 4 12B for Hindi.
The problem
Out of the box, Gemma 4 12B writes good, accurate Hindi, but it often adds English in brackets
and gives long markdown answers. Example (asking how the monsoon forms):
तापमान में अंतर (Temperature Difference) … 'निम्न दबाव' (Low Pressure) …
For Hindi-first users (voice apps, education, rural users) the bracketed English is noise.
The fix: a short system prompt
हमेशा केवल शुद्ध हिंदी में उत्तर दें। कोष्ठक में अंग्रेज़ी शब्द न लिखें।
जितना पूछा जाए, ठीक उतना ही दें। Markdown का उपयोग न करें।
Optional third line, so maths answers show a short working step instead of only the final number:
गणित या तर्क के प्रश्नों में अंतिम उत्तर से पहले एक-दो पंक्तियों में हल दिखाएँ।
How I tested it
25 Hindi prompts across 5 categories: facts, reasoning/maths, "give N items" tasks, short writing
(messages, a leave letter, a poem), and everyday questions (UPI, monsoon health, phone battery).
I compared two setups and read every answer:
- Base + system prompt:
gemma-4-12B-it, temperature 1.0 - LoRA fine-tune: the same model fine-tuned on 12.5k Hindi instruction pairs (mostly machine-translated), temperature 0.3, no system prompt
Results
| Base + system prompt | LoRA fine-tune | |
|---|---|---|
| Fully correct (manual review) | 24 / 25 | 20 / 25 |
| English words in brackets | 0 | 0 |
| Requested list counts met | 7/7 | 7/7 |
| Translation artifacts (visarga as a colon, "हैः") | 2 | 45 |
| Average answer length | 219 chars | 628 chars |
The fine-tune also picked up habits from its translated training data: chatbot filler
("नमस्ते! मुझे खुशी है…"), advice to wear a sweater in the Indian monsoon, and expanding UPI as
"Universal Payments Interface".
Things to know
- With the prompt, the base model can be very terse on maths (just "150 किलोमीटर"). The optional third line fixes this.
- Recommended sampling settings (temperature 1.0, top_p 0.95, top_k 64) worked well for the base model with the prompt.
- Tested on 25 prompts only. It's a small evaluation, so please share if you see different behaviour.
Links
- 💬 Try it (chat, with a toggle to switch the system prompt off) + compare all 25 answers: https://huggingface.co/spaces/pankajpandey-dev/gemma-4-12b-hindi-chat
- 📂 All answers + metrics: https://huggingface.co/datasets/pankajpandey-dev/gemma-4-12b-hindi-eval25
Feedback welcome, especially from anyone using Gemma 4 in other Indian languages. 🙏