Kriti Telephony v2
Nepali speech recognition for real phone calls that stays strong on clean speech. 119M parameters, by Naamche Labs. Full recipe, outputs and scorer: github.com/Naamche-Labs/kriti-telephony.
Results
Word error rate (WER), lower is better.
NepTel 0.2: real 8 kHz Nepali phone audio, references corrected by a human against the audio, scored with the benchmark's own score_v02.py.
| System | Decoding | NepTel 0.2 WER |
|---|---|---|
| Kriti Telephony v2 | beam search, width 4 | 32.49 |
| Kriti Telephony v2 | greedy | 33.09 |
| Rani (Ampixa Labs), published output | greedy, no language model | 33.51 |
| Kriti Telephony v1 | greedy | 37.89 |
FLEURS Nepali test (726 clean read sentences, Google, human references; not used in training):
| Model | WER | CER |
|---|---|---|
| Kriti (base) | 24.33 | 8.28 |
| Kriti Telephony v1 | 31.53 | 10.54 |
| Kriti Telephony v2 | 24.77 | 8.81 |
v1 traded clean-speech accuracy for phone accuracy. v2 keeps the phone gain and is back to base-model level on clean speech.
Speed (one CPU thread, Apple M5, batch 1, greedy, mainline NeMo 3.0, the 25 NepTel clips / 622.72 s used in Ampixa's speed table): 40.6 times real time, first text for a 10 s clip in 0.24 s.
Files
| File | Runtime | Notes |
|---|---|---|
kriti_telephony.nemo |
AI4Bharat NeMo fork | v2, the checkpoint behind the numbers above |
kriti_telephony_v2_nemo3.nemo |
mainline NeMo 3.0 | same weights reduced to a plain Nepali RNNT; 33.62 greedy / 33.47 beam on NepTel 0.2 |
v1 is kept at the v1 tag of this repository.
Usage
AI4Bharat fork (exact pins in the GitHub repo's ENVIRONMENT.md):
from huggingface_hub import hf_hub_download
import nemo.collections.asr as nemo_asr
path = hf_hub_download("Aarjan/kriti-telephony", "kriti_telephony.nemo")
model = nemo_asr.models.ASRModel.restore_from(path, strict=False)
model.cur_decoder = "rnnt"
print(model.transcribe(["call.wav"], language_id="ne"))
Mainline NeMo 3.0:
path = hf_hub_download("Aarjan/kriti-telephony", "kriti_telephony_v2_nemo3.nemo")
model = nemo_asr.models.ASRModel.restore_from(path)
print(model.transcribe(["call.wav"]))
Beam search with width 4 (the 32.49 result) uses NeMo's maes decoder; see train/v2/decode_lm.py in the GitHub repo.
How v2 was built
Starting from Kriti Telephony v1, full fine-tune in two rounds:
- Round 1: IndicVoices Nepali (458 h of human-transcribed speech from 1,076 speakers, standard-spelling transcripts), half of it passed through a phone-line simulation (300 to 3400 Hz, 8 kHz, GSM / G.711 / G.726 / Opus-NB codecs), plus the v1 training mix.
- Round 2: about 36 h of extra audio, Nepali parliament speech with its human sentences and Nepali YouTube (YODAS2), labelled by the round-1 model and kept only where labels agreed, 70% phone-simulated. Cosine learning-rate decay.
Checkpoints were chosen on a phone-simulated IndicVoices validation set, not on NepTel. NepTel audio and references were never used for training or checkpoint selection. No Rani weights or Rani outputs were used.
Kriti Telephony v1 was distilled from Ampixa's NepaliConformer offline model (disclosed in METHODOLOGY.md); v2 is initialised from v1.
Limitations
- NepTel is small (78 segments). The paired bootstrap of v2 (beam) against Rani is -1.0 points, 95% interval (-3.0, +1.0).
- The two runtimes differ by about half a point on NepTel; the headline numbers are from the AI4Bharat fork.
- English loanwords in Devanagari spelling variants (for example लिंक / लिङ्क) remain a large share of errors.
License and attribution
MIT for this model. Training data keeps its own licences: IndicVoices (CC-BY-4.0, AI4Bharat), YODAS2 (CC-BY-3.0, ESPnet), Nepali Parliament DS v2 (CC-BY-4.0), OpenSLR-54. NepTel references and scorer by Ampixa Labs (CC-BY-4.0).
- Downloads last month
- 258