NE-Embed

Multilingual sentence embeddings for Northeast India's low-resource languages.

---

No publicly available embedding model produced reliable representations for languages like Garo, Khasi, Nyishi, or Kokborok, until now. NE-Embed is a LaBSE fine-tune trained on 201,738 English-to-NE-language parallel pairs across 10 languages, producing 768-dim L2-normalized embeddings for cross-lingual retrieval.

Install

pip install ne-embed

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MWirelabs/ne-embed")
embeddings = model.encode(["Khasi sentence", "English sentence"])

Results

R@1 on a 6,000-vector mixed-language FAISS retrieval index (500/language), vs. zero-shot LaBSE:

Language	LaBSE (zs)	NE-Embed	Gain
Bodo	55.8	99.8	+44.0
Assamese	95.6	97.4	+1.8
Khasi	28.6	95.6	+67.0
Garo	13.2	90.8	+77.6
Mizo	46.6	91.8	+45.2
Nagamese	77.0	88.0	+11.0
Pnar	27.2	86.2	+59.0
Kokborok	26.4	71.6	+45.2
Nyishi	10.2	75.0	+64.8
Meitei	13.6	34.2	+20.6

Also cuts cross-language retrieval interference (CLRI) sharply, e.g. Garo from 88.8% to 3.0%, Khasi from 65.0% to 3.4%.

Training

• Base: sentence-transformers/LaBSE
• Loss: MultipleNegativesRankingLoss
• Data: AI4Bharat BPCC (Assamese, Bodo, Meitei) + MWire Labs WMT 2026 corpus (Garo, Khasi, Mizo, Nyishi, Kokborok, Pnar, Nagamese)
• Max sequence length: 128 tokens

Used in

NE-Agent — retrieval-augmented multilingual terminal agent

Citation

@misc{nyalang2026neembed,
  title  = {NE-Embed: Multilingual Sentence Embeddings for Northeast Indian Languages},
  author = {Nyalang, Badal},
  year   = {2026},
  note   = {MWire Labs},
  url    = {https://huggingface.co/MWirelabs/ne-embed}
}

License

CC-BY-4.0, MWire Labs, Shillong, Meghalaya.