Bahasa Indonesia benchmark for AI models (NS-ID v1)
NS-ID is our own Bahasa Indonesia benchmark. It gives each AI model the same 60 tasks, from summarising a news text to writing a formal letter in EYD Edisi V and translating idioms, and scores them the same way. The results below come from our run of October 25, 2026.
Short answer
The top tier, models within 3 points of the best score of 99.3 out of 100, is Kimi K3.1 Pro, Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Grok 4.7, Claude Sonnet 5.5, Qwen3.8 Max (0902), GPT-6 Luna, DeepSeek V4.1 Flash, Perplexity Sonar, DeepSeek V4 Pro and ChatGPT (chat-latest). 16 of 20 models scored 90 or more, so the suite separates weaker models well but not the strongest ones. Writing tasks are model-graded.
Models tested
20
Tasks (public sample)
60 (12)
Run date
October 25, 2026
Run cost
$11.10
Results per model
model-gradedScroll the table sideways to see every column.
| Tier | Model | Total | Comprehension | Formal writing | Register | Translation | Indonesia knowledge | Instructions | Failed tasks | Status |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Kimi K3.1 ProMoonshot AI | 99.3 | 100.0 | 100.0 | 100.0 | 95.5 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | Claude Opus 5.5Anthropic | 99.0 | 98.8 | 97.5 | 98.4 | 99.1 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | GPT-6.1 SolOpenAI | 98.5 | 96.3 | 98.8 | 98.4 | 97.3 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | GPT-6 AstraOpenAI | 98.0 | 95.0 | 98.8 | 96.9 | 97.3 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | Grok 4.7xAI (SpaceXAI) | 98.0 | 97.5 | 97.5 | 100.0 | 92.9 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | Claude Sonnet 5.5Anthropic | 97.6 | 96.3 | 98.8 | 95.3 | 95.5 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | Qwen3.8 Max (0902)Alibaba (Qwen) | 97.6 | 95.0 | 98.8 | 96.9 | 94.6 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | GPT-6 LunaOpenAI | 97.4 | 97.5 | 98.8 | 95.3 | 92.9 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | DeepSeek V4.1 FlashDeepSeek | 97.1 | 93.8 | 97.5 | 96.9 | 94.6 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | Perplexity SonarPerplexity | 97.1 | 95.0 | 97.5 | 100.0 | 90.2 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | DeepSeek V4 ProDeepSeek | 96.8 | 91.3 | 96.3 | 96.9 | 96.4 | 100.0 | 100.0 | 0 | complete model-graded |
| 1 | ChatGPT (chat-latest)OpenAI | 96.4 | 96.3 | 93.8 | 93.8 | 94.6 | 100.0 | 100.0 | 0 | complete model-graded |
| 2 | Gemini 4 ArgonGoogle | 94.6 | 88.8 | 91.3 | 93.8 | 93.8 | 100.0 | 100.0 | 0 | complete model-graded |
| 2 | Claude Haiku 5.5Anthropic | 93.2 | 88.8 | 96.3 | 98.4 | 92.0 | 90.0 | 93.8 | 0 | complete model-graded |
| 2 | Gemini 3.8 FlashGoogle | 93.0 | 82.5 | 90.0 | 89.1 | 96.4 | 100.0 | 100.0 | 0 | complete model-graded |
| 3 | Gemini 3.5 Flash-LiteGoogle | 90.7 | 85.0 | 85.0 | 87.5 | 86.6 | 100.0 | 100.0 | 0 | complete model-graded |
| 3 | Mistral Medium 3.5Mistral AI | 89.6 | 92.5 | 86.3 | 89.1 | 90.2 | 90.0 | 89.6 | 0 | complete model-graded |
| 3 | Llama 4 MaverickMeta | 88.2 | 85.0 | 86.3 | 89.1 | 84.8 | 90.0 | 93.8 | 0 | complete model-graded |
| 3 | Command A+Cohere | 87.8 | 90.0 | 83.8 | 93.8 | 85.7 | 80.0 | 93.8 | 0 | complete model-graded |
| 4 | Mistral Small 4Mistral AI | 84.5 | 87.5 | 87.5 | 85.9 | 84.8 | 80.0 | 81.3 | 0 | complete model-graded |
Model-graded until human raters are appointed
Exact answers and format checks are scored by code. The 27 writing tasks are scored 1 to 5 by AI judges with a public rubric. No human has re-scored a sample yet, so every writing score carries the label model-graded.
Ceiling effect
16 of 20 models scored 90 or more. NS-ID v1 tells weaker models apart well but leaves little room between the strongest ones, and a difference of two or three points can come from normal run-to-run variation. The next version adds harder tasks.
Who judged the writing tasks
Each writing answer was scored by two AI judges from vendors other than the tested model's vendor. When the two differed by more than 1 point, a third judge decided (median of three). Judges never see the model name, vendor, price or sponsorship.
Judge pool: Gemini 3.8 Flash, GPT-6.1 Sol, Mistral Medium 3.5, Claude Sonnet 5.5
Reproducibility check
We ran the full suite a second time on 2 models. Our target is a total score within plus or minus 3 points.
| Model | First run | Rerun | Difference | Within target |
|---|---|---|---|---|
| Claude Haiku 5.5 | 93.19 | 93.30 | +0.11 | yes |
| GPT-6 Luna | 97.40 | 96.99 | -0.41 | yes |
How each model was reached
| Model | Access path | Model version returned by the API | Tested |
|---|---|---|---|
| Kimi K3.1 Pro | OpenRouter intermediary | moonshotai/kimi-k3.1-pro / fpv0_24af6fd9 | October 25, 2026 |
| Claude Opus 5.5 | Anthropic API direct | claude-opus-5-5 | October 25, 2026 |
| GPT-6.1 Sol | OpenAI API direct | gpt-6.1-sol | October 25, 2026 |
| GPT-6 Astra | OpenAI API direct | gpt-6-astra | October 25, 2026 |
| Grok 4.7 | OpenRouter intermediary | x-ai/grok-4.7 | October 25, 2026 |
| Claude Sonnet 5.5 | Anthropic API direct | claude-sonnet-5-5 | October 25, 2026 |
| Qwen3.8 Max (0902) | OpenRouter intermediary | qwen/qwen3.8-max-0902 | October 25, 2026 |
| GPT-6 Luna | OpenAI API direct | gpt-6-luna | October 25, 2026 |
| DeepSeek V4.1 Flash | DeepSeek API direct | deepseek-flash / e72d3c7f663d3e1383b00b3884cb4501 | October 25, 2026 |
| Perplexity Sonar | Perplexity API direct | sonar | October 25, 2026 |
| DeepSeek V4 Pro | DeepSeek API direct | deepseek-v4-pro / a307abda487cd1b463329ccb945ce396 | October 25, 2026 |
| ChatGPT (chat-latest) | OpenAI API direct | chat-latest | October 25, 2026 |
| Gemini 4 Argon | OpenRouter intermediary | google/gemini-4-argon | October 25, 2026 |
| Claude Haiku 5.5 | Anthropic API direct | claude-haiku-5-5 | October 25, 2026 |
| Gemini 3.8 Flash | OpenRouter intermediary | google/gemini-3.8-flash | October 25, 2026 |
| Gemini 3.5 Flash-Lite | OpenRouter intermediary | google/gemini-3.5-flash-lite | October 25, 2026 |
| Mistral Medium 3.5 | OpenRouter intermediary | mistralai/mistral-medium-3-5 | October 25, 2026 |
| Llama 4 Maverick | OpenRouter intermediary | meta-llama/llama-4-maverick | October 25, 2026 |
| Command A+ | Cohere API direct | command-a-plus-05-2026 | October 25, 2026 |
| Mistral Small 4 | OpenRouter intermediary | mistralai/mistral-small-2603 | October 25, 2026 |
Limitations
- Judges differ in strictness. Google models cannot be judged by Gemini, so their writing was scored by the two stricter judges, which likely lowers their writing scores a little. Scores were not adjusted for this.
- Writing scores come from AI judges, which can share blind spots. Two judges from other vendors plus a third on disagreement reduce this but do not remove it.
- 60 tasks is a small suite. A gap of a few points is within normal variation and should not be read as a ranking difference.
- Exact-answer tasks are strict about format. A model that explains its working instead of giving the final answer loses the point.
- Perplexity Sonar searches the web for every answer, as its product does, so it can look up knowledge questions the others answer from memory.
- Gemini and Mistral models were reached through OpenRouter in this run, because our direct Gemini key hit its spending cap and our Mistral key had no request quota.
Public sample tasks
12 of the 60 tasks are public under CC BY 4.0. The other 80% stay private so models cannot be trained on them.
cmp-01: Comprehension, exact answer
Bacalah teks berikut.
Koperasi Tani Makmur di Desa Sukamaju didirikan pada tahun 2015 oleh 24 petani padi. Pada awalnya koperasi hanya menyediakan pupuk dan benih dengan harga grosir bagi anggotanya. Tiga tahun kemudian, koperasi membuka unit penggilingan gabah sehingga anggota tidak perlu lagi menjual gabah kering kepada tengkulak. Pada akhir 2024, jumlah anggota tercatat 180 orang, dan dua pertiga di antaranya adalah perempuan. Rapat anggota tahunan memutuskan bahwa 40 persen sisa hasil usaha dibagikan kepada anggota, 30 persen disimpan sebagai dana cadangan, dan sisanya dipakai untuk pelatihan pertanian organik. Pengurus berharap koperasi dapat mulai menjual beras kemasan ke pasar swalayan di ibu kota kabupaten pada tahun depan.
Pertanyaan: Pada tahun berapa koperasi membuka unit penggilingan gabah?
Tulis jawaban akhir saja, tanpa penjelasan.
cmp-07: Comprehension, writing, rubric 1 to 5
Ringkaslah teks berikut dalam paling banyak dua kalimat bahasa Indonesia baku.
Koperasi Tani Makmur di Desa Sukamaju didirikan pada tahun 2015 oleh 24 petani padi. Pada awalnya koperasi hanya menyediakan pupuk dan benih dengan harga grosir bagi anggotanya. Tiga tahun kemudian, koperasi membuka unit penggilingan gabah sehingga anggota tidak perlu lagi menjual gabah kering kepada tengkulak. Pada akhir 2024, jumlah anggota tercatat 180 orang, dan dua pertiga di antaranya adalah perempuan. Rapat anggota tahunan memutuskan bahwa 40 persen sisa hasil usaha dibagikan kepada anggota, 30 persen disimpan sebagai dana cadangan, dan sisanya dipakai untuk pelatihan pertanian organik. Pengurus berharap koperasi dapat mulai menjual beras kemasan ke pasar swalayan di ibu kota kabupaten pada tahun depan.
eyd-01: Formal writing, exact answer
Apa bentuk baku kata "resiko" menurut KBBI?
Tulis jawaban akhir saja, tanpa penjelasan.
SourceAnswer checked against the source on October 25, 2026
eyd-04: Formal writing, exact answer
Kalimat manakah yang penulisannya sesuai dengan EYD Edisi V?
A. Dia tinggal disebuah rumah di pinggir kota.
B. Dia tinggal di sebuah rumah dipinggir kota.
C. Dia tinggal di sebuah rumah di pinggir kota.
D. Dia tinggal disebuah rumah dipinggir kota.
Jawab hanya dengan satu huruf pilihan (A, B, C, atau D), tanpa penjelasan.
SourceAnswer checked against the source on October 25, 2026
ins-01: Instructions, format checks
Sebutkan tepat 5 nama buah tropis yang tumbuh di Indonesia dalam bentuk daftar bernomor (1. sampai 5.). Jangan menulis kalimat pembuka, kalimat penutup, atau penjelasan apa pun.
ins-02: Instructions, format checks
Kembalikan HANYA sebuah objek JSON yang valid (tanpa blok kode, tanpa teks lain) untuk ide artikel tentang cara menghemat listrik di rumah, dengan kunci persis berikut: "judul" (string), "kata_kunci" (array berisi tepat 3 string), dan "perkiraan_menit_baca" (bilangan bulat).
kno-01: Indonesia knowledge, exact answer
Menurut Keputusan Presiden Nomor 33 Tahun 2009, tanggal berapa ditetapkan sebagai Hari Batik Nasional? Jawab dengan format tanggal dan bulan.
Tulis jawaban akhir saja, tanpa penjelasan.
SourceAnswer checked against the source on October 25, 2026
kno-02: Indonesia knowledge, exact answer
Pasal berapa dalam UUD 1945 yang menyatakan bahwa bahasa negara ialah Bahasa Indonesia? Jawab dengan angka saja.
Tulis jawaban akhir saja, tanpa penjelasan.
SourceAnswer checked against the source on October 25, 2026
reg-03: Register, exact answer
Manakah padanan ragam baku yang paling tepat untuk kalimat "Gue udah kelar ngerjain laporannya"?
A. Saya sudah selesai mengerjakan laporannya.
B. Aku udah selesai mengerjakan laporan.
C. Saya telah kelar menggarap laporan.
D. Gue sudah menyelesaikan laporannya.
Jawab hanya dengan satu huruf pilihan (A, B, C, atau D), tanpa penjelasan.
reg-05: Register, writing, rubric 1 to 5
Ubahlah pesan WhatsApp santai berikut menjadi surel resmi kepada bagian HRD. Pertahankan semua informasinya.
"Halo kak, aku Dinda yg kemarin interview posisi admin gudang. Mau nanya dong hasilnya udah keluar belum ya? Soalnya aku juga lagi ditawarin kerja di tempat lain, jadi butuh kepastian sebelum tgl 15. Makasih banyak kak!"
tr-en-03: Translation, writing, rubric 1 to 5
Terjemahkan kalimat berikut ke bahasa Indonesia yang wajar. Jangan menerjemahkan ungkapannya kata demi kata. Tulis hasil terjemahan saja.
"It's raining cats and dogs, so let's call it a day and finish the rest tomorrow."
tr-id-01: Translation, writing, rubric 1 to 5
Translate the following Indonesian text into natural English. Do not translate the proverb word for word; convey its meaning. Output the translation only.
"Pendiriannya bagai air di daun talas: hari ini setuju, besok sudah berubah lagi."
Questions about NS-ID
What does NS-ID measure?
How well an AI model reads, writes and follows instructions in Bahasa Indonesia: comprehension and summary, formal writing in EYD Edisi V, switching between casual and formal register, translation in both directions, Indonesia-context facts and strict output formats.
Why are most tasks secret?
If every task were public, models could be trained on them and the scores would stop meaning anything. We publish 20% as a sample and the full method, and keep 80% private.
How often is NS-ID run?
On the first Monday of every month, with a hard monthly budget. We aim to review and publish results within 7 days. If the budget would be passed or a provider is down, the run stops and nothing partial is published.