NS-ID methodology (Bahasa Indonesia benchmark)

NS-ID is Newly Stack's own test of how well AI models handle Bahasa Indonesia. We run it ourselves, with API keys we pay for, and we publish how it works so anyone can check the method. Payment, sponsorship, affiliate deals and listing status never change who is tested or what score a model gets.

Which models we test

How we pick the 20 models for NS-ID v1: (1) We rank text AI assistant brands by the combined monthly search volume of every keyword that names the brand in our two keyword sets (global English and Indonesia), using bot-spike corrected volumes. (2) We keep only brands we can test through an official API with keys we already hold; brands without an API, such as GitHub Copilot, wait for manual testing in phase 2. (3) For each kept brand we test the model behind or closest to its consumer app, and, for brands that offer several current API tiers (in v1: ChatGPT, Gemini, Claude, DeepSeek and Mistral), also their flagship or low-cost tiers, until we reach 20 models. (4) Remaining slots go to other vendors whose API we can already reach. Payment, sponsorship and affiliate deals never affect selection or scores. The list is reviewed before every monthly run.

NS-ID v1 tests 20 models from 11 vendors. Two brands with large search interest are not tested yet: GitHub Copilot and Microsoft Copilot have no public text API for their consumer products, so they wait for manual testing in phase 2.

How each model was reached in this run:

• Direct vendor APIs: OpenAI (4 models), Anthropic (3), DeepSeek (2), Perplexity (1), Cohere (1).
• Through OpenRouter: Meta Llama 4 Maverick, Grok 4.7, Kimi K3.1 Pro and Qwen3.8 Max (no direct key), plus the three Gemini models and the two Mistral models. On 25 October 2026 our direct Gemini key answered "monthly spending cap exceeded" and our direct Mistral key had a request quota of zero per minute, so the same models were called through OpenRouter, which forwards to Google and Mistral as providers. The model version string returned by each API is stored with the results.

What the suite contains

60 tasks in six categories, all in Bahasa Indonesia (translation tasks also use English):

• 48 tasks (80%) stay private so models cannot be trained on them. 12 tasks (20%) are published as a sample under CC BY 4.0.
• Every knowledge answer was checked against an official source, mostly the original text of Indonesian laws on peraturan.go.id and KBBI Daring, with the URL and check date stored per task.
• Spelling and standard words follow EYD Edisi V and KBBI Daring.
• Reference answers were written for this suite by the Newly Stack editorial team with AI assistance, not copied from any tested model's output. A human review of every reference is planned before v2.

How scores are calculated

• Exact answers (25 tasks) are scored automatically: correct is 100, wrong is 0. Answers are normalised before comparison (case, punctuation, citation markers such as [1], number formats such as 1.250). The prompt asks for the final answer only; an answer that shows its working instead of a final answer counts as wrong.
• Instruction tasks (8 tasks) are scored by programmatic checks, such as exactly five list items, valid JSON with the requested keys, a word range, all capital letters, or a 4-line pantun with an a-b-a-b rhyme. The score is the share of checks passed.
• Writing tasks (27 tasks) are scored 1 to 5 with a public rubric (rubric.md). Each writing task runs 3 times and the median counts. Scores map to 0 to 100.
• Category score = mean of its tasks. Total score (0 to 100) = mean of the six category scores, so each category weighs the same.
• Failed tasks (API error after 5 attempts, empty output, or no judge verdict) score 0. A model with more than 5% failed tasks (more than 3 of 60) is marked incomplete.

Judges

• Each writing output is scored by two AI judges from two vendors that differ from the tested model's vendor. Judges see the task, rubric, criteria, reference answer and anonymous outputs only, never the model name, vendor, price or sponsorship.
• If the two judges differ by more than 1 point, a third judge from a third vendor scores that output and the median of three counts. Otherwise the mean of the two counts.
• Judge pool in v1: Gemini 3.8 Flash (Google), GPT-6.1 Sol (OpenAI), Claude Sonnet 5.5 (Anthropic) and Mistral Medium 3.5 (Mistral AI). Default pair: Gemini 3.8 Flash and GPT-6.1 Sol. OpenAI models are judged by Gemini 3.8 Flash and Claude Sonnet 5.5; Google models by GPT-6.1 Sol and Claude Sonnet 5.5. The third judge is the cheapest remaining eligible judge (Mistral Medium 3.5, or Claude Sonnet 5.5 when a Mistral model is tested).
• Judges run with low reasoning settings to keep cost predictable (Gemini reasoning effort low, GPT-6.1 Sol reasoning effort low, Claude Sonnet 5.5 without extended thinking). DeepSeek V4 Pro was tried as a judge in a pilot and dropped: its default thinking used the whole 6,000-token budget without returning a verdict.
• In this run the judges scored 1,620 writing outputs; the two primary judges differed by more than 1 point on 30 of them (1.9%), which triggered a third judge. Judge cost: USD 4.25.
• All writing scores carry the label "model-graded". Human review of a 10% sample starts when Newly Stack appoints raters.

Settings and reproducibility

• Temperature 0 is requested on every API that accepts it. The OpenAI and Anthropic APIs reject a non-default temperature for the tested models, so their vendor default is used. The setting actually used is stored per model.
• Models run with their vendor's default reasoning settings, close to how the consumer apps behave. Maximum output is 6,000 tokens per answer.
• Stored per run: suite version and its SHA-256 hash, requested model ID, model version string returned by the API, route, temperature used, date and time, every raw output with its SHA-256 hash, token counts and cost.
• Before publication we corrected the automatic scorer (version nsid-v1-auto-1.1: Indonesian end rhyme for pantun is now the last vowel plus trailing consonants, answers written as LaTeX fractions or the character for two thirds are read as 2/3, and short framing up to 12 words is accepted). All automatic scores were recomputed from the stored raw outputs without new API calls; 11 task scores changed and each change is logged in the results file.
• Reproducibility test: the full suite was run a second time on two low-cost models. Our target is a total score within plus or minus 3 points. Result of the October 2026 rerun: GPT-6 Luna: 97.4 then 96.99 (delta -0.41); Claude Haiku 5.5: 93.19 then 93.3 (delta +0.11). Both stayed within the plus or minus 3 point target.

Budget

• Test costs are paid with API keys Newly Stack already holds. Monthly cap: USD 50 across all model testing (this benchmark plus the practical test lab), with USD 30 reserved for NS-ID.
• Before every run the runner projects the full cost from measured token usage per model, plus a 15% margin, and checks it against the costs already recorded this month in a shared cost ledger. If month-to-date spend plus the projection would pass either cap, the run stops with status budget_exceeded and publishes nothing. The guard also stops a run mid-way if actual spend reaches a cap; partial scores are never published.
• October 2026 spend for NS-ID (pilot, main run, rerun and speed runs): USD 13.72 of the USD 30 NS-ID budget. Main run: USD 11.10 (models USD 6.85, judges USD 4.25), projected beforehand at USD 26.80. All model testing this month, including the test lab: USD 30.58 of the USD 50 cap.

Independence

• Scores are computed only from task definitions and model outputs. The scoring code has no input for sponsorship, plan, affiliate deal or ownership; an automated test proves the score is identical with a sponsor flag switched on or off.
• Newly Stack's own apps are labelled as ours wherever they appear. Sponsored placements are labelled "Sponsored" and never change benchmark scores or order.

Schedule

• NS-ID runs on the first Monday of every month. We aim to review and publish results within 7 days of the run.
• The model list is reviewed before every run with the selection rule above. Prices come from each vendor's official pricing page with the check date shown.

Limitations

• Ceiling effect: most models scored above 90. The v1 suite separates weaker models well but leaves little room between the strongest ones; the next version will add harder tasks.
• Judges differ in strictness. On the same outputs, Gemini 3.8 Flash scored on average 0.17 points (1 to 5 scale) higher than GPT-6.1 Sol and 0.34 points higher than Claude Sonnet 5.5. Google models cannot be judged by Gemini, so their writing tasks were scored by the two stricter judges, which likely lowers their writing scores compared with the other models. Scores were not adjusted for this in v1.
• The Cohere API does not return a model version string, so only the requested model ID is recorded for Command A+.
• Writing scores come from AI judges, which can share blind spots. Using two vendors other than the tested model's vendor, plus a third judge on disagreement, reduces but does not remove this.
• 60 tasks is a small suite. Differences of a few points between models are within normal run-to-run variation and should not be read as a ranking gap.
• Exact-answer tasks are strict about format. A model that explains its reasoning instead of giving only the final answer loses the point, which measures instruction following as much as knowledge.
• Perplexity Sonar searches the web for every answer, as its product does. It can look up knowledge questions that other models answer from memory.
• Models whose API does not accept temperature 0 vary more between runs.
• Cohere does not publish a per-token price for Command A+ (checked 25 October 2026), so its cost is estimated with an assumed price for budget control only and no price is shown for it.
• DeepSeek costs are counted at peak-hour prices, which can overstate the actual bill.
• Gemini and Mistral models were reached through OpenRouter in this run, which can add latency but should not change answers.
• Speed figures include the network distance from where they are measured; each speed table states its location. The October 2026 run was measured from a laptop in Indonesia on a Jakarta consumer internet connection; later scheduled runs may come from our app server, and the table then says so.

Related work

Other public evaluations that cover Indonesian, checked on 25 October 2026. They measure different things from NS-ID, and we link them so you can compare.

• SEA-HELM by AI Singapore: a holistic evaluation suite for Southeast Asian languages, including Indonesian, described in the paper SEA-HELM: Southeast Asian Holistic Evaluation of Language Models.
• IndoMMLU: 14,981 multiple-choice questions from Indonesian school and university entrance exams across 64 tasks (paper).
• Artificial Analysis multilingual benchmark: compares leading models by language, Indonesian among them.