Suggestion: a Text-to-SQL task (English / MSA / Gulf dialect)

#34
by usmanbilytica - opened

Hi OALL team, thank you for the leaderboard. It has become the reference point for Arabic LLMs in the region.

Suggestion: add a Text-to-SQL task. In Gulf enterprises and government, one of the most common LLM use cases is asking a database a question in Arabic, and current Arabic suites don't measure it.

To make this easy to try, I published a small open benchmark:

Dataset: https://huggingface.co/datasets/usmanbilytica/gulf-text-to-sql-bench
Code: https://github.com/usmanbilytica/arabic-text-to-sql-bench
Live leaderboard: https://www.kaggle.com/benchmarks/usmanahmad/gulf-text-to-sql-bench

  • 30 questions, each written in English, Modern Standard Arabic and Gulf dialect (90 items), over a deterministic SQLite database of Saudi public-sector data
  • Scored by execution accuracy (result rows compared, not SQL strings), so no judge model is needed
  • MIT code, CC BY 4.0 data

First results show why the dialect split matters. gpt-oss-120b scores 100% in English but 86.7% in MSA and 76.7% in Gulf dialect. The leading proprietary models (Gemini 4 Argon, GPT-6.1 Sol) stay between 96% and 100% on all three.

Happy to adapt it to your harness (lighteval task, prompt template, splits) or grow it to a few hundred questions if useful.

Usman Ahmad, Bilytica (Riyadh)

Sign up or log in to comment