The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs Using Indian Riddles

This is the official repository for the paper "The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs Using Indian Riddles."

This repository contains the curated datasets and prompt templates used to test the reasoning capabilities and self-awareness of Large Language Models (LLMs) across seven major Indian languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Tamil, and Telugu.

🗂️ Repository Structure

Based on our three-phase methodology, the repository is organized as follows:

• data/: Contains the core multilingual riddle datasets.
  • 96_riddles/: .tsv files containing the 96 curated traditional original riddles and their answers for each of the 7 Indian languages.
  • context_reconstructed_riddles/: .json files containing the high-quality, human-curated context-reconstructed variants used for our context-reconstructed few-shot prompting.
  • semantic_similar_riddles/: Datasets organized for the semantic similarity few-shot prompting experiments.
• prompts/: Contains the exact prompt templates used for evaluating the models.
  • Language-specific directories (e.g., bengali, gujarati, etc.) housing the 0-shot, random few-shot, semantic few-shot, and context-reconstructed few-shot prompts.
  • yes_or_no_prompts/: The single-token output prompts used for the Phase 3 Self-Evaluation task.

📊 About the Task

Solving riddles requires complex multi-step commonsense reasoning, metaphorical interpretation, and deep cultural knowledge. This project evaluates models not only on their ability to generate the correct answer (Riddle-Solving) but also on their ability to correctly identify if their own generated answer was right or wrong (Self-Awareness).

Key Finding: Our evaluation reveals a crucial "Self-Awareness Paradox" in modern LLMs—top-performing models (like Gemini 2.5 Pro) are highly overconfident and fail to recognize their own mistakes, whereas lower-performing models exhibit substantially better self-awareness.

📝 Citation

If you use this dataset or code in your research, please consider citing our paper:

@inproceedings{p-m-etal-2026-riddle,
    title = "The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual {LLM}s Using {I}ndian Riddles",
    author = "P M, Abhinav  and
      Saxena, Ojasva  and
      C, Oswald  and
      Krishnamurthy, Parameswari",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    booktitle = "Proceedings of the Fifteenth Language Resources and Evaluation Conference",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://aclanthology.org/2026.lrec-1.430/",
    doi = "10.63317/2pgfbjkdofoe",
    pages = "5516--5527",
    abstract = "The extent to which large language models (LLMs) can perform culturally grounded reasoning across non-English languages remains underexplored. This paper examines the reasoning and self-assessment abilities of LLMs across seven major Indian languages- Bengali, Gujarati, Hindi, Kannada, Malayalam, Tamil, and Telugu. We introduce a multilingual riddle dataset combining traditional riddles with context-reconstructed variants and evaluate five LLMs- Gemini 2.5 Pro, Gemini 2.5 Flash, Mistral-Saba, LLaMA-4-Scout, and LLaMA-4-Maverick under seven prompting strategies. In the first stage, we assess riddle-solving performance and find that while Gemini 2.5 Pro performs best overall, few-shot methods yield only marginal gains, and accuracy varies notably across languages. In the second stage, we conduct a self-evaluation experiment to measure reasoning consistency. The results reveal a key finding: a model{'}s initial accuracy is inversely correlated with its ability to identify its own mistakes. Top-performing models such as Gemini 2.5 Pro are overconfident (4.34{\%} True Negative Rate), whereas lower-performing models like LLaMA-4-Scout are substantially more self-aware (42.09{\%} True Negative Rate). These results point to clear gaps in multilingual reasoning and highlight the need for models that not only reason effectively but also recognize their own limitations."
}