IndicBankBench: Assessing the Safety and Reliability of Language Model Assistants for Indian Retail Banking

Meta description: IndicBankBench tests retail banking assistants using 799 synthetic cases across 20 axes, checking safety, tool use, confirmation, and advisory quality.

Tags: IndicBankBench, Banking AI, AI Benchmarks, Tool Use, Evaluation

Featured image: Abstract technology style, no real people, no company logos, no text, showing a digital banking workflow divided into verification, tool evidence, confirmation, and advisory-quality checkpoints.

Background: Why final-answer checks can miss banking failures

IndicBankBench is a benchmark for retail banking assistants that moves evaluation beyond the final customer reply. The problem it targets is specific: an assistant can produce a fluent, plausible answer while still failing the operational steps that make a banking interaction safe. It might look at the wrong account, rely on stale information, perform an action without confirmation, or satisfy a request that the system cannot support. In a mock banking environment, those process failures can be observed directly, rather than being hidden by a polished response.

This matters because retail banking requests often connect language understanding with account state, tool calls, and customer consent. A correct-looking answer does not prove that the assistant found the right account, refreshed the evidence used to decide, or paused before an action. The benchmark therefore asks whether the assistant follows a safe path through a simulated customer case.

Development: 799 cases built around 20 primary axes

The released set contains 799 synthetic retail-banking cases organized across 20 primary axes. The cases are designed to test several distinct behaviors. An assistant must identify the right account, use fresh tool evidence, obtain confirmation before taking an action, and decline a request that lacks sufficient support.

Because the cases are synthetic, the benchmark evaluates behavior in a controlled setting rather than replaying a specific real customer record. The 20 axes provide a structured way to examine different failure modes. The mock banking environment gives evaluators a place to check whether the assistant's steps line up with the situation, not just whether its final sentence sounds reasonable.

Latest: Ordered evaluation and repeated runs

The evaluator checks four dimensions in sequence: safety, action/tool use, response adequacy, and advisory quality. The order is important. If an assistant fails an earlier step, the evaluation can locate the point where the case went wrong instead of treating the transcript as a single pass or fail output.

Each case is run three times. This repetition is meant to separate occasional success from reliable behavior. A model that passes one run may still be unstable on confirmation, tool evidence, or refusal decisions, so the benchmark looks for consistency across multiple attempts.

The project also invites feedback on case design and grading rules. The supporting materials are published through a paper at https://arxiv.org/abs/2609.29167, code at https://github.com/npci/IndicBankBench, and a dataset at https://huggingface.co/datasets/NPCI/IndicBankBench.

What the benchmark makes visible

IndicBankBench shifts the question from "did the assistant say the right thing?" to "did it do the right thing under banking-like constraints?" The 799 cases, 20 primary axes, and three-run structure make that shift concrete. For readers tracking AI assistants, the key takeaway is that evaluation can now focus on account selection, fresh evidence, confirmation, and refusal behavior as measurable steps.

The next signal to watch is how feedback changes the case design and grading rules. As the benchmark is examined and extended, the value will come from whether its ordered checks and repeated runs expose reliability gaps that single-answer evaluations can miss.