Introducing the Telenor AI Bench: A New Standard for Trust and Transparency
Published
Currently, the industry relies on generic, often academic benchmarks to evaluate AI models. While useful, these benchmarks rarely reflect the specific challenges of Telenor and our region - such as understanding Nordic languages -or the complex, real-world use cases we face every day in the telecommunications industry. So, how do we ensure that the AI models we use are not just powerful, but also the rightones for the job? How do we compare them in a way that is scientific, transparent, and relevant to our unique needs?
These are the questions that led to the creation of the Telenor AI Bench.
With Telenor AI Bench we have created a suite of standardized, in-house evaluation tools designed to test AI models on tasks that truly matter to us and our customers. It’s not just about measuring accuracy; it’s about measuring performance in a context that is meaningful for Telenor.
From ensuring that a customer service bot understands a local dialect (ASR) to testing a model's resilience against malicious inputs (Jailbreak defense) and its ability to retrieve accurate information (RAG), the AI Bench aims to provide a comprehensive and realistic assessment.
A sneak peek on our AI Bench
Why the AI Bench Matters
The importance of the AI Bench extends beyond internal decision-making. It represents our commitment to a responsible and transparent approach to AI. By creating our own standards, we can:
- Make Informed Choices: We can objectively compare different models, whether built in-house or from external vendors, and select the best one for a specific task.
- Ensure Accountability: We can clearly document why a certain model was chosen and demonstrate its performance based on our own testing.
- Drive the Industry Forward: By sharing our methodologies in peer-reviewed research papers, we aim to inspire a move towards more domain-specific and transparent evaluation standards across the industry.
We believe that the combination of our internal AI competences, the technological infrastructure of AIFactory and the AI Bench's rigorous evaluation framework is what will enable Telenor to build the next generation AI services with confidence. It’s how we ensure that our AI-powered future is not only innovative but also trustworthy, secure, and built to last.
The AI Bench is one of the first tools launched in collaboration with the Telenor AI Factory to establish an AI Lab in Telenor. The ambition of the AI Lab is to provide resources and tools not only to accelerate responsible adoption of AI internally, but also to builda community where we can co-create the agentic future together with partners.
