Research

How we test.

Every figure on this site comes from our own written tests. This page says what each model was tested on, how each result is scored, and where the tests fall short.

The rules every test follows

Same prompts, same machine. Each model is compared with the open model it starts from, on the same prompts and the same computer. The Text models are also compared with EuroLLM 1.7B, and the two Vision models with each other.
Written rules, fixed in advance. A check is either a pass or fail rule applied to one reply, or a measured rate such as character error or word error. The rule does not change between the two models.
Laptops, not servers. Every model was tested on a 16 GB Apple silicon Mac. Most results come from a MacBook Air M4 or M5, and Gati Voice was also timed on an iMac M1. Speed figures say which machine they come from.
Every result is shown. Each model page lists all of its published results, including the ones where our model scores below its starting point.

Reading the numbers

A pass rate is the share of prompts or messages that passed one check. An error rate counts mistakes against a reference, so lower is better. Character error can go above 100% when a model adds text that is not on the page, because every added character counts as a mistake.

Some checks use small sets, as few as 8 prompts, so one reply can move a result by several points. Speed comes from a small number of runs on one machine.

Each model

Balsa Omni v1

Tested on: 214 text prompts written for a UK bank support chat, covering payment status, scam requests, amounts, dates and spelling.
How it is scored: Each check is a fixed pass or fail rule applied to one reply. A result is the share of prompts that passed.
Compared with: Base model
Hardware: Runs on: Apple silicon Macs, on device
Results published: 15, all on the model page

Balsa Text v1

Tested on: 100 public English questions, 28 identity checks with and without our prompt, 55 replies checked for naming another maker, and a check on writing £250,000.
How it is scored: Questions are marked right or wrong against the published answer. Identity replies are marked by fixed rules.
Compared with: Base model, EuroLLM 1.7B
Hardware: Tested on: MacBook Air M4, 16 GB
Results published: 9, all on the model page

Balsa Vision v1

Tested on: 103 real scanned English pages from printed books, each proofread by a person, plus 40 Hindi and Marathi pages.
How it is scored: Character error against the proofread text, so lower is better. Each page was read again to check the output did not change.
Compared with: Gati Vision v1
Hardware: Ran on: MacBook Air M4, 16 GB
Results published: 16, all on the model page

Balsa Voice v1

Tested on: UK customer service sentences with amounts, dates, times, card digits and reference codes, read in several voices.
How it is scored: The audio is turned back into text by an open speech to text model, then compared with the sentence. The result is the word error rate, so lower is better.
Compared with: Base model
Hardware: Runs on: Apple silicon Macs, on device. Speed: 21.6× real time on a MacBook Air M5
Results published: 21, all on the model page

Balsa Decision v1

Tested on: UK customer messages in sets of 15 or 30 per check, with the same messages and question wording for both models.
How it is scored: A check passes when the model scores the right option highest. A result is the share of messages that passed.
Compared with: Base model
Hardware: Ran on: MacBook Air M4, 16 GB
Results published: 6, all on the model page

Gati Omni v1

Tested on: 224 text prompts for banking support in eight languages used in India, covering scam requests, payment status, next steps and amounts.
How it is scored: Each check is a fixed pass or fail rule applied to one reply, including whether it answers in the customer's language and script.
Compared with: Base model
Hardware: Runs on: Apple silicon Macs, on device
Results published: 13, all on the model page

Gati Text v1

Tested on: 818 public questions in five languages, 52 identity checks, 55 replies checked for false claims about itself and 12 Hinglish questions.
How it is scored: Questions are marked right or wrong against the published answer. Identity and Hinglish replies are marked by fixed rules.
Compared with: Base model, EuroLLM 1.7B
Hardware: Tested on: MacBook Air M4, 16 GB
Results published: 17, all on the model page

Gati Vision v1

Tested on: 309 real scanned pages from printed books, 103 each in Hindi, Marathi and English, each proofread by a person.
How it is scored: Character error against the proofread text, so lower is better. Each page was read again on the exact file we ship.
Compared with: Balsa Vision v1
Hardware: Ran on: MacBook Air M4, 16 GB
Results published: 23, all on the model page

Gati Voice v1

Tested on: UPI payment sentences with rupee amounts, dates and digits, in Hindi, Marathi and English, read in several voices.
How it is scored: The audio is turned back into text by open speech to text models, then compared with the sentence. Results are word and character error rates, so lower is better. Hindi and Marathi clips were also judged by listeners.
Compared with: Base model
Hardware: Runs on: Apple silicon Macs, on device. Real-time factor: 0.76 on a MacBook Air M5
Results published: 25, all on the model page

Gati Decision v1

Tested on: Hindi customer messages in sets of 14 to 30 per check, with the same messages and question wording for both models.
How it is scored: A check passes when the model scores the right first step highest. A result is the share of messages that passed.
Compared with: Base model
Hardware: Ran on: MacBook Air M4, 16 GB
Results published: 6, all on the model page

What is not published yet

The test prompts and the replies each model gave are not on this site yet. The results are internal testing and have not been independently verified.

To test a model on your own data, write to research@f-r.co.