A review sorter for Quran apps, built on Laya

What it does

When we study what people say about Quran apps, we often have thousands of reviews to read, and asking an AI assistant to read every one of them is slow, while counting keywords misses too much. The review sorter reads a large batch of reviews on our own computer and sorts each one into one of ten themes by its main complaint.

For each review it gives an answer and says how sure it is. An answer with a confidence of 0.7 or more stands, and the rest are passed on to Claude to read.

Built on an open model

The model at the heart of the sorter is Laya, made by NandhaKishorM and released under the Apache 2.0 licence, and we use version 0.3.22. We did not build Laya. What we added is the ten themes, the reviews we labelled by hand, the training on those examples, the test, and the skill that runs the sorter and hands the unsure reviews on.

Laya picks from a fixed set of answers rather than writing text, so it cannot make anything up. The sorter runs offline on our own computer and never prints the full text of a review. It was trained for reviews of Quran and Islamic apps only, and other kinds of review would need new labels and a new round of training.

How well it does

We labelled 899 Quran app reviews by hand and kept 301 of them back to test the model. On those 301 reviews the trained model was right 73% of the time, against 47% for keyword counting, and it was right 66% of the time on reviews written in Arabic.

When it was sure of its answer, which was the case for about 58.5% of the reviews, it was right 92% of the time.

Where it fell short

Before the test we set a pass mark of 75%, and the trained model reached 73%, so it did not pass. We kept it for one job only, which is a first sort, because it roughly halves the reading Claude has to do but cannot be trusted on its own.

On its first real run, on 23 October 2026, it sorted 300 reviews in 1.4 minutes, or 1.8 minutes counting the time to load. It was sure about 54% of them and unsure about 46%, and Claude read the 139 unsure reviews and changed the theme on 60 of them.

Where AI is involved

Both steps of the sorter use AI. Laya is a small AI model that chooses a theme for each review without writing any text of its own, and Claude, a large AI assistant, reads the reviews Laya is unsure about and corrects them where it disagrees.