Mapping Semantic Search: The Alif2Ayah Concept Index

Traditional search engines rely on a simple premise: if a user types a word, the system finds that exact word in a database. While this works for finding a specific stock ticker in ShariahSignal or a particular habit in a tracker, it often fails when applied to scripture. A user might search for 'gratitude,' but the text uses the word 'thankfulness' or 'shukr.' A user might search for 'mental health,' but the relevant verses speak of 'tranquility' or the 'expansion of the chest.' In developing the beta for Alif2Ayah, we realized that to help users move from 'Alif to Ayah,' we had to bridge the gap between abstract human intent and specific revelatory language. This required moving beyond keyword matching and building a robust semantic search index.

The Limitations of Keyword-Based Search

Standard SQL LIKE queries or even Full-Text Search (FTS) modules in SQLite are limited by the vocabulary of the translation. If you search for 'resilience,' you might get zero results because the translator chose the word 'patience' or 'steadfastness.' For a tool designed to be a companion through life’s journeys, this is a significant friction point.

We encountered a similar logic problem when building Salah Companion. In that app, accuracy is binary—the prayer time is either correct for a coordinate or it isn't. You can read more about that precision in our piece on Unit Testing Prayer Time Logic. However, semantic search is not binary; it is probabilistic. It involves calculating how 'close' a search query is to a verse in a multi-dimensional conceptual space.

How Semantic Search Works: The Vector Approach

Instead of treating a verse as a string of characters, we treat it as a vector—a long list of numbers that represent its meaning. This process, known as embedding, places verses in a high-dimensional mathematical space. Verses with similar meanings are placed closer together geographically in this space, even if they share no common words.

For Alif2Ayah, our engineering process follows a three-step pipeline:

1. 1.Preprocessing: Normalizing the text of various translations to ensure consistent intent.
2. 2.Vectorization: Converting these texts into numerical embeddings using a pre-trained transformer model.
3. 3.Similarity Indexing: Creating a local index on the device that can quickly calculate the 'distance' between a user's search query and the 6,236 verses of the Quran.

Designing the Concept-to-Verse Index

One of the primary challenges was ensuring this logic could run efficiently on a mobile device without requiring a massive cloud backend. As a studio, we favor Local-Only Data for privacy and speed. However, generating embeddings on-device for every search is computationally expensive.

Our solution for the Alif2Ayah beta was to pre-calculate the 'Concept Index.' We mapped thousands of modern English concepts—ranging from 'burnout' to 'parenting'—to their most relevant verses. When a user types a query, the app doesn't just search the Quran; it searches the index of concepts first, which then points to the relevant verses.

The 'Ambiguity' Problem in Religious Texts

Designing search for Alif2Ayah presented a unique ethical challenge: we cannot allow the search algorithm to 'interpret' the text. There is a fine line between helping a user find a verse about 'patience' and having the AI suggest a verse that it thinks applies to a complex legal situation.

To mitigate this, we implemented a 'Confidence Threshold.' If the mathematical similarity between a search term and a verse falls below a certain score (e.g., 0.75), we do not return the result. This prevents 'hallucinated' connections where the algorithm might link a search for 'investment' to a verse about 'planting seeds' in a strictly metaphorical sense that doesn't fit the user's context.

Engineering for Performance

Running high-dimensional vector searches on a mid-range Android device can lead to significant UI lag. We applied lessons learned from Optimizing SQLite for Speed to ensure the search remains snappy. We use a quantized version of the embedding model, which reduces the precision of the numbers from 32-bit floats to 8-bit integers. This shrinks the index size by 75% while maintaining roughly 98% of the search accuracy.

Furthermore, we use an asynchronous search worker. When the user starts typing, the vector calculation happens on a background thread. As soon as the results are ready, they are pushed to the UI via a StateFlow in Jetpack Compose, ensuring the keyboard and animations never stutter.

Implementation Checklist for Semantic Search

If you are building a similar system for a productivity app, here is the framework we used for Alif2Ayah:

• Define the Domain: Don't try to map every concept in the English language. Limit your embedding space to the specific context of your app (e.g., productivity, finance, or scripture).
• Choose a Model: We tested several Sentence-BERT models before settling on a lightweight version that could be bundled within the APK.
• Benchmarking: Measure the 'Recall at 10'—how often the most relevant verse appears in the top 10 results.
• User Feedback Loops: In the beta, we allow users to flag results that feel irrelevant. This data helps us tune the 'Confidence Threshold' without collecting identifying user data.
• Fallback Logic: Always include a standard keyword search as a fallback. If a user is looking for a specific verse number (e.g., 2:255), semantic search is useless.

Moving Toward 'Intent-Based' Interaction

Alif2Ayah is part of a broader shift in our studio's philosophy. We want to move away from apps that wait for you to tell them exactly what to do, toward apps that understand the context of your life. Just as Mindful Assistant protects your focus by understanding when you are in a deep work state, Alif2Ayah aims to provide guidance by understanding the why behind your search.

Building the concept-to-verse index taught us that engineering is often about managing trade-offs between 'smart' features and 'reliable' performance. By using pre-calculated embeddings and quantized models, we’ve managed to create a search experience that feels intuitive and human, without sacrificing the privacy and speed that our users expect from AIA Innovations.

What to do next

1. 1.Test the Beta: Download the Alif2Ayah beta and try searching for abstract concepts like 'longing,' 'doubt,' or 'ambition' to see the semantic index in action.
2. 2.Evaluate Your Search: If you are a developer, audit your own app's search function. Try three synonyms for your core features; if they return zero results, it is time to consider a basic synonym map or a vector embedding.
3. 3.Read on Performance: To understand how we handle the backend of these data-heavy apps, read our guide on Optimizing Android Asset Delivery to see how we manage large index files.