Mapping Semantic Search: The Alif2Ayah Concept Index
We detail the engineering process of building a semantic index that maps natural language concepts to Quranic verses without relying on exact keywords.
Traditional keyword search often fails when the user’s vocabulary doesn’t align with the specific translation of a text. In the context of the Quran, a user might search for "kindness to parents," but if the translation they are searching uses the word "filial piety" or simply "goodness," the result might be missed. For our app Alif2Ayah, we recognized that building a useful discovery tool required moving beyond literal string matching. We needed a semantic index—a system that understands the underlying concepts within a verse and connects them to the messy, varied way humans actually speak and search.
The Problem with Keyword-Only Search
Keyword search relies on exact matches or, at best, fuzzy matching (like Levenshtein distance). While this works for finding specific phrases you already know, it is remarkably poor for exploration. If you are building a tool for learners, you cannot assume they know the exact terminology. In our experience building ShariahSignal, we saw a similar challenge: users don't always know the specific financial ratio names, so they look for broader concepts like "debt levels" or "interest income."
In Alif2Ayah, the challenge is compounded by translation. The Quranic Arabic is static, but English translations vary wildly in their choice of vocabulary. A search index built on one translation becomes useless if the user prefers another. A semantic index solves this by mapping concepts to the Verse ID itself, creating a layer of abstraction that sits between the search query and the text.
Designing the Concept Architecture
Our engineering process for the Alif2Ayah concept index involved three distinct layers: the Vocabulary Layer, the Mapping Layer, and the Query Layer. Instead of a flat list of words, we built a relational structure that prioritizes intent.
#### 1. The Vocabulary Layer (Synonym Sets) We began by identifying core thematic concepts found throughout the text. We then expanded these into "synonym sets." For example, the concept of Gratitude isn't just the word "thanks." It includes related terms like "appreciation," "shukr," and "acknowledgment."
#### 2. The Mapping Layer (Manual Tagging vs. NLP) To ensure high precision, we avoided fully automated AI tagging, which often hallucinates connections or misses subtle nuances. Instead, we used a semi-supervised approach. We utilized Natural Language Processing (NLP) to suggest potential verse-to-concept mappings, which were then reviewed for accuracy. This ensured that a verse discussing the "balance of the scales" was correctly mapped to the concept of Justice even if the word "justice" did not appear in that specific verse.
#### 3. The Query Layer (Embedding and Retrieval) In the app, when a user types a query, we don't just search the database for that string. We transform the query into a vector—a mathematical representation of its meaning—and compare it against our indexed concepts. This allows "treating people well" to surface verses tagged with Excellence in Conduct.
Comparison: Keyword Search vs. Semantic Indexing
| Feature | Keyword Search (Traditional) | Semantic Index (Alif2Ayah) |
|---|---|---|
| Match Logic | Exact string or character match | Meaning and conceptual proximity |
| Translation Sensitivity | High (Highly dependent on specific words) | Low (Focuses on the core message) |
| Discovery | Must know what you are looking for | Can explore by theme or intent |
| Accuracy | High for specific quotes | High for general topics and themes |
| Latency | Extremely low (SQL LIKE) | Moderate (Requires vector calculation) |
Technical Implementation in Android
Building this for a mobile environment presented unique constraints. Unlike a web-based search engine with unlimited server resources, Alif2Ayah needs to be performant on a variety of Android devices. We had to decide between a cloud-based inference model and a local-only approach.
Following our studio's philosophy in Implementing Local-Only Data, we opted for a hybrid strategy. The heavy lifting of the semantic mapping (the relationship between Concept A and Verse B) is pre-computed and stored in a compressed SQLite database within the app.
When the user performs a search, the app uses a lightweight local dictionary to map their keywords to our pre-defined concept IDs. This avoids the latency of a network call and ensures the app remains functional even when the user is offline. It also protects user privacy, as their search queries never leave their device.
Handling Thematic Ambiguity
One of the hardest engineering hurdles was handling verses that touch on multiple concepts simultaneously. A single verse might address Patience, Prayer, and Success all at once.
If we weighted every concept equally, the search results would become cluttered. We implemented a "primary and secondary" tagging system. Each verse has one or two primary concepts that represent its main focus, and several secondary tags for supporting themes. In the UI, primary matches are elevated, while secondary matches are shown as "Related Verses." This hierarchy helps the user stay focused instead of being overwhelmed by a wall of data—a design principle we also utilize in our prayer tracker Salah Companion to keep the interface clean and actionable.
The Workflow: Concept-to-Verse Pipeline
For those building similar discovery-focused apps, here is the technical pipeline we used for Alif2Ayah:
- 1.Data Extraction: We extracted the text of the Quran and standardized it into a unique Verse ID format (Sura:Ayah).
- 2.Concept Seeding: We created a master list of 500+ fundamental concepts derived from classical thematic indexes.
- 3.Vectorization: We ran each concept and its synonyms through a transformer model to generate a high-dimensional vector.
- 4.Clustering: We identified which verses shared similar linguistic and conceptual patterns, generating a list of candidate mappings.
- 5.Validation: We manually filtered the candidates to remove weak associations.
- 6.SQLite Optimization: We indexed the final mappings in a relational table with full-text search (FTS5) enabled on the concept labels to ensure millisecond retrieval times.
Lessons in Precision and Recall
In search engineering, there is a constant trade-off between precision (how many results are relevant) and recall (how many relevant results you actually found).
During the beta phase of Alif2Ayah, we found that our initial semantic engine had high recall but low precision. A search for "water" returned every verse mentioning rain, rivers, the sea, and ritual washing. While technically accurate, it was useless for a user trying to find a specific parable.
We refined this by introducing "Intent Filters." If a user searches for a broad term, we prompt them with sub-concepts (e.g., "Did you mean: Water as a blessing, or Water in the context of creation?"). This turns search from a one-way command into a two-way dialogue, helping the user narrow down their intent without needing to know the technical terms.
What to do next
If you are building a search-heavy application or want to improve how you interact with complex texts, consider these steps:
- 1.Audit your current search logs: Identify the "zero-result" queries. These are often cases where users used the "wrong" word for the "right" concept. This is your roadmap for building a synonym set.
- 2.Decouple your search index from your display text: Don't just search the strings you show the user. Create a hidden metadata layer (tags or concepts) that powers the search engine independently of the UI.
- 3.Prioritize local indexing: If you are building for mobile, look at SQLite FTS5 or Room-based indexing. Keeping the search logic on-device significantly improves the "feel" of the app by removing network lag, which is critical for productivity tools.