RAG Development in Doha, Qatar

Doha's RAG demand: (1) Government: Qatar government knowledge systems. Government RAG: policy (government policy RAG — ministry knowledge management. Legislation: legal knowledge — RAG over Qatari legislation, decrees, and regulatory guidance in Arabic, enabling staff to query legal requirements conversationally. Precedent: policy precedent — retrieving relevant previous policy decisions, ministerial directives, and cabinet resolutions. Citizen: citizen inquiry — RAG-powered chatbot answering citizen questions using official government documentation as source), operations (government operations RAG — internal knowledge management. Procedures: procedure lookup — employees querying operational procedures, forms, and workflows conversationally. HR: HR knowledge — staff querying HR policies, leave entitlements, and benefits in Arabic. Training: training knowledge — onboarding and continuous learning supported by organisational knowledge base), and archive (government archive RAG — historical document intelligence. Heritage: cultural heritage — Qatar National Library, Qatar Museums, and Heritage Foundation digitised archives made searchable through RAG. Historical: historical documents — Arabic manuscripts, treaties, and historical records accessible through natural language query. Research: government research — policy researchers querying across decades of government publications and studies)). (2) Energy: Qatar energy knowledge systems. Energy RAG: operations (energy operations RAG — QatarEnergy ecosystem knowledge. Technical: technical documentation — RAG over equipment manuals, operating procedures, and maintenance records for LNG processing. Safety: safety knowledge — retrieving relevant safety procedures, incident reports, and best practices for specific operational scenarios. Regulatory: regulatory compliance — querying HSE regulations, environmental requirements, and industry standards), engineering (engineering RAG — technical knowledge management. Design: design documentation — retrieving relevant engineering drawings, specifications, and design basis documents. Standards: standards query — querying applicable engineering standards (API, ASME, IEC) for specific technical situations. Lessons: lessons learned — accessing project lessons learned, technical reviews, and engineering change records), and exploration (exploration RAG — geological knowledge. Geological: geological reports — RAG over decades of geological survey data, well reports, and reservoir studies. Seismic: seismic interpretation — retrieving relevant seismic data interpretations and analysis for specific geological features. Research: petroleum research — querying research publications and technical papers relevant to Qatar's geology)). (3) Legal: Qatar legal knowledge systems. Legal RAG: firm (law firm RAG — legal knowledge management. Qatar law: Qatari law — RAG over Qatar Civil Code, Commercial Companies Law, Labour Law, and other Qatari legislation in Arabic. Opinions: legal opinions — retrieving relevant previous opinions, advices, and memoranda from firm's knowledge base. Cases: case law — accessing QFC Regulatory Tribunal decisions, Court of Cassation judgments, and arbitration awards), corporate (corporate legal RAG — in-house legal knowledge. Contracts: contract intelligence — querying across company's contract portfolio for specific terms, obligations, and expiry dates. Compliance: compliance knowledge — retrieving regulatory requirements relevant to specific business activities. Policy: corporate policy — employees querying company policies, procedures, and guidelines conversationally), and regulatory (regulatory RAG — regulatory body knowledge. Guidance: regulatory guidance — RAG over Qatar Central Bank circulars, QFC rules, and CRA (Communications Regulatory Authority) regulations. Reporting: reporting requirements — querying regulatory reporting obligations and deadlines. Updates: regulatory updates — tracking and querying recent regulatory changes and their implications)). (4) Healthcare: medical knowledge systems. Health RAG: clinical (clinical RAG — medical knowledge management. Guidelines: clinical guidelines — RAG over treatment protocols, clinical pathways, and medical guidelines used in Hamad Medical Corporation. Drug: drug information — querying drug interactions, dosages, and contraindications from approved formulary. Research: medical research — retrieving relevant research publications for clinical decision support), patient (patient knowledge RAG — patient-facing information. Health: health information — patients querying health conditions, treatments, and preventive care in Arabic and English. Services: service information — querying available health services, clinic locations, and specialist availability. Procedure: procedure preparation — patients querying pre-procedure instructions, recovery information, and follow-up requirements), and admin (healthcare admin RAG — operational knowledge. Policy: hospital policy — staff querying hospital policies, procedures, and protocols. Coding: medical coding — querying ICD-10, procedure codes, and billing rules. Procurement: medical procurement — querying approved vendor lists, product specifications, and procurement procedures)). (5) Education: academic knowledge systems. Education RAG: university (university RAG — Education City academic knowledge. Research: research knowledge — RAG over university research publications, enabling cross-disciplinary knowledge discovery. Library: library intelligence — making university library collections searchable through natural language query in Arabic and English. Admin: academic administration — querying academic policies, degree requirements, and procedural information), K-12 (K-12 education RAG — school knowledge systems. Curriculum: curriculum knowledge — teachers querying curriculum standards, learning objectives, and resource materials. Parent: parent information — parents querying school policies, events, and student requirements in Arabic. Special: special education — querying special education resources, accommodations, and support strategies), and corporate (corporate training RAG — enterprise learning. Onboarding: onboarding knowledge — new employees querying company information, procedures, and organisational knowledge. Expertise: expertise location — finding internal experts based on project history, publications, and skills. Best practice: best practice — querying organisational best practices and lessons learned)).

Our Doha RAG development follows methodology for Arabic-English bilingual retrieval: (1) Knowledge audit (weeks 1-2): understanding your knowledge landscape. Documents: document inventory — cataloguing all documents by language (Arabic, English, bilingual), format, volume, and topic. Quality: content quality assessment — evaluating document quality, completeness, and currency. Structure: knowledge structure — understanding how information is organised, categorised, and related. Users: user needs analysis — who will query the system, what questions they ask, and what answers they need. Arabic: Arabic content assessment — evaluating Arabic document quality, identifying dialect variation, and assessing OCR needs for scanned Arabic documents. (2) Architecture (weeks 2-3): designing the RAG system. Embedding: embedding model selection — choosing models that handle Arabic effectively (Arabic-specific or multilingual embeddings). Chunking: chunking strategy — document segmentation optimised for Arabic text structure (different paragraph and sentence patterns from English). Retrieval: retrieval strategy — hybrid search (semantic + keyword) with Arabic-specific tokenisation. Generation: LLM selection — choosing generation model with Arabic capability and appropriate deployment (cloud versus on-premise). Pipeline: RAG pipeline design — ingestion, embedding, storage, retrieval, and generation workflow. (3) Development (weeks 3-6): building the system. Ingestion: document ingestion pipeline — processing Arabic and English documents (PDF, Word, scanned documents with Arabic OCR). Embedding: embedding generation — creating vector representations of document chunks in bilingual embedding space. Storage: vector database — Pinecone, Weaviate, or pgvector for vector storage with metadata filtering. Retrieval: retrieval implementation — query processing, embedding, search, and re-ranking. Generation: response generation — LLM generating answers from retrieved context with source citations. Interface: user interface — Arabic-English bilingual query interface with source document links. (4) Testing (weeks 5-6): Arabic-specific validation. Accuracy: retrieval accuracy — testing that correct documents are retrieved for Arabic and English queries. Cross-language: cross-language retrieval — testing that Arabic query retrieves relevant English documents and vice versa. Quality: answer quality — evaluating generated answers for accuracy, completeness, and appropriate source citation. Hallucination: hallucination testing — verifying system does not generate unsupported claims. Arabic: Arabic quality — testing Arabic response quality, grammar, and cultural appropriateness. (5) Deployment (weeks 6-7): production deployment. Infrastructure: Qatar hosting — deploying on local or regional infrastructure per data residency requirements. Security: NICS compliance — security controls, access management, and audit logging. Monitoring: system monitoring — retrieval quality, response latency, and usage analytics. Update: document update pipeline — process for ingesting new and updated documents. Training: user training — training users on effective querying in Arabic and English.