What Documents Do You Have?

Walk through your organization and inventory every place documents live. The typical small business has documents scattered across: a shared network drive (the "Z: drive"), individual employee computers, email attachments, a cloud storage service (Google Drive, Dropbox, OneDrive), an HR system, a CRM, a document management system (if you have one), physical filing cabinets (not digitized), and the institutional knowledge that exists only in people's heads. List every source. For each, note the approximate volume and whether it is digitized.

How Many Documents?

Count, or estimate, the number of documents and their total size. A small business typically has 1,000-50,000 documents totaling 1-50 GB of text. This is not the same as file storage size -- scanned PDFs with images are large files but small text content. Focus on text volume. If you have 10,000 PDFs that are scanned images, the text content after OCR might be 2-5 GB. If you have 10,000 Word documents and spreadsheets, the text content might be 500 MB-2 GB. The text volume determines embedding compute time (indexing) and vector database storage. A rough rule: 1 GB of text produces roughly 200,000-500,000 chunks at 512-token chunk size, which is 200,000-500,000 vectors in Qdrant, occupying roughly 1-2 GB of vector storage.

What Formats?

Document the file formats in your estate. The most common in small businesses:

Who Needs Access?

Map the people who will use the system and what they need to access. A typical small business has 3-5 access tiers: everyone (public policies, general SOPs), department-specific (engineering docs, sales materials, HR records), management-only (financial reports, strategic plans, salary data), and individual-only (personal performance reviews, individual employment agreements). Document who is in each tier and which documents belong to which tier. This mapping drives the role-based access control (RBAC) design in Step 5. If you skip this and load everything into one shared collection, employees can query documents they should not see.

Option A: Reuse an Existing Server ($0)

If you have a server that is not fully utilized -- an old file server, a decommissioned domain controller, a backup server with spare capacity -- you can repurpose it for RAG. Minimum specs: 16 GB RAM, 4-core CPU, 500 GB SSD storage, gigabit network. This handles up to 10,000 documents and 5-10 concurrent users with a CPU-only LLM (Llama 3 8B quantized to 4-bit, or Phi-3 / Qwen 2.5 3B for faster responses). Response time: 8-15 seconds per query. Cost: $0 if you already own the hardware. Limitation: CPU-only inference is slow for larger LLMs and limits you to smaller models with lower answer quality on complex questions.

Option B: Buy a Mini PC ($500-$1,200)

A modern mini PC (Beelink, Minisforum, Intel NUC) with 32-64 GB RAM, an 8-core CPU, and a 1-2 TB NVMe SSD provides a clean, low-power dedicated RAG server. This handles up to 30,000 documents and 10-20 concurrent users with a CPU-only LLM. The fast NVMe storage makes a significant difference for Qdrant vector search latency. Power draw: 15-65 watts, so it can run 24/7 inexpensively. Cost: $500-$1,200. Limitation: still CPU-only for the LLM, so model size and response speed are constrained.

Option C: Workstation with GPU ($1,500-$3,000)

A workstation with a dedicated GPU is the recommended path for any business that can afford the upfront cost. A single NVIDIA RTX 3060 (12 GB VRAM) or RTX 4060 (8 GB VRAM) transforms the system: the LLM runs on GPU, response times drop to 3-6 seconds, and you can run a larger, smarter model (Llama 3 8B at 8-bit, or Llama 3.1 8B at 4-bit with more context). The GPU also accelerates embedding generation during indexing, cutting a 10,000-document ingestion from hours to minutes. Specs: 32-64 GB RAM, Ryzen 7 / Core i7 or better, 1-2 TB NVMe SSD, RTX 3060 12GB or better. Cost: $1,500-$3,000 for the complete machine.

Qdrant -- Vector Database

Qdrant stores the vector representations (embeddings) of your document chunks and performs fast similarity search when a query comes in. It is written in Rust, runs as a single Docker container, handles millions of vectors, and supports metadata filtering (so you can restrict searches to "HR documents only" or "documents the current user can access"). Qdrant is the heart of the retrieval system. Cost: Free, open source (Apache 2.0).

Ollama -- Local LLM Runtime

Ollama runs the language model that generates answers from retrieved context. It handles model downloading, quantization, GPU/CPU management, and exposes a clean REST API. For a business deployment, use Llama 3 8B (good balance of quality and speed on a 12 GB GPU) or Qwen 2.5 7B (strong multilingual support). For CPU-only systems, use Phi-3 mini (3.8B, very fast on CPU) or Llama 3 8B at 4-bit quantization (slower but higher quality). Ollama makes switching models trivial -- one command. Cost: Free, open source (MIT).

sentence-transformers -- Embedding Models

The embedding model converts document chunks and user queries into vectors (lists of numbers) that capture semantic meaning. When the user asks "remote work policy," the embedding model produces a vector close to the vector for the document passage about "work from home eligibility" -- even though the words differ, the meaning is the same. Recommended models: BGE-large-en-v1.5 (best quality for English), BGE-small-en-v1.5 (faster, smaller, good for CPU-only), or Nomic-embed-text (long context, open data). These run through the sentence-transformers Python library. Cost: Free, open source.

PostgreSQL -- Metadata and Access Control

Postgres stores document metadata (title, author, department, access tier, ingestion date, source path), user accounts and roles, and provides full-text search capabilities that complement Qdrant's vector search. The combination of Qdrant (semantic search) and Postgres (keyword search) enables hybrid retrieval, which is more accurate than either method alone. Postgres also enforces access control -- when a user queries, the system checks Postgres for their role and filters results to documents they are authorized to see. Cost: Free, open source (PostgreSQL License).

Docker Compose -- Orchestration

Docker Compose defines all five services (Qdrant, Ollama, embedding server, Postgres, and the web application) in a single YAML file and starts them with one command: docker compose up -d. It handles networking between services, persistent storage volumes, GPU passthrough, restart policies, and logging. When the server reboots, Docker Compose restarts everything automatically. No manual service management required. Cost: Free, open source (Apache 2.0).

Phase 1: Collect and Organize

Gather documents from all sources identified in Step 1. Copy them to a staging directory on the RAG server (or a mounted network share). Organize by department and access tier using a clear folder structure:

This folder structure maps directly to the access control system in Step 5. The folder name becomes the document's access tier in Postgres, and Qdrant's metadata filter ensures users only retrieve documents from tiers they can access.

Phase 2: Batch Import

Run the ingestion pipeline over the staged documents. The pipeline performs five operations per document: (1) parse the file to extract text (Apache Tika handles PDF, DOCX, XLSX, PPTX, HTML; Tesseract OCR handles scanned images; Whisper handles audio), (2) clean the text (remove headers/footers, fix encoding, normalize whitespace), (3) chunk the text into 400-600 token passages with 50-100 token overlap between chunks, (4) generate embeddings for each chunk using the embedding model, and (5) store the vectors in Qdrant with metadata in Postgres.

Ingestion time depends on document count, text volume, and whether you have a GPU. A CPU-only system processes roughly 50-100 documents per minute (embedding is the bottleneck). A GPU system processes 500-2,000 documents per minute. For 10,000 documents: 1.5-3 hours on CPU, 5-20 minutes on GPU. The ingestion runs as a background job -- users can start querying as soon as the first batch is loaded.

Phase 3: Ongoing Ingestion Workflow

Documents do not stop being created after the initial import. New policies get written, new products launch, new customer agreements are signed. You need a workflow for adding new documents to the RAG system without re-ingesting everything. Three approaches, in order of sophistication:

• Manual add: An admin drops new documents into the staging directory and runs a single command: python ingest.py /documents/hr-policies/new-policy.pdf. The pipeline processes just that file. Takes 5-30 seconds per document. Appropriate for businesses that add a handful of documents per week.
• Watch folder: The system monitors a designated "inbox" folder. Any new file placed there is automatically ingested within 60 seconds. Appropriate for businesses that add documents daily. Set up with a simple file-watcher script running as a systemd service.
• API integration: The RAG system exposes an ingestion API. Your document management system, CRM, or HR platform pushes new documents to the RAG system automatically when they are created or updated. Appropriate for businesses with an existing DMS or workflow automation. Requires custom integration work.

Web Interface

The RAG web application runs on the server and is accessed at http://[server-ip]:8080 from any browser on the local network. The interface is deliberately simple -- a single search box, like Google, where users type their question in plain English. The system returns the answer with the source citations listed below it. Each citation shows the document title, the relevant passage, and a link to open the source document. For internal deployment, no internet access is required. For remote access (work-from-home employees), use a VPN or a reverse proxy with authentication -- do not expose the RAG server directly to the internet.

User Accounts

Each employee gets a user account with a username (typically their email address) and a password. Accounts are stored in Postgres with bcrypt-hashed passwords. For businesses with an existing identity provider (Active Directory, Google Workspace, Okta), the system can be configured to authenticate against that provider via LDAP or OAuth -- employees log in with their existing credentials and do not need a separate password. For smaller businesses without an identity provider, local accounts are fine.

Role-Based Access Control (RBAC)

Access control is the most important security feature of a business RAG system. Without it, every employee can query every document -- including salary data, customer contracts, and strategic plans. The RBAC system uses four standard roles, each mapped to document access tiers:

The access control enforcement happens at two levels. First, when a user asks a question, the system queries Qdrant with a metadata filter: access_tier IN [user_authorized_tiers]. This means the vector search only retrieves chunks from documents the user can access. Second, the LLM generates its answer only from the retrieved (authorized) chunks -- it never sees documents the user cannot access. This dual enforcement ensures that even if the LLM tried to reference a restricted document, it would not have that context in its prompt.

Query Testing

Gather 30-50 questions that employees actually ask. Get these from department heads, from your helpdesk tickets, from the questions new hires ask during onboarding. Include a mix of factual ("What is the PTO accrual rate for year 3?"), procedural ("How do I submit an expense report?"), comparative ("What is the difference between the Pro and Enterprise pricing tiers?"), and ambiguous ("What is the dress code?") questions. Run each question through the system and evaluate the answer against the known-correct answer from the source document. Record: did it find the right document? Was the answer accurate? Were the citations correct? Was anything hallucinated (stated confidently but wrong)?

Accuracy Evaluation

Score each test question on a simple 3-point scale:

Target: 80%+ correct, 15% partial, 5% or less wrong on your test set after tuning. If you are seeing more than 10% wrong answers, there is a configuration problem -- usually the wrong embedding model, chunk size that is too large or too small, or a document that was not ingested correctly (OCR failure on a scanned PDF is the most common culprit).

Tuning Chunk Size and Retrieval Parameters

Two parameters have the biggest impact on answer quality: chunk size and retrieval count. Chunk size is how many tokens of text each vector represents. Too small (128 tokens) and the chunk loses context -- the system retrieves fragments that are too short to answer the question. Too large (1024+ tokens) and the chunk dilutes the semantic signal -- the vector represents too many topics and matches poorly. For most business documents, 400-600 tokens with 50-100 tokens of overlap between chunks is the sweet spot. Retrieval count is how many chunks the system retrieves before passing them to the LLM. Too few (3-5) and you miss relevant context. Too many (20+) and you overwhelm the LLM with noise, increasing hallucination risk and slowing response time. Start with 8-10 retrieved chunks, then adjust based on test results.

Team Training Session

Run a 45-60 minute training session for all users. Cover: what the system does (answers questions from company documents, with citations), what it does not do (it is not a general chatbot, it will not answer questions about topics not in your documents), how to access it (bookmark the URL, log in with credentials), a live demo of 5-10 real questions, and 10 minutes for employees to try it themselves with their own questions while you are there to help. Schedule the session during paid work hours, not as optional after-hours training. If the CEO or owner attends, adoption rates are significantly higher.

Cheat Sheet

Provide a one-page cheat sheet (printed and posted in common areas, plus a PDF in the system itself). Include:

Best Practices for Asking Questions

The quality of the answer depends heavily on the quality of the question. RAG systems retrieve documents based on semantic similarity, so vague or overly broad questions retrieve vague or overly broad passages. Teach employees these principles:

• Be specific: "What is the PTO accrual rate for an employee in their third year?" is better than "How does PTO work?"
• Name entities: Include product names, client names, department names, or document titles if you know them. "What warranty terms did we agree to with Acme Corp?" is better than "What are our warranty terms?"
• Ask one question at a time: "What is the PTO policy and how do I submit an expense report?" is two questions. Ask them separately.
• Rephrase if the first answer is wrong: If the system did not find the right document, try different words. "Remote work policy" might find it when "work from home" did not.
• Check the citations: The answer includes source links. If the answer seems off, click through to the source document to verify.

Adding New Documents

New documents arrive constantly -- updated policies, new product specs, new customer agreements. Use the ongoing ingestion workflow from Step 4 (manual add, watch folder, or API integration). Each new document takes 5-30 seconds to ingest. The admin should also periodically review whether old documents should be removed -- if a policy was replaced, the old version should be deleted from the system (or the system will answer questions with outdated information). Schedule a monthly document review: what is new, what has changed, what is outdated.

Re-Indexing

If you change the embedding model, chunk size, or chunk overlap, you must re-index the entire document corpus. Re-indexing re-processes every document with the new settings and replaces the old vectors in Qdrant. For 10,000 documents on a GPU system, re-indexing takes 5-20 minutes. On CPU-only, 1-3 hours. Schedule re-indexing during off-hours (lunchtime, evening, or weekend) so users are not affected. Re-indexing is also a good time to clean up -- remove documents that are no longer needed, update metadata, fix any ingestion errors from the initial load.

Monitoring

The system ships with Prometheus and Grafana for monitoring. Key metrics to watch:

• Query volume: How many questions per day? If it drops to zero, adoption has stalled and you need to investigate.
• Response latency: Average time from question to answer. Should be 3-8 seconds with a GPU, 8-15 seconds CPU-only. If it creeps up, check GPU utilization and Qdrant memory.
• Retrieval success rate: What percentage of queries return at least one relevant chunk? If this drops, documents may have been removed or the embedding model is struggling with new document types.
• Disk usage: Qdrant and Postgres data grow over time. Alert at 80% disk capacity.
• GPU utilization: If consistently above 90%, you are compute-bound and may need a second GPU or a larger model is overloading the current one.

Check the Grafana dashboard once a week. Set up email alerts for disk usage and service downtime. Most issues are caught by simply looking at the dashboard for two minutes every Monday morning.

Backup

Three things need backup: Qdrant vectors, Postgres metadata, and the original document files. Qdrant supports snapshot backups -- a single API call creates a snapshot of the entire collection. Postgres uses standard pg_dump. The original documents are already on your file server (or should be). Run backups nightly, automatically, to a separate disk or NAS. Keep at least 7 days of backups. Test a restore once -- an untested backup is not a backup, it is a hope. The restore procedure: stop services, replace data volumes with backup, restart services. Takes 10-15 minutes for a small business deployment.

Single Server (Small Business)

One machine runs everything. Qdrant, Ollama, the embedding server, Postgres, and the web application all share a single Docker Compose deployment on one box. This handles up to 30,000 documents and 20 concurrent users. It is the recommended starting point for any business under 25 employees. The entire stack fits on a mini PC the size of a paperback book or a workstation under a desk. Setup time: 1-2 days. No networking complexity -- one IP address, one port (8080) to remember.

Multi-User Server (Mid-Size Business)

For 25-100 employees, use a dedicated server with a GPU and 64 GB RAM. The architecture is the same (single Docker Compose, all services on one machine), but the hardware is beefier. At this scale, add a reverse proxy (Nginx or Caddy) for SSL termination and to handle 20-50 concurrent connections. Consider adding a second GPU if response latency creeps above 8 seconds under load. The Postgres database may benefit from running on a separate NVMe disk from Qdrant to avoid I/O contention during concurrent queries and ingestion.

On-Premise Private Cloud (Enterprise)

For 100+ employees or 100,000+ documents, distribute the stack across multiple servers. Qdrant runs as a 3-node cluster with sharding and replication. Postgres uses streaming replication with read replicas. Ollama runs on a dedicated GPU server. The embedding and re-ranking services run on a second GPU server. The web application runs on a separate application server behind a load balancer. Monitoring and backup run on a management node. This is a Kubernetes deployment, not Docker Compose. Setup time: 2-4 weeks. This is where Avondale.AI's professional deployment services are strongly recommended -- the complexity is real, and misconfiguration at this scale is costly.

HR Policies

Typically PDF or DOCX, well-structured with numbered sections (4.1, 4.2, etc.). These are the highest-value documents for RAG -- they answer the questions employees ask most frequently. Ingest with 400-token chunks and 50-token overlap. Preserve section numbers in metadata so the LLM can cite "Section 4.2" in its answers. Access tier: public (all employees) for general policies, HR-private for personnel-specific policies.

Standard Operating Procedures (SOPs)

Step-by-step process documents, usually with numbered lists and sub-steps. Chunk at 512 tokens with 64-token overlap. The chunker should be aware of list structure -- do not split a step across two chunks. If an SOP has a diagram, the text around the diagram should reference it ("see Figure 3"). Access tier: department-specific (the relevant team) or public if it is a company-wide process.

Training Manuals

Longer documents, often 20-100 pages, with chapters and sections. Chunk at 512 tokens with 64-token overlap. Preserve chapter and section titles in metadata so the LLM can cite "Chapter 3, Section 2." If the manual has quizzes or exercises, those can be ingested separately or excluded (they add noise). Access tier: public or department-specific, depending on the training content.

Product Catalogs

Often spreadsheets (XLSX, CSV) with structured data -- SKU, name, description, price, specifications. Do not chunk these like text documents. Instead, convert each product (or row group) into a text representation: "Product: Widget Pro, SKU: WP-100, Price: $29.99, Description: High-performance widget for industrial applications, Specs: 100mm x 50mm, aluminum, 200N load capacity." Then embed that text. This gives the embedding model a complete product description per vector, which retrieves much better than splitting a spreadsheet into arbitrary row chunks. Access tier: public (if the catalog is for internal reference) or sales-only (if it contains pricing not meant for all employees).

Customer Agreements

Contracts, MSAs, NDAs, SOWs. Usually PDF, sometimes DOCX. Dense legal text with defined terms, clauses, and exhibits. Chunk at 600 tokens with 100-token overlap (the longer overlap prevents splitting a clause across chunks). Preserve the client name, contract type, and effective date in metadata so users can ask "What warranty terms did we give Acme Corp in 2024?" Access tier: management-only or sales-department-only. These are the most sensitive documents -- ensure RBAC is correctly configured before ingesting.

Financial Reports

P&L statements, balance sheets, cash flow reports. Usually XLSX or PDF. For spreadsheets, convert to text representation similar to product catalogs (one row or section per chunk). For PDF financial reports, chunk at 512 tokens. These documents are strictly management-only or owner-only access. Be careful with the LLM's ability to answer arithmetic questions from financial data -- RAG is good at finding the right report but may not compute derived figures ("What was our Q3 revenue growth?"). The answer should cite the source numbers and let the user do the math, or use a tool-calling LLM that can compute.

Email Archives

PST or MBOX exports from Outlook, or individual EML files. Extract the subject, body, sender, recipient, and date. Thread related emails together so the system understands the conversation context. Chunk at 400 tokens with 50-token overlap. Email archives are valuable for answering "What did we discuss with the client about X?" but they are also the noisiest document type -- full of signatures, quoted replies, and off-topic content. Clean aggressively before ingesting. Access tier: management-only or department-specific, depending on the email content. Individual personal emails should not be ingested at all.

Public Documents (All Employees)

Company-wide policies, general SOPs, safety procedures, product catalogs (non-sensitive), organizational charts, holiday calendars, benefit summaries. Access tier: public. Every user -- including read-only accounts -- can query these. This is typically 30-50% of your document corpus and accounts for the majority of queries.

Department-Specific

Engineering specs, sales proposals, marketing plans, customer service scripts. Access tier: engineering, sales, marketing, customer_service, etc. Users are assigned to one or more departments in Postgres. A salesperson can access public + sales. A sales manager can access public + sales + management. This is the most common access pattern and typically covers 40-60% of the corpus.

Management-Only

Financial reports, strategic plans, board minutes, salary structures, customer contract terms, legal correspondence. Access tier: management. Only users with the manager role (or admin) can query these. This is typically 5-15% of the corpus but contains the most sensitive information. Double-check that every document in this tier truly belongs here -- it is better to over-restrict than to expose sensitive data.

Individual-Only

Personnel files, individual performance reviews, individual employment agreements, personal compensation data. Access tier: individual:[user_id]. Only the specific user and HR admins can access these. This pattern uses a dynamic filter -- the access tier includes the user's own ID. In practice, most small businesses do not ingest individual personnel files into the RAG system at all, because the risk outweighs the benefit and HR systems already handle individual queries. If you do ingest them, use the strictest access tier and test it extensively.

Time Saved Per Employee

Measure the average time to find an answer before RAG (baseline) and after RAG (post-deployment). Before deployment, time 10 employees answering 5 common questions each using traditional methods (shared drive search, asking colleagues, checking email). Record the total time. After deployment, have the same employees answer the same questions using RAG. The difference is time saved per question. Multiply by estimated questions per week per employee. A typical result: 15 minutes per question pre-RAG, 1 minute per question post-RAG, 4-10 questions per week per employee = 1-2.5 hours saved per employee per week.

Faster Onboarding

New hires are the heaviest users of "where do I find..." and "how do we..." questions. A RAG system lets new hires self-serve answers without interrupting colleagues. Measure onboarding time: the number of days from hire date to "productive" (defined as the employee being able to complete core tasks independently). Companies deploying RAG report 20-40% faster onboarding because new hires find answers in seconds instead of waiting for a colleague to respond. For a business that hires 5-10 people per year, each saving 2-4 weeks of onboarding ramp time, this is significant.

Reduced Repeat Questions

Track the number of times the same question (or semantically similar questions) is asked across the system. Before RAG, employees ask colleagues the same questions repeatedly -- "How do I submit PTO?" gets asked 50 times a year to different people. After RAG, these questions go to the system instead of to people. The system logs show the query volume; compare against a pre-deployment survey of "how many times per week do colleagues ask you questions about company information?" A 50-70% reduction in repeat questions to colleagues is typical. This frees senior staff from being human search engines and lets them focus on their actual work.

Knowledge Retention When Staff Leave

When a key employee leaves, they take institutional knowledge with them -- the unwritten understanding of how things work, where specific information lives, and the context behind decisions. A RAG system captures this knowledge in the documents it indexes and the answers it generates. When a replacement hire asks the same questions, the system has the answers. This is the hardest metric to quantify but the most valuable for businesses with high turnover or aging workforce. The cost of re-creating lost knowledge (re-discovering what a departed employee knew) is estimated at 3-6 months of salary for that role. A RAG system does not prevent knowledge loss, but it captures and makes queryable the documented portion of that knowledge, which for many businesses is 60-80% of it.

Poor Retrieval Quality

Symptom: The system returns answers that are irrelevant, miss the obvious document, or cite the wrong source. Diagnosis: Check the retrieved chunks before they go to the LLM. If the right document is not in the top 10 retrieved chunks, the problem is in retrieval, not generation. Fixes: (1) Try a different embedding model -- BGE-large instead of BGE-small, or Nomic-embed-text for longer context. (2) Enable hybrid search (combine Qdrant vector search with Postgres full-text search using reciprocal rank fusion). (3) Check if the document was ingested correctly -- open it in the system and verify the text was extracted properly (OCR failure on scanned PDFs is the #1 cause). (4) Add a cross-encoder re-ranker (BGE-reranker-large) to re-score the top 20 retrieved chunks and select the best 10.

Slow Responses

Symptom: Answers take more than 10 seconds (GPU) or more than 20 seconds (CPU). Diagnosis: Check which stage is slow -- embedding the query, retrieving from Qdrant, or LLM generation. Use the Grafana dashboard to isolate the bottleneck. Fixes: (1) If embedding is slow on CPU, switch to a smaller embedding model (BGE-small-en instead of BGE-large). (2) If Qdrant retrieval is slow, enable scalar quantization in Qdrant (reduces memory and speeds search at a small recall cost). (3) If the LLM is slow, use a smaller or more aggressively quantized model (Llama 3 8B at 4-bit instead of 8-bit), reduce num_predict to 512 tokens, or add a GPU if you are running CPU-only. (4) If multiple users are concurrent and the system serializes, check that Ollama is configured for concurrent requests (OLLAMA_NUM_PARALLEL=4).

Missing Documents

Symptom: A question about a document you know exists returns no answer or the wrong answer. Diagnosis: Check if the document was ingested. Search the Qdrant collection for the document title or a unique phrase from it. If it is not there, it was not ingested -- check the ingestion logs for errors. Fixes: (1) If the document failed ingestion (parse error, unsupported format, corrupt file), fix the source file and re-ingest. (2) If the document was ingested but is not retrieved, check the access tier -- the user may not have access and the filter is excluding it. (3) If the document is there and accessible but not retrieved, the chunk size may be too large (the relevant passage is diluted) or the embedding model does not capture the semantic match -- try rephrasing the question or adjusting chunk size.

Hallucinated Answers

Symptom: The system states something confidently that is not in the source documents, or conflates information from different documents incorrectly. Diagnosis: Check the system prompt -- does it instruct the LLM to only use the provided context and say "I don't know" when the context does not contain the answer? Fixes: (1) Strengthen the system prompt: "Answer the question using ONLY the provided context. If the context does not contain the answer, say 'I could not find this information in the company documents.' Do not speculate or use outside knowledge." (2) Reduce the LLM temperature to 0.1 or 0.2 (lower temperature = less creative = less hallucination). (3) Increase retrieval count from 8 to 12-15 so the LLM has more context to draw from. (4) Switch to a larger or more capable LLM (Llama 3 8B instead of Phi-3 mini) -- larger models follow instructions more reliably. (5) Add a re-ranker to ensure the most relevant chunks are presented to the LLM. (6) If the question is outside the scope of the document corpus, the system should say so -- this is correct behavior, not a failure.

Setup Challenges

If you do not have someone on staff who is comfortable with Linux, Docker, and command-line administration, the initial setup will be frustrating and time-consuming. Docker installation, GPU driver configuration, Docker Compose networking, and Postgres configuration all require comfort with the terminal. A professional setup takes 1-2 days. A first-time DIY attempt can take 1-2 weeks of evenings and weekends, with the risk of giving up before the system is working. If your time is worth more than $200/hour, professional setup pays for itself in saved time alone.

Custom Integrations

If you need the RAG system to integrate with your existing systems -- pulling documents from SharePoint, syncing with your CRM, authenticating against Active Directory, embedding in your intranet portal, or triggering workflows when certain documents are added -- that is custom development work. It requires API integration, webhooks, potentially middleware, and testing against your specific systems. This is well beyond a Docker Compose deployment and is a natural fit for professional services.

Security Requirements

If your business operates in a regulated industry -- healthcare (HIPAA), finance (SOC 2, GLBA), legal (attorney-client privilege), defense (ITAR, CMMC) -- the RAG system must meet specific compliance requirements. Access logging, audit trails, data retention policies, encryption at rest, and penetration testing may be required. A professional can configure these correctly and document the compliance posture. Getting this wrong is not a technical inconvenience -- it is a legal liability.

Scaling Issues

When you outgrow the single-server deployment -- 50+ concurrent users, 100,000+ documents, or the need for high availability -- the architecture must change. Distributed Qdrant, Postgres replication, load balancing, Kubernetes orchestration, and multi-GPU management are significantly more complex than the single-machine Docker Compose deployment. A professional deployment at this scale takes 2-4 weeks and involves architecture decisions that are difficult to reverse once in production.