Building Personal Document Chat Agent: A Per-User RAG Platform on AWS Bedrock

The application is an AI chat and document workspace. Users can upload PDFs, spreadsheets, presentations, images, and meeting recordings, select the files they want to discuss, and receive grounded answers with citations to the retrieved source chunks.

Retrieval-augmented generation sounds simple on paper: store documents, retrieve relevant passages, and give those passages to a language model. Building the application was less about connecting an LLM to a vector database and more about answering a series of architectural questions:

• How should documents be isolated between users?
• When should retrieval happen?
• How do different file formats enter the same pipeline?
• How do we make exact identifiers and semantic questions work equally well?
• What happens when a question spans several documents?

This engineering journal entry is a record of that journey. It is not a blueprint that every system should copy. It describes the decisions that worked for the application, the failures behind them, and the trade-offs I would consider again.

TL;DR

The design keeps retrieval explicit and user-scoped: each user has a separate Bedrock Knowledge Base, while vector infrastructure is shared to control fixed costs. Originals stay in S3 and ingestion creates isolated, searchable derivatives. The RAG pipeline narrows context through selected files, query decomposition, hybrid retrieval, per-file fan-out, reranking, citation checks, bounded tools, and a fixed no-results path. Conversation memory is compacted only as an optimization, and DynamoDB remains the source of truth. The main lesson is to improve the evidence boundary before tuning prompts; reliability comes from making isolation, retrieval decisions, and failure behavior explicit.

What the application does

Three application surfaces share the same backend:

Surface	Role in the workspace
Chat	Streams model responses with selected-file tagging, attachments, citations, model choice, and reusable saved prompts
Drive	Manages the user's S3-backed files and folders, including previews and downloads
Editors	Opens documents, spreadsheets, and PDFs inside the application

The RAG design lets users choose how broad the search should be.

The four ideas that shaped the design

Four decisions became the foundation of the system.

Retrieval is explicit

Users can ask broadly and retrieve across everything added to the knowledge base, or type @ to focus retrieval on one or more files. The explicit file signal usually improves accuracy by narrowing the search boundary and keeping unrelated sources out of the context.

Each user has a separate knowledge base

Instead of placing every user's documents in one knowledge base and depending on a tenant filter for every query, the system creates separate Amazon Bedrock Knowledge Bases for each user's drive and legal content.

This moves an important security property into the resource topology. A retrieval request is already scoped to the user's knowledge base before metadata filters are applied. Tenant isolation therefore does not depend solely on remembering the correct predicate in every search path.

The individual knowledge bases still share one OpenSearch Serverless collection and vector index. That distinction matters: logical isolation is per user, while the expensive vector infrastructure is shared. Creating a separate OpenSearch collection for every user would introduce an unacceptable fixed-cost multiplier. A future alternative would be Amazon S3 Vectors, which could materially reduce vector-storage costs for this workload; it would still need to be evaluated against query latency, filtering, indexing, and operational requirements before replacing OpenSearch Serverless.

Ingestion and retrieval are separate concerns

Users upload originals into their normal storage paths. A preprocessing pipeline produces knowledge-base-ready derivatives under a separate prefix. The Bedrock data source scans only that derived prefix.

This keeps original files available for viewing and download while allowing the retrieval pipeline to normalize images, presentations, audio, and other formats independently.

Retrieval quality comes before prompt tuning

Several of the largest answer-quality improvements came from changing what entered the context window, not from rewriting the system prompt. Hybrid search, query decomposition, per-file fan-out, reranking, and an explicit no-results path were more valuable than increasingly elaborate instructions to the model.

High-level AWS architecture

The application ships from one React 19 codebase as a web application and, through Tauri 2, as a desktop application for macOS and Windows. FastAPI and the RAG orchestration layer run in a containerized backend on ECS Fargate. AWS manages identity, object storage, operational data, sessions, model inference, embeddings, and vector retrieval.

Only the load balancer is publicly reachable; application tasks and Valkey remain private. Gateway endpoints keep S3 and DynamoDB traffic off NAT gateways, security groups define service-to-service access, and Cognito plus HttpOnly Valkey sessions handle identity. Long-lived infrastructure is separated from application releases, with configuration discovered through Parameter Store and Secrets Manager. Nested-stack parameters also keep CloudFormation deployment order explicit and avoid circular cross-stack patches.

How the RAG pipeline evolved

The RAG pipeline started as a simple retrieve-and-generate flow, which made the retrieval boundary visible and gave the system a clear request state. As real questions became more complex, it evolved into a guarded, multi-step pipeline with query decomposition, retrieval, reranking, bounded tools, grounding checks, memory management, and response streaming. The chat flow below shows how those steps work together.

Rendering diagram…

Each stage has one job and can fail open where blocking the answer would be worse: decomposition falls back to the original question, reranking keeps the vector-score order if its API fails, and memory compaction never blocks a response. When the user selected files but no citable chunk survives, the workflow fails closed and skips generation rather than asking the model to improvise.

Grounding checks changed how streaming works. The critique step needs a complete answer before it can decide whether to retry, so the graph now runs to completion first and then replays the accepted response through the existing server-sent-event contract. Time to first byte became longer, but an ungrounded draft is no longer streamed before the system can reject it.

The streaming layer still follows a few ordering rules that were easy to overlook:

• The assistant response is persisted before the completion event is sent. The client refetches immediately after completion, and that request may reach another container.
• A disconnected client should not leave partially created messages behind.
• Proxy buffering must be disabled, otherwise the server appears to stream while the reverse proxy holds the tokens until the response ends.
• Nonessential work such as title generation and chat summarization belongs outside the response path.

Compact long conversations without changing the source of truth

Long chats eventually consume most of a model's context window. The application keeps a rolling summary and the latest ten turns in Valkey, then swaps that compact representation into the graph once estimated usage passes 80% of the selected model's context limit.

DynamoDB remains the complete conversation record. The cached summary only changes what enters the current retrieval and generation turn. If Valkey is unavailable, the node quietly keeps the raw history, because memory optimization should never make chat unavailable.

A bounded tool-calling loop, layered onto the fixed pipeline

The original two-node graph — retrieve, then generate — was deliberately deterministic: the same question with the same selected files walks the same nodes in the same order. That determinism is valuable for debugging and for the fail-open/fail-closed guarantees described above, but it cannot help with a question that only becomes answerable after seeing an intermediate result, such as "find the file that mentions the vendor contract, then tell me its renewal date."

Rather than replacing the deterministic graph with a fully agentic loop, I inserted one bounded tool-calling step between reranking and generation. On its first pass, the model is offered a small set of read-only tools — searching or summarizing the user's own files — and may call one. If it does, the tool executes and the result is fed back to the model, up to a small fixed number of iterations and a short wall-clock budget. If the model never asks for a tool, the step costs exactly one extra model call and changes nothing else.

Three constraints keep this from turning into an unbounded agent:

1. A hard iteration and time cap. The loop cannot run indefinitely waiting for the model to stop calling tools.
2. A closed, per-request tool set. Tools are built fresh for each request and closed over the requesting user's identity, so a tool call can never reach another user's files. The set of available tools also changes depending on which part of the workspace the conversation is in, so a general document chat and a specialized document-review workspace never share the same tool surface.
3. Tool output never enters the citation scheme. Retrieved chunks that back a [N] marker and tool-call results are kept in separate prompt sections. A tool result can inform the answer, but it cannot masquerade as cited evidence — that distinction matters because tool calls are not scored, reranked, or grounding-checked the way retrieval evidence is.

The user-visible effect is a live transcript: each retrieval and each tool call renders as its own step, moving from running to done or error, and that transcript is persisted alongside the message so it survives a page reload instead of only existing for the lifetime of the stream.

Route whole-document questions around chunk search

Chunk-similarity search has a structural weakness: a request like "summarize this document" or "what's the gist of this file" has almost no lexical or semantic overlap with any single chunk, because the request is about the whole document rather than about any specific passage in it. Tuning score thresholds cannot fix a mismatch that fundamental.

When exactly one file is tagged and the question matches a small set of summary-style patterns, the workflow now skips vector search entirely and uses the file's already-computed, cached summary as the sole context item, falling back to normal retrieval if no summary is available yet. This is the same lesson as the rest of the retrieval design: when an answer is systematically wrong, the fix is usually to change what enters the prompt, not to tune the prompt itself.

Designing the ingestion pipeline

Bedrock Knowledge Bases accepts a defined set of source formats, while real users upload much more varied material. The ingestion pipeline normalizes those files into forms the knowledge base can process.

Rendering diagram…

Preserve originals and isolate derivatives

Original files remain where the user uploaded them. Normalized copies go under a kb/ prefix, and the knowledge-base data source includes only that prefix.

For example, an image might become:

Original: docs/users/{user-id}/photo.jpg
Derived:  kb/docs/users/{user-id}/photo.jpg.kb.pdf

The important part is that the derivative retains the full original filename and appends a suffix. This gives retrieval a stable relationship between a selected original and everything indexed on its behalf. Images are converted to PDF so the multimodal parser can interpret scanned pages and screenshots instead of relying only on text extraction.

Treat ingestion conflicts as normal

Bedrock permits only one ingestion job at a time for a data source. Closely spaced uploads can therefore collide even when nothing is wrong.

Rather than treating that collision as a terminal failure, the preprocessor records a deferred-sync flag. A scheduled reconciliation job revisits those records and starts ingestion when the data source becomes available. Structured audit events distinguish an initial conflict from a later recovery.

This changed the mental model from "the upload event must complete everything" to "the system must eventually converge." The latter is much more resilient to duplicate events, concurrency, and lost notifications. I also kept a repair endpoint for files that never reached kb/, because an event-driven path still needs a reconciliation path.

Prevent recursive processing mistakes

The preprocessor writes a new object to the same bucket that triggered it, so one upload can invoke the function again for its derivative. The handler must recognize generated keys and normalize them back to the original before updating metadata. Without that rule, derived objects can create phantom file records that the user never uploaded and the application never displays.

One knowledge base per user

Knowledge bases are provisioned lazily. Creating one for every registered account in advance would allocate resources for people who may never upload a document or use retrieval.

The first retrieval-related action follows a guarded workflow:

1. Write a mapping row with the state CREATING.
2. If another request sees that state, it waits or reports that setup is in progress instead of starting a second creation flow.
3. Create the Bedrock Knowledge Base using Cohere's multilingual embedding model.
4. Attach an S3 data source restricted to kb/docs/users/{user-id}/.
5. Store the knowledge-base and data-source identifiers, then set the state to ACTIVE.
6. If provisioning fails, set the state to FAILED so the problem is visible and recoverable.

Rendering diagram…

The mapping row acts as both a directory and a concurrency guard. This is important because knowledge-base creation is a runtime operation; it cannot be fully represented as static infrastructure when the resource appears only after a user needs it.

Isolation without multiplying vector-store cost

The architecture separates two concerns:

• Bedrock Knowledge Base: one per user, providing a clear retrieval boundary.
• OpenSearch Serverless collection and index: shared, providing common vector infrastructure.

This gives me a stronger tenant boundary without paying the minimum OpenSearch capacity for every user. It is still essential to secure the mapping table, S3 key prefixes, IAM roles, and provisioning endpoints. A per-user knowledge base reduces the number of places where tenant filtering can fail; it does not remove the need for authorization around the resources.

Chunking for precision and context

I used hierarchical chunking with small child chunks and larger parent chunks. The child chunks are embedded and matched, while their parents are supplied to the language model.

A representative configuration is:

• Parent chunks: about 1,500 tokens
• Child chunks: about 300 tokens
• Overlap: about 60 tokens

The smaller child makes retrieval precise. The larger parent gives the model enough surrounding material to understand the matched passage. This avoids choosing between tiny fragments that match well but lack context and large chunks that contain context but produce less specific embeddings.

There is still a token-cost trade-off. Every returned parent may add up to roughly 1,500 tokens to the prompt. Retrieval limits and score thresholds therefore matter not just for quality but for predictable inference cost.

The search mechanism

The search design combines several mechanisms. None is particularly complex by itself, but together they determine whether the model receives useful evidence.

Decompose only complex questions

Questions that are long, contain several clauses, or ask multiple things can be split into two to four focused sub-questions before retrieval. Each sub-question runs through the same file filters, then the results are merged and deduplicated by source and content.

The decomposition call is heuristic-gated. A short, single-topic question keeps the original query and pays no additional model latency. If decomposition fails, retrieval also falls back to the original question rather than failing the turn.

Hybrid retrieval

Every document retrieval uses hybrid search: dense vector similarity together with keyword matching.

Semantic search is useful when a question paraphrases the source. Keyword matching is essential for exact strings such as invoice numbers, section names, identifiers, dates, and people's names. Document questions often contain both, so choosing only one search mode created avoidable blind spots.

Filter by selected file

Bedrock records the source S3 URI in chunk metadata. A selected file becomes a startsWith filter against its derivative path:

startsWith(
  x-amz-bedrock-kb-source-uri,
  s3://{bucket}/kb/docs/users/{user-id}/report.pdf
)

Because converted derivatives preserve the original filename, this prefix matches both a direct copy and a derived object such as photo.jpg.kb.pdf. The caller does not need to know how the file was normalized.

When several files are selected, their clauses can be joined with orAll. However, a single blended retrieval is not always the best way to answer a multi-file question.

Fan out multi-file questions

This was the most useful retrieval change I made.

Suppose the user asks:

Compare the quarterly revenue in one report with the forecast in another spreadsheet.

A single embedding represents the combined question. In one top-K search, chunks from the slightly better-matching file can occupy every result slot. The model then receives evidence for only half the comparison and may still produce a confident answer.

The solution is one retrieval per selected file, executed concurrently.

Rendering diagram…

Each file gets a chance to contribute evidence. The system merges all results and sorts them by score, but the documents no longer compete for the same initial top-K slots.

I use a slightly lower score threshold for multi-file retrieval. A question that spans several topics tends to score less strongly against each individual file, and explicit file selection already provides a meaningful relevance signal. The exact values are configuration, not universal constants; they should be tuned against a representative evaluation set.

Apply the score floor defensively

The minimum confidence threshold is enforced by the retriever and checked again in application code. This guards against a retriever object created under an older configuration continuing to return results after thresholds change.

Cached retrievers are keyed by the knowledge-base identifier, retrieval configuration, and minimum score. Configuration therefore participates in cache identity rather than becoming hidden mutable state.

Rerank before generation

Vector similarity is a useful first pass, but it is not a final judgment about whether a chunk answers the question. After the file and sub-query results are merged, the pipeline sends the citable chunks to the Bedrock Rerank API and removes anything below a relevance floor.

Reranking is an optional quality layer. If the API is unavailable, the workflow keeps the original vector-score order and continues. This gives the application a measurable retrieval improvement without making one extra managed service a hard dependency for every answer.

That relevance floor caused a subtle regression for tagged files. Vector retrieval already applies a deliberately relaxed score threshold when a file is explicitly selected, on the theory that user intent is itself a strong relevance signal. But a generic query — "summarize the document," for instance — can still score near zero against the correct chunk on Bedrock Rerank's semantic-overlap scale, even though the chunk is exactly right. Reranking's own hard cutoff was silently discarding chunks that the relaxed retrieval threshold had deliberately let through. The fix was to keep reranking's ordering for tagged files while disabling its filtering floor for them: tagged-file evidence still gets reordered by relevance, but it is no longer dropped by a second, stricter threshold that was never meant to apply to it. It is a reminder that two independently reasonable thresholds, applied one after the other, can silently cancel each other's intent.

Validate citations

Retrieved chunks are assembled into a numbered context block. The model refers to them with markers such as [1] and [2].

Before storing the answer, the application removes markers that point beyond the available context. If five chunks exist, a generated [7] should not become a broken citation in the interface. This does not prove that every remaining citation is semantically perfect, but it prevents references to nonexistent evidence.

Stop when selected files return no evidence

An empty context is ambiguous to a language model. When the user selected a file but no citable chunk survives retrieval and reranking, the graph returns a fixed, helpful no-results response and skips generation entirely. Likely reasons include:

• no file was selected;
• the selected file is still processing or failed to process; or
• no chunk was relevant enough to pass the score threshold.

This turns a probabilistic instruction—"do not guess when evidence is missing"—into a deterministic branch in the workflow.

Critique grounded answers with a bounded retry

Citation validation can remove a marker that points nowhere, but it cannot detect an unsupported claim with no marker. After generation, a second model call checks whether the factual claims are grounded in the numbered context. An ungrounded draft is discarded and the workflow retries retrieval and generation, at most twice.

The limit matters. A grounding check should improve reliability, not create an unbounded loop or an unpredictable bill. If the critique service itself fails, the pipeline keeps the original answer; the check is a quality gate rather than an availability gate.

Cost lessons from the architecture

The fixed cost of this design is not evenly distributed. S3, Lambda, and DynamoDB can stay inexpensive at low traffic, while OpenSearch Serverless has a capacity floor even when the index is small. Per-user knowledge bases therefore share one vector collection to preserve isolation without multiplying that baseline. Variable costs come mainly from model tokens, embeddings, parsing and transcription, retrieved context, reranking, and bounded retries. Explicit file selection and result limits control both context size and spending.

What I learned

Isolation is stronger when it is part of the topology

A metadata predicate is useful, but it is easy to omit or misconfigure. Giving every user a distinct knowledge base makes the user boundary a property of the primary retrieval resource. Sharing only the underlying vector infrastructure preserves the economic benefit of consolidation.

Retrieval problems often look like model problems

When an answer is incomplete, my first question is now: what evidence actually entered the prompt? Multi-file competition, overly strict filters, poor decomposition, stale score thresholds, and weak reranking cannot be repaired by asking the model to reason more carefully.

User intent is valuable retrieval metadata

Selecting a file is a high-quality signal. It justifies narrowing the search space, can support a modestly lower confidence threshold, and tells the system which documents deserve independent retrieval attempts.

Event-driven ingestion needs reconciliation

Object-created notifications are useful triggers, but they do not guarantee that every downstream step completes exactly once. Deferred-sync records, scheduled recovery, idempotent key handling, and a repair path make the pipeline dependable.

Bound agentic behavior instead of avoiding it or embracing it fully

A fixed retrieve-then-generate graph is easy to reason about, but some questions genuinely need an intermediate lookup before they can be answered. Rather than choosing between a rigid pipeline and a fully open-ended agent, a small, capped, per-request tool loop captures most of the benefit: it only activates when the model asks for it, it cannot run away in iterations or wall-clock time, and its output is kept structurally separate from cited evidence. The determinism of the original pipeline is preserved for the common case; the flexibility is opt-in and bounded for the rest.

Serverless does not always mean scale-to-zero

Some managed services have a fixed capacity floor. That fact can change a tenancy design completely. I would model the vector-store baseline before committing to an isolation strategy, especially for a low-traffic or early-stage system.

Record the reason behind unusual code

The most useful comments in this system explain why an apparently unnecessary behavior exists: why persistence precedes the final stream event, why generated keys are normalized, why a data-source filter may be omitted, and why multi-file search fans out. Those notes preserve lessons that would otherwise be rediscovered through failures.