DocSage

I chose pgvector on AWS RDS instead of introducing a dedicated vector database. At the project's scale, keeping document metadata, chunks, and embeddings in PostgreSQL reduced operational complexity without creating a meaningful retrieval-performance constraint.

The design still uses an HNSW index for approximate nearest-neighbor search. If the corpus grew to millions of chunks with strict latency requirements, a specialized vector service could become the better tradeoff. For the current workload, one database meant fewer services, SDKs, costs, and failure modes.

The documents are dynamic, and users need citations. Fine-tuning would make updates slower and would not provide reliable source attribution. RAG keeps the knowledge outside the model, allows documents to change independently, and makes the supporting evidence available to the user.

The API does not return only an answer. It also returns the chunks used, similarity scores, filters, embedding and LLM metadata, token counts, and estimated cost.

This was a deliberate product decision: when an answer looks questionable, a user should be able to inspect the retrieval process rather than guess why the model responded that way.

The LLM response itself is also treated as untrusted input. DocSage requests JSON output, validates it against a Pydantic schema, and retries malformed responses with exponential backoff before returning a result.

An initial similarity threshold of 0.5 worked poorly for short documents and sometimes returned no context at all. Manual testing suggested that 0.3 improved recall, but a later code audit revealed that the application did not have one authoritative default: the retrieval service and API route used 0.3 while the PostgreSQL adapter still defaulted to 0.5. The API documentation also described a different value from the parameter it exposed.

That changed the lesson. A plausible threshold — even one tested against representative documents — is still only a hypothesis until the system has one source of truth and a repeatable evaluation harness. Consolidating the retrieval parameters and measuring them against a curated dataset must come before claiming that any value is optimal.

The audit also found a subtle sweep-breaking bug: an explicit threshold of 0.0 was replaced by the service default because the fallback used Python's truthiness operator. Left unfixed, an evaluation could label a run as 0.0 while actually testing 0.3. This is exactly the kind of silent metric error that makes an apparently rigorous baseline less trustworthy than no baseline at all.

I selected text-embedding-3-small because its 1,536-dimensional vectors fit the existing pgvector design and its lower cost matched the project's scale. The system does not yet have a benchmark that justifies claiming the larger model would provide no meaningful quality improvement; that comparison belongs in the same repeatable evaluation pipeline as threshold and top_k tuning.

Some PDFs contained null bytes that PostgreSQL text columns rejected. The ingestion pipeline now sanitizes extracted text before any database write.

Moving to direct asyncpg connections exposed serialization differences hidden by the previous data-access layer. UUIDs, JSONB fields, and vectors required explicit conversion at system boundaries. I added centralized parsing and serialization so Pydantic models, JSON, and PostgreSQL received the types they expected.

Creating a TLS connection pool to AWS RDS sometimes delayed application startup long enough for Render to mark the deployment unhealthy. I moved pool initialization into a background task so the server could bind its port immediately, while real requests wait for database readiness with a bounded timeout.

Schema isolation placed the application schema ahead of public in PostgreSQL's search path. That prevented pgvector's distance operator from resolving correctly. Qualifying the vector type and operator explicitly made the search function reliable under the least-privilege application role.

Originally, uploading a document and generating its embeddings were separate actions. Users naturally uploaded a document and queried it immediately, receiving no results without understanding why. I changed the workflow so embedding begins automatically after upload.

That was not an algorithmic failure. It was a mismatch between the system's internal states and the user's mental model.