Decision page
Private internal document assistant stack for Legal
What should a legal team use for a private assistant over internal documents?
ZBS editorial starting point. Start with Docling, pgvector, vLLM and LangGraph when controlled inference is a hard requirement.
Staff can search approved material without treating local hosting as proof of access control or quality.
Editorial starting point
Controlled internal retrieval
A concrete starting configuration that keeps source facts, model output and operational authority separate.
Choose this when: External model inference is prohibited and the firm can operate the stack.
-
Convert approved documents locally. The public quickstart documents local conversion and structured export.
Limit: Scans, unusual tables, annotations and source metadata require a representative parsing test.
Evidence: source 1
-
Search with access metadata. It keeps vector search beside relational metadata and access filters in PostgreSQL.
Limit: Approximate retrieval can miss results and needs recall measurement against an exact baseline.
Evidence: source 1
-
Serve the selected local model. It provides a documented self-operated model-serving layer.
Limit: Serving a model does not prove its task accuracy, safe tool use or secure operation.
Evidence: source 1
-
Coordinate retrieval and refusal. Stateful workflow steps keep model output separate from acceptance and operational action.
Limit: Policy, persistence, access control and recovery remain application responsibilities.
Evidence: source 1
Private / local
Controlled reasoning path
Keep parsing, retrieval or model inference in controlled infrastructure while retaining the same source-of-truth and approval rules.
Choose this when: Sensitive inputs cannot be sent to an external model API and the team can operate the additional infrastructure.
-
Ingestion
Docling
source backed inference
Use a separate sensitive-file profile. The public quickstart documents local conversion and structured export.
Limit: Scans, unusual tables, annotations and source metadata require a representative parsing test.
Evidence: source 1
-
Retrieval
pgvector
source backed inference
Keep exact and approximate search. It keeps vector search beside relational metadata and access filters in PostgreSQL.
Limit: Approximate retrieval can miss results and needs recall measurement against an exact baseline.
Evidence: source 1
-
Inference
vLLM
source backed inference
Serve an isolated smaller tier. It provides a documented self-operated model-serving layer.
Limit: Serving a model does not prove its task accuracy, safe tool use or secure operation.
Evidence: source 1
-
Route privileged requests. Stateful workflow steps keep model output separate from acceptance and operational action.
Limit: Policy, persistence, access control and recovery remain application responsibilities.
Evidence: source 1
Budget alternative
Lower-cost external model path
Keep the workflow and source integration explicit while evaluating a lower-cost model candidate on the same acceptance set.
Choose this when: External processing is acceptable and measured model spend is a leading constraint.
-
Ingestion
Docling
source backed inference
Convert documents locally. The public quickstart documents local conversion and structured export.
Limit: Scans, unusual tables, annotations and source metadata require a representative parsing test.
Evidence: source 1
-
Retrieval
pgvector
source backed inference
Select authorized source chunks. It keeps vector search beside relational metadata and access filters in PostgreSQL.
Limit: Approximate retrieval can miss results and needs recall measurement against an exact baseline.
Evidence: source 1
-
Provide an external cost comparison. It is a concrete lower-cost external model candidate for the same acceptance set.
Limit: Price alone is not task fitness; output structure, languages, availability and data terms need testing.
Evidence: source 1
-
Keep refusal rules explicit. Stateful workflow steps keep model output separate from acceptance and operational action.
Limit: Policy, persistence, access control and recovery remain application responsibilities.
Evidence: source 1
Trade-offs that change the choice
Implementation path
1. Define the boundary for files, chunks, prompts, logs and backups.
2. Carry permissions and deletion through the index.
3. Test cross-matter leakage and stale access.
4. Measure recall, grounding, exposure and operating cost.
Known limits
Private means a verified boundary, not merely local hosting.
The external budget path is incompatible when external inference is forbidden.
No product on this page is a universal winner; the configuration still needs a task-specific acceptance test.
EU and US routes stay consolidated with Global until evidence changes the answer.
Validate this stack on your data
A recommendation is a starting point. Practice Lab can test the same workflow on representative inputs, constraints and failure cases.
Request a real-data evaluation
