Controlled knowledge system

Give people faster answers without asking them to trust a guessing machine.

Transform approved pages, policies, manuals, and internal knowledge into cited answers with freshness metadata, evidence thresholds, access boundaries, and a clear path to a person.

Outcome strip

  • Faster lookup: Reduce the time staff or prospects spend searching scattered sources.
  • Traceable answers: Show which approved source supports each material answer.
  • Visible gaps: Turn unanswered and escalated questions into a prioritized content queue.

The business problem

Teams repeatedly answer the same questions while policies, service details, product information, and procedures live across websites, PDFs, drives, inboxes, and people’s memory. Search is slow, answers vary, and a generic chatbot can sound confident even when the source is absent or outdated.

The AI Knowledge and Support System treats answer generation as a controlled retrieval process. It knows which sources are approved, which version is current, what a user may see, how strong the evidence must be, and when to refuse or escalate.

What changes

Before:

  • Staff repeat common answers or hunt through multiple folders.
  • Website visitors receive inconsistent guidance.
  • Old policies continue to surface after a revision.
  • Chatbot answers cannot be traced to a source.
  • Content gaps are anecdotal and never prioritized.

After:

  • Approved knowledge is versioned and searchable through one retrieval layer.
  • Answers cite the supporting page or document section.
  • Weak or missing evidence triggers a refusal or human handoff.
  • Permissions and channel policy constrain what can be retrieved.
  • Evaluations and feedback reveal accuracy, freshness, and missing knowledge.

Best fit

This system is designed for:

  • Training institutes answering program, schedule, requirement, tuition, and admissions questions
  • Ecommerce teams supporting product discovery, shipping, returns, and policy questions
  • Service businesses with repeat pre-sale questions and documented operating policies
  • Agencies or internal teams searching playbooks, SOPs, specifications, and client-approved knowledge
  • Organizations willing to define an approved source set and accountable content owners

It is not the right first project when:

  • Source content is missing, contradictory, or has no owner.
  • The requirement is unrestricted web search presented as company policy.
  • The assistant would make medical, legal, financial, employment, admissions, or other high-impact decisions without expert review.
  • There is no way to test answer quality or accept human escalation.
  • The expected result is “replace the support team” rather than improve access and triage.

Architecture

Caption: Approved, versioned sources are extracted and indexed with metadata. A request retrieves evidence under channel and permission rules. The answer cites evidence or refuses; low-confidence and sensitive questions transfer to a person. Evaluation and feedback expose knowledge gaps and drift.

Text walkthrough

  1. Approve sources: Select authoritative pages, documents, policies, manuals, and owners.
  2. Ingest: Extract content, retain source and section references, tag audience/permission, and record version/freshness.
  3. Index: Split content into retrievable units and store embeddings plus metadata in the selected vector/search layer.
  4. Retrieve: Filter by permissions and source status, retrieve relevant evidence, and apply a relevance threshold.
  5. Answer or refuse: Generate within an agreed response schema, attach citations, and explicitly decline when evidence is insufficient.
  6. Handoff: Create a ticket or route context to a person for sensitive, unresolved, or high-value questions.
  7. Evaluate: Run a maintained test set and review user feedback, unsupported answers, citation correctness, latency, and gaps.
  8. Improve knowledge: Assign missing or stale content to an owner and re-index only after approval.

Core capabilities

Approved-source ingestion

Each source receives an owner, audience, status, version or modified date, and canonical URL or file ID. Deleted or superseded content is removed or deprecated deliberately rather than remaining silently retrievable.

Permission-aware retrieval

Public, client, role, and internal content can be separated using metadata and identity rules. The system defaults to the least information required for the channel. A public web assistant must never retrieve internal-only material simply because it shares the same index.

Cited answers and calibrated refusal

Material statements link to their supporting source. If retrieval does not meet the defined threshold, the assistant says what it cannot confirm and offers the next safe action. A confident tone is never treated as evidence.

Human handoff with context

Handoff includes the user’s question, safe conversation context, sources considered, confidence/reason, priority, and expected owner. The person does not have to restart the conversation.

Evaluation and knowledge-gap reporting

A maintained set of representative, edge, adversarial, and unanswerable questions tests retrieval, citation, refusal, permissions, and latency. Production feedback identifies questions the source set cannot answer and routes them to a content owner.

Human decisions and controls

People approve:

  • Which sources are authoritative and which version is active
  • Audience and permission classification
  • Policy-sensitive answer templates and escalation categories
  • Evaluation questions, expected sources, and acceptance thresholds
  • Material prompt/model/index changes
  • Whether a newly identified gap becomes official knowledge

People also review sampled answers and all high-impact complaints or access incidents.

Security, privacy, and failure handling

  • Separate public and restricted retrieval paths where risk justifies it.
  • Apply authentication/authorization before retrieval, not only after generation.
  • Treat retrieved documents as data, not trusted instructions; bound tools and system behavior against prompt injection.
  • Log source IDs, retrieval scores, decision path, model/version, citation IDs, and outcome without retaining unnecessary sensitive content.
  • Define conversation and log retention, deletion, and export behavior.
  • Refuse when no approved evidence meets the threshold.
  • If the index or model is unavailable, return a safe fallback and handoff route.
  • Test cross-permission leakage, stale sources, malicious documents, missing citations, contradictory sources, and unanswerable questions.

Typical integrations

  • Sources: WordPress, approved websites, Google Drive/SharePoint, help centers, PDFs, databases, product catalogs
  • Ingestion/orchestration: n8n or controlled application jobs
  • Retrieval: pgvector, Qdrant, or an existing supported search/vector platform
  • Channels: website, help desk, Slack/Teams, internal portal, CRM workspace
  • Handoff: ticketing, email, Slack/Teams, CRM task
  • Evaluation/analytics: versioned test set, structured logs, dashboard, feedback queue

Specific products are selected for source permissions, volume, hosting, latency, and current client systems—not to maximize the technology list.

Measurement plan

MetricDefinitionMethodReview cadence
Retrieval successQuestions retrieving expected source in top-kEvaluation setEach release
Citation correctnessMaterial claims supported by cited sourceHuman/automated rubricEach release + sample
Unsupported-answer rateAnswers containing unsupported material / answered testsEvaluation setEach release
Correct refusal rateUnanswerable tests refused safely / unanswerable testsEvaluation setEach release
Permission leakageRestricted-source disclosures in negative testsSecurity testEach release
Escalation rateHanded-off conversations / eligible conversationsProduction logWeekly
Answer latencyMedian and p90 request-to-response timeRuntime logDaily/weekly
Staff lookup timeTime to answer representative tasks before/afterTimed studyMonthly/quarterly
Knowledge gapsUnique unanswered themes assigned and resolvedGap queueWeekly

Publish the evaluation dataset or a safe representative subset with exact scoring rules. Do not publish a single “accuracy” percentage without defining the question set and rubric.

Delivery plan

1. Knowledge and risk map - 3 to 5 working days

Inventory sources, owners, audiences, permissions, update cadence, recurring questions, and high-impact categories. Define the first evaluation set and refusal policy.

2. Retrieval prototype - 1 to 2 weeks

Implement ingestion, metadata, retrieval, citations, refusal, and a restricted test interface. Run the evaluation and correct source/content defects before adding channels.

3. Channel and handoff pilot - 1 to 2 weeks

Connect one channel and one human handoff destination. Test permissions, retention, abuse, monitoring, downtime behavior, and owner workflow with a limited audience.

4. Controlled expansion - ongoing or handed off

Add sources and channels only after evaluation coverage and content ownership expand with them. Review gaps, drift, cost, and incidents on a defined cadence.

What the buyer receives

  • Approved-source and permission map
  • Versioned ingestion and indexing pipeline
  • Retrieval/answer flow with citations and refusal behavior
  • One agreed user channel and one human handoff path
  • Evaluation dataset, rubric, baseline, and release results
  • Logs/dashboard for quality, latency, escalation, and gaps
  • Source update and deprecation procedure
  • Security/retention boundaries, test report, runbook, and handoff

Limitations

  • The system cannot make weak or contradictory source material authoritative.
  • Retrieval evaluation reduces risk; it does not prove every future answer correct.
  • Model, embedding, and source changes can alter behavior and require re-evaluation.
  • Permission-aware systems depend on correct identity and source classification.
  • Fast answers do not guarantee lower support cost; volume, resolution, handoff, and adoption must be measured.

Related proof

Cited Knowledge Assistant - Demonstration

Uses approved public AH Shagar site and service documents. Publish the evaluation set, expected citations, refusal tests, unsupported-answer rate, latency, version, and last-tested date. Do not describe it as production client support.

Trade-School Admissions Automation - Demonstration

Shows how program FAQs and document requirements can support admissions triage while eligibility and unusual cases remain human decisions.

FAQ

Is this just a chatbot?

No. The visible chat or search interface is only one channel. The core system is source approval, versioning, permission-aware retrieval, citation, refusal, evaluation, handoff, and a content-improvement loop.

Can it answer from private documents?

Yes, if identity, authorization, source classification, retention, and hosting requirements are explicitly designed and tested. Private and public data should not share an unrestricted retrieval path.

How do we know the answers are reliable?

Define representative questions and expected sources, test retrieval/citations/refusals, sample production answers, and re-run evaluation after material changes. Reliability is measured behavior, not a model-brand promise.

Will it replace our support staff?

The safe objective is faster self-service, better staff lookup, consistent triage, and fewer repetitive answers. People still own exceptions, judgment, complaints, and source quality.

What happens when policy changes?

The content owner approves the new version, the old source is deprecated, affected tests are updated, the index is refreshed, and evaluation runs before the change is released.

Next step

Find the knowledge bottleneck worth fixing first.

Bring the current process, tools, approximate volume, and result that matters. The first conversation will identify the smallest useful next step.