Outcome strip
- Faster lookup: Reduce the time staff or prospects spend searching scattered sources.
- Traceable answers: Show which approved source supports each material answer.
- Visible gaps: Turn unanswered and escalated questions into a prioritized content queue.
The business problem
Teams repeatedly answer the same questions while policies, service details, product
information, and procedures live across websites, PDFs, drives, inboxes, and people’s
memory. Search is slow, answers vary, and a generic chatbot can sound confident even when the
source is absent or outdated.
The AI Knowledge and Support System treats answer generation as a controlled retrieval
process. It knows which sources are approved, which version is current, what a user may see,
how strong the evidence must be, and when to refuse or escalate.
What changes
- Staff repeat common answers or hunt through multiple folders.
- Website visitors receive inconsistent guidance.
- Old policies continue to surface after a revision.
- Chatbot answers cannot be traced to a source.
- Content gaps are anecdotal and never prioritized.
- Approved knowledge is versioned and searchable through one retrieval layer.
- Answers cite the supporting page or document section.
- Weak or missing evidence triggers a refusal or human handoff.
- Permissions and channel policy constrain what can be retrieved.
- Evaluations and feedback reveal accuracy, freshness, and missing knowledge.
Best fit
This system is designed for:
- Training institutes answering program, schedule, requirement, tuition, and admissions
questions
- Ecommerce teams supporting product discovery, shipping, returns, and policy questions
- Service businesses with repeat pre-sale questions and documented operating policies
- Agencies or internal teams searching playbooks, SOPs, specifications, and client-approved
knowledge
- Organizations willing to define an approved source set and accountable content owners
It is not the right first project when:
- Source content is missing, contradictory, or has no owner.
- The requirement is unrestricted web search presented as company policy.
- The assistant would make medical, legal, financial, employment, admissions, or other
high-impact decisions without expert review.
- There is no way to test answer quality or accept human escalation.
- The expected result is “replace the support team” rather than improve access and triage.
Architecture
Caption: Approved, versioned sources are extracted and indexed with metadata. A request
retrieves evidence under channel and permission rules. The answer cites evidence or refuses;
low-confidence and sensitive questions transfer to a person. Evaluation and feedback expose
knowledge gaps and drift.
Text walkthrough
- Approve sources: Select authoritative pages, documents, policies, manuals, and owners.
- Ingest: Extract content, retain source and section references, tag audience/permission,
and record version/freshness.
- Index: Split content into retrievable units and store embeddings plus metadata in the
selected vector/search layer.
- Retrieve: Filter by permissions and source status, retrieve relevant evidence, and
apply a relevance threshold.
- Answer or refuse: Generate within an agreed response schema, attach citations, and
explicitly decline when evidence is insufficient.
- Handoff: Create a ticket or route context to a person for sensitive, unresolved, or
high-value questions.
- Evaluate: Run a maintained test set and review user feedback, unsupported answers,
citation correctness, latency, and gaps.
- Improve knowledge: Assign missing or stale content to an owner and re-index only after
approval.
Core capabilities
Approved-source ingestion
Each source receives an owner, audience, status, version or modified date, and canonical URL
or file ID. Deleted or superseded content is removed or deprecated deliberately rather than
remaining silently retrievable.
Permission-aware retrieval
Public, client, role, and internal content can be separated using metadata and identity
rules. The system defaults to the least information required for the channel. A public web
assistant must never retrieve internal-only material simply because it shares the same
index.
Cited answers and calibrated refusal
Material statements link to their supporting source. If retrieval does not meet the defined
threshold, the assistant says what it cannot confirm and offers the next safe action. A
confident tone is never treated as evidence.
Human handoff with context
Handoff includes the user’s question, safe conversation context, sources considered,
confidence/reason, priority, and expected owner. The person does not have to restart the
conversation.
Evaluation and knowledge-gap reporting
A maintained set of representative, edge, adversarial, and unanswerable questions tests
retrieval, citation, refusal, permissions, and latency. Production feedback identifies
questions the source set cannot answer and routes them to a content owner.
Human decisions and controls
- Which sources are authoritative and which version is active
- Audience and permission classification
- Policy-sensitive answer templates and escalation categories
- Evaluation questions, expected sources, and acceptance thresholds
- Material prompt/model/index changes
- Whether a newly identified gap becomes official knowledge
People also review sampled answers and all high-impact complaints or access incidents.
Security, privacy, and failure handling
- Separate public and restricted retrieval paths where risk justifies it.
- Apply authentication/authorization before retrieval, not only after generation.
- Treat retrieved documents as data, not trusted instructions; bound tools and system
behavior against prompt injection.
- Log source IDs, retrieval scores, decision path, model/version, citation IDs, and outcome
without retaining unnecessary sensitive content.
- Define conversation and log retention, deletion, and export behavior.
- Refuse when no approved evidence meets the threshold.
- If the index or model is unavailable, return a safe fallback and handoff route.
- Test cross-permission leakage, stale sources, malicious documents, missing citations,
contradictory sources, and unanswerable questions.
Typical integrations
- Sources: WordPress, approved websites, Google Drive/SharePoint, help centers, PDFs,
databases, product catalogs
- Ingestion/orchestration: n8n or controlled application jobs
- Retrieval: pgvector, Qdrant, or an existing supported search/vector platform
- Channels: website, help desk, Slack/Teams, internal portal, CRM workspace
- Handoff: ticketing, email, Slack/Teams, CRM task
- Evaluation/analytics: versioned test set, structured logs, dashboard, feedback queue
Specific products are selected for source permissions, volume, hosting, latency, and current
client systems—not to maximize the technology list.
Measurement plan
| Metric | Definition | Method | Review cadence |
|---|
| Retrieval success | Questions retrieving expected source in top-k | Evaluation set | Each release |
| Citation correctness | Material claims supported by cited source | Human/automated rubric | Each release + sample |
| Unsupported-answer rate | Answers containing unsupported material / answered tests | Evaluation set | Each release |
| Correct refusal rate | Unanswerable tests refused safely / unanswerable tests | Evaluation set | Each release |
| Permission leakage | Restricted-source disclosures in negative tests | Security test | Each release |
| Escalation rate | Handed-off conversations / eligible conversations | Production log | Weekly |
| Answer latency | Median and p90 request-to-response time | Runtime log | Daily/weekly |
| Staff lookup time | Time to answer representative tasks before/after | Timed study | Monthly/quarterly |
| Knowledge gaps | Unique unanswered themes assigned and resolved | Gap queue | Weekly |
Publish the evaluation dataset or a safe representative subset with exact scoring rules.
Do not publish a single “accuracy” percentage without defining the question set and rubric.
Delivery plan
1. Knowledge and risk map - 3 to 5 working days
Inventory sources, owners, audiences, permissions, update cadence, recurring questions, and
high-impact categories. Define the first evaluation set and refusal policy.
2. Retrieval prototype - 1 to 2 weeks
Implement ingestion, metadata, retrieval, citations, refusal, and a restricted test
interface. Run the evaluation and correct source/content defects before adding channels.
3. Channel and handoff pilot - 1 to 2 weeks
Connect one channel and one human handoff destination. Test permissions, retention, abuse,
monitoring, downtime behavior, and owner workflow with a limited audience.
4. Controlled expansion - ongoing or handed off
Add sources and channels only after evaluation coverage and content ownership expand with
them. Review gaps, drift, cost, and incidents on a defined cadence.
What the buyer receives
- Approved-source and permission map
- Versioned ingestion and indexing pipeline
- Retrieval/answer flow with citations and refusal behavior
- One agreed user channel and one human handoff path
- Evaluation dataset, rubric, baseline, and release results
- Logs/dashboard for quality, latency, escalation, and gaps
- Source update and deprecation procedure
- Security/retention boundaries, test report, runbook, and handoff
Limitations
- The system cannot make weak or contradictory source material authoritative.
- Retrieval evaluation reduces risk; it does not prove every future answer correct.
- Model, embedding, and source changes can alter behavior and require re-evaluation.
- Permission-aware systems depend on correct identity and source classification.
- Fast answers do not guarantee lower support cost; volume, resolution, handoff, and adoption
must be measured.
Related proof
Cited Knowledge Assistant - Demonstration
Uses approved public AH Shagar site and service documents. Publish the evaluation set,
expected citations, refusal tests, unsupported-answer rate, latency, version, and last-tested
date. Do not describe it as production client support.
Trade-School Admissions Automation - Demonstration
Shows how program FAQs and document requirements can support admissions triage while
eligibility and unusual cases remain human decisions.
FAQ
Is this just a chatbot?
No. The visible chat or search interface is only one channel. The core system is source
approval, versioning, permission-aware retrieval, citation, refusal, evaluation, handoff,
and a content-improvement loop.
Can it answer from private documents?
Yes, if identity, authorization, source classification, retention, and hosting requirements
are explicitly designed and tested. Private and public data should not share an unrestricted
retrieval path.
How do we know the answers are reliable?
Define representative questions and expected sources, test retrieval/citations/refusals,
sample production answers, and re-run evaluation after material changes. Reliability is
measured behavior, not a model-brand promise.
Will it replace our support staff?
The safe objective is faster self-service, better staff lookup, consistent triage, and fewer
repetitive answers. People still own exceptions, judgment, complaints, and source quality.
What happens when policy changes?
The content owner approves the new version, the old source is deprecated, affected tests are
updated, the index is refreshed, and evaluation runs before the change is released.