AI Answer Library

What state does your data need to be in before starting an AI project?

Short answer

The minimum is three properties: findable, extractable, and attributable. Findable means the documents live in one known place with a way to search them. Extractable means they are not scanned images or locked files and can be reliably converted to text. Attributable means every document carries a version, an effective date and a visibility scope. Without these three, no model will answer correctly. But you do not need a data platform first — cleaning up the corpus for your first scenario is enough to start.

Key points

  • 01Corpus quality sets the accuracy ceiling; the model only determines how close you get to it. Stale documents, conflicting versions and scanned images will drag any model down to unusable.
  • 02Extractability is a hard gate: scans need OCR, protected files need their restrictions lifted, and complex tables and charts need a dedicated parsing strategy. Otherwise that content simply does not exist as far as the system is concerned.
  • 03Every document must answer three questions: which version, effective from when, and visible to whom. Without version and date the system will confidently cite a superseded policy; without visibility scope, permission isolation cannot be built at all.
  • 04Coverage matters far more than volume. Three hundred accurate documents covering the frequent questions beat thirty thousand files mixed with drafts, old versions and irrelevant material.
  • 05Internal jargon needs a glossary. The same thing goes by different names in different departments, and abbreviations mean different things in different contexts. Retrieval cannot solve this; a glossary and synonym mapping must.
  • 06Do not defer the project indefinitely to "get the data right first". The correct order is: choose the first scenario, clean only the corpus it needs, make it work, then extend to the same standard.

What "ready" means for each data type

The table breaks down the data types common in an enterprise, with a definition of "usable" and an owner for each. The criteria are written concretely so they can serve as a pre-kickoff checklist: for every row, sample twenty items at random and count how many pass. The ownership column matters just as much — most of this work sits with the business side, not the technical team, which has neither the authority nor the judgement to decide which version of a policy is currently in force. If that is not settled at kickoff, the project stalls mid-build.

Data typeTypical problemsWhat counts as usableWho owns it
Policies, processes and regulationsMultiple live versions, no effective dates, superseded copies still circulatingOne current version each, with an effective date and a superseded flag; archived versions excluded from retrievalThe owning department — the technical team cannot judge which version is in force
Product material and technical manualsScanned and image-only PDFs, complex tables, specifications trapped in figuresReliably extractable as text, table structure preserved, key parameters searchableTechnical side runs OCR and parsing; business side spot-checks correctness
Historical tickets and support transcriptsFull of personal information, uneven answer quality, and mishandled cases recorded as-isRedacted, and filtered down to cases a human confirmed were handled correctlyBusiness side curates correct samples; legal signs off on the redaction standard
Structured data in business systemsNo callable API, field semantics nobody can explain, and stale dataA stable API or view, documented field semantics, and a defined freshness guaranteeThe system owner exposes the interface; the business side documents field semantics
Internal terminology and abbreviationsSame thing named differently, same name meaning different things, department-specific abbreviationsA glossary with synonym mappings and disambiguation rules for overloaded termsBusiness side leads, extending it continuously from usage feedback
Permissions and visibilityNo visibility labels, and a company-wide share drive containing everythingEvery document labelled with visible departments or roles, aligned with retrieval-layer filtersBusiness side defines the boundary; technical side enforces it in retrieval

A start-up checklist you can run as-is

Walk this list at the kickoff review; any item you cannot answer is a risk signal. One: which documents does the first scenario touch — can you produce an explicit list rather than "whatever is on the share drive"? Two: sample twenty at random — how many yield clean text? Below fifteen, schedule OCR and format remediation first. Three: does the set contain multiple versions of the same subject, and if so, who has the authority to declare which is current? Four: does it contain personal or sensitive data, and who signs off on the redaction standard? Five: can the business team write 50–100 questions they have actually asked, with reference answers? If not, the scenario is usually not yet thought through. Six: can every one of those answers be found in the documents from item one? Whatever cannot must either get a new document or move out of phase one. Seven: who is allowed to see each document? Eight: how often does this data change, and how does the system learn that it changed? Complete those eight and most of the technical uncertainty disappears.

Two things people commonly get wrong

First, "we must build a data platform first" is an expensive misconception. A data platform solves company-wide data supply — long timeline, heavy investment — while one AI scenario typically needs only a small slice of it. The right order is to make one scenario work, let real usage expose the data gaps, then extend governance on demand. Done the other way round, the platform can absorb two years while not a single AI scenario ships. Second, "more data is better" does not hold either. Dumping every historical file into the knowledge base buries retrieval under drafts, superseded versions and irrelevant material, and accuracy falls. Prefer less over more: load only confirmed, currently effective documents and add as needed. Both points come under pressure early in a project, because "build the foundation first" always sounds safer than "make one small scenario work" — yet in practice the former fails more often.

Where this applies

When this answer does not hold

  • This is written for RAG-style knowledge-base projects. Supervised fine-tuning or training has entirely different requirements — paired input/output samples and annotation consistency, not document retrievability.
  • There is no universal answer to how much data is enough; it depends on question coverage, not item count. Test it with the evaluation set: if the answers to real questions exist in the corpus, it is enough — if not, more irrelevant documents will not help.
  • Corpora containing personal information need a compliance assessment before ingestion. The redaction standard, retention period and deletion capability are legal questions and should not be decided unilaterally by the technical team.
  • Data governance is not a one-off task. Documents keep changing, being superseded and being added, and unless "how updates flow into the knowledge base" is designed as a standing process, accuracy decays over time on its own.

People also ask

  • How do you prepare data for an enterprise knowledge base?
  • What data quality does an AI project actually require?
  • Can scanned PDFs go straight into a knowledge base?
  • Do we need a data platform in place before we start with AI?
  • How much data is enough to start an AI project?
Written by: YGG Technology Solutions TeamPublished: 2026-08-01Last reviewed: 2026-08-01