When an AI assistant gives a distributor the wrong voltage or cites an expired certificate, teams often reach for a larger model or a fine-tuning project. That response skips the most important step: identifying where the failure entered the system.
AWS published an eight-step generative AI customization framework on September 14, moving from an unmodified model through prompt engineering, retrieval-augmented generation, caching, distillation, fine-tuning, continued pre-training and custom model development. Its practical principle is to start with the lightest intervention and move up only when the current layer cannot meet a defined requirement.
For product knowledge, the same visible error can come from very different causes. A missing voltage may reflect a prompt that never required field-by-field verification. An old certificate may mean the retrieval layer lacks version filters. A response that combines two similar models may result from document chunks that lost their product identity. Inconsistent tone or repeated failure to follow a stable classification rule is closer to a behavioral problem.
Replay the failure before selecting a method
Take twenty recent questions and keep the user input, retrieved passages, answer, approved answer and source version. If the correct evidence never reached the model, improve document ownership, indexing, access control and retrieval. If the evidence was present but the instruction was ambiguous, repair the prompt and output schema. Consider fine-tuning only when clearly instructed examples still produce a repeated behavioral failure.
Fast-changing facts usually belong outside model weights. Product catalogs, prices, stock, certification status and market policies need frequent updates and visible citations. A retrieval layer can replace or re-index an approved source as it changes. Fine-tuning is more suitable for stable behavior, such as consistently turning an RFQ into application, material, dimensions, quantity and delivery fields.
Large data volume is not by itself a reason for continued pre-training. A historical drive may contain discontinued models, draft quotations, duplicated manuals and certificates that are no longer valid. Training on that collection can make conflict harder to detect. Establishing an authoritative source, effective date and document owner usually produces more value than adding another thousand files.
Spend according to the failure distribution
Label each defect as an instruction gap, missing source, retrieval failure, reasoning or behavior failure, or undefined business rule. Review the distribution after a month. If most defects come from stale documents, invest in product master data. If approved evidence is consistently retrieved but the model still violates a stable rule, compare prompt, model and fine-tuning options.
The evaluation set should include near-identical models, missing fields, expired documents, multilingual questions and situations that require refusal or escalation. A good answer must trace back to an approved current source. When the evidence is insufficient, the assistant should route the question to product or sales instead of inventing a plausible specification.
This sequence also helps international teams explain investment. Better prompts address instruction problems. Retrieval addresses changing knowledge. Fine-tuning targets repeatable behavior. Continued pre-training addresses deeper domain comprehension at substantially higher cost. Naming the failure before choosing the method prevents an expensive model project from becoming a substitute for basic information governance.
The most mature system is not the one with the heaviest customization. It is the one that can show why an answer failed, which layer owns the correction and how the correction will be tested against the same buyer question.
The same method improves vendor selection. Ask suppliers to run the same failure set and explain which evidence was retrieved, how a newly approved document becomes available, and what happens when the source cannot support an answer. A polished demonstration that contains only easy questions says little about production reliability.
Keep ownership visible after launch. Product teams approve specifications, operations maintain source versions, and sales identifies questions that need commercial judgment. The AI layer can connect those responsibilities, but it should not silently replace them. A correction is complete only when the source, retrieval behavior and regression test agree.
Measure progress by failure category rather than by the number of documents loaded. Fewer unsupported answers and faster source updates are meaningful; a larger index is not. This keeps the program attached to buyer accuracy instead of technical volume.
Sources
Amazon Web Services, The generative AI customization spectrum: From prompt engineering to custom models on AWS, September 14, 2026.