The useful test for a product assistant is not simply whether it returns a document. It is whether it returns a relevant document that meets the requested market, version and document-state conditions. Removing a filter may produce a fluent answer while making the result less suitable for the buyer who asked the question.

That distinction matters to distributors and international marketing teams relying on a Chinese supplier's technical library. An older specification can contain very similar language to a current one. A search that retrieves more material is not necessarily better at selecting the material applicable to the product being discussed.

AWS's September 30 explanation of metadata pre-filtering in S3 Vectors offers a concrete mechanism to examine. In ENHANCED mode, metadata conditions are resolved first and similarity search runs over the matching vectors. AWS describes CLASSIC differently: vector search and filtering operate in tandem, validating candidates as the search proceeds. It should not be reduced to a claim that CLASSIC always takes a fixed top set and then filters it afterwards.

Establish which documents are eligible

When an answer disappears after a filter is added, first choose a document known to be applicable. Confirm that it is present in the index, then inspect the metadata actually stored on that record. A request can use the intended condition while the record has a missing field, an inconsistent state label or the wrong version value. In that case, a highly similar document may correctly be excluded by the condition that was sent.

Market and product-version fields are typically part of the application's own metadata design, not universal meanings supplied by a vector service. The catalogue owner and the retrieval developer need to agree on what each value means. A label that an overseas team reads as “current” may not mean the same thing as the value used during ingestion. Changing the model's instructions cannot resolve that disagreement in the stored data.

Next, inspect the actual index mode rather than assuming the announcement changed it. AWS says existing indexes remain in CLASSIC until updated. For buckets created before September 30, the earlier default also remains relevant to newly created indexes until the bucket's default is changed. The age of an individual document is therefore not enough to infer which search behaviour is active.

These checks separate three different situations: the document was never indexed; it was indexed but does not satisfy the supplied metadata condition; or it is eligible but the filtered retrieval behaviour needs further examination. Each calls for different work. Treating all three as an assistant that “does not know the product” encourages prompt changes that do not address the cause.

Include a document that must stay out

A small comparison set can reveal more than a large collection of undirected questions. Include one known in-scope document and another with similar wording that must be excluded. Check whether the first can be found and whether the second stays out. Looking only for a non-empty response would miss an apparent improvement achieved by mixing in the wrong version.

This is especially important for technical sales. A document for a different configuration may look convincing enough to be reused in a response. Relaxing a market or version condition to avoid an empty result can move uncertainty from the assistant to the buyer, who may have no way to notice the mismatch. If the eligible evidence is absent, identify the gap rather than silently broadening the answer's basis.

An embedding change raises a separate question. Checking an existing index before changing the embedding model concerns the configuration behind similarity comparisons. Metadata constraints determine which records are eligible for those comparisons. A failure at either layer can leave an answer missing, but changing both at once makes the original cause harder to isolate.

Metadata filtering also does not, by itself, establish correct access control. A tenant or market value may constrain a query, but the application still needs to decide who is allowed to issue it and under which conditions. Better retrieval recall is not a reason to loosen document access. Test scope and authorisation as distinct properties instead of treating an empty or populated answer as proof of either.

For teams already operating a product knowledge base, record the failed query, its filter and the identifier of a document expected to qualify. Include the metadata stored on that document, not just the label shown in the catalogue. The developer can then distinguish a record that fails the condition from one that qualifies but is not retrieved, before deciding which configuration or data to change.

Sources