An August 14 AWS Machine Learning Blog post on custom reward functions for multi-turn reinforcement learning with Amazon Nova Forge makes a practical warning: the reward function determines what a model actually learns. A subtly defective metric can teach the wrong behavior while the overall training curve still appears healthy. AWS describes composite evaluation using outcome rewards, intermediate behavior signals and penalties, and emphasizes that each component needs to create meaningful variation. For an export company building AI lead qualification, the connection is immediate. An agent will pursue the behavior that earns a score, which may be very different from understanding a real buying task.

A complete form is not a verified buying situation

Suppose an agent receives a higher score whenever it fills company name, country, quantity, email and expected delivery date. It may learn to ask repetitive questions, infer missing details or force an ambiguous message into a convenient category. The CRM record looks complete, yet the end use, decision role, technical fit and timing may still be unknown.

Define the endpoint as a reviewable buying-task brief, not a field-completion percentage. The brief should preserve the buyer's wording, identify the evidence supporting each conclusion, mark unknowns and recommend a next step. “Unknown” is often the correct answer. Guessing should not be rewarded merely because it eliminates a blank field.

This is particularly important across languages. A distributor's phrase may be a regional category label rather than a precise model requirement. A quantity may refer to samples, cartons or annual demand. The system should surface the ambiguity instead of translating it into false precision.

Separate outcome, behavior and penalty signals

An outcome score can test whether the final brief supports an appropriate route to sales, engineering, compliance or rejection. A behavior score can check whether the agent cited the original message, asked a necessary clarification, preserved units and recognized contradictory statements. Penalties can discourage invention, repeated questions, unauthorized promises and loops that make no progress.

Keep the components visible. One high-weight score should not conceal the fact that another component contributes nothing. As AWS notes in its technical context, a reward term that takes the same value across all candidates does not provide a useful learning signal. In a business evaluation, a “politeness” score that is always perfect may add no decision value, while a rarely triggered contradiction check may be poorly calibrated.

Inspect score distributions and example conversations for every component. Ask whether higher-scored outputs actually help a person make the intended routing decision. A rising aggregate metric is insufficient if the agent has discovered a cheaper way to satisfy the grader.

Test the disagreements found in export sales

The evaluation set should include messages that mix a genuine inquiry with a service pitch, uncertain buyer-versus-trader identity, conflicting quantity units, multiple languages, missing attachments, market-compliance questions and urgent delivery requests. Include cases where the correct outcome is “insufficient evidence,” “manual review required” or “outside our product scope.”

Use a fixed regression set when the prompt, model, tool or reward changes. Add newly discovered failure patterns without removing older difficult cases. Preserve examples that challenge the preferred narrative; otherwise the evaluation set can become a collection of demonstrations the agent already knows how to pass.

Human reviewers also need a disagreement rule. If sales and engineering assess the same lead differently, record why. The purpose is not to force instant consensus but to improve the definition of routing evidence and clarify which owner decides technical fit.

What this means for Chinese exporters

An export follow-up agent optimized only for reply speed, field count or automatic classification can move the organization toward impressive but commercially weak metrics. The useful outcome is fewer incorrect handoffs, faithful preservation of buyer intent, visible information gaps and a clear reason for the next question.

Business owners must therefore participate in reward and evaluation design. Engineering can implement the measurements, but sales and delivery teams define a usable buying task. Compliance and product owners define the conditions that an automated system must escalate rather than decide.

The same discipline supports management reporting. Separate the percentage of messages processed, the percentage with sufficient evidence, the manual-review rate and the later handoff correction rate. None of these alone proves commercial success, but together they reveal whether the agent is reducing ambiguity or merely producing fuller records.

Action checklist

Sources