← All perspectives

Perspective / UX engineering

The UX debt hidden in AI-generated prototypes—and how to measure it before launch

AI-generated prototypes accelerate visible output, while usability risks can remain unresolved. A technical evaluation framework connects task evidence, failure severity and release decisions to the cost of turning a prototype into a usable service.

10 min readReviewed

Hands testing a simple mobile interface beside a paper prototype.
Conceptual editorial image.

AI-assisted prototyping can shorten the distance between an idea and an interactive demonstration. The commercial risk appears when that demonstration is treated as evidence that the underlying customer journey works. Usability depends on whether intended users can complete meaningful tasks under realistic conditions, including errors, interruptions and unusual cases. A structured evaluation programme makes those uncertainties visible before launch. It also helps leadership distinguish valuable acceleration from work that has simply moved into validation, remediation and support.

The executive view

  • Treat an interactive prototype as a testable proposition, with its fidelity and limitations documented.
  • Use observed task behaviour and explicit measures; synthetic participants are not evidence from real users.
  • Prioritise UX debt by consequence, exposure and recoverability, then retest material changes before release.
01

Separate generation speed from validated progress

NNGroup’s October 2026 survey of 521 UX practitioners reports time savings alongside concerns about output quality and verification. Those are practitioner reports, not a controlled measurement of product outcomes. They support a useful question: what evidence shows that faster production has created something people can use?

Measure progress through resolved uncertainty. A prototype can help test navigation, terminology or the sequence of a task. It cannot establish every property of the eventual service, particularly when its data, permissions and error states are simulated.

Create an assumption register before evaluation. Each entry should identify a proposed behaviour, why it matters, the evidence required and the decision that depends on it. This turns an impressive demonstration into a testable proposition. It also stops a polished interaction from silently approving unrelated questions about operational feasibility or customer understanding.

There is a difference between reducing the effort to explore an option and reducing the effort to certify a service. A generated checkout can make it cheap to compare interaction sequences. It does not settle whether an account can purchase on credit, whether a price includes an agreed discount or whether a failed payment can be retried safely. Put those decisions into the product backlog with named owners rather than allowing visible fidelity to stand in for commercial policy.

Track a small number of evidence milestones: task definition agreed, uncertain interaction evaluated, major issue corrected and changed behaviour retested. These milestones give a sponsor a clearer view than screen counts. They also reveal when generation creates more options than the team can evaluate. A sensible response is to constrain the experiment to the deciding question, not simply increase the rate of production. The resulting prototype becomes a learning asset with a known scope instead of a progressively larger set of unexamined commitments.

02

Specify the prototype’s fidelity and its gaps

Fidelity has several dimensions. Visual accuracy, interaction completeness, realistic content, data behaviour and organisational rules can vary independently. A high-quality screen may contain a simplified workflow; a rough interface may support a realistic end-to-end task.

For an illustrative expense-approval service, state which behaviours are real. Can the prototype enforce role restrictions, retain a draft, represent a rejected expense and show a changed approval state? Does it simulate slow responses and unavailable documents? Mark unimplemented behaviour as a limitation in the evaluation plan.

Keep a coverage map from task to supported states. If a participant cannot reach a necessary state because the prototype lacks it, record that as a limitation rather than a usability success or failure. The resulting evidence becomes more useful because decision-makers can see exactly what was tested and what remains an assumption.

A fidelity record should also specify where the prototype borrows its apparent intelligence. A static response, a live language model, a human-operated simulation and a production integration can all produce a convincing interaction, but they support different conclusions. If a researcher manually supplies the perfect answer, the study may test presentation while saying little about the reliability of retrieval or generation.

For AI features, include uncertain and contradictory outputs deliberately. Can a user inspect the source, challenge a suggestion, continue without it and understand what was actually committed? A prototype that always returns a plausible response removes the conditions that distinguish an assistive service from an unpredictable one. Label simulated failure cases in the research record, while presenting them to participants in a way that preserves the intended task. Preserve the prompt, model configuration and representative inputs when live generation is material to the result. Without that configuration record, a later team may be unable to reproduce the behaviour that informed the design decision.

03

Evaluate representative tasks with appropriate participants

Define tasks through user goals, using language that does not reveal the intended navigation. Asking someone to select a named button tests recognition of an instruction; asking them to resolve an incorrect claim tests whether the service supports their work.

Recruit participants whose experience and responsibilities reflect the decision being evaluated. New users and experienced operators may need different sessions. Include relevant accessibility needs and working conditions in the plan. Determine sample size from research purpose and risk, rather than presenting one universal participant count as sufficient for every service.

AI can assist with preparation and analysis, but generated personas and simulated responses should not be represented as evidence from real users. Record how evidence was obtained and who reviewed interpretations. The objective is to understand behaviour in context, including where participants hesitate, misunderstand a term or recover using an unexpected route.

The GOV.UK Service Manual on moderated usability testing describes observing actual or likely users attempting specific tasks. Apply that principle to commercial software by recruiting across the roles whose work and authority differ. An expense claimant, line manager and finance administrator should not be treated as interchangeable participants merely because they share an employer.

Control task order where learning could distort later results. Someone who has already discovered a hidden menu may succeed on the next task for that reason, rather than because the revised wording is clear. Separate exploratory qualitative work from a comparative experiment intended to estimate a performance difference. Record prior experience, assistance and relevant environment differences. A handful of sessions can expose serious design problems without supporting a population-wide completion-rate claim. State the inference the evidence supports, then identify what extra research would be required for a larger investment decision. That precision is particularly valuable when a senior team is tempted to generalise from a persuasive demonstration.

04

Define measures before observing the results

Establish what independent completion means for each task. Distinguish success without assistance, success with assistance, partial completion and abandonment. Define the start and end points for timing, and record interruptions that make a timing comparison unsuitable.

Capture errors that matter to the business: choosing the wrong account, approving the wrong amount, losing a draft or misunderstanding whether a request was submitted. Confidence and satisfaction can add context, but they should not substitute for observed completion.

Use a consistent evidence table linking an observation to its interpretation and recommendation. A participant’s comment is evidence of what that person reported; the team’s conclusion about its cause is an interpretation. Preserve that distinction. It makes the analysis easier to challenge and prevents a persuasive quotation from carrying more weight than repeated behavioural evidence or a high-severity failure.

Suppose a small illustrative formative study contains eight task attempts and six unaided completions. Reporting 75% is arithmetically correct, but presenting it as the expected production conversion rate would overstate what this sample establishes. The useful finding may be that two participants misunderstood the same approval state. Preserve the observations, investigate the cause and test the proposed correction. Larger quantitative conclusions need an appropriate sampling and analysis design.

Timing also needs a clear interpretation. A shorter task can mean better interaction, omitted checking or a user who has misunderstood the consequence. Pair duration with correct final state and comprehension. For an AI recommendation, ask whether users can distinguish source evidence from model interpretation, without prompting them towards the desired answer. Retain contradictory evidence rather than averaging it away. A successful fast path and an unsafe failure path can coexist in one product. A decision-maker needs both findings because a single mean score cannot describe the release consequence of each.

05

Work through an illustrative approval journey

Consider the hypothetical expense service. Its main demonstration shows a successful claim moving from submission to approval. A fuller evaluation deliberately introduces conditions outside that path.

Task conditionBehaviour to evaluateRelevant evidence
Incorrect amountUser recognises and corrects the error before submissionIndependent correction and final submitted value
Missing receiptUser understands the requirement and can recoverError comprehension and successful continuation
Interrupted sessionUser can identify what was retainedRecovery path and confidence matched to actual state
Approver lacks authorityService explains the next permitted actionNo unintended approval and a usable alternative
Claim changes during reviewUser understands the current versionDecision made against the correct information

The table is an illustrative test framework, not reported research. It shows how a product team can extend evaluation beyond the attractive central path. Each condition needs an expected behaviour and an explicit decision about the adequacy of the evidence.

06

Prioritise UX debt by consequence and reach

NNGroup’s analysis of AI-related UX debt highlights the risk of producing interfaces faster than they are evaluated. For delivery governance, translate that concern into a remediation backlog with explicit consequences.

Assess severity, likely exposure, recoverability and the confidence of the evidence. A rare error that causes an irreversible commercial decision may deserve attention before a frequent but minor inconvenience. Do not collapse those dimensions into a single unexplained score.

For each issue, state whether it belongs to content, interaction design, a shared component or the underlying service rule. Estimate remediation effort with uncertainty visible. Repairing a local label and reworking an approval model are different investments. Evaluate whether a shared fix will address several journeys or whether it risks changing behaviour elsewhere. That analysis helps leadership fund the cause rather than repeatedly treating symptoms.

Translate exposure into operating consequences without pretending uncertain inputs are precise. As a hypothetical planning example, assume 2,000 monthly claims, a suspected 4% rate of avoidable support contact and ten minutes per contact. That implies roughly 13.3 hours of support effort per month. It is a scenario for deciding whether to investigate the issue, not a forecast established by a prototype study. Replace the assumed contact rate with live evidence when available and keep financial approval errors outside this simple time-saving model.

Some issues require a release block even when their frequency is unknown. A control that can approve a claim against the wrong legal entity deserves a different decision from a confusing secondary label. Record the affected authority, the potential consequence and the available recovery route. Make the accountable business owner part of that decision. This prevents a general UX score from accepting a consequential workflow defect and keeps remedial work connected to the risk it actually addresses.

07

Make release readiness an evidence-based decision

Agree exit criteria before the final demonstration. Identify the critical tasks and failure types that must be resolved, the evidence that needs review and the limitations that require further work. Avoid retrospectively redefining success to fit the results available.

Retest material changes using the relevant tasks. A fix can remove one problem while introducing another, particularly when a shared pattern or workflow rule changes. Preserve the earlier observation and the new result so the decision remains traceable.

The executive artefact is a concise readiness record containing task coverage, unresolved risks, evidence quality and the rationale for proceeding or delaying. This does not eliminate judgement. It gives judgement a clear foundation. AI-assisted prototyping creates most value when the time saved is used to explore and evaluate better options, with the resulting service assessed on what people can reliably accomplish.

Further questions

Questions this raises

What is UX debt in an AI-generated prototype?

It is unresolved usability work carried forward as the prototype develops, including unclear journeys, missing states and untested assumptions. Generation speed can conceal that work when visible output is treated as validated progress. The debt can sit beneath a visually complete interface. A prototype may omit recovery from an interrupted session, use terminology that only its designer understands or imply that a generated recommendation has already been approved. Catalogue those assumptions against actual tasks. The useful unit of debt is an unresolved behaviour with a consequence and an owner, rather than the number of screens awaiting polish.

Can AI-generated personas replace user testing?

They can support ideation or preparation, but they do not provide observations from intended users. Any simulated evidence should be labelled and kept separate from research with actual participants. A simulated participant can generate hypotheses for researchers to investigate, but it cannot establish whether an intended user understood a label, completed a task or trusted an outcome appropriately. Record simulated responses as synthetic material. Use suitable actual or likely users for the decisions that need behavioural evidence, and preserve the recruitment and task context when reporting results to leadership.

How should a team decide whether a prototype is ready to become a product?

Use explicit task coverage, realistic state evaluation, severity-based remediation and evidence-based exit criteria. Technical production readiness needs separate assessment alongside usability. The readiness record should identify the decisions supported by the study and the important gaps. A realistic usability result does not establish production security, performance or transaction integrity. Conversely, technically sound software can still support a confusing journey. Bring those evidence streams together at the release decision, assigning unresolved conditions to accountable owners rather than treating one successful demonstration as universal approval.

References

Sources & further reading

  1. NNGroup: AI in UX—Work Is Faster, But Practitioners Have Mixed Feelings 23 October 2026
  2. NNGroup: The Custodial Era of UX—Cleaning Up After AI 28 August 2026
  3. GOV.UK Service Manual: Using moderated usability testing

Connected capabilities

Brand & digital experienceSoftware & CRMAI solutions

From thinking to doing

What could this mean
for your business?

Your next chapter

What could your
business do better?

Tell us where you want to go, what’s getting in the way, or what you need to deliver. We’ll start there.

Start a conversation

Start a conversation

Tell us what
you have in mind.

Interactive preview. Your information stays in this page and is not sent.