The Common AI Failure Pattern

How AI agents fail by testing logic when the bug is visual.

The Pattern

The most frequent testing failure in AI-assisted development follows a predictable pattern:

1. User reports: "It's not showing" or "It's still broken"

2. AI agent writes a unit test or checks logic

3. Logic test passes -- the data is correct, the component returns the right JSX

4. AI agent declares: "Fixed! The test passes."

5. User reports: "It's still not showing."

6. AI agent re-runs the same test, gets the same passing result

7. Cycle repeats until the user loses trust

Why It Happens

The AI agent chose Level 1 (unit test) or Level 2 (DOM test) for a problem that requires Level 5 (visual verification).

The data and logic are correct. The component renders the right elements in the DOM tree. But a CSS rule somewhere in the ancestor chain makes the element invisible.

Concrete Example

A developer asks the AI to add a notification banner. The AI creates the component:

// NotificationBanner.tsx
export function NotificationBanner({ message }: { message: string }) {
  return (
    <div className="notification-banner">
      <p>{message}</p>
    </div>
  );
}

And writes a test:

// NotificationBanner.test.tsx
it("renders the message", () => {
  render(<NotificationBanner message="Update available" />);
  expect(screen.getByText("Update available")).toBeTruthy();
});

The test passes. But the user sees nothing on screen. Why?

/* layout.css -- inherited from the page layout */
.main-content {
  overflow: hidden;
  max-height: 0;
  transition: max-height 0.3s ease;
}

.main-content.expanded {
  max-height: 1000px;
}

The notification banner is rendered inside .main-content, which has max-height: 0 and overflow: hidden by default. The element exists in the DOM (Level 2 passes), but it is visually clipped to zero height (Level 5 would catch this).

The Fix

Level 5 verification would reveal:

verify-ui result:
  .main-content {
    overflow: hidden     // <-- clipping children
    max-height: 0px      // <-- zero height
  }
  .notification-banner {
    display: block       // present in DOM
    // but parent clips it to invisible
  }

Other Variants of This Pattern

The overflow: hidden + height: 0 pattern is just one variant. Other common causes:

* Exception: jest-dom's toBeVisible() does catch display: none set inline (style="display: none") or via a hidden attribute — jsdom resolves these directly, and its CSSOM can even match simple <style>-tag selectors when that stylesheet is actually present in the test's DOM. What defeats Level 2 in practice is usually scope: a component test renders the component alone, so an ancestor stylesheet like .main-content above is never loaded into that DOM at all. And for geometry-driven hiding — clipping, off-screen positioning, stacking — jsdom has no layout engine, so Level 2 cannot see it regardless of how the CSS was authored.

The Hydration Mis-Nesting Variant

The CSS variants above are one family of "lower levels pass, page is broken." Islands architectures have another, structural one: hydration mis-nesting. When an island hydrates, its runtime re-emits the marker wrapper and re-parents the server-rendered children one level off -- a "double-wrap." The children are all still there; they are just nested one element too deep.

What makes this class dangerous is that every level below Level 4 passes:

• Level 1 / Level 2 (including render-to-string SSG-presence tests) pass -- the pre-hydration markup is correct.

• Level 3 passes -- the built dist HTML is correct.

The breakage appears only after hydration runs in a real browser, so nothing that inspects static output can see it. This is a structural, deterministic bug -- not a hydration race (a flake). It also recurs: one project hit this exact class twice, in two different island components.

Only a Level 4 test with a hydration wait catches it: trigger hydration (scroll the island into view, or otherwise activate it), wait for the hydration signal, then assert the post-hydration DOM structure -- child counts and vertical stacking, not just presence.

// island.hydration.spec.ts (Playwright)
test("island children are not double-wrapped after hydration", async ({ page }) => {
  await page.goto("/article-with-island");

  // Trigger + wait for hydration, not a fixed timeout.
  await page.locator("[data-island]").scrollIntoViewIfNeeded();
  await page.locator("[data-island][data-hydrated]").waitFor();

  // Assert STRUCTURE, not just presence: the items must be direct
  // children of the list, and they must stack vertically (a wrong
  // nesting level collapses or re-flows them).
  const items = page.locator("[data-island] > ul > li");
  await expect(items).toHaveCount(3);

  const first = await items.nth(0).boundingBox();
  const second = await items.nth(1).boundingBox();
  expect(second!.y).toBeGreaterThan(first!.y);
});

Key Takeaway

What If Level 5 Cannot Reach the Surface?

Most visibility bugs are caught by Level 5 because the element has a DOM bounding rect, computed styles, and pixel content a screenshot diff can compare. But canvas-based and photo-editor-style surfaces sometimes don't expose any of that — the rendered image lives inside a <canvas> element with no stable DOM child to assert on, and screenshot pixel-diff is too noisy to be reliable.

When both of these are true at once:

• L4 is intractable to write (canvas drop targets, multi-camera transforms, resizable layers)

• L5 cannot reach the assertion (no DOM rect, computed styles don't apply, pixel-diff too noisy)

…then Level 6 (AI-based) is the final resort. It is not the next thing to try when L5 is merely awkward — L6 is non-deterministic, cost-bearing, and not appropriate for CI. Reach for it only when there is no deterministic path to the assertion.