Owning Your LLM UI Layer After It Ships — Server‑Driven UI iOS Frameworks & Hover/Tap Patterns (SwiftUI)

Meta title: Owning Your LLM UI Layer After It Ships — Server‑Driven UI iOS Frameworks & Hover/Tap Patterns (SwiftUI)

Macro close-up of fractured ice surface symbolizing structured LLM UI layers in SwiftUI.

Meta title & description

Meta title: Owning Your LLM UI Layer After It Ships — Server‑Driven UI iOS Frameworks & Hover/Tap Patterns (SwiftUI)

Meta description: Tutorial for building server-driven UI iOS frameworks with SwiftUI — configurable event handlers, hover & tap image interactions, mapping LLM responses to native SwiftUI screens, plus safety and testing patterns.

When you generate native iOS UI from LLM responses, shipping the first feature is the easy part.

Owning that LLM UI layer after it ships—versioning, validating, and evolving hover/tap interactions—is where most teams get surprised.

This tutorial walks through how to:

  • Turn LLM responses into SwiftUI screens in a structured, server-driven way.
  • Add hover and tap interactions to "div‑wrapped" images, including text/data reveals.
  • Expose event handling patterns as configuration in a server-driven UI iOS framework.
  • Validate interaction schemas on the server for safety and long‑term maintainability.

It’s written for iOS engineers building AI‑native experiences with SwiftUI and server‑driven UI.

For a deeper product UX example—how AI answers should shape concrete flows—see our companion guide “What are my options now? — rebooking, refunds and the shape of a disruption answer”, which explores AI-native disruption workflows.

Prerequisites

Before you start, you should have:

  • A SwiftUI iOS app (Xcode 15+ recommended).
  • A backend capable of serving JSON (OpenAPI or similar).
  • Basic familiarity with:
    • Codable in Swift.
    • Server‑driven UI concepts.
    • LLM tools / structured outputs.

We’ll focus on a single pattern: an image block that reveals extra layers on hover or tap, driven by JSON config.

Step 1 – Define a server-driven ImageBlock JSON contract

To own your LLM UI layer, start by constraining what the model can output.

We’ll define a minimal JSON contract for ImageBlock with configurable interactions.

Sample JSON for an interactive image

{
"type": "imageBlock",
"id": "fare-card-1",
"imageUrl": "https://cdn.example.com/images/fare-card.png",
"altText": "Flex fare card with rebooking and refund options",
"interactions": {
"hover": {
"effect": "overlay",
"overlay": {
"text": "Includes free same-day rebooking",
"dataLayer": {
"rows": [
{ "label": "Rebooking window", "value": "24 hours" },
{ "label": "Refund fee", "value": "$50" }
]
}
}
},
"tap": {
"effect": "panel",
"panel": {
"title": "Fare details",
"dataLayer": {
"rows": [
{ "label": "Fare type", "value": "Flex" },
{ "label": "Changes", "value": "Allowed with fee" }
]
}
}
}
}
}

Key ideas:

  • type identifies the block kind.
  • interactions.hover.effect and interactions.tap.effect are enumerated strings (overlay, panel, etc.).
  • overlay and panel shape the text and data layers that will be rendered on interaction.

Minimal JSON Schema for ImageBlock & interactions

You should validate responses server‑side using a JSON Schema (or equivalent OpenAPI fragment).

Here’s a minimal JSON Schema you can adapt:

{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://example.com/schemas/image-block.json",
"title": "ImageBlock",
"type": "object",
"required": ["type", "id", "imageUrl", "altText"],
"properties": {
"type": { "const": "imageBlock" },
"id": { "type": "string" },
"imageUrl": { "type": "string", "format": "uri" },
"altText": { "type": "string" },
"interactions": {
"type": "object",
"properties": {
"hover": {
"type": "object",
"required": ["effect"],
"properties": {
"effect": {
"type": "string",
"enum": ["overlay"]
},
"overlay": {
"type": "object",
"required": ["text"],
"properties": {
"text": { "type": "string" },
"dataLayer": {
"type": "object",
"properties": {
"rows": {
"type": "array",
"items": {
"type": "object",
"required": ["label", "value"],
"properties": {
"label": { "type": "string" },
"value": { "type": "string" }
}
}
}
}
}
}
}
}
},
"tap": {
"type": "object",
"required": ["effect"],
"properties": {
"effect": {
"type": "string",
"enum": ["panel"]
},
"panel": {
"type": "object",
"required": ["title"],
"properties": {
"title": { "type": "string" },
"dataLayer": {
"type": "object",
"properties": {
"rows": {
"type": "array",
"items": {
"type": "object",
"required": ["label", "value"],
"properties": {
"label": { "type": "string" },
"value": { "type": "string" }
}
}
}
}
}
}
}
}
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}

This schema:

  • Enforces enumerated effects (overlay, panel).
  • Requires label/value pairs in dataLayer.rows.
  • Rejects unknown properties via additionalProperties: false.

You can embed this in OpenAPI via components.schemas.ImageBlock if you prefer.

Step 2 – Server-side validation & safety rules

With a schema in place, validate every LLM‑produced UI block before it reaches the app.

Example validation pseudocode

In a typical Node/TypeScript backend using ajv:

import Ajv from "ajv";
import schema from "./image-block.schema.json";

const ajv = new Ajv({ allErrors: true });
const validateImageBlock = ajv.compile(schema);

export function validateUiPayload(payload: unknown) {
if (!validateImageBlock(payload)) {
// Log and fall back
console.error("ImageBlock validation failed", validateImageBlock.errors);
return { type: "error", reason: "invalid_image_block" };
}

// Additional safety: only allow https image URLs
const imageUrl = (payload as any).imageUrl as string;
if (!imageUrl.startsWith("https://")) {
console.warn("Blocked non-https image URL", imageUrl);
return { type: "error", reason: "insecure_image_url" };
}

return payload;
}

Recommended safety rules:

  • Protocol: Require https:// for remote image URLs.
  • Domain allowlist: Restrict to known CDNs or asset domains.
  • Action allowlist: If tap/hovers trigger actions, constrain them to a set of known identifiers (show_detail_panel, navigate_to_offer).

These checks sit alongside schema validation.

Step 3 – Map JSON to SwiftUI types (Codable + naming alignment)

Once the server validates the payload, the app needs a clean Swift model that matches JSON.

Here we align field names via CodingKeys so the JSON contract is stable but Swift stays idiomatic.

Swift models for ImageBlock & interactions

import Foundation

struct ImageBlock: Codable, Identifiable {
let id: String
let imageUrl: URL
let altText: String
let interactions: Interactions?

enum CodingKeys: String, CodingKey {
case id
case imageUrl
case altText
case interactions
}
}

struct Interactions: Codable {
let hover: HoverInteraction?
let tap: TapInteraction?
}

enum HoverEffect: String, Codable {
case overlay
}

enum TapEffect: String, Codable {
case panel
}

struct HoverInteraction: Codable {
let effect: HoverEffect
let overlay: HoverOverlay?
}

struct TapInteraction: Codable {
let effect: TapEffect
let panel: TapPanel?
}

struct HoverOverlay: Codable {
let text: String
let dataLayer: DataLayer?
}

struct TapPanel: Codable {
let title: String
let dataLayer: DataLayer?
}

struct DataLayer: Codable {
let rows: [DataRow]
}

struct DataRow: Codable {
let label: String
let value: String
}

Because the JSON field names already match (overlay, panel, dataLayer, rows), we only need simple CodingKeys.

Concrete Codable decode example

Given the sample JSON from Step 1, decoding is straightforward:

let jsonData = sampleJsonString.data(using: .utf8)!

do {
let decoder = JSONDecoder()
let block = try decoder.decode(ImageBlock.self, from: jsonData)
print("Decoded ImageBlock id: \(block.id)")
} catch {
print("Failed to decode ImageBlock: \(error)")
}

If you add more nested structures later, keep JSON and Swift names aligned—or use explicit CodingKeys whenever you diverge.

Step 4 – Configurable event handlers for server-driven iOS

Now we’ll wire interactions into SwiftUI without hard‑coding behavior per screen.

Instead, we build a generic ServerDrivenImageBlockView that reads the interactions config and exposes event handling patterns.

Configurable event handlers for server-driven iOS

We’ll:

  • Use closures (onPanelRequested, onOverlayChanged) to plug into wider navigation or analytics.
  • Interpret interaction enums (HoverEffect, TapEffect) to decide which UI to show.

import SwiftUI

struct ServerDrivenImageBlockView: View {
let block: ImageBlock
let onPanelRequested: (TapPanel) -> Void
let onHoverOverlayChanged: (HoverOverlay?) -> Void

@State private var isHovering: Bool = false

var body: some View {
AsyncImage(url: block.imageUrl) { phase in
switch phase {
case .empty:
ProgressView()
case .success(let image):
image
.resizable()
.scaledToFit()
.modifier(hoverModifier)
.modifier(tapModifier)
case .failure:
placeholderView
@unknown default:
placeholderView
}
}
.accessibilityLabel(block.altText)
}

private var placeholderView: some View {
Rectangle().overlay(Text("Image unavailable"))
}

@ViewBuilder
private var hoverModifier: some ViewModifier {
// We’ll fill this in Step 5 for platforms that support hover.
EmptyModifier()
}

@ViewBuilder
private var tapModifier: some ViewModifier {
EmptyModifier()
}
}

struct EmptyModifier: ViewModifier {
func body(content: Content) -> some View { content }
}

We’ll implement hoverModifier and tapModifier next, with platform‑specific behavior.

Step 5 – add hover interaction to image SwiftUI (platform notes)

Hover is not universally available across Apple platforms.

From Apple’s SwiftUI documentation:

  • onHover(perform:) is available on:
    • macOS 10.15+
    • iPadOS 13.4+ (when a pointer device is present)
  • onContinuousHover(perform:) is available on:
    • macOS 13+
    • iPadOS 16+
  • iPhone has no hover concept—there is no pointer; hover handlers won’t fire.

Apple’s docs emphasize using platform‑appropriate APIs and native controls for interaction semantics (see SwiftUI View.onHover).

Implementing hover in SwiftUI with platform checks

Here’s a hover overlay implementation that only activates on platforms that support hover:

extension ServerDrivenImageBlockView {
private var supportsHover: Bool {
#if os(macOS)
return true
#elseif os(iOS)
// iPadOS pointer: treat all iOS as potentially hover-capable,
// but overlay will simply never be triggered on iPhone.
return true
#else
return false
#endif
}

private var hoverOverlay: HoverOverlay? {
guard
supportsHover,
let hover = block.interactions?.hover,
hover.effect == .overlay
else { return nil }
return hover.overlay
}

private var hoverModifier: some ViewModifier {
guard let hoverOverlay = hoverOverlay else {
return EmptyModifier()
}

return HoverOverlayModifier(
overlay: hoverOverlay,
isHovering: _isHovering,
onHoverOverlayChanged: onHoverOverlayChanged
)
}
}

struct HoverOverlayModifier: ViewModifier {
let overlay: HoverOverlay
@Binding var isHovering: Bool
let onHoverOverlayChanged: (HoverOverlay?) -> Void

func body(content: Content) -> some View {
content
.onHover { isInside in
isHovering = isInside
onHoverOverlayChanged(isInside ? overlay : nil)
}
.overlay(alignment: .bottomLeading) {
if isHovering {
VStack(alignment: .leading, spacing: 4) {
Text(overlay.text)
.font(.caption)
if let dataLayer = overlay.dataLayer {
ForEach(dataLayer.rows, id: \ .label) { row in
HStack {
Text(row.label)
Spacer()
Text(row.value)
}
.font(.caption2)
}
}
}
.padding(8)
.background(.ultraThinMaterial)
}
}
}
}

This pattern:

  • Reads configuration from hover.overlay.
  • Calls onHoverOverlayChanged so a higher‑level coordinator can react (analytics, state).
  • Uses overlay to render text and a data layer when hovering.

On iPhone, onHover simply never fires, and users only see tap semantics (we’ll configure those next).

Step 6 – tap interactions div-wrapped images SwiftUI

Tap is available on all iOS devices, and Apple recommends using Button or ButtonStyle for button‑like interactions to get native semantics (hit testing, accessibility).

For an image card that opens a detail panel, we’ll treat the entire image as a button.

Implementing tap-based panel reveals in SwiftUI

extension ServerDrivenImageBlockView {
private var tapPanel: TapPanel? {
guard let tap = block.interactions?.tap, tap.effect == .panel else {
return nil
}
return tap.panel
}

private var tapModifier: some ViewModifier {
guard let tapPanel = tapPanel else {
return EmptyModifier()
}
return TapPanelModifier(panel: tapPanel, onPanelRequested: onPanelRequested)
}
}

struct TapPanelModifier: ViewModifier {
let panel: TapPanel
let onPanelRequested: (TapPanel) -> Void

func body(content: Content) -> some View {
Button(action: {
onPanelRequested(panel)
}) {
content
.contentShape(Rectangle())
}
.buttonStyle(.plain)
}
}

This implementation:

  • Treats the image as a plain button for better semantics and accessibility.
  • Uses contentShape(.interaction) semantics via contentShape(Rectangle()) so the whole image hit‑tests.
  • Delegates behavior to onPanelRequested, which can:
    • Present a sheet.
    • Push a navigation destination.
    • Log an analytics event.

This is a clean example of configurable event handlers for server-driven iOS—the UI component doesn’t know or care what the panel does, only that tap triggers a structured event.

Best tools for server-driven UI iOS development

If you’re building a server-driven UI iOS framework around these patterns, you’ll need a stack that can:

  • Constrain LLM output via JSON Schema or OpenAPI.
  • Validate and version interaction patterns server‑side.
  • Render safe, native SwiftUI on the client.

Common components:

  • LLM layer with structured outputs
    • OpenAI Structured Outputs ensure responses adhere to a JSON Schema and make refusals programmatically detectable (source: developers.openai.com).
  • Schema validation
    • ajv (Node), jsonschema (Python), or OpenAPI‑native validators.
  • Server-driven UI orchestration
    • Your backend broker that chooses which blocks (image, panel, list, wizard) to send.
  • Native rendering layer
    • SwiftUI views tailored to your design system.

Uzori sits at this intersection by providing an AI interface layer that lets agents compose SwiftUI screens directly, validated by your server, and streamed into your app.

Step 7 – Handling unknown or future interaction effects safely

Your LLM and backend will evolve.

You need a clear strategy for unsupported effects so you don’t break shipped apps.

Express enumerated effects in JSON Schema

We already used:

  • enum: ["overlay"] for hover.
  • enum: ["panel"] for tap.

To prepare for future values, you can:

  • Add new enum values (e.g., "overlay", "badge", "blur").
  • Keep clients forward‑compatible by:
    • Ignoring unknown effects.
    • Falling back to a safe default.

Safe client-side handling of unknown effects

In Swift, treat enum decoding carefully:

enum HoverEffect: String, Codable {
case overlay

init(from decoder: Decoder) throws {
let container = try decoder.singleValueContainer()
let raw = try container.decode(String.self)
guard let effect = HoverEffect(rawValue: raw) else {
// Unknown effect: log and default to overlay, or ignore entirely.
print("Unknown HoverEffect: \(raw). Falling back to overlay.")
self = .overlay
return
}
self = effect
}
}

Alternatively, decode effect as a plain String and switch:

if hover.effectString == "overlay" {
// render overlay
} else {
// unsupported: no hover, log for telemetry
}

Server‑side, you should reject unknown effects via schema to keep the contract strict, or introduce a versioned schema (ImageBlockV2) when adding new ones.

Step 8 – Versioning interaction patterns in a server-driven UI layer

Owning an LLM UI layer means owning versions.

Practical patterns:

  • Embed a UI version field in payloads: { "type": "imageBlock", "uiVersion": "1.0", "id": "fare-card-1", "imageUrl": "https://...", "altText": "...", "interactions": { ... } }
  • Server‑side routing by version
    • 1.x → old schema, safe for legacy clients.
    • 2.x → new effects, fields, layouts.
  • Client feature flags
    • Guard new behaviors behind flags or capabilities reported to the server.
For disruption flows and complex decision trees, versioning is especially important; see “What are my options now? — rebooking, refunds and the shape of a disruption answer” for how answer shapes drive multi‑step UI.

Step 9 – Testing server-provided interactions

To keep quality high, treat server‑driven interaction patterns as testable units, not just data.

Recommended tests:

  • Schema validation tests (backend)
    • Given sample LLM responses, assert that invalid effects/URLs fail.
  • Snapshot tests (client)
    • Feed known JSON payloads into your SwiftUI views.
    • Use UI snapshot frameworks to verify layout.
  • Interaction tests (client)
    • For hover (macOS/iPadOS):
      • Simulate pointer entering/leaving the image area.
      • Assert overlay appears/disappears.
    • For tap (iOS):
      • Tap the image.
      • Assert onPanelRequested is called with expected TapPanel data.
  • End‑to‑end tests
    • Run against a staging backend returning real LLM‑generated payloads.

The goal: you can change the server schema or LLM prompts without breaking shipped apps.

Best tools for server-driven UI iOS development (summary)

To recap the best tools for server-driven UI iOS development with an LLM UI layer:

  • JSON Schema / OpenAPI
  • LLM with structured outputs (e.g., OpenAI Structured Outputs)
  • Backend validators and routers
  • SwiftUI rendering layer
  • Telemetry & logging for unknown interactions

Uzori’s SDK focuses on this bridge, letting you turn LLM responses into SwiftUI screens while keeping structure and safety on the server side.

AI interface layers beyond chat UI

Most production teams now treat UI as an operational layer with security, performance, and versioning concerns.

Examples from public engineering posts:

  • Shopify’s remote-rendering architecture separates UI definition from rendering via message passing to preserve look/feel, performance, and accessibility across platforms (source: shopify.engineering).
  • Meta’s Messenger rewrite used native OS primitives plus dynamic templates and a server broker to make the app smaller and faster at massive scale (source: engineering.fb.com).
  • Google’s A2UI and related protocol‑based interfaces emphasize that agents should send declarative UI, state, and actions rather than plain text (source: developers.googleblog.com).

These patterns all support the same idea: AI UX shouldn’t be just a chat box.

It should be a layer of server‑driven, native UI your team can own and evolve.

Configurable event handlers for server-driven iOS (recap)

The core pattern from this tutorial:

  • Server defines:
    • What interactions exist (hover, tap).
    • How they behave (overlay, panel).
    • What data is revealed (dataLayer.rows).
  • Client renders:
    • Native SwiftUI components (images, overlays, panels).
    • Native interactions (onHover, Button, contentShape).
  • Event handlers are configurable:
    • onPanelRequested(panel).
    • onHoverOverlayChanged(overlay?).

This is how you own your LLM UI layer after it ships: by treating interactions as structured, versioned, testable configuration—not one‑off code.

FAQ

Can I hover on iPhone?

No.

iPhone does not have a pointer or hover concept, so SwiftUI’s onHover and onContinuousHover callbacks do not fire in practice.

You can still compile code that uses onHover on iOS, but hover interactions will only be meaningful on:

  • macOS (mouse/trackpad).
  • iPadOS with a pointer device.

For iPhone, rely on tap and other touch gestures.

How should I version interaction patterns in a server-driven UI layer?

Use explicit versioning:

  • Add a uiVersion field to your UI payloads.
  • Maintain separate schemas per version (e.g., ImageBlockV1, ImageBlockV2).
  • Route responses server‑side based on client capabilities.

Avoid silent contract changes—treat new interaction types (badge, blur, etc.) as versioned additions.

How do I test server-provided interactions?

Combine:

  • Backend tests: ensure schema validation rejects bad payloads.
  • SwiftUI unit/snapshot tests: render blocks from fixed JSON examples.
  • Interaction tests: simulate hover/tap and assert overlays/panels behave correctly.
  • E2E tests: hit staging endpoints that return LLM‑generated UI blocks.

This gives you confidence that changing prompts or schemas won’t break your shipped app.

What are the security considerations for remote image URLs and actions?

Key considerations:

  • Image URLs:
    • Enforce https://.
    • Maintain domain allowlists.
    • Limit query parameters if needed.
  • Actions triggered by interactions:
    • Use a small, enumerated set of action identifiers.
    • Validate parameters server‑side.
    • Avoid arbitrary URLs or executable code from LLM output.

Always treat LLM‑generated UI as untrusted input that must be validated before rendering.

How should I handle unsupported hover/tap effects in the client?

Use safe fallbacks:

  • Decode effect as string.
  • Switch on known values (overlay, panel).
  • For unknown values:
    • Log for telemetry.
    • Skip the effect or fall back to a simple, non‑interactive render.

Server‑side, prefer rejecting payloads with unknown effects during validation to keep your contract strict.

By combining schema-constrained LLM output, server-driven validation, and native SwiftUI rendering, you get an AI interface layer you can actually own—long after the first feature ships.

Uzori’s iOS SDK is built around exactly these principles: one screen to integrate, infinite AI‑generated flows, all validated and rendered as native SwiftUI.

← All posts