Programmable 3D Avatar Faces

Org Status: 🟢 Active Cloudflare: N/A Last Audited: 2026-04-28

You want your AI assistant to have a face. Not a static icon, not a pulsing circle — a real animated face that blinks, lip-syncs to speech, and shifts expression based on system state. Think VIKI from I, Robot, or a VTuber-style avatar that lives in your macOS menu bar. This article maps every viable path from “I want a programmable face” to “it’s rendering at 60fps in a 50x60pt SwiftUI view.”

What you’ll learn:

• How the major face animation systems work (Live2D, SceneKit, Three.js, sprite sheets) and which ones you can actually control programmatically
• The ARKit blend shape standard and why it matters even when you’re not doing face tracking
• How to map TTS output to mouth shapes in real-time (Rhubarb, OVR Visemes, Azure, Apple AVSpeechSynthesizer)
• Which cloud avatar APIs (HeyGen, Simli, D-ID) offer embeddable streaming and which are just video generators
• Why a 2D sprite-sheet approach might beat full 3D for a notch-sized display
• Production-ready code for every approach: SceneKit blend shapes, SpriteKit state machines, WKWebView + Three.js, and pure SwiftUI animation

Every AI assistant looks the same: a colored circle that pulses when it’s “thinking.” Siri has one. Alexa has one. ChatGPT has one. This is a missed opportunity. Humans are wired to read faces — micro-expressions, gaze direction, mouth movement. A face creates presence in a way that no amount of smooth gradient animation can match.

But building a programmable face is hard for several specific reasons:

1. Most face engines assume face tracking, not face generation. ARKit’s blend shapes are designed to read your face via a TrueDepth camera, not to drive a synthetic face from code. You need to invert the pipeline.

2. Game engines are overkill. Unity and Unreal can absolutely render a talking face. But embedding a game engine runtime in a macOS notch HUD — a 50x60pt overlay window — is absurd. You need something lightweight.

3. Lip sync is a separate problem from expression. Getting the mouth to match speech requires phoneme-to-viseme mapping with sub-frame timing. Most “avatar” products handle this internally but don’t expose the control surface.

4. The display is tiny. At 50x60 points (100x120 pixels on Retina), you don’t need subsurface scattering or physically-based skin shading. You need readable expression at low resolution. This changes the entire technical calculus.

Here’s the concrete scenario: you’re building “Jane,” an AI assistant that lives in a macOS notch-area HUD. The HUD is a borderless NSPanel that overlays the MacBook screen. Jane’s face occupies roughly 50x60pt in the left ear of the notch. She needs to:

• Show 4-6 discrete expression states (calm, thinking, alert, alarmed, happy, speaking)
• Lip sync to TTS audio output
• Blink and perform idle animations (subtle head movement, gaze shifts)
• Transition smoothly between states
• Render at 60fps with negligible CPU/GPU impact
• Be driven entirely by code — no camera, no tracking, no user input

// The target API — what we want to be able to do
protocol AvatarController {
    func setExpression(_ expression: Expression, duration: TimeInterval)
    func setViseme(_ viseme: Viseme)
    func startIdleAnimation()
    func stopIdleAnimation()
    func blink()
    func setGazeDirection(_ direction: CGPoint) // -1...1 range
}

enum Expression: String, CaseIterable {
    case calm, thinking, alert, alarmed, happy, speaking, listening
}

enum Viseme: String, CaseIterable {
    case sil   // silence
    case pp    // p, b, m
    case ff    // f, v
    case th    // th
    case dd    // t, d
    case kk    // k, g
    case ch    // ch, j, sh
    case ss    // s, z
    case nn    // n, l
    case rr    // r
    case aa    // a
    case ee    // e
    case ih    // i
    case oh    // o
    case ou    // u
}

If you get this right, your assistant goes from a colored dot to a character with presence. Users will anthropomorphize it. They’ll say “she looks worried” or “she’s thinking about it.” That’s the goal.

Blend Shapes (Morph Targets)

A blend shape is a named deformation of a base mesh. You have a neutral face mesh, and a set of target meshes that represent extremes — mouth fully open, left eyebrow fully raised, eyes fully closed. Each blend shape has a weight from 0.0 to 1.0. At runtime, you interpolate between the neutral mesh and the target meshes based on weights.

// The blend shape concept as a type
interface BlendShapeState {
  // Mouth shapes (visemes)
  jawOpen: number;        // 0.0 - 1.0
  mouthFunnel: number;
  mouthPucker: number;
  mouthLeft: number;
  mouthRight: number;
  mouthSmileLeft: number;
  mouthSmileRight: number;
  mouthFrownLeft: number;
  mouthFrownRight: number;

  // Eyes
  eyeBlinkLeft: number;
  eyeBlinkRight: number;
  eyeLookUpLeft: number;
  eyeLookDownLeft: number;
  eyeLookInLeft: number;
  eyeLookOutLeft: number;

  // Brows
  browDownLeft: number;
  browDownRight: number;
  browInnerUp: number;
  browOuterUpLeft: number;
  browOuterUpRight: number;
}

// An expression is just a preset blend shape configuration
const EXPRESSIONS: Record<string, Partial<BlendShapeState>> = {
  calm: {
    mouthSmileLeft: 0.1,
    mouthSmileRight: 0.1,
    eyeBlinkLeft: 0.05,
    eyeBlinkRight: 0.05,
  },
  thinking: {
    browInnerUp: 0.3,
    eyeLookUpLeft: 0.2,
    eyeLookUpRight: 0.2,
    mouthPucker: 0.15,
  },
  alarmed: {
    eyeBlinkLeft: 0.0,
    eyeBlinkRight: 0.0,
    browInnerUp: 0.7,
    browOuterUpLeft: 0.5,
    browOuterUpRight: 0.5,
    jawOpen: 0.2,
    mouthFrownLeft: 0.3,
    mouthFrownRight: 0.3,
  },
};

Key insight: ARKit defines 52 blend shape locations. This has become the de facto standard — not because ARKit is the only system, but because every major avatar platform (Ready Player Me, VRChat, MetaHuman) supports ARKit-compatible blend shapes. If your face model uses ARKit naming, it will work with virtually any animation pipeline.

The ARKit blend shape set is documented at ARFaceAnchor.BlendShapeLocation. The critical thing to understand is that these values are just numbers. You don’t need a TrueDepth camera to use them. You can set jawOpen = 0.6 from code and render the result in SceneKit.

Visemes

A viseme is the visual representation of a phoneme — the mouth shape that corresponds to a speech sound. The most widely used standard is the Oculus OVR LipSync set of 15 visemes:

// The OVR viseme set — 15 mouth shapes cover all English speech
interface VisemeSet {
  sil: BlendShapeState;  // Silence / neutral
  PP: BlendShapeState;   // p, b, m — lips pressed together
  FF: BlendShapeState;   // f, v — lower lip to upper teeth
  TH: BlendShapeState;   // th — tongue between teeth
  DD: BlendShapeState;   // t, d — tongue to palate
  kk: BlendShapeState;   // k, g — back of tongue up
  CH: BlendShapeState;   // ch, j, sh — wide mouth
  SS: BlendShapeState;   // s, z — teeth together, lips slightly apart
  nn: BlendShapeState;   // n, l — tongue to palate, mouth relaxed
  RR: BlendShapeState;   // r — lips slightly rounded
  aa: BlendShapeState;   // a — wide open
  E: BlendShapeState;    // e — wide, mid-height
  ih: BlendShapeState;   // i — narrow, high
  oh: BlendShapeState;   // o — rounded, mid-height
  ou: BlendShapeState;   // u — rounded, small
}

// Mapping visemes to ARKit blend shapes
const VISEME_TO_BLENDSHAPE: Record<string, Partial<BlendShapeState>> = {
  sil: { jawOpen: 0.0, mouthFunnel: 0.0, mouthPucker: 0.0 },
  PP:  { jawOpen: 0.0, mouthPucker: 0.3, mouthFunnel: 0.0 },
  FF:  { jawOpen: 0.05, mouthFunnel: 0.2, mouthSmileLeft: 0.1 },
  TH:  { jawOpen: 0.1, mouthFunnel: 0.1 },
  DD:  { jawOpen: 0.15, mouthSmileLeft: 0.05, mouthSmileRight: 0.05 },
  kk:  { jawOpen: 0.2, mouthFunnel: 0.1 },
  CH:  { jawOpen: 0.15, mouthSmileLeft: 0.2, mouthSmileRight: 0.2 },
  SS:  { jawOpen: 0.05, mouthSmileLeft: 0.15, mouthSmileRight: 0.15 },
  nn:  { jawOpen: 0.1, mouthSmileLeft: 0.05, mouthSmileRight: 0.05 },
  RR:  { jawOpen: 0.15, mouthPucker: 0.2, mouthFunnel: 0.15 },
  aa:  { jawOpen: 0.5, mouthSmileLeft: 0.0, mouthSmileRight: 0.0 },
  E:   { jawOpen: 0.3, mouthSmileLeft: 0.2, mouthSmileRight: 0.2 },
  ih:  { jawOpen: 0.15, mouthSmileLeft: 0.3, mouthSmileRight: 0.3 },
  oh:  { jawOpen: 0.35, mouthPucker: 0.15, mouthFunnel: 0.25 },
  ou:  { jawOpen: 0.2, mouthPucker: 0.35, mouthFunnel: 0.3 },
};

Key insight: 15 visemes is the sweet spot. Fewer and the mouth looks robotic. More and you can’t interpolate smoothly at real-time speeds. The OVR set is language-agnostic — the same 15 shapes cover English, Japanese, French, and most other languages because visemes map to mouth shapes, not linguistic units.

Expression State Machine

Expressions aren’t instantaneous — you transition between them. A state machine manages this:

interface ExpressionTransition {
  from: string;
  to: string;
  duration: number;        // seconds
  easing: EasingFunction;
  blendShapeDeltas: Partial<BlendShapeState>;
}

type EasingFunction = 'linear' | 'easeIn' | 'easeOut' | 'easeInOut' | 'spring';

// Expression layers — allow lip sync and expression to coexist
interface AnimationLayer {
  name: string;
  weight: number;
  blendShapes: Partial<BlendShapeState>;
  priority: number; // higher priority layers override lower ones
}

// Layer system: expression + lip sync + idle can all run simultaneously
// Layer 0: Idle animation (blinks, micro-movements)
// Layer 1: Expression state (calm, thinking, alarmed)
// Layer 2: Lip sync (viseme animation — overrides mouth-related shapes only)
// Layer 3: Overrides (forced blink, gaze direction)

function composeLayers(layers: AnimationLayer[]): BlendShapeState {
  const sorted = layers.sort((a, b) => a.priority - b.priority);
  const result: Partial<BlendShapeState> = {};

  for (const layer of sorted) {
    for (const [key, value] of Object.entries(layer.blendShapes)) {
      const existing = result[key as keyof BlendShapeState] ?? 0;
      // Blend based on layer weight
      result[key as keyof BlendShapeState] =
        existing * (1 - layer.weight) + (value as number) * layer.weight;
    }
  }

  return result as BlendShapeState;
}

Key insight: Never apply lip sync and expression on the same blend shapes without a layering system. If “alarmed” sets jawOpen: 0.2 and the current viseme sets jawOpen: 0.5, you need a rule for resolving the conflict. The standard approach is to let lip sync override mouth shapes while expression controls brows and eyes.

Before diving into specific engines, you need to make a fundamental choice. At 50x60pt (100x120 pixels on Retina), do you even need 3D?

Option A: Full 3D (SceneKit / RealityKit / Metal)

A 3D face mesh with blend shapes, lit and rendered in real-time.

Pros:

• Smooth interpolation between any expression
• Continuous viseme blending (not discrete frames)
• Head rotation, lighting changes, particle effects
• Looks sophisticated even at small sizes — 3D depth cues read well

Cons:

• Requires a 3D model with properly weighted blend shapes
• GPU overhead (small but nonzero)
• More complex pipeline — model authoring, rigging, shader setup

Option B: 2D Sprite Sheets (SpriteKit / SwiftUI Animation)

Pre-rendered frames for each expression and viseme, played as sprite animations.

Pros:

• Zero GPU overhead — it’s just image swapping
• Perfect quality at target size (pre-rendered at exact resolution)
• Dead simple implementation
• Artist-friendly — any illustrator can make the frames

Cons:

• Combinatorial explosion: 7 expressions x 15 visemes x 3 blink states = 315 frames minimum
• Transitions between expressions are stepped, not smooth
• No continuous head rotation or gaze direction
• Adding a new expression means drawing more frames

Option C: Hybrid (2D layers with programmatic composition)

Separate layers for face base, eyes, mouth, brows — each independently animated.

Pros:

• Reduces combinatorial explosion (separate layers multiply, don’t explode)
• Smooth animation via layer-level transforms
• Good quality at small sizes
• Moderate complexity

Cons:

• Parallax and depth cues are limited
• Layer registration requires precision
• Still need pre-rendered assets per layer

Option D: WebView-based (WKWebView + Three.js / TalkingHead)

A web-based renderer embedded in a native macOS app via WKWebView.

Pros:

• Enormous ecosystem of Three.js face animation tools
• Ready Player Me avatars work out of the box
• TalkingHead library provides complete lip sync pipeline
• Cross-platform compatible

Cons:

• WKWebView has overhead (separate process, IPC)
• WebGL performance in WKWebView is inconsistent on macOS
• Bridge between Swift and JavaScript adds latency
• Memory footprint is much larger than native approaches

Recommendation for the Jane Use Case

For a 50x60pt notch display, Option C (Hybrid 2D layers) is the best starting point, with Option A (SceneKit) as the upgrade path. Here’s why:

At this size, 3D depth cues are barely perceptible. What matters is readable expression. A well-designed set of 2D layers — stylized, high-contrast, designed for the target resolution — will read better than a realistic 3D face shrunk to 100x120 pixels. The VIKI aesthetic (translucent, holographic, stylized) actually plays to 2D’s strengths.

If you later want the avatar to appear in a larger context (a floating panel, a full-screen takeover), that’s when you upgrade to SceneKit with blend shapes.

This is the gold standard for native macOS 3D face rendering. SceneKit is Apple’s high-level 3D framework, built on Metal, with first-class SwiftUI integration.

Setup: Loading a Face Model with Morph Targets

You need a 3D model (GLTF/USDZ/DAE) with blend shape targets. The model should have ARKit-compatible blend shape names. You can create one in Blender using the ARKit Blendshape Helper addon, or use a Ready Player Me avatar exported as GLB.

import SceneKit
import SwiftUI

class FaceSceneController: ObservableObject {
    let scene = SCNScene()
    private var faceNode: SCNNode?
    private var morpher: SCNMorpher?

    // Blend shape target indices — mapped from ARKit names
    private var blendShapeIndices: [String: Int] = [:]

    // Animation layers
    private var expressionLayer: [String: CGFloat] = [:]
    private var visemeLayer: [String: CGFloat] = [:]
    private var idleLayer: [String: CGFloat] = [:]

    // Idle animation state
    private var idleTimer: Timer?
    private var blinkTimer: Timer?
    private var nextBlinkTime: TimeInterval = 0

    func loadFaceModel(named modelName: String) {
        guard let modelScene = SCNScene(named: modelName) else {
            fatalError("Could not load face model: \(modelName)")
        }

        // Find the face mesh node (assumes it's named "Face" or "Head")
        guard let face = modelScene.rootNode.childNode(
            withName: "Face", recursively: true
        ) else {
            fatalError("No 'Face' node found in model")
        }

        faceNode = face
        scene.rootNode.addChildNode(face)

        // Access the morpher (blend shape controller)
        guard let m = face.morpher else {
            fatalError("Face node has no morpher — ensure model has blend shapes")
        }
        morpher = m
        m.calculationMode = .additive

        // Build index map from target names
        for (index, target) in m.targets.enumerated() {
            if let name = target.name {
                blendShapeIndices[name] = index
            }
        }

        setupCamera()
        setupLighting()
    }

    private func setupCamera() {
        let camera = SCNCamera()
        camera.fieldOfView = 30
        camera.zNear = 0.1
        camera.zFar = 100

        let cameraNode = SCNNode()
        cameraNode.camera = camera
        cameraNode.position = SCNVector3(0, 0, 0.5) // Close-up on face
        cameraNode.look(at: SCNVector3Zero)
        scene.rootNode.addChildNode(cameraNode)
    }

    private func setupLighting() {
        // Soft ambient + directional for readable features at small sizes
        let ambient = SCNLight()
        ambient.type = .ambient
        ambient.color = NSColor(white: 0.4, alpha: 1)
        let ambientNode = SCNNode()
        ambientNode.light = ambient
        scene.rootNode.addChildNode(ambientNode)

        let directional = SCNLight()
        directional.type = .directional
        directional.color = NSColor(white: 0.8, alpha: 1)
        directional.castsShadow = false
        let dirNode = SCNNode()
        dirNode.light = directional
        dirNode.eulerAngles = SCNVector3(-Float.pi / 4, Float.pi / 6, 0)
        scene.rootNode.addChildNode(dirNode)
    }

    // MARK: - Blend Shape Control

    func setBlendShape(_ name: String, weight: CGFloat, animated: Bool = true) {
        guard let index = blendShapeIndices[name] else { return }

        if animated {
            SCNTransaction.begin()
            SCNTransaction.animationDuration = 0.1
            morpher?.setWeight(weight, forTargetAt: index)
            SCNTransaction.commit()
        } else {
            morpher?.setWeight(weight, forTargetAt: index)
        }
    }

    func setExpression(_ expression: Expression, duration: TimeInterval = 0.3) {
        let targets = expression.blendShapeTargets

        SCNTransaction.begin()
        SCNTransaction.animationDuration = duration
        SCNTransaction.animationTimingFunction =
            CAMediaTimingFunction(name: .easeInEaseOut)

        // Reset all expression-controlled shapes
        for name in Expression.expressionShapeNames {
            if let index = blendShapeIndices[name] {
                let visemeValue = visemeLayer[name] ?? 0
                let targetValue = targets[name] ?? 0
                // Expression sets base, viseme overrides mouth shapes
                if Expression.mouthShapeNames.contains(name) {
                    morpher?.setWeight(max(targetValue, visemeValue), forTargetAt: index)
                } else {
                    morpher?.setWeight(targetValue, forTargetAt: index)
                }
            }
        }

        SCNTransaction.commit()
        expressionLayer = targets
    }

    func setViseme(_ viseme: Viseme, weight: CGFloat = 1.0) {
        let targets = viseme.blendShapeTargets

        // Visemes update immediately — no animation duration
        // The calling code handles interpolation timing
        for (name, value) in targets {
            guard let index = blendShapeIndices[name] else { continue }
            let scaledValue = value * weight
            morpher?.setWeight(scaledValue, forTargetAt: index)
            visemeLayer[name] = scaledValue
        }
    }

    // MARK: - Idle Animation

    func startIdleAnimation() {
        // Blink timer — humans blink every 2-10 seconds
        blinkTimer = Timer.scheduledTimer(withTimeInterval: 0.1, repeats: true) {
            [weak self] _ in
            self?.updateBlink()
        }
        scheduleNextBlink()

        // Micro-movement timer — subtle head sway
        idleTimer = Timer.scheduledTimer(withTimeInterval: 1.0 / 30.0, repeats: true) {
            [weak self] _ in
            self?.updateIdleMovement()
        }
    }

    func stopIdleAnimation() {
        blinkTimer?.invalidate()
        idleTimer?.invalidate()
        blinkTimer = nil
        idleTimer = nil
    }

    private func scheduleNextBlink() {
        nextBlinkTime = CACurrentMediaTime() + Double.random(in: 2.0...6.0)
    }

    private func updateBlink() {
        let now = CACurrentMediaTime()
        guard now >= nextBlinkTime else { return }

        // Blink animation: close over 0.05s, hold 0.05s, open over 0.1s
        SCNTransaction.begin()
        SCNTransaction.animationDuration = 0.05
        setBlendShape("eyeBlinkLeft", weight: 1.0, animated: false)
        setBlendShape("eyeBlinkRight", weight: 1.0, animated: false)
        SCNTransaction.commit()

        DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) { [weak self] in
            SCNTransaction.begin()
            SCNTransaction.animationDuration = 0.1
            self?.setBlendShape("eyeBlinkLeft", weight: 0.0, animated: false)
            self?.setBlendShape("eyeBlinkRight", weight: 0.0, animated: false)
            SCNTransaction.commit()
        }

        scheduleNextBlink()
    }

    private func updateIdleMovement() {
        let time = CACurrentMediaTime()
        // Perlin-noise-like subtle head movement
        let yaw = sin(time * 0.3) * 0.02
        let pitch = sin(time * 0.5 + 1.0) * 0.01
        faceNode?.eulerAngles.y = Float(yaw)
        faceNode?.eulerAngles.x = Float(pitch)
    }

    func setGazeDirection(_ direction: CGPoint) {
        // Map -1...1 to blend shape weights
        let lookRight = max(0, direction.x)
        let lookLeft = max(0, -direction.x)
        let lookUp = max(0, direction.y)
        let lookDown = max(0, -direction.y)

        setBlendShape("eyeLookOutLeft", weight: lookLeft * 0.5)
        setBlendShape("eyeLookInRight", weight: lookLeft * 0.5)
        setBlendShape("eyeLookInLeft", weight: lookRight * 0.5)
        setBlendShape("eyeLookOutRight", weight: lookRight * 0.5)
        setBlendShape("eyeLookUpLeft", weight: lookUp * 0.3)
        setBlendShape("eyeLookUpRight", weight: lookUp * 0.3)
        setBlendShape("eyeLookDownLeft", weight: lookDown * 0.3)
        setBlendShape("eyeLookDownRight", weight: lookDown * 0.3)
    }
}

Embedding in SwiftUI

import SwiftUI
import SceneKit

struct AvatarSceneView: NSViewRepresentable {
    @ObservedObject var controller: FaceSceneController

    func makeNSView(context: Context) -> SCNView {
        let scnView = SCNView()
        scnView.scene = controller.scene
        scnView.backgroundColor = .clear
        scnView.allowsCameraControl = false
        scnView.preferredFramesPerSecond = 60
        scnView.antialiasingMode = .multisampling4X
        scnView.isJitteringEnabled = false

        // For small views, reduce rendering quality for performance
        scnView.antialiasingMode = .none // At 50x60pt, AA is invisible
        scnView.preferredFramesPerSecond = 30 // 30fps is sufficient for expressions

        return scnView
    }

    func updateNSView(_ nsView: SCNView, context: Context) {
        // Scene updates happen through the controller, not through SwiftUI diffing
    }
}

// Usage in the HUD
struct AvatarZone: View {
    @StateObject private var faceController = FaceSceneController()

    var body: some View {
        AvatarSceneView(controller: faceController)
            .frame(maxWidth: .infinity, maxHeight: .infinity)
            .clipShape(UnevenRoundedRectangle(
                topLeadingRadius: 12,
                bottomLeadingRadius: 12,
                bottomTrailingRadius: 0,
                topTrailingRadius: 0
            ))
            .onAppear {
                faceController.loadFaceModel(named: "jane_face.usdz")
                faceController.startIdleAnimation()
                faceController.setExpression(.calm)
            }
    }
}

Key insight: At 50x60pt, turn off antialiasing (antialiasingMode = .none) and drop to 30fps. The visual difference is imperceptible at this size, but the GPU savings are real. You can always render at higher quality when the avatar appears in a larger context.

For the absolute lightest-weight approach, use pre-rendered sprite frames driven by a state machine. This is ideal for the Jane notch HUD.

import SpriteKit
import SwiftUI

// MARK: - Avatar Sprite Sheet Definition

struct AvatarSpriteSheet {
    // Each expression has a set of mouth frames (one per viseme)
    // and eye states (open, half, closed)
    struct ExpressionSet {
        let baseFace: String           // Asset name for the base face
        let visemeFrames: [Viseme: String] // Asset name per viseme
        let eyeOpen: String
        let eyeHalf: String
        let eyeClosed: String
        let browTexture: String        // Expression-specific brow
    }

    static let expressions: [Expression: ExpressionSet] = [
        .calm: ExpressionSet(
            baseFace: "face_calm",
            visemeFrames: Viseme.allCases.reduce(into: [:]) { $0[$1] = "mouth_calm_\($1.rawValue)" },
            eyeOpen: "eyes_calm_open",
            eyeHalf: "eyes_calm_half",
            eyeClosed: "eyes_calm_closed",
            browTexture: "brow_calm"
        ),
        .thinking: ExpressionSet(
            baseFace: "face_thinking",
            visemeFrames: Viseme.allCases.reduce(into: [:]) { $0[$1] = "mouth_thinking_\($1.rawValue)" },
            eyeOpen: "eyes_thinking_open",
            eyeHalf: "eyes_thinking_half",
            eyeClosed: "eyes_thinking_closed",
            browTexture: "brow_thinking"
        ),
        .alarmed: ExpressionSet(
            baseFace: "face_alarmed",
            visemeFrames: Viseme.allCases.reduce(into: [:]) { $0[$1] = "mouth_alarmed_\($1.rawValue)" },
            eyeOpen: "eyes_alarmed_open",
            eyeHalf: "eyes_alarmed_half",
            eyeClosed: "eyes_alarmed_closed",
            browTexture: "brow_alarmed"
        ),
        // ... more expressions
    ]
}

// MARK: - SpriteKit Avatar Scene

class AvatarSpriteScene: SKScene {

    // Sprite layers (back to front)
    private let faceSprite = SKSpriteNode()
    private let mouthSprite = SKSpriteNode()
    private let eyesSprite = SKSpriteNode()
    private let browSprite = SKSpriteNode()
    private let overlaySprite = SKSpriteNode() // holographic effects

    // State
    private var currentExpression: Expression = .calm
    private var currentViseme: Viseme = .sil
    private var isBlinking = false
    private var nextBlinkTime: TimeInterval = 0
    private var gazeOffset: CGPoint = .zero

    override func didMove(to view: SKView) {
        backgroundColor = .clear
        scaleMode = .aspectFill

        // Layer setup — all centered
        let layers: [SKSpriteNode] = [faceSprite, mouthSprite, eyesSprite, browSprite, overlaySprite]
        for (index, layer) in layers.enumerated() {
            layer.position = CGPoint(x: size.width / 2, y: size.height / 2)
            layer.zPosition = CGFloat(index)
            layer.size = size
            addChild(layer)
        }

        setExpression(.calm)
        scheduleNextBlink()
    }

    // MARK: - Expression Control

    func setExpression(_ expression: Expression, animated: Bool = true) {
        guard let set = AvatarSpriteSheet.expressions[expression] else { return }
        currentExpression = expression

        let update = {
            self.faceSprite.texture = SKTexture(imageNamed: set.baseFace)
            self.eyesSprite.texture = SKTexture(imageNamed: set.eyeOpen)
            self.browSprite.texture = SKTexture(imageNamed: set.browTexture)
            self.updateMouthForCurrentViseme()
        }

        if animated {
            // Cross-fade transition
            let fadeOut = SKAction.fadeAlpha(to: 0.7, duration: 0.1)
            let changeTextures = SKAction.run(update)
            let fadeIn = SKAction.fadeAlpha(to: 1.0, duration: 0.15)
            let sequence = SKAction.sequence([fadeOut, changeTextures, fadeIn])

            faceSprite.run(sequence)
            eyesSprite.run(sequence)
            browSprite.run(sequence)
        } else {
            update()
        }
    }

    // MARK: - Viseme Control

    func setViseme(_ viseme: Viseme) {
        currentViseme = viseme
        updateMouthForCurrentViseme()
    }

    private func updateMouthForCurrentViseme() {
        guard let set = AvatarSpriteSheet.expressions[currentExpression] else { return }
        if let textureName = set.visemeFrames[currentViseme] {
            mouthSprite.texture = SKTexture(imageNamed: textureName)
        }
    }

    // MARK: - Blink System

    private func scheduleNextBlink() {
        nextBlinkTime = CACurrentMediaTime() + Double.random(in: 2.5...5.5)
    }

    override func update(_ currentTime: TimeInterval) {
        // Blink check
        if !isBlinking && CACurrentMediaTime() >= nextBlinkTime {
            performBlink()
        }

        // Subtle eye position from gaze
        eyesSprite.position = CGPoint(
            x: size.width / 2 + gazeOffset.x * 2,  // 2pt max eye shift
            y: size.height / 2 + gazeOffset.y * 1.5
        )
    }

    private func performBlink() {
        isBlinking = true
        guard let set = AvatarSpriteSheet.expressions[currentExpression] else { return }

        let halfClose = SKAction.run {
            self.eyesSprite.texture = SKTexture(imageNamed: set.eyeHalf)
        }
        let fullClose = SKAction.run {
            self.eyesSprite.texture = SKTexture(imageNamed: set.eyeClosed)
        }
        let fullOpen = SKAction.run {
            self.eyesSprite.texture = SKTexture(imageNamed: set.eyeOpen)
            self.isBlinking = false
            self.scheduleNextBlink()
        }

        let sequence = SKAction.sequence([
            halfClose,
            SKAction.wait(forDuration: 0.03),
            fullClose,
            SKAction.wait(forDuration: 0.06),
            halfClose,
            SKAction.wait(forDuration: 0.03),
            fullOpen,
        ])

        eyesSprite.run(sequence)
    }

    // MARK: - Gaze

    func setGazeDirection(_ direction: CGPoint) {
        gazeOffset = direction
    }
}

// MARK: - SwiftUI Integration

struct SpriteAvatarView: View {
    let scene: AvatarSpriteScene

    init() {
        let s = AvatarSpriteScene(size: CGSize(width: 100, height: 120))
        s.scaleMode = .aspectFill
        scene = s
    }

    var body: some View {
        SpriteView(scene: scene, options: [.allowsTransparency])
            .frame(maxWidth: .infinity, maxHeight: .infinity)
    }
}

Asset production for the sprite approach:

For a layered 2D avatar, you need these assets per expression:

At 100x120px (Retina), each PNG is about 5-15KB. Total asset size: ~1-2MB. Trivial.

Key insight: The layered sprite approach separates the combinatorial problem. Instead of 7 x 15 x 3 = 315 pre-composed frames, you have 140 layer textures that combine at runtime. The SpriteKit compositor handles the layering with zero effort.

If you want a full 3D avatar with minimal native code, embed a web-based renderer. The TalkingHead library is the most complete open-source solution — it handles Ready Player Me avatars, lip sync, expressions, and idle animations out of the box.

import SwiftUI
import WebKit

class AvatarWebViewController: NSObject, ObservableObject, WKScriptMessageHandler {
    let webView: WKWebView

    init(frame: CGRect = .zero) {
        let config = WKWebViewConfiguration()
        config.preferences.setValue(true, forKey: "developerExtrasEnabled")

        // Enable WebGL
        let prefs = WKWebpagePreferences()
        prefs.allowsContentJavaScript = true
        config.defaultWebpagePreferences = prefs

        let contentController = WKUserContentController()
        config.userContentController = contentController

        webView = WKWebView(frame: frame, configuration: config)
        webView.setValue(false, forKey: "drawsBackground") // Transparent background

        super.init()

        contentController.add(self, name: "avatarBridge")
        loadAvatarPage()
    }

    private func loadAvatarPage() {
        let html = Self.avatarHTML
        webView.loadHTMLString(html, baseURL: Bundle.main.resourceURL)
    }

    // MARK: - Control API

    func setExpression(_ expression: String, duration: Double = 0.3) {
        callJS("setExpression('\(expression)', \(duration))")
    }

    func setViseme(_ viseme: String, weight: Double = 1.0) {
        callJS("setViseme('\(viseme)', \(weight))")
    }

    func speak(_ text: String) {
        let escaped = text.replacingOccurrences(of: "'", with: "\\'")
        callJS("speak('\(escaped)')")
    }

    func setMood(_ mood: String, intensity: Double = 0.5) {
        callJS("setMood('\(mood)', \(intensity))")
    }

    private func callJS(_ script: String) {
        webView.evaluateJavaScript(script) { _, error in
            if let error = error {
                print("JS error: \(error.localizedDescription)")
            }
        }
    }

    // MARK: - WKScriptMessageHandler

    func userContentController(
        _ userContentController: WKUserContentController,
        didReceive message: WKScriptMessage
    ) {
        // Receive events from the web avatar (e.g., animation complete, error)
        guard let body = message.body as? [String: Any] else { return }
        if let event = body["event"] as? String {
            print("Avatar event: \(event)")
        }
    }

    // MARK: - HTML Template

    static let avatarHTML = """
    <!DOCTYPE html>
    <html>
    <head>
        <meta charset="utf-8">
        <style>
            * { margin: 0; padding: 0; }
            body { background: transparent; overflow: hidden; }
            canvas { width: 100%; height: 100%; }
        </style>
    </head>
    <body>
        <script type="importmap">
        {
            "imports": {
                "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js",
                "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/",
                "talkinghead": "https://cdn.jsdelivr.net/npm/talkinghead@latest/modules/talkinghead.mjs"
            }
        }
        </script>
        <script type="module">
            import { TalkingHead } from 'talkinghead';

            let head;

            async function init() {
                const container = document.body;
                head = new TalkingHead(container, {
                    ttsEndpoint: null, // We'll drive TTS from Swift
                    cameraView: 'head',
                    cameraDistance: 0.4,
                    cameraRotateX: 0,
                    cameraRotateY: 0,
                    lightAmbientColor: 0x404040,
                    lightDirectColor: 0xcccccc,
                    background: { color: 'transparent' },
                });

                // Load a Ready Player Me avatar
                await head.showAvatar({
                    url: 'https://models.readyplayer.me/64bfa15f0e72c63d7c3934a6.glb?morphTargets=ARKit&textureAtlas=1024',
                    body: 'F',
                    avatarMood: 'neutral',
                    lipsyncLang: 'en',
                });

                // Notify Swift that avatar is ready
                window.webkit.messageHandlers.avatarBridge.postMessage({
                    event: 'ready'
                });
            }

            // API exposed to Swift via evaluateJavaScript
            window.setExpression = (name, duration) => {
                if (!head) return;
                const moods = {
                    calm: { mood: 'neutral', intensity: 0.3 },
                    thinking: { mood: 'concern', intensity: 0.4 },
                    alert: { mood: 'surprise', intensity: 0.5 },
                    alarmed: { mood: 'anger', intensity: 0.6 },
                    happy: { mood: 'joy', intensity: 0.7 },
                    listening: { mood: 'interest', intensity: 0.4 },
                };
                const m = moods[name] || moods.calm;
                head.setMood(m.mood);
            };

            window.setViseme = (viseme, weight) => {
                if (!head) return;
                // TalkingHead handles viseme interpolation internally
                // For manual control, directly set morph targets
                head.setMorphTarget(viseme, weight);
            };

            window.speak = (text) => {
                if (!head) return;
                head.speakText(text);
            };

            window.setMood = (mood, intensity) => {
                if (!head) return;
                head.setMood(mood);
            };

            init().catch(console.error);
        </script>
    </body>
    </html>
    """;
}

// MARK: - SwiftUI Wrapper

struct WebAvatarView: NSViewRepresentable {
    @ObservedObject var controller: AvatarWebViewController

    func makeNSView(context: Context) -> WKWebView {
        controller.webView
    }

    func updateNSView(_ nsView: WKWebView, context: Context) {}
}

Key insight: The WKWebView approach trades performance for ecosystem access. TalkingHead gives you Ready Player Me compatibility, built-in lip sync for multiple languages, emotion-to-blend-shape mapping, and Mixamo animation support — all for free. The cost is ~50MB of memory overhead and potential WebGL rendering inconsistencies. For a notch HUD, this may be too heavy. For a floating panel or full-screen mode, it’s excellent.

For the absolute simplest approach — no SpriteKit, no SceneKit, no WebView — you can build a face entirely in SwiftUI with shapes and animations.

import SwiftUI

// MARK: - Stylized Face Components

struct StylizedEye: View {
    let openness: CGFloat   // 0 = closed, 1 = fully open
    let lookDirection: CGPoint // -1...1

    var body: some View {
        ZStack {
            // Eye socket
            Ellipse()
                .fill(.white.opacity(0.9))
                .frame(width: 14, height: 10 * openness)
                .animation(.easeInOut(duration: 0.05), value: openness)

            // Iris
            Circle()
                .fill(
                    RadialGradient(
                        colors: [.cyan, .blue.opacity(0.8)],
                        center: .center,
                        startRadius: 0,
                        endRadius: 4
                    )
                )
                .frame(width: 6, height: 6)
                .offset(
                    x: lookDirection.x * 2,
                    y: lookDirection.y * 1.5
                )
                .opacity(openness > 0.3 ? 1 : 0)
                .animation(.easeOut(duration: 0.1), value: lookDirection)

            // Pupil
            Circle()
                .fill(.black)
                .frame(width: 3, height: 3)
                .offset(
                    x: lookDirection.x * 2,
                    y: lookDirection.y * 1.5
                )
                .opacity(openness > 0.3 ? 1 : 0)
        }
    }
}

struct StylizedMouth: View {
    let viseme: Viseme
    let expressionSmile: CGFloat // -1 (frown) to 1 (smile)

    private var mouthShape: some Shape {
        MouthPath(viseme: viseme, smile: expressionSmile)
    }

    var body: some View {
        mouthShape
            .fill(.white.opacity(0.7))
            .frame(width: 16, height: 10)
            .animation(.easeOut(duration: 0.05), value: viseme)
            .animation(.easeInOut(duration: 0.2), value: expressionSmile)
    }
}

struct MouthPath: Shape {
    var viseme: Viseme
    var smile: CGFloat

    var animatableData: CGFloat {
        get { smile }
        set { smile = newValue }
    }

    func path(in rect: CGRect) -> Path {
        var path = Path()
        let w = rect.width
        let h = rect.height
        let cx = rect.midX
        let cy = rect.midY

        switch viseme {
        case .sil:
            // Closed mouth — slight line
            let smileY = -smile * h * 0.1
            path.move(to: CGPoint(x: cx - w * 0.3, y: cy + smileY))
            path.addQuadCurve(
                to: CGPoint(x: cx + w * 0.3, y: cy + smileY),
                control: CGPoint(x: cx, y: cy - smileY * 2)
            )
        case .aa:
            // Wide open mouth
            path.addEllipse(in: CGRect(
                x: cx - w * 0.25, y: cy - h * 0.3,
                width: w * 0.5, height: h * 0.6
            ))
        case .oh:
            // Rounded O shape
            path.addEllipse(in: CGRect(
                x: cx - w * 0.15, y: cy - h * 0.25,
                width: w * 0.3, height: h * 0.5
            ))
        case .ee:
            // Wide smile
            path.addEllipse(in: CGRect(
                x: cx - w * 0.3, y: cy - h * 0.1,
                width: w * 0.6, height: h * 0.25
            ))
        case .pp:
            // Lips pressed together
            path.move(to: CGPoint(x: cx - w * 0.2, y: cy))
            path.addLine(to: CGPoint(x: cx + w * 0.2, y: cy))
        default:
            // Generic slightly open mouth for other visemes
            path.addEllipse(in: CGRect(
                x: cx - w * 0.2, y: cy - h * 0.15,
                width: w * 0.4, height: h * 0.3
            ))
        }

        return path
    }
}

// MARK: - Holographic Face Composite

struct HolographicFace: View {
    @StateObject private var animator = FaceAnimator()

    var body: some View {
        ZStack {
            // Scanline overlay (VIKI aesthetic)
            ScanlineOverlay()
                .opacity(0.15)

            // Face outline — holographic glow
            FaceOutline()
                .stroke(
                    LinearGradient(
                        colors: [.cyan.opacity(0.6), .blue.opacity(0.3)],
                        startPoint: .top,
                        endPoint: .bottom
                    ),
                    lineWidth: 1
                )

            // Eyes
            HStack(spacing: 10) {
                StylizedEye(
                    openness: animator.eyeOpenness,
                    lookDirection: animator.gazeDirection
                )
                StylizedEye(
                    openness: animator.eyeOpenness,
                    lookDirection: animator.gazeDirection
                )
            }
            .offset(y: -8)

            // Brows
            HStack(spacing: 14) {
                BrowShape(raise: animator.browRaise, furrow: animator.browFurrow)
                    .rotation3DEffect(.degrees(180), axis: (x: 0, y: 1, z: 0))
                BrowShape(raise: animator.browRaise, furrow: animator.browFurrow)
            }
            .offset(y: -16)

            // Mouth
            StylizedMouth(
                viseme: animator.currentViseme,
                expressionSmile: animator.mouthSmile
            )
            .offset(y: 10)
        }
        .frame(width: 50, height: 60)
        .background(
            RoundedRectangle(cornerRadius: 8)
                .fill(.black.opacity(0.8))
        )
        .onAppear {
            animator.startIdleLoop()
        }
    }
}

struct ScanlineOverlay: View {
    @State private var offset: CGFloat = 0

    var body: some View {
        GeometryReader { geo in
            VStack(spacing: 2) {
                ForEach(0..<30, id: \.self) { _ in
                    Rectangle()
                        .fill(.white.opacity(0.05))
                        .frame(height: 1)
                }
            }
            .offset(y: offset)
            .onAppear {
                withAnimation(.linear(duration: 3).repeatForever(autoreverses: false)) {
                    offset = 4
                }
            }
        }
    }
}

struct FaceOutline: Shape {
    func path(in rect: CGRect) -> Path {
        Path(ellipseIn: rect.insetBy(dx: 6, dy: 3))
    }
}

struct BrowShape: View {
    let raise: CGFloat  // 0 to 1
    let furrow: CGFloat // 0 to 1

    var body: some View {
        Path { path in
            path.move(to: CGPoint(x: 0, y: 4 - raise * 3 + furrow * 2))
            path.addQuadCurve(
                to: CGPoint(x: 10, y: 2 - raise * 2)),
                control: CGPoint(x: 5, y: 0 - raise * 4 + furrow * 3)
            )
        }
        .stroke(.white.opacity(0.6), lineWidth: 1.5)
        .frame(width: 10, height: 8)
    }
}

// MARK: - Face Animator

@MainActor
class FaceAnimator: ObservableObject {
    @Published var currentViseme: Viseme = .sil
    @Published var eyeOpenness: CGFloat = 1.0
    @Published var gazeDirection: CGPoint = .zero
    @Published var browRaise: CGFloat = 0
    @Published var browFurrow: CGFloat = 0
    @Published var mouthSmile: CGFloat = 0.1

    private var blinkTimer: Timer?
    private var idleTimer: Timer?

    func startIdleLoop() {
        // Random blinks
        scheduleNextBlink()

        // Subtle gaze drift
        idleTimer = Timer.scheduledTimer(withTimeInterval: 2.0, repeats: true) { [weak self] _ in
            Task { @MainActor in
                guard let self = self else { return }
                withAnimation(.easeInOut(duration: 1.5)) {
                    self.gazeDirection = CGPoint(
                        x: CGFloat.random(in: -0.3...0.3),
                        y: CGFloat.random(in: -0.2...0.2)
                    )
                }
            }
        }
    }

    private func scheduleNextBlink() {
        let delay = Double.random(in: 2.0...5.0)
        blinkTimer = Timer.scheduledTimer(withTimeInterval: delay, repeats: false) { [weak self] _ in
            Task { @MainActor in
                self?.performBlink()
            }
        }
    }

    private func performBlink() {
        withAnimation(.easeIn(duration: 0.05)) {
            eyeOpenness = 0.0
        }
        DispatchQueue.main.asyncAfter(deadline: .now() + 0.08) { [weak self] in
            withAnimation(.easeOut(duration: 0.1)) {
                self?.eyeOpenness = 1.0
            }
            self?.scheduleNextBlink()
        }
    }

    func setExpression(_ expression: Expression) {
        withAnimation(.easeInOut(duration: 0.3)) {
            switch expression {
            case .calm:
                browRaise = 0; browFurrow = 0; mouthSmile = 0.1
            case .thinking:
                browRaise = 0.3; browFurrow = 0.2; mouthSmile = 0
            case .alert:
                browRaise = 0.5; browFurrow = 0; mouthSmile = -0.1
            case .alarmed:
                browRaise = 0.7; browFurrow = 0.5; mouthSmile = -0.3
            case .happy:
                browRaise = 0.2; browFurrow = 0; mouthSmile = 0.5
            case .speaking:
                browRaise = 0.1; browFurrow = 0; mouthSmile = 0
            case .listening:
                browRaise = 0.2; browFurrow = 0.1; mouthSmile = 0.05
            }
        }
    }
}

Key insight: The pure SwiftUI approach has a unique advantage for the VIKI aesthetic — you can draw a stylized, holographic face with scan lines, glow effects, and translucent geometry that would be harder to achieve with a pre-made 3D model. The downside is that complex path animations at 60fps can cause SwiftUI body recomputation overhead. Keep the animated paths simple.

The lip sync pipeline is independent of the rendering approach. Whether you’re using SceneKit, SpriteKit, or SwiftUI, you need to convert speech audio into a stream of viseme events.

Approach A: Apple AVSpeechSynthesizer with Word-Level Timing

Apple’s TTS gives you word-level callbacks but not phoneme-level. You need to estimate visemes from the text.

import AVFoundation

class TTSVisemeDriver: NSObject, AVSpeechSynthesizerDelegate, ObservableObject {
    private let synthesizer = AVSpeechSynthesizer()
    private var currentUtterance: AVSpeechUtterance?
    private var fullText: String = ""

    // Callback for each viseme
    var onViseme: ((Viseme, TimeInterval) -> Void)?
    var onExpressionHint: ((Expression) -> Void)?

    override init() {
        super.init()
        synthesizer.delegate = self
    }

    func speak(_ text: String) {
        fullText = text
        let utterance = AVSpeechUtterance(string: text)
        utterance.voice = AVSpeechSynthesisVoice(language: "en-US")
        utterance.rate = AVSpeechUtteranceDefaultSpeechRate
        utterance.pitchMultiplier = 1.0
        currentUtterance = utterance
        synthesizer.speak(utterance)
    }

    // MARK: - Delegate

    func speechSynthesizer(
        _ synthesizer: AVSpeechSynthesizer,
        willSpeakRangeOfSpeechString characterRange: NSRange,
        utterance: AVSpeechUtterance
    ) {
        let nsString = fullText as NSString
        let word = nsString.substring(with: characterRange)

        // Convert word to phoneme sequence (simplified)
        let visemes = Self.wordToVisemes(word)

        // Schedule visemes across the estimated word duration
        let estimatedWordDuration = 0.15 * Double(word.count) // rough estimate
        let visemeDuration = estimatedWordDuration / Double(max(visemes.count, 1))

        for (index, viseme) in visemes.enumerated() {
            let delay = Double(index) * visemeDuration
            DispatchQueue.main.asyncAfter(deadline: .now() + delay) { [weak self] in
                self?.onViseme?(viseme, visemeDuration)
            }
        }
    }

    func speechSynthesizer(
        _ synthesizer: AVSpeechSynthesizer,
        didFinish utterance: AVSpeechUtterance
    ) {
        onViseme?(.sil, 0.1)
    }

    // MARK: - Phoneme-to-Viseme Mapping

    /// Simplified English text-to-viseme mapping
    /// For production use, integrate Rhubarb Lip Sync or a phoneme dictionary
    static func wordToVisemes(_ word: String) -> [Viseme] {
        var visemes: [Viseme] = []
        let lower = word.lowercased()
        var i = lower.startIndex

        while i < lower.endIndex {
            let char = lower[i]
            let next = lower.index(after: i) < lower.endIndex
                ? lower[lower.index(after: i)]
                : nil

            switch char {
            case "a": visemes.append(.aa)
            case "e": visemes.append(.ee)
            case "i": visemes.append(.ih)
            case "o": visemes.append(.oh)
            case "u": visemes.append(.ou)
            case "p", "b", "m": visemes.append(.pp)
            case "f", "v": visemes.append(.ff)
            case "t", "d":
                if next == "h" {
                    visemes.append(.th)
                    i = lower.index(after: i)
                } else {
                    visemes.append(.dd)
                }
            case "k", "g": visemes.append(.kk)
            case "s", "z": visemes.append(.ss)
            case "n", "l": visemes.append(.nn)
            case "r": visemes.append(.rr)
            case "c":
                if next == "h" {
                    visemes.append(.ch)
                    i = lower.index(after: i)
                } else {
                    visemes.append(.kk)
                }
            case "j": visemes.append(.ch)
            case "w": visemes.append(.ou)
            case "y": visemes.append(.ih)
            default: break // Skip non-letter characters
            }

            i = lower.index(after: i)
        }

        return visemes.isEmpty ? [.sil] : visemes
    }
}

Approach B: Rhubarb Lip Sync (Offline Pre-Processing)

Rhubarb Lip Sync is a command-line tool that analyzes audio files and produces frame-accurate viseme data. It uses speech recognition for better accuracy than frequency analysis alone.

import Foundation

struct RhubarbVisemeEvent: Decodable {
    let start: Double
    let end: Double
    let value: String // A-H mouth shape codes
}

struct RhubarbOutput: Decodable {
    let mouthCues: [RhubarbVisemeEvent]
}

class RhubarbLipSync {
    private let rhubarbPath: String

    init(rhubarbPath: String = "/usr/local/bin/rhubarb") {
        self.rhubarbPath = rhubarbPath
    }

    /// Process an audio file and return viseme timeline
    func process(audioFile: URL, dialogText: String? = nil) async throws -> [RhubarbVisemeEvent] {
        var args = [
            audioFile.path,
            "--machineReadable",
            "-f", "json",
            "--extendedShapes", "GHX" // Include extended mouth shapes
        ]

        // Providing dialog text dramatically improves accuracy
        if let text = dialogText {
            let textFile = FileManager.default.temporaryDirectory
                .appendingPathComponent("dialog.txt")
            try text.write(to: textFile, atomically: true, encoding: .utf8)
            args.append(contentsOf: ["-d", textFile.path])
        }

        let process = Process()
        process.executableURL = URL(fileURLWithPath: rhubarbPath)
        process.arguments = args

        let pipe = Pipe()
        process.standardOutput = pipe

        try process.run()
        process.waitUntilExit()

        let data = pipe.fileHandleForReading.readDataToEndOfFile()
        let output = try JSONDecoder().decode(RhubarbOutput.self, from: data)
        return output.mouthCues
    }

    /// Map Rhubarb mouth shape codes to OVR visemes
    static func rhubarbToViseme(_ code: String) -> Viseme {
        switch code {
        case "A": return .pp   // Closed mouth (m, b, p)
        case "B": return .ee   // Slightly open (most consonants)
        case "C": return .ee   // Open (e, ae)
        case "D": return .aa   // Wide open (a)
        case "E": return .oh   // Rounded (o)
        case "F": return .ou   // Tight rounded (u, w)
        case "G": return .ff   // Upper teeth on lower lip (f, v)
        case "H": return .nn   // Tongue behind teeth (l, th)
        case "X": return .sil  // Idle / silence
        default:  return .sil
        }
    }
}

/// Drives viseme playback from a Rhubarb timeline
class VisemePlayer {
    private var timeline: [RhubarbVisemeEvent] = []
    private var displayLink: CVDisplayLink?
    private var startTime: Double = 0
    private var currentIndex = 0

    var onViseme: ((Viseme) -> Void)?

    func play(timeline: [RhubarbVisemeEvent]) {
        self.timeline = timeline
        currentIndex = 0
        startTime = CACurrentMediaTime()

        // Use a timer instead of CVDisplayLink for simplicity
        Timer.scheduledTimer(withTimeInterval: 1.0/60.0, repeats: true) { [weak self] timer in
            guard let self = self else { timer.invalidate(); return }

            let elapsed = CACurrentMediaTime() - self.startTime

            // Find current viseme based on elapsed time
            while self.currentIndex < self.timeline.count {
                let event = self.timeline[self.currentIndex]
                if elapsed >= event.start && elapsed < event.end {
                    let viseme = RhubarbLipSync.rhubarbToViseme(event.value)
                    self.onViseme?(viseme)
                    break
                } else if elapsed >= event.end {
                    self.currentIndex += 1
                } else {
                    break
                }
            }

            if self.currentIndex >= self.timeline.count {
                timer.invalidate()
                self.onViseme?(.sil)
            }
        }
    }
}

Approach C: Azure Speech Service Viseme API

For the highest quality lip sync, Azure Speech Service provides real-time blend shape output — 55 facial positions per frame, streamed alongside TTS audio.

// Azure Speech SDK — TypeScript example for the pipeline concept
import * as SpeechSDK from 'microsoft-cognitiveservices-speech-sdk';

interface AzureVisemeFrame {
  frameIndex: number;
  blendShapes: number[][]; // 55 blend shape values per frame
}

class AzureVisemeDriver {
  private synthesizer: SpeechSDK.SpeechSynthesizer;
  private visemeQueue: AzureVisemeFrame[] = [];

  constructor(subscriptionKey: string, region: string) {
    const speechConfig = SpeechSDK.SpeechConfig.fromSubscription(
      subscriptionKey,
      region
    );
    speechConfig.speechSynthesisVoiceName = 'en-US-JennyNeural';

    const audioConfig = SpeechSDK.AudioConfig.fromDefaultSpeakerOutput();
    this.synthesizer = new SpeechSDK.SpeechSynthesizer(speechConfig, audioConfig);

    // Subscribe to viseme events
    this.synthesizer.visemeReceived = (sender, event) => {
      if (event.animation) {
        const animationData = JSON.parse(event.animation);
        // animationData.BlendShapes is an array of frames
        // Each frame is an array of 55 float values
        for (const frame of animationData.BlendShapes) {
          this.visemeQueue.push({
            frameIndex: this.visemeQueue.length,
            blendShapes: frame,
          });
        }
      }
    };
  }

  async speak(text: string): Promise<void> {
    this.visemeQueue = [];

    // Use SSML to request blend shapes
    const ssml = `
      <speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"
             xmlns:mstts="http://www.w3.org/2001/mstts" xml:lang="en-US">
        <voice name="en-US-JennyNeural">
          <mstts:viseme type="FacialExpression"/>
          ${text}
        </voice>
      </speak>
    `;

    return new Promise((resolve, reject) => {
      this.synthesizer.speakSsmlAsync(
        ssml,
        result => {
          if (result.reason === SpeechSDK.ResultReason.SynthesizingAudioCompleted) {
            resolve();
          } else {
            reject(new Error(`Speech synthesis failed: ${result.errorDetails}`));
          }
        },
        error => reject(error)
      );
    });
  }

  // Azure's 55 blend shapes map to ARKit names
  // See: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-speech-synthesis-viseme
  static readonly AZURE_TO_ARKIT: Record<number, string> = {
    0: 'eyeBlinkLeft',
    1: 'eyeLookDownLeft',
    2: 'eyeLookInLeft',
    3: 'eyeLookOutLeft',
    4: 'eyeLookUpLeft',
    5: 'eyeSquintLeft',
    6: 'eyeWideLeft',
    7: 'eyeBlinkRight',
    8: 'eyeLookDownRight',
    9: 'eyeLookInRight',
    10: 'eyeLookOutRight',
    11: 'eyeLookUpRight',
    12: 'eyeSquintRight',
    13: 'eyeWideRight',
    14: 'jawForward',
    15: 'jawLeft',
    16: 'jawRight',
    17: 'jawOpen',
    18: 'mouthClose',
    19: 'mouthFunnel',
    20: 'mouthPucker',
    21: 'mouthLeft',
    22: 'mouthRight',
    23: 'mouthSmileLeft',
    24: 'mouthSmileRight',
    25: 'mouthFrownLeft',
    26: 'mouthFrownRight',
    27: 'mouthDimpleLeft',
    28: 'mouthDimpleRight',
    29: 'mouthStretchLeft',
    30: 'mouthStretchRight',
    31: 'mouthRollLower',
    32: 'mouthRollUpper',
    33: 'mouthShrugLower',
    34: 'mouthShrugUpper',
    35: 'mouthPressLeft',
    36: 'mouthPressRight',
    37: 'mouthLowerDownLeft',
    38: 'mouthLowerDownRight',
    39: 'mouthUpperUpLeft',
    40: 'mouthUpperUpRight',
    41: 'browDownLeft',
    42: 'browDownRight',
    43: 'browInnerUp',
    44: 'browOuterUpLeft',
    45: 'browOuterUpRight',
    46: 'cheekPuff',
    47: 'cheekSquintLeft',
    48: 'cheekSquintRight',
    49: 'noseSneerLeft',
    50: 'noseSneerRight',
    51: 'tongueOut',
    52: 'headRoll',
    53: 'leftEyeRoll',
    54: 'rightEyeRoll',
  };
}