hz-unity-face-tracking

Unity Face Tracking for ARKit-Rigged Models (Meta Movement SDK)

End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Meta VR. Uses the public OVRCustomFace extension hook — no OVR_INTERNAL_CODE, ships to 3P.

The model's blendshape names must follow the ARKit naming convention (camelCase, _L/_R suffixes, e.g. eyeBlink_L, jawOpen, mouthSmile_R). An optional prefix like blendShape2.eyeBlink_L is automatically stripped.

When to use

• User has an FBX/GLB/mesh with ARKit-named blendshapes and wants it driven by Meta VR face tracking.
• User asks: "animate this head with my face", "drive these blendshapes from face tracking", "use Movement SDK A2E with my model", "wire ARKit shapes to Meta VR".
• Target device: Quest Pro, Quest 3, Quest 3S (Quest 2 is no-op — no face cameras).

Prerequisites checklist

1. Quest face tracking-capable headset (Pro / 3 / 3S).
2. Packages in Packages/manifest.json:
   • com.meta.xr.sdk.core (Meta XR Core — provides OVRFaceExpressions, OVRCustomFace)
   • com.meta.xr.sdk.movement (Meta Movement SDK — A2E + retargeting helpers)
3. Assets/Oculus/OculusProjectConfig.asset (verify via Project Settings → Meta XR):
   • faceTrackingSupport: 1 (Supported) or 2 (Required)
   • eyeTrackingSupport: 1 if the rig has gaze
4. Android manifest (Assets/Plugins/Android/AndroidManifest.xml) permissions:
   • <uses-feature android:name="oculus.software.face_tracking" android:required="false" />
   • <uses-permission android:name="com.oculus.permission.FACE_TRACKING" />
   • <uses-permission android:name="android.permission.RECORD_AUDIO" /> (required for A2E)
   • For eye gaze: oculus.software.eye_tracking + com.oculus.permission.EYE_TRACKING
5. OVRCameraRig in the scene with OVRManager.FaceTrackingDataSources including Audio (A2E) — if you skip Audio, mouth motion is visual-only.

After any change to OculusProjectConfig, regenerate the manifest (see Regenerating the manifest below).

Approach (high level)

1. Drop the ARKitOVRCustomFace script (in references/ARKitOVRCustomFace.cs) into the project.
2. Add an OVRFaceExpressions component to the OVRCameraRig (or anywhere in the scene).
3. On the GameObject that has the model's SkinnedMeshRenderer, add ARKitOVRCustomFace. Adding the component triggers Reset(), which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names.
4. Wire the component's FaceExpressions field to the OVRFaceExpressions instance.
5. For eye gaze: drop ARKitEyeGazeBlendshapeDriver in (see Eye tracking below).
6. Press Play with Meta XR Link, or build APK and side-load. On first launch, accept the Face Tracking / Eye Tracking / Microphone permission prompts.

That's the whole flow. Details below.

Eye tracking (DO THIS — face expressions alone don't work)

Don't rely on OVRFaceExpressions.EyesLook* for eye gaze. Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is OVREyeGaze, which taps the eye tracker directly.

For ARKit rigs with eyeLook* blendshapes (no eye bones), this skill ships ARKitEyeGazeBlendshapeDriver:

• Reads gaze rotation from two OVREyeGaze components (one per eye, TrackingMode = HeadSpace).
• Computes rotation relative to a head reference (e.g. CenterEyeAnchor).
• Decomposes pitch → eyeLookUp/Down_{L,R}, yaw → eyeLookIn/Out_{L,R} (with the ARKit "_In = toward nose" convention).
• Writes weights in LateUpdate, so it overrides whatever ARKitOVRCustomFace wrote in Update.

For rigs with eye bones (no eyeLook* blendshapes), skip the blendshape decomposition and just parent OVREyeGaze to each eye bone with ApplyRotation = true — the component will rotate the bone directly.

Eye gaze setup (blendshape rigs)

1. Create two empty GameObjects under CenterEyeAnchor (or your head transform): LeftEyeGaze, RightEyeGaze.
2. Add OVREyeGaze to each. Set Eye = Left / Right, TrackingMode = HeadSpace, ApplyPosition = false, ApplyRotation = true, ConfidenceThreshold = 0.5.
3. Add ARKitEyeGazeBlendshapeDriver to the head's SkinnedMeshRenderer GameObject (alongside ARKitOVRCustomFace). Wire leftEye, rightEye, and referenceFrame (= CenterEyeAnchor).
4. Tweak maxAngleDeg (default 30°) and smoothing (default 0.4) to taste.

Step-by-step

1. Install the script

Copy references/ARKitOVRCustomFace.cs into Assets/Scripts/ARKitOVRCustomFace.cs. It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a MapBlendshapes() method that scans SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i), strips any prefix before the last ., lowercases, and matches against the table. Unmatched mesh blendshapes are set to OVRFaceExpressions.FaceExpression.Max (sentinel — skipped at runtime).

2. Enable Movement SDK + permissions

Project settings:

Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)

Then regenerate the manifest.

Regenerating the manifest

Against a live Editor, call the SDK's generator through run_script — the type resolves directly, so no reflection is needed:

// AgentScripts/UpdateManifest.cs
public static class UpdateManifest
{
    public static string Run()
    {
        OVRManifestPreprocessor.GenerateOrUpdateAndroidManifest(true);   // silentMode: true
        return "ok";
    }
}

unity command run_script --file AgentScripts/UpdateManifest.cs --entry UpdateManifest.Run --format json

silentMode: true is required — without it the generator opens a modal dialog that blocks the request. Then confirm the face-tracking entries from the checklist above actually landed in Assets/Plugins/Android/AndroidManifest.xml. See hz-unity-meta-core-sdk for the full manifest workflow and the rule against hand-editing managed entries.

Alternatives: the Editor menu Meta → Tools → Update AndroidManifest.xml, or the meta_update_android_manifest tool if you are on a Unity MCP server with the Meta Unity extension.

3. Scene setup

Scene Hierarchy
├── OVRCameraRig                         (OVRQuickActionsAPI.AddOVRInteractionRig(), or the
│   │                                     OVRCameraRig prefab; meta_add_camerarig under MCP)
│   └── (add) OVRFaceExpressions         component
└── YourHeadModel
    └── ...SkinnedMeshRenderer GO...
        ├── SkinnedMeshRenderer          (existing)
        └── (add) ARKitOVRCustomFace     component
             └── FaceExpressions = the OVRFaceExpressions ref
             └── retargetingType = Custom (set automatically by base when overriding)
             └── Mappings[] = auto-filled on Reset()
             └── BlendShapeStrengthMultiplier = 100 (default; OVR weights are 0–1, mesh wants 0–100)

If the head has multiple SkinnedMeshRenderers (e.g. separate teeth/tongue meshes), add ARKitOVRCustomFace to each one.

4. OVRManager — enable Audio as a data source (A2E)

On OVRCameraRig's OVRManager component:

• Face Tracking Data Sources → check Visual AND Audio (Audio = A2E; produces mouth shapes from microphone when the visual face cameras can't see something — talking, occlusion, etc.).

5. Trigger the mapping if the component already existed

MapBlendshapes() runs automatically when the component is added (Reset()) and from OnValidate() when Mappings is empty. If you need to remap manually (e.g. after re-importing the FBX), use the component's inspector context menu → Map Blendshapes, or from a script:

go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();

Against a live Editor via run_script, the project script's type is directly referenceable (it lives in Assets/), so just call it:

// AgentScripts/Remap.cs
using UnityEngine;

public static class Remap
{
    public static string Run(string modelName)
    {
        var smr = GameObject.Find(modelName).GetComponentInChildren<SkinnedMeshRenderer>();
        smr.gameObject.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();
        return $"Remapped {smr.sharedMesh.blendShapeCount} blendshapes on {smr.name}";
    }
}

unity command run_script --file AgentScripts/Remap.cs --entry Remap.Run --args '["YourHeadModel"]' --format json
unity command save_scene --format json

Under a Unity MCP server the OVR/MSDK types are not visible to the dynamic assembly, so the reflection-free workaround there is SendMessage:

GameObject.Find("YourHeadModel")
    .GetComponentInChildren<SkinnedMeshRenderer>().gameObject
    .SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);

6. Verify

• In Editor, with Meta XR Link / Quest Link, enter Play mode and make faces. The model should mirror them. Driving that loop from the CLI:

  unity command clear_console --format json
  unity command editor_play --format json          # blocks until play mode is live
  # --- the wearer now makes faces; ask the user to confirm before continuing ---
  unity command get_console_logs --severity error --format json
  unity command editor_stop --format json
  

  Do not chain these four back to back. editor_stop immediately after editor_play leaves no time for anyone to make a face, so the check proves nothing. Enter Play mode, tell the user to make faces and report what they see, and only then read the log and stop.

• On device, build APK, side-load (metavr app install <apk>), grant Face Tracking + Microphone permissions on first launch, and stream metavr adb logcat --follow --tag Unity while making faces to catch mapping errors.

• Check the mapping at edit time: inspect the ARKitOVRCustomFace component. Mappings.Length should equal SkinnedMeshRenderer.sharedMesh.blendShapeCount. The Console log from MapBlendshapes() reports mapped X/N blendshapes — X should be 50 (or 52 if the FBX has all of them). Read the serialized array length without opening the Inspector — --field takes a SerializedProperty path, so it wants the [SerializeField] backing field _mappings on OVRCustomFace, not the public Mappings property, which is not serialized:

  unity command get_serialized_fields --target <head GO> --component ARKitOVRCustomFace --field _mappings --format json
  

Troubleshooting

Symptom	Cause / fix
mapped 0/N in Console	Mesh blendshape names don't follow ARKit convention. Verify with mesh.GetBlendShapeName(i). Names must match e.g. eyeBlink_L, jawOpen — case-insensitive, prefix before last . is stripped.
Face is frozen	OVRFaceExpressions not assigned, or scene has no OVRCameraRig/OVRManager with face tracking enabled. Check OVRFaceExpressions.FaceTrackingEnabled and ValidExpressions at runtime.
Mouth doesn't move when speaking	A2E disabled. Enable Audio under OVRManager → Face Tracking Data Sources, and ensure RECORD_AUDIO permission is granted on device.
Eyes don't blink	Mesh's eyelid shapes aren't named eyeBlink_L/_R. Either rename, or add a custom row to ARKitTable.
Eyes don't move (look around)	You're relying on OVRFaceExpressions.EyesLook* instead of OVREyeGaze — switch to ARKitEyeGazeBlendshapeDriver.
Eye gaze in wrong direction	Note the deliberate ARKit ↔ OVR swap: ARKit's eyeLookIn_L (eye looking nose-ward, i.e. right) maps to OVR EyesLookRightL. The table already does this — don't "fix" it.
Weights look half-strength or clamped	BlendShapeStrengthMultiplier defaults to 100 because Unity blendshapes are 0–100 while OVR is 0–1. Don't lower this unless intentional.
cannot change access modifiers when overriding compile error	Base method is protected internal in another assembly. The override must use protected (not protected internal) — already correct in the supplied script.
_mappings out of sync with shared mesh assertion at Start	Mesh changed since mapping was generated. Re-run MapBlendshapes() via the component context menu.

ARKit ↔ OVR FaceExpression mapping (reference)

The 52 ARKit shapes don't 1:1 a FACS-based OVR enum. Notable choices baked into ARKitTable:

• Centrally-named ARKit shapes that have L+R OVR pairs (browInnerUp, cheekPuff, mouthFunnel, mouthPucker, mouthRollLower, mouthRollUpper) pick only the L-side OVR expression. If your model has obviously asymmetric mouth/brow when the user uses these expressions and you want true symmetric drive, subclass and additively sum L+R in a custom MapBlendshapes (use SlothARKitFaceDriver-style per-blendshape sum). For most heads the L-only choice is fine because the face is roughly symmetric and the asymmetry is below visual threshold.
• eyeLookIn/Out are deliberately swapped per side relative to OVR's left/right semantics (see Troubleshooting).
• mouthClose → LipsToward (closest FACS analogue).
• tongueOut → TongueOut (requires HorizonOS ≥ 65 + tongue tracking; otherwise stays at 0).

Files in this skill

• SKILL.md — this file
• references/ARKitOVRCustomFace.cs — drop-in OVRCustomFace subclass
• references/ARKitEyeGazeBlendshapeDriver.cs — OVREyeGaze → eyeLook* blendshape driver (LateUpdate)

Why this approach (vs. alternatives)

• Custom MonoBehaviour driver that reads OVRFaceExpressions[expr] and writes blendshape weights directly: works, but doesn't integrate with MSDK's correctives, eye constraints, or future retargeting upgrades. Use only if you can't extend OVRCustomFace.
• OVRCustomFace + RetargetingType.ARKitBlendshapes: clean, but guarded behind #if OVR_INTERNAL_CODE in the public Oculus Integration package — not 3P-shippable.
• OVRCustomFace + RetargetingType.Custom + GetCustomBlendShapeNameAndExpressionPairs override (this skill): fully public API, shippable, and gets all base-class behavior (data validity gating, weight scaling, mesh assertion).