# Recording & playback

<Badge tone="type">namespace · AR51.Unity.SDK.Recorder / .Playback</Badge> <Badge>Unity / C#</Badge>

The recording & playback surface lets you persist incoming mocap frames to a file and replay them back through the live consumers. A recording is a sequence of timed **keyframes**; each parsed line becomes a [`KeyframeCache`](#keyframecache) holding one or more [`KeyframeTask`](#keyframetask)s, and executing a task feeds its payload into the matching consumer ([`SkeletonConsumer`](/docs/sdk-api/unity-api/classes/skeletonconsumer), [`ObjectTransformConsumer`](/docs/sdk-api/unity-api/classes/objecttransformconsumer), [`ObjectDetectionConsumer`](/docs/sdk-api/unity-api/classes/objectdetectionconsumer)) exactly as if it had arrived live. [`SkeletonWebStreamer`](#skeletonwebstreamer) is the related component that pushes visible bone transforms out to a browser viewer.

**Units:** Unity world-space positions are `Vector3` in **meters**, rotations are `Quaternion`, and all keyframe/cache times are **seconds** (capture times are seconds since the Unix epoch, UTC).

:::caution[Meters, not centimeters]
Every position/length on this page is in **meters** (Unity convention). AR 51 mocap data is already meters — no conversion is applied on the Unity side.
:::

## Vector3Keyframe / QuaternionKeyframe

<Badge tone="type">struct</Badge> Namespace: `AR51.Unity.SDK.Recorder`.

Simple timed sample structs for recorded animation curves — one timestamped value each.

```csharp
public struct Vector3Keyframe    { public float Time; public Vector3 Value;    public Vector3Keyframe(float time, Vector3 value); }
public struct QuaternionKeyframe { public float Time; public Quaternion Value; public QuaternionKeyframe(float time, Quaternion value); }
```

<div className="apiPropTable">

| Field | Type | Description |
|---|---|---|
| `Time` | `float` | sample time, in **seconds** |
| `Value` | `Vector3` / `Quaternion` | the sampled value — positions in **meters**, rotations as a Unity quaternion |

</div>

## IKeyframeSource

<Badge tone="type">interface</Badge> Namespace: `AR51.Unity.SDK.Playback`.

Implement this to **remap recorded keyframes during playback** — prepend a skeleton-id prefix, apply a spatial transform to joint positions, and offset capture timestamps. Use it to relocate or retime a recorded performance (e.g. replay a recording into a different anchor space, or merge several recordings under distinct id prefixes). A [`KeyframeTask`](#keyframetask) applies its assigned `Source` (when set) before executing.

| Member | Returns | Description |
|---|---|---|
| `Prefix` | `string` | prepended to skeleton ids coming from this source |
| `Transform(Vector3[] positions)` | `Vector3[]` | remap an array of joint positions (meters) |
| `Transform(Vector3 position)` | `Vector3` | remap a single position (meters) |
| `Offset(double time)` | `double` | remap a capture timestamp (seconds) |

```csharp
string Prefix { get; }
Vector3[] Transform(Vector3[] positions);
Vector3 Transform(Vector3 position);
double Offset(double time);
```

## KeyframeTask

<Badge tone="type">abstract class</Badge> Namespace: `AR51.Unity.SDK.Playback`.

A single unit of playback work. `Execute()` replays the task into the live consumers — applying its optional [`Source`](#ikeyframesource) remap first. The concrete subclasses each wrap one payload type.

```csharp
public abstract class KeyframeTask
{
    public IKeyframeSource Source { get; set; }
    public abstract void Execute();
}
```

<div className="apiPropTable">

| Property | Type | Description |
|---|---|---|
| `Source` | [`IKeyframeSource`](#ikeyframesource) | optional remap applied before `Execute()` (null = replay as recorded) |

</div>

### Subclasses

Each subclass holds one payload and routes it to the matching consumer on `Execute()`.

| Task | Payload | `Execute()` replays into |
|---|---|---|
| [`ActionKeyframeTask`](#actionkeyframetask) | `Action` | runs an arbitrary delegate (markers, side-effects) |
| [`SkeletonTask`](#skeletontask) | [`Skeleton`](/docs/sdk-api/unity-api/classes/skeleton) | [`SkeletonConsumer.Instance.OnSkeleton`](/docs/sdk-api/unity-api/classes/skeletonconsumer) |
| [`ObjectTransformTask`](#objecttransformtask) | [`ObjectTransform`](/docs/sdk-api/unity-api/classes/objecttransform) | [`ObjectTransformConsumer.Instance.OnObject`](/docs/sdk-api/unity-api/classes/objecttransformconsumer) |
| [`ObjectDetectionTask`](#objectdetectiontask) | `ObjectDetectionReply` *(transport)* | [`ObjectDetectionConsumer.Instance.OnObjectDetected`](/docs/sdk-api/unity-api/classes/objectdetectionconsumer) |

```csharp
public class ActionKeyframeTask : KeyframeTask         { public ActionKeyframeTask(Action action); public override void Execute(); }
public sealed class SkeletonTask : KeyframeTask        { public Skeleton Skeleton { get; }        public SkeletonTask(Skeleton s); public override void Execute(); }
public sealed class ObjectTransformTask : KeyframeTask { public ObjectTransform ObjectTransform { get; } public ObjectTransformTask(ObjectTransform t); public override void Execute(); }
public sealed class ObjectDetectionTask : KeyframeTask { public ObjectDetectionReply ObjectDetectionReply { get; } public ObjectDetectionTask(ObjectDetectionReply t); public override void Execute(); }
```

#### ActionKeyframeTask

Wraps an arbitrary `Action`; `Execute()` invokes it. Use to schedule custom side-effects (logging, marker callbacks) on the playback timeline.

#### SkeletonTask

Holds a [`Skeleton`](/docs/sdk-api/unity-api/classes/skeleton). `Execute()` applies the (optionally `Source`-transformed) skeleton to `SkeletonConsumer.Instance.OnSkeleton`, driving characters exactly as a live frame would.

#### ObjectTransformTask

Holds an [`ObjectTransform`](/docs/sdk-api/unity-api/classes/objecttransform). `Execute()` feeds it to `ObjectTransformConsumer.Instance.OnObject`.

#### ObjectDetectionTask

Holds an `ObjectDetectionReply` (transport/proto type). `Execute()` feeds it to `ObjectDetectionConsumer.Instance.OnObjectDetected`.

## KeyframeCache

<Badge tone="type">sealed class</Badge> Namespace: `AR51.Unity.SDK.Playback`.

One parsed line of a recording: timing, source device, the raw JSON, and the decoded [`KeyframeTask`](#keyframetask)s. [`KeyframeParser`](#keyframeparser) produces these when loading a recording; you can also build one directly from a [`Skeleton`](/docs/sdk-api/unity-api/classes/skeleton), an `ObjectDetectionReply`, or a timestamped `Action`.

<div className="apiPropTable">

| Member | Type | Description |
|---|---|---|
| **Timing** | | |
| `time` | `double` | absolute capture time, **seconds since the Unix epoch** |
| `timeSinceStart` | `double` | **seconds** since the recording started |
| **Source** | | |
| `Source` | [`IKeyframeSource`](#ikeyframesource) | optional remap applied to this line's tasks |
| `device_name` | `string` | originating device name |
| `device_ip` | `string` | originating device IP |
| `json` | `string` | the raw recorded JSON for this line |
| **Decoded payload** | | |
| `Tasks` | [`KeyframeTask[]`](#keyframetask) | the decoded tasks (populated by `ParseTasks`) |
| `Skeletons` | [`Skeleton[]`](/docs/sdk-api/unity-api/classes/skeleton) | skeletons decoded from this line |
| `ObjectDetectionReplies` | `ObjectDetectionReply[]` | detection replies decoded from this line *(transport, read-only)* |

</div>

```csharp
public KeyframeCache(ReadOnlySpan<char> line);
public KeyframeCache(string line);
public KeyframeCache(Skeleton s, double? overrideTimestamp = null);
public KeyframeCache(ObjectDetectionReply objectDetection, double? overrideTimestamp = null);
public KeyframeCache(double timestamp, Action action);

public void ParseTasks(bool parseBinary = false);   // decode json → Tasks
public override string ToString();
```

`ParseTasks(parseBinary)` decodes the line's `json` into the `Tasks` array; pass `parseBinary: true` to also decode packed binary payloads.

## KeyframeParser

<Badge tone="type">static class</Badge> Namespace: `AR51.Unity.SDK.Playback`.

Loads and parses recording files into [`KeyframeCache`](#keyframecache) lists — multi-threaded, sorting the result by time.

| Method | Returns | Description |
|---|---|---|
| [`EstimateKeyframeCount`](#estimatekeyframecount) | `long` | rough keyframe count from a byte length (pre-size buffers) |
| [`Parse`](#parse) | `List<KeyframeCache>` | parse streams synchronously |
| [`ParseAsync`](#parseasync) | `IEnumerator` | coroutine wrapper around `Parse` with progress reporting |

### EstimateKeyframeCount

```csharp
public static long EstimateKeyframeCount(long length);
```

Estimates how many keyframes a recording of `length` bytes contains — use it to pre-size lists/progress bars before parsing.

### Parse

```csharp
public static List<KeyframeCache> Parse(Stream[] fileStreams, int length,
    CancellationToken? token = null, Action<string, float> onProgress = null, IKeyframeSource[] sources = null);
```

Parses the given file streams into a time-sorted list of [`KeyframeCache`](#keyframecache). Runs multi-threaded.

<Params>
<Param name="fileStreams" type="Stream[]">the recording file stream(s) to read</Param>
<Param name="length" type="int">expected keyframe count (e.g. from `EstimateKeyframeCount`)</Param>
<Param name="token" type="CancellationToken?">optional cancellation</Param>
<Param name="onProgress" type="Action<string, float>">progress callback — message + a **0–1** fraction</Param>
<Param name="sources" type="IKeyframeSource[]">optional per-stream [`IKeyframeSource`](#ikeyframesource) remaps</Param>
</Params>

### ParseAsync

```csharp
public static IEnumerator ParseAsync(Stream[] fileStreams, int length, List<KeyframeCache> keyframes,
    CancellationToken? token = null, Action<string, float> onProgress = null, IKeyframeSource[] sources = null);
```

Coroutine wrapper around [`Parse`](#parse): `yield`-return it from a Unity coroutine and read the results from the `keyframes` list you pass in. `onProgress` reports a **0–1** fraction so you can drive a loading bar.

<Example inferred>

```csharp
var keyframes = new List<KeyframeCache>();
using var fs = File.OpenRead(recordingPath);
int count = (int)KeyframeParser.EstimateKeyframeCount(fs.Length);
StartCoroutine(KeyframeParser.ParseAsync(
    new Stream[] { fs }, count, keyframes,
    onProgress: (msg, f) => Debug.Log($"{msg} {f:P0}")));   // f is 0..1
```

</Example>

:::note
On **WebGL** the async signature differs: `ParseAsync(StringReader reader, int length, List<KeyframeCache> keyframes, CancellationToken? token, Action<float> onProgress)` — a `StringReader` source and a float-only progress callback.
:::

## SkeletonWebStreamer

<Badge tone="type">class · Singleton</Badge> <Badge>Unity component</Badge> <Badge>singleton</Badge>

```csharp
public class SkeletonWebStreamer : Singleton<SkeletonWebStreamer>
```

Namespace: *global* (no namespace). A **Unity component** (singleton) that streams the currently-visible characters' bone transforms over **TCP** to a web skeleton server, for browser-based viewers. Enable it — and configure the target — via [`SkeletonConsumer.StreamToWeb`](/docs/sdk-api/unity-api/classes/skeletonconsumer) / `StreamToWebUri`, or set the fields directly and enable the component.

<div className="apiPropTable">

| Member | Type | Default | Description |
|---|---|---|---|
| `HostUri` | `string` | `"skeleton-web-server-aci.westeurope.azurecontainer.io"` | target web skeleton server host |
| `Port` | `int` | `13000` | target TCP port |

</div>

```csharp
public string HostUri = "skeleton-web-server-aci.westeurope.azurecontainer.io";
public int Port = 13000;
public Task<bool> TryConnectAsync(string hostname = "127.0.0.1", int port = 13000);
```

Enable the component to start streaming; set `HostUri` / `Port` first. The usual path is to flip [`SkeletonConsumer.StreamToWeb`](/docs/sdk-api/unity-api/classes/skeletonconsumer) on, which configures and enables this streamer for you.

:::caution
⚠️ needs confirmation: serialization runs automatically while the component is enabled — `TryConnectAsync` is normally driven **internally**, so app code rarely calls it directly.
:::

## See also

- [`SkeletonConsumer`](/docs/sdk-api/unity-api/classes/skeletonconsumer) — the consumer that [`SkeletonTask`](#skeletontask) replays into; owns `StreamToWeb` / `StreamToWebUri`
- [`Skeleton`](/docs/sdk-api/unity-api/classes/skeleton) — the per-frame payload carried by [`SkeletonTask`](#skeletontask) and decoded into [`KeyframeCache`](#keyframecache)
- [`ObjectTransform`](/docs/sdk-api/unity-api/classes/objecttransform) — payload for [`ObjectTransformTask`](#objecttransformtask)
- [`ObjectTransformConsumer`](/docs/sdk-api/unity-api/classes/objecttransformconsumer) · [`ObjectDetectionConsumer`](/docs/sdk-api/unity-api/classes/objectdetectionconsumer) — replay targets for the object tasks
- [`extensions`](/docs/sdk-api/unity-api/classes/extensions) — `IsIdentified` and the `Core`↔Unity conversion helpers used around recordings
- [Class index](/docs/sdk-api/unity-api/classes) — all Unity SDK types