Operator-side State Reconstruction

What

Operator-side State Reconstruction runs after a feedback package has crossed the downlink. It receives the released package, queues it until the next module step, and applies its payload to a local representation of the remote world. A pose package updates a local pose representation sampled remotely; a camera or lidar package updates the corresponding delayed sensor view; map or landmark packages can update a local environment model.

The module reconstructs received evidence, not the authoritative robot. It must not read the Vehicle / Robot Model, the live robot transform, or the current remote camera. The local world therefore remains at the latest remote time for which the required packages have arrived. Interpolation, prediction, filtering, and visual overlays are separate choices made by later operator-side modules.

Why

Communication delay makes the operator's view a different state from the robot's current state. Reconstruction gives downstream modules a coherent local representation of that delayed world: the remote robot can be placed in a local scene, delayed sensor data can be attached to it, and a study can update a map or SLAM model as new observations arrive. The shipped TurtleBot implementation currently stores delayed pose and video; it does not yet build a map or move a separate local robot representation.

How

Operator-side State Reconstruction turns released feedback into the latest local representation of the remote world. It consumes only what has crossed the downlink; it does not read the live vehicle.

Downlink Communication supplies the reconstruction stage. Reconstruction explicitly publishes usable operator-side results and may expose owned local buffers; it never automatically forwards received payloads. Presentation reads the reconstructed video directly, while trajectory assistance consumes copied motion values. A study can decode binary observations or reconstruct a map here; already usable observations may require only validation and adoption into a local representation.

• Receive. ReceivePackage only enqueues; it does not update operator state immediately.
• Process. On the next module step, every pending package is passed to ReconstructPackage in queue order.
• Apply. The implementation updates the remote-world model for the payload type. Each accepted arrival updates its representation and preserves its sample time, and older camera frames are released when replaced.

Boundary

Contract

The base class implements IPackageConsumer and IPackageSource. Your implementation provides ReconstructPackage and explicitly calls PublishReconstructedFeedback with a ReconstructedFeedbackContract payload when usable output is available. The base class owns the queue:

using System;
using System.Collections.Generic;
using Unite.Core;
using UnityEngine;

namespace Unite.Kernel
{
    /// <summary>
    /// Reconstructs operator-side information from packages released by the
    /// downlink and exposes reconstructed information to presentation and assistance.
    /// Implementations update local data representations only; they do not render
    /// Unity objects or compute assistance.
    /// </summary>
    public abstract class OperatorSideStateReconstructionModule
        : TeleroboticsModule, IPackageSource, IPackageConsumer
    {
        private readonly Queue<Package> pendingPackages = new Queue<Package>();

        public event Action<Package> PackageProduced;

        internal sealed override int ExecutionOrder => 850;

        public void ReceivePackage(Package package)
        {
            if (package == null)
            {
                Debug.LogError($"{GetType().Name} received a null package.", this);
                return;
            }

            pendingPackages.Enqueue(package);
        }

        internal sealed override void Step(double timestampSeconds)
        {
            while (pendingPackages.Count > 0)
            {
                Package package = pendingPackages.Dequeue();
                try
                {
                    ReconstructPackage(package, timestampSeconds);
                }
                finally
                {
                    package.Dispose();
                }
            }
        }

        /// <summary>
        /// Publishes an explicitly reconstructed result. The kernel never
        /// forwards received transport packages automatically. Implementations
        /// may publish zero or more results and expose owned local buffers.
        /// </summary>
        protected void PublishReconstructedFeedback(
            ReconstructedFeedbackContract information,
            double sourceTimestampSeconds,
            string streamId)
        {
            if (information == null || string.IsNullOrWhiteSpace(streamId))
            {
                Debug.LogError(
                    $"{GetType().Name} requires reconstructed information and a stream id.",
                    this);
                return;
            }

            PackageProduced?.Invoke(new Package(
                information, sourceTimestampSeconds, streamId));
        }

        /// <summary>
        /// Updates a local representation from one released feedback package.
        /// Publish only usable reconstructed results; unknown or incomplete
        /// inputs need not produce output. The input is disposed after this hook;
        /// retain a payload lease if a local buffer needs its resources longer.
        /// </summary>
        protected abstract void ReconstructPackage(
            Package package,
            double timestampSeconds);

        protected virtual void OnDestroy()
        {
            while (pendingPackages.Count > 0)
                pendingPackages.Dequeue().Dispose();
        }
    }
}

• ReceivePackage is called by Downlink Communication when a package is released.
• timestampSeconds is the current kernel time when reconstruction runs; it is not the package's sample time.
• The payload carries its own capture metadata, such as SampledAt. Preserve that metadata when creating a representation.
• The base module never forwards input automatically. Publish zero or more explicitly reconstructed results; unknown or incomplete inputs may produce none. Consumers can also read declared local buffers.
• The base module disposes each received package after ReconstructPackage, including ignored inputs. If a buffer keeps its resources, call package.RetainPayload() and dispose the returned lease when replacing or destroying that buffer. Override OnDestroy only with a call to base.OnDestroy(), which cleans unprocessed inputs.

Examples

The TurtleBot implementation is in Assets/UNITE/Demo/every-move-you-make/Runtime/EveryMoveStateReconstruction.cs. The second tab shows an EnvironmentUpdatePackage published by Remote Observation & State Capture and consumed by the reconstruction component.

using System;
using Unite.Core;
using Unite.Kernel;
using UnityEngine;

namespace Unite.Demo.EveryMoveYouMake
{
    /// <summary>
    /// Module 5.9 implementation for the study. It owns the latest released local
    /// pose and video-frame representations and never accesses authoritative state.
    /// </summary>
    public sealed class EveryMoveStateReconstruction
        : OperatorSideStateReconstructionModule
    {
        private IDisposable ownedViewFrame;

        public ReconstructedPose LatestPose { get; private set; }
        public ReconstructedVideoFrame LatestViewFrame { get; private set; }
        public ReconstructedMotionState LatestViewState { get; private set; }
        public Texture CurrentViewFrame { get; private set; }
        public double CurrentViewSampledAt { get; private set; }
        public long CurrentViewSequence { get; private set; } = -1;

        protected override void ReconstructPackage(Package package, double timestampSeconds)
        {
            if (package.Payload is PosePackage pose)
            {
                LatestPose = new ReconstructedPose(pose.Position, pose.Heading, pose.SampledAt);
                PublishReconstructedFeedback(
                    LatestPose, package.SourceTimestampSeconds, EveryMoveStreams.Pose);
                return;
            }

            if (!(package.Payload is ViewFramePackage view))
                return;

            if (!view.Frame)
            {
                return;
            }

            IDisposable previous = ownedViewFrame;
            ownedViewFrame = package.RetainPayload();
            LatestViewFrame = new ReconstructedVideoFrame(
                view.Frame, view.SampledAt, view.Sequence,
                view.WorldToCameraMatrix, view.ProjectionMatrix);
            CurrentViewFrame = view.Frame;
            CurrentViewSampledAt = view.SampledAt;
            CurrentViewSequence = view.Sequence;

            // Copy just the received motion values needed by prediction. Do not
            // expose the transport package or its nested remote sensor handles.
            TurtleBotState captured = view.StateAtCapture;
            LatestViewState = captured == null ? null : new ReconstructedMotionState(
                captured.Position, captured.Heading,
                captured.LeftVelocity, captured.RightVelocity,
                view.SampledAt, view.Sequence);
            if (LatestViewState != null)
                PublishReconstructedFeedback(
                    LatestViewState, package.SourceTimestampSeconds,
                    EveryMoveStreams.ReconstructedViewState);

            previous?.Dispose();
        }

        protected override void OnDestroy()
        {
            base.OnDestroy();
            ownedViewFrame?.Dispose();
            ownedViewFrame = null;
            CurrentViewFrame = null;
            LatestViewFrame = null;
            LatestViewState = null;
        }
    }
}

using System.Collections.Generic;
using Unite.Core;
using Unite.Demo.EveryMoveYouMake;
using Unite.Kernel;
using UnityEngine;

public sealed class EnvironmentUpdatePackage
{
    public double SampledAt { get; }
    public IReadOnlyList<Vector3> OccupiedPoints { get; }

    public EnvironmentUpdatePackage(
        double sampledAt,
        IReadOnlyList<Vector3> occupiedPoints)
    {
        SampledAt = sampledAt;
        OccupiedPoints = occupiedPoints;
    }
}

public sealed class RemoteWorldModel
{
    private readonly List<Vector3> occupiedPoints = new();

    public Vector3 RobotPosition { get; private set; }
    public float RobotHeading { get; private set; }
    public double LastSampledAt { get; private set; }
    public IReadOnlyList<Vector3> OccupiedPoints => occupiedPoints;

    public void ApplyPose(PosePackage pose)
    {
        RobotPosition = pose.Position;
        RobotHeading = pose.Heading;
        LastSampledAt = pose.SampledAt;
    }

    public void ApplyEnvironment(EnvironmentUpdatePackage update)
    {
        occupiedPoints.Clear();
        occupiedPoints.AddRange(update.OccupiedPoints);
        LastSampledAt = update.SampledAt;
    }
}

public sealed class StudyStateReconstruction
    : OperatorSideStateReconstructionModule
{
    public RemoteWorldModel RemoteWorld { get; } = new();

    protected override void ReconstructPackage(
        Package package,
        double timestampSeconds)
    {
        if (package.Payload is PosePackage pose)
        {
            RemoteWorld.ApplyPose(pose);
            return;
        }

        if (package.Payload is EnvironmentUpdatePackage update)
            RemoteWorld.ApplyEnvironment(update);
    }
}

Config

1. Implement OperatorSideStateReconstructionModule and handle the pose, sensor, and environment payloads the study sends through Downlink Communication.
2. Assign one implementation to the agent's Operator-side State Reconstruction field. The Kernel connects Downlink Communication to it.
3. Keep the reconstructed world separate from the Vehicle / Robot Model. Do not store a live Transform, camera, or authoritative state reference in the reconstruction module.
4. Reconstruction retains a payload lease for its video buffer and releases that lease when replaced or destroyed. The texture is recycled only after all owners, including any local observation, have released it. Presentation reads LatestViewFrame each step; it must not retain or dispose replaced frame references. Queued motion results contain only copied values, so replacing a texture cannot invalidate prediction input.