DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Humanoid Robot Software — Ch.09: Human Robot Interaction [EN]

STATERA Library · Series 08 · Volume 22 · Chapter 09 · EN
Source: https://github.com/Statera-Library/08_Vol_22_Humanoid_Robot_Software/blob/main/Chapter_09_Human_Robot_Interaction/Chapter_09_Human_Robot_Interaction_EN.md

Volume 22. Humanoid Robot Software

Chapter 09. Human Robot Interaction

09.01. HRI Principles Proximity Safety Social Norms

Human--Robot Interaction (HRI) for humanoid robots concerns the principles that allow humans and physically capable autonomous machines to share spaces, exchange intentions, and cooperate without creating unacceptable risk or social discomfort. Unlike conventional industrial robots isolated behind barriers, humanoids may walk beside people, manipulate common objects, communicate through speech and gesture, and enter personal spaces. HRI therefore combines physical safety, perception, behavioral predictability, communication, and social norms into a unified interaction framework. Volume_22_Humanoid_Robot_Softwa...

Proximity is one of the fundamental variables governing human--humanoid interaction. A robot must continuously estimate not only the geometric distance to nearby people but also how that distance relates to motion, orientation, activity, and environmental context. The same separation that is acceptable while two agents stand still may become dangerous when a humanoid approaches rapidly, carries a large object, or extends an arm toward a person. Proximity must therefore be interpreted dynamically rather than as a fixed distance threshold.

A practical HRI system can represent the space surrounding a person as multiple interaction zones whose boundaries vary with context. Distant regions generally permit unrestricted navigation, while closer regions require progressively greater awareness, lower velocity, and more conservative motion. When the robot enters personal or contact-level space, its behavior should become explicitly interaction-aware. Such zones should not be treated as rigid circles because human orientation, gaze, motion direction, obstacles, and the purpose of the interaction significantly influence acceptable separation.

Safety begins with the principle that uncertainty must reduce robot aggressiveness rather than increase it. Human detection, pose estimation, tracking, depth perception, tactile sensing, and force sensing provide complementary evidence about the interaction state. When perception confidence decreases, the humanoid should enlarge safety margins, reduce velocity or force, avoid unnecessary approach motions, and transition toward a safe state when required. This principle connects HRI directly with the humanoid perception architecture, which includes human detection, pose estimation, tracking, audio processing, and whole-body contact perception. Volume_22_Humanoid_Robot_Softwa...

Collision avoidance alone is insufficient for safe interaction because a mathematically collision-free trajectory can still appear threatening to a human. Rapid arm movements near the face, sudden changes of walking direction, approaching from behind, or passing through a narrow gap beside a person may satisfy geometric constraints while producing anxiety or surprise. Human-aware motion planning therefore considers visibility, predictability, velocity, acceleration, body orientation, and interpersonal distance in addition to conventional obstacle clearance.

Predictability is particularly important for humanoids because their human-like morphology naturally encourages people to interpret their movements as intentional behavior. Motion should communicate what the robot is likely to do next. Before walking across a person\'s path, reaching toward an object, or initiating physical collaboration, the humanoid can provide anticipatory cues through gaze direction, head orientation, body posture, hand motion, speech, or deliberate trajectory shaping. These cues reduce ambiguity and give people time to understand and respond to the robot\'s intended action.

Social norms introduce another layer beyond physical safety. Humans follow implicit conventions concerning personal space, turn taking, yielding, gaze, approach direction, conversational distance, and acceptable physical contact. A socially competent humanoid should model these conventions sufficiently to avoid behavior that is technically safe but socially inappropriate. For example, it should generally avoid interrupting a conversation by physically entering the space between participants, standing unnecessarily close to a stranger, blocking a doorway, or continuously staring at a person.

Context determines how these norms should be interpreted. Appropriate interaction behavior in a factory may differ significantly from behavior in a hospital, retail store, office, warehouse, or home. Industrial workers may accept close approaches during coordinated material handling, whereas patients or customers may expect greater personal distance and clearer communication. HRI policies therefore require contextual parameters rather than one universal social-distance model, allowing the same humanoid platform to adapt its interaction strategy to different environments and user populations.

Human motion must also be treated as uncertain and potentially unpredictable. People can stop suddenly, reverse direction, extend an arm, drop an object, or unintentionally enter the robot\'s planned trajectory. HRI planning should therefore estimate possible future human states rather than relying exclusively on current positions. Short-horizon prediction can produce probabilistic occupancy regions around humans, enabling the humanoid to select trajectories that remain robust across several plausible movements instead of narrowly avoiding the most likely prediction.

Interaction safety becomes more demanding during physical collaboration. Object handover, co-carrying, guided assistance, and shared manipulation deliberately reduce separation between human and robot. In these situations, safe behavior depends on compliant control, force limitation, contact detection, stable posture, and rapid reaction to unexpected resistance. The humanoid should distinguish intended collaborative contact from accidental collision while ensuring that uncertainty in this classification does not permit hazardous force. The volume consequently treats handover and co-carry as dedicated HRI functions. Volume_22_Humanoid_Robot_Softwa...

Whole-body behavior must be considered because a humanoid is not merely an arm mounted on a fixed base. A reaching movement may shift the torso, alter balance, move an elbow toward another person, or require a compensating step. Similarly, walking while carrying an object changes both the robot\'s physical footprint and the consequences of collision. HRI safety should therefore evaluate the predicted swept volume, contact energy, balance state, manipulated object geometry, and surrounding people across the complete body rather than monitoring only the end effector.

Communication provides another safety channel. Voice, gaze, gestures, displays, and body motion can expose robot state and intention before physical action occurs. A humanoid may indicate that it has recognized a command, request confirmation when intent is ambiguous, signal before approaching, or explain why it cannot safely execute an instruction. Multimodal communication is especially valuable because no single channel is universally reliable: speech can fail in noise, gaze may not be observed, and gestures may be ambiguous or culturally dependent.

The robot must also recognize that human attention cannot be assumed. A person looking away, wearing headphones, concentrating on another task, or standing behind an obstruction may not perceive the robot\'s signals. Interaction logic should therefore estimate whether communication has likely been received before executing actions that depend on human awareness. When acknowledgment is uncertain, the robot can slow down, increase separation, repeat the cue through another modality, or wait rather than assuming successful coordination.

Socially appropriate behavior should never override fundamental safety constraints. A robot might normally yield politely, maintain conversational distance, or continue assisting a user, but these preferences remain subordinate to collision prevention, actuator limits, balance stability, emergency stopping, and other protective functions. This creates a hierarchy in which hard safety constraints define permissible behavior while social norms optimize behavior within the safe region. Such separation prevents learned or generative interaction policies from compromising fundamental protective mechanisms.

HRI design should additionally account for vulnerable users and individual differences. Children, older adults, people with reduced mobility, trained operators, and first-time users can respond differently to identical robot behavior. Walking speed, communication timing, personal-space preference, reaction time, and tolerance for physical proximity vary substantially. A robust humanoid should therefore avoid assuming a single idealized human model and should adapt cautiously when reliable evidence about an individual user\'s preferences becomes available.

Privacy and dignity are also relevant interaction principles because humanoids may continuously observe faces, voices, gestures, and activities to understand nearby people. Perception should collect and retain only information required for legitimate robot functions, while interaction behavior should avoid creating the impression of unnecessary surveillance. Visible indicators of sensing or recording states, understandable explanations of robot actions, and appropriate control over stored interaction data can improve trust without requiring users to understand the underlying AI architecture.

Trust should emerge from consistent competence rather than from anthropomorphic appearance alone. A humanoid that behaves reliably, respects boundaries, communicates uncertainty, and stops safely when conditions become ambiguous is more trustworthy than one that attempts to appear excessively human while behaving unpredictably. Designers should therefore avoid interaction mechanisms that imply capabilities the robot does not possess. Appropriate transparency about perception failures, task limitations, and confidence helps users form accurate expectations about the system.

These principles form the foundation for the more specialized HRI mechanisms developed throughout the chapter. Voice interfaces, gesture and gaze interaction, expression rendering, physical collaboration, proxemics, conversational agents, intent estimation, safety requirements, and user evaluation can be understood as different mechanisms operating over the same core relationship between human state, robot state, environmental context, safety constraints, and socially acceptable behavior. The chapter structure explicitly develops these functions from foundational principles toward evaluation and safety. Volume_22_Humanoid_Robot_Softwa...

Ultimately, effective humanoid HRI can be viewed as a continuous closed-loop process of perceiving people, estimating interaction context, predicting possible human behavior, selecting physically safe and socially appropriate actions, communicating robot intent, observing the human response, and adapting subsequent behavior. Proximity safety defines where and how the robot may move, physical safeguards constrain what it may do, and social norms shape how acceptable actions should be expressed. Their integration allows humanoids to become understandable and cooperative participants in human environments rather than merely autonomous machines operating near people.

09.02. Voice Command Interface ASR NLU TTS [w/Code]

A voice command interface allows a humanoid robot to receive human instructions through natural speech and convert them into actions that can be interpreted by its task-planning and control systems. Within the Human--Robot Interaction architecture, voice complements gesture, gaze, physical interaction, and proactive assistance. A practical pipeline typically combines Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), dialogue or intent management, and Text-to-Speech (TTS) to create a closed communication loop between the user and the humanoid.

The voice interaction pipeline begins with acoustic sensing. Microphones mounted on the head or upper body capture speech together with environmental noise, robot actuator sounds, reverberation, and competing voices. Because humanoids operate in dynamic environments rather than controlled recording rooms, raw audio normally requires preprocessing such as beamforming, echo cancellation, noise suppression, automatic gain control, and voice activity detection. These stages improve the signal presented to ASR and help determine when a person is actually addressing the robot.

Microphone-array processing is particularly useful when several people or noise sources surround the humanoid. Direction-of-arrival estimation can identify the approximate location of an active speaker, while beamforming emphasizes sound arriving from that direction. This acoustic information can also be fused with visual human detection, face orientation, lip motion, or gaze estimation. Multimodal speaker association helps the robot determine not only what was spoken but also who issued the command and whether that person was intentionally addressing the robot.

Automatic Speech Recognition converts the processed audio waveform into text or another linguistic representation. Modern ASR systems typically use neural acoustic and language modeling to recognize continuous speech rather than requiring isolated predefined command words. For humanoid applications, recognition accuracy must be balanced against latency and computational cost because commands often need to be interpreted on an onboard GPU, NPU, or CPU. Streaming ASR is therefore useful because partial hypotheses can be generated while the user is still speaking.

ASR uncertainty should remain visible to downstream components instead of being discarded after selecting the most likely transcript. Recognition confidence, alternative hypotheses, word timing, and detected language can provide valuable information to the interaction manager. If the difference between plausible transcripts could cause significantly different physical actions, the robot should request clarification rather than executing the highest-probability interpretation automatically. This becomes especially important for commands involving locomotion, manipulation, or close human contact.

Natural Language Understanding transforms recognized language into a representation that the robot can reason about. A command such as "bring me the bottle from the table" contains an intended action, an object, a source location, and an implicit recipient. NLU may extract these components as intents and slots, semantic frames, structured task descriptions, or grounded language representations. The objective is not merely to classify sentences but to connect linguistic expressions to entities, actions, constraints, and relationships in the robot\'s perceived environment.

Language grounding connects NLU with humanoid perception. References such as "that box," "the cup beside the laptop," or "the person near the door" cannot be resolved from text alone. The interaction system must associate linguistic descriptions with objects and people detected in the current scene. Object detection, 3D scene understanding, tracking, open-vocabulary perception, and semantic memory can provide candidate entities, while spatial relationships and interaction history help determine which entity the user most likely intended.

Ambiguity management is therefore a central function of voice-based HRI. If two similar bottles are visible, executing "pick up the bottle" immediately may produce an incorrect action even when ASR and NLU are individually accurate. The robot can instead ask, "Do you mean the bottle on the left?" and use the response to refine the task representation. Clarification should be triggered according to action risk and uncertainty, allowing harmless ambiguities to be resolved efficiently while applying stricter confirmation to safety-critical actions.

Dialogue management maintains interaction state across multiple conversational turns. Human instructions frequently depend on previous context, as in "pick up the red box," followed by "put it over there." Understanding the second command requires retaining the referenced object and resolving the new destination. Dialogue state may therefore include active users, referenced objects, current tasks, unresolved ambiguities, confirmations, robot capabilities, execution status, and relevant interaction history.

The resulting semantic command must be translated into an executable robot task rather than directly into joint commands. A task interface can map the interpreted instruction into symbolic actions, behavior-tree nodes, planner goals, manipulation skills, navigation goals, or higher-level agent requests. This architectural separation is important because speech understanding should specify what the user intends while locomotion, manipulation, whole-body control, and safety systems determine how the physical action can safely be performed.

Before execution, the command should pass through capability and safety validation. A linguistically valid instruction is not necessarily physically feasible or safe. The requested object may exceed payload limits, the destination may be unreachable, the path may cross a restricted region, or another person may be too close to the planned motion. The HRI layer should therefore communicate with planning and safety components before acknowledging that a command will be executed, preventing conversational confidence from being mistaken for physical capability.

Text-to-Speech closes the interaction loop by allowing the humanoid to communicate its interpretation, status, questions, warnings, and task results. TTS should provide intelligible speech with sufficiently low latency to maintain natural turn taking. Prosody, speaking rate, volume, and phrasing can be adapted to environmental conditions and interaction context. The objective is not simply to produce human-like speech but to communicate robot state clearly and predictably so that users can understand what the system is doing.

Spoken feedback can be organized around important task transitions. The robot may acknowledge that it heard a request, ask for missing information, confirm a potentially hazardous operation, report that execution has begun, or explain why the task cannot be completed. Excessive verbal feedback, however, can make interaction slow and distracting. Effective systems therefore combine speech with nonverbal signals such as head motion, gaze, lights, displays, or gestures so that routine state changes do not require a full spoken response.

Turn-taking behavior determines when the robot listens and when it speaks. Voice activity detection, endpoint detection, interruption handling, and dialogue state are needed to prevent the robot from talking over users or waiting unnecessarily after a command has ended. The system should also support barge-in, allowing a person to interrupt synthesized speech with a new command or an urgent stop request. Safety-related interruption channels should receive higher priority than normal conversational dialogue.

Robot-generated noise creates a distinctive challenge for humanoid voice interfaces. Cooling fans, joint actuators, footsteps, impacts, and synthesized speech can contaminate microphone signals. Acoustic echo cancellation can subtract the known TTS output from captured audio, while adaptive filtering and noise suppression reduce predictable robot noise. During locomotion or manipulation, the system may dynamically adjust recognition thresholds or combine audio with visual evidence to maintain reliable command detection.

Wake-word detection or explicit interaction-state management can prevent surrounding conversation from being interpreted as robot commands. However, requiring a wake word before every instruction can make prolonged collaboration unnatural. A practical design can use an activation phrase to establish an interaction session and then maintain temporary conversational engagement while the user remains nearby and attentive. The session can expire when attention shifts, the user leaves, or a configurable timeout occurs.

Multilingual operation introduces additional requirements for language identification, ASR models, NLU representations, and TTS voices. A humanoid deployed in public or international environments may encounter users with different accents, speaking rates, vocabulary, and code-switching behavior. The system should treat unfamiliar speech as uncertainty rather than forcing it into a misleading interpretation. Domain-specific terminology, object names, workplace vocabulary, and user-defined expressions can also be incorporated without sacrificing general-language capability.

Latency across the complete ASR--NLU--TTS pipeline strongly affects perceived interaction quality. A highly accurate system that waits several seconds before responding can appear unresponsive and can disrupt natural conversation. Streaming audio processing, incremental ASR, early intent estimation, cached language resources, and onboard inference can reduce response delay. More computationally expensive reasoning may be performed asynchronously when the command requires complex planning, while immediate acknowledgment informs the user that processing is continuing.

Large language models can extend conventional NLU by interpreting flexible instructions, resolving conversational context, and transforming natural language into structured task representations. Nevertheless, an LLM output should not be treated as an unrestricted physical command. Generated plans must be grounded in perceived objects, validated against robot capabilities, checked by deterministic safety constraints, and converted through controlled task interfaces. This separation is essential when language models participate in embodied systems capable of exerting physical force.

Failure handling should be designed as part of normal voice interaction rather than as an exceptional condition. The robot may fail to hear a command, recognize the wrong words, misunderstand an object reference, lose network connectivity, or discover during execution that the requested task is impossible. The interface should identify which stage failed whenever possible and communicate a useful recovery path. Repeating the entire interaction is less effective than asking specifically for the missing or uncertain information.

Evaluation of a humanoid voice interface should therefore extend beyond conventional word error rate. Relevant measures include command recognition accuracy, intent accuracy, grounding success, clarification frequency, end-to-end latency, task completion rate, false activation rate, interruption response, recovery success, and user comprehension of robot feedback. Testing should include quiet and noisy environments, multiple speakers, robot self-noise, varying distances, accents, ambiguous commands, and safety-critical interruption scenarios.

A production-quality voice interface ultimately functions as a bidirectional bridge between human language and embodied robot behavior. ASR transforms speech into linguistic evidence, NLU converts that evidence into grounded intent, dialogue management preserves context and resolves uncertainty, task interfaces connect intent to robot capabilities, and TTS communicates interpretation and execution state back to the human. Integrated with perception, planning, manipulation, locomotion, and safety mechanisms, this pipeline enables natural voice interaction without allowing linguistic flexibility to bypass the constraints required for dependable humanoid operation.

09.03. Gesture and Gaze Based Interaction [w/Code]

Gesture- and gaze-based interaction enables a humanoid robot to communicate with people through nonverbal signals that naturally accompany human behavior. Hand gestures, arm movements, body orientation, head pose, and gaze can indicate commands, attention, references, intentions, or conversational state without requiring continuous speech. In Human--Robot Interaction, these channels complement voice interfaces and become especially valuable in noisy environments, shared workspaces, and situations where spoken communication is inconvenient or ambiguous.

A gesture interaction pipeline begins by detecting and tracking the human body. RGB cameras, depth cameras, or multimodal perception systems can estimate the positions of the head, torso, arms, hands, and major skeletal joints. Temporal tracking is important because a gesture is usually defined by motion over time rather than by a single body pose. The system must therefore distinguish intentional communicative movements from ordinary actions such as walking, reaching, scratching, carrying objects, or adjusting clothing.

Human pose estimation provides a structured representation from which larger body gestures can be recognized. Joint coordinates, relative limb angles, hand trajectories, body orientation, and motion velocity can be transformed into features for gesture classification. A humanoid may recognize commands such as stop, come here, move left, follow me, or wait. However, the same physical movement can have different meanings depending on context, so gesture recognition should be connected to interaction state rather than treated as an isolated visual classification problem.

Hand gestures provide a more detailed interaction channel. Hand detection and keypoint estimation can represent palm orientation, finger configuration, pointing direction, and dynamic finger motion. Simple symbolic gestures can trigger predefined robot functions, while pointing can establish spatial references to objects or locations. Reliable recognition requires sufficient image resolution and visibility, because hands may be small in the camera image, partially occluded by objects, or temporarily hidden behind the user\'s body during natural interaction.

Dynamic gestures require temporal reasoning. A waving hand, beckoning motion, directional sweep, or repeated stop signal cannot be reliably understood from one image. The perception system can maintain a sequence of skeletal or hand states and classify motion patterns over a temporal window. Recurrent models, temporal convolution, transformers, or learned action-recognition policies can perform this mapping, but practical systems must also determine when a gesture begins and ends so that continuous human movement does not produce repeated false commands.

Pointing is particularly important because it connects human communication with spatial grounding. When a user says "pick up that object" while pointing, the robot can estimate a pointing ray from the shoulder, elbow, wrist, hand, or index finger and project it into the perceived 3D environment. Candidate objects near this direction can then be ranked using geometric proximity, semantic information, gaze, and spoken descriptions. This multimodal grounding reduces ambiguity compared with interpreting either language or gesture independently.

Gaze provides information about human attention and can support both reference resolution and conversational coordination. A humanoid can estimate head orientation, facial landmarks, eye direction, or combined head-eye pose to infer where a person is likely looking. Exact eye tracking may be difficult at long distances, so head pose often provides a useful coarse approximation. The resulting gaze direction can be associated with objects, other people, robot body regions, or locations represented in the robot\'s scene model.

Gaze should normally be interpreted probabilistically rather than as an exact geometric ray. Camera resolution, eye visibility, glasses, lighting, head movement, and calibration errors introduce uncertainty. Instead of assuming that a person is looking at one precise point, the system can represent a gaze cone or probability distribution over candidate targets. Combining this distribution with object positions, dialogue context, pointing gestures, and recent interaction history produces more robust estimates of the user\'s intended reference.

Joint attention occurs when the human and robot coordinate attention toward the same entity or event. This capability is fundamental to natural HRI because many instructions rely on shared visual context. The robot may follow a person\'s gaze toward an object, orient its head toward that object, and then return its gaze toward the user to indicate successful reference resolution. Such behavior provides a visible confirmation that the robot has understood what the person is attending to before beginning physical action.

Robot gaze is itself an interaction output. By rotating its head or directing visual sensors toward a person, the humanoid can indicate attention and readiness to interact. Looking toward an intended destination before walking or toward an object before reaching can also communicate upcoming behavior. These anticipatory gaze cues make robot actions easier to predict. However, prolonged or mechanically fixed staring may appear unnatural, so gaze behavior should balance task requirements, sensing needs, and socially appropriate attention.

Gesture and gaze become particularly effective when integrated with speech. A user may say "put this there" while holding one object and pointing toward a destination, or say "follow him" while looking at another person. Language supplies semantic relationships while gesture and gaze provide spatial grounding. A multimodal fusion layer can combine ASR confidence, NLU intent, skeletal observations, pointing geometry, gaze likelihood, object detections, and dialogue context to estimate the most plausible user intention.

Fusion should account for timing because multimodal signals are rarely perfectly synchronized. A person may point before speaking, maintain a gesture while describing an object, or shift gaze after completing a sentence. The interaction system therefore needs a temporal association window that relates observations occurring within a meaningful interval. Maintaining short-term multimodal memory allows the robot to connect words such as "there," "that," or "him" with gestures and gaze signals that occurred slightly earlier or later.

Intentionality is another major challenge. Humans constantly move their hands, heads, and eyes without issuing robot commands. Treating every detected gesture as intentional interaction would produce frequent false activations. The robot should therefore estimate whether the user is engaged with it by considering body orientation, distance, gaze toward the robot, dialogue state, previous activation, gesture persistence, and environmental context. Explicit engagement states can greatly reduce accidental commands in crowded or collaborative environments.

When confidence is insufficient, the humanoid should request confirmation rather than silently selecting an uncertain interpretation. If pointing intersects several nearby objects, the robot can orient toward the most likely candidate and ask whether it is correct. If a hand signal is partially occluded, it can request repetition. Confirmation can itself be multimodal, allowing the user to answer verbally, nod, point again, or use a simple approval gesture. This creates a closed-loop interaction rather than a one-way recognition pipeline.

Safety-related gestures require special treatment. A stop gesture, retreat signal, or human indication of discomfort should be recognized with low latency and routed through a high-priority interaction path. Such commands should not depend on lengthy conversational reasoning before affecting robot behavior. Where appropriate, gesture-based stop requests can complement physical emergency-stop mechanisms, voice interruption, proximity monitoring, and collision avoidance, while dedicated safety functions remain responsible for enforcing the final protective response.

Gesture recognition must remain robust across different users. Body dimensions, dominant hand, mobility, clothing, cultural conventions, gesture amplitude, and individual communication styles can vary considerably. A system trained only on exaggerated laboratory gestures may perform poorly during natural interaction. Training and validation data should therefore contain diverse users, viewing angles, distances, partial occlusions, lighting conditions, backgrounds, and realistic transitions between communicative and noncommunicative human activity.

Social conventions also influence gaze interpretation. Direct eye contact can indicate engagement, but the acceptable duration and meaning of gaze vary across individuals and cultural settings. Similarly, pointing with a finger, whole hand, head, or gaze may be preferred in different situations. A humanoid should therefore avoid assuming that one gesture vocabulary represents universal human behavior. Configurable interaction policies and learned adaptation can improve usability while preserving conservative behavior when user preferences remain unknown.

Real-time performance is essential because delayed responses weaken the relationship between a human signal and robot behavior. Human detection, pose estimation, hand tracking, gaze estimation, temporal gesture recognition, and multimodal fusion must operate with sufficiently low end-to-end latency. Computational scheduling may prioritize coarse but fast recognition during motion and activate more detailed hand or eye analysis when an interaction is detected, allowing onboard resources to be allocated according to current HRI requirements.

Occlusion handling is particularly important for humanoids operating close to people. Hands can disappear behind objects, arms may leave the camera field of view, and the robot\'s own manipulators can obstruct visual observations during collaborative tasks. Multiple cameras, depth sensing, temporal tracking, and prediction can preserve interaction state through short observation gaps. The system should nevertheless reduce confidence when evidence disappears rather than continuing to assume that an earlier gesture remains valid indefinitely.

Evaluation should measure more than frame-level gesture classification accuracy. Relevant metrics include gesture detection precision and recall, false activation rate, pointing-target accuracy, gaze-target accuracy, engagement detection, multimodal grounding success, recognition latency, clarification frequency, task completion, and safety-stop response. Testing should include natural background motion, multiple people, simultaneous speech and gesture, partial visibility, ambiguous references, and interactions performed while the humanoid itself is walking or manipulating objects.

A production gesture- and gaze-based HRI system ultimately forms a continuous perception, interpretation, and feedback loop. Human pose, hand motion, pointing, head orientation, and gaze provide nonverbal evidence of attention and intent; multimodal reasoning combines these signals with speech and environmental perception; and the humanoid responds through movement, gaze, gesture, or voice. When uncertainty, timing, safety, and social context are handled explicitly, nonverbal interaction becomes a powerful mechanism for intuitive and dependable cooperation between humans and humanoid robots.

09.04. Facial Expression and Emotion Rendering [w/Code]

Facial expression and emotion rendering provide a humanoid robot with nonverbal mechanisms for communicating internal interaction states to people. A face, animated display, eyes, eyelids, mouth, head motion, or combinations of these elements can indicate attention, acknowledgment, uncertainty, task completion, warning, or conversational engagement. The objective is not necessarily to reproduce human emotion perfectly, but to make the robot\'s state easier to interpret during Human--Robot Interaction.

Humanoid facial interfaces can range from mechanically actuated faces to simplified displays with animated eyes and mouths. Highly articulated faces may provide expressive flexibility but require additional actuators, control complexity, maintenance, and safety consideration. Simplified graphical faces can communicate essential states with fewer mechanical components. The appropriate design depends on robot morphology, application environment, interaction distance, expected users, and the amount of social communication required.

Expression rendering should begin from an explicit interaction state rather than directly from arbitrary actuator commands. The HRI system may maintain states such as neutral, attentive, acknowledging, processing, successful, uncertain, warning, or unavailable. A rendering layer converts these semantic states into coordinated facial, gaze, head, and voice behaviors. This separation allows higher-level interaction logic to specify communicative intent while the rendering subsystem determines how that intent is physically or visually expressed.

Human facial expressions are produced through coordinated changes in regions such as the eyebrows, eyes, cheeks, and mouth. A humanoid with an articulated face can approximate these changes through controllable facial degrees of freedom. Instead of independently commanding every actuator, the system can define reusable expression primitives that specify coordinated target positions, motion amplitudes, velocities, and durations. Blending multiple primitives enables transitions between expressions without abrupt mechanical changes.

For robots using display-based faces, expression parameters can control eye shape, pupil direction, blinking, eyebrow-like graphics, mouth curvature, and animation timing. Display rendering provides considerable flexibility because expressions can be modified through software without changing mechanical hardware. However, excessive visual complexity can make robot behavior distracting or cartoon-like. Consistent and easily interpretable cues are generally more valuable than maximizing the number of available facial animations.

Temporal behavior is essential because an expression is not simply a static facial configuration. Its onset, duration, intensity, and decay affect how people interpret it. An acknowledgment expression may appear rapidly and return smoothly to neutral, while an uncertainty cue may persist until clarification is obtained. Expression trajectories should therefore be generated over time with controlled transitions so that facial behavior remains synchronized with dialogue, gaze, task execution, and physical motion.

Blinking provides a simple example of how small temporal behaviors influence perceived naturalness. A robot that never blinks may appear mechanically fixed, while constant or perfectly periodic blinking can also look unnatural. Blink timing can be generated with moderate variability and coordinated with gaze shifts or conversational transitions. When cameras are located behind or near animated eyes, however, cosmetic eye behavior must not interfere with sensing or create unnecessary mechanical motion.

Gaze and facial expression should be coordinated because people interpret them together. A humanoid that verbally acknowledges a person while looking elsewhere may appear inattentive even when speech recognition is functioning correctly. During conversation, the robot can orient toward the active speaker, provide brief acknowledgment cues, shift gaze toward referenced objects, and then return attention to the user. Facial rendering should therefore operate as part of a broader multimodal behavior system rather than as an isolated animation module.

Speech and expression also require temporal synchronization. A spoken confirmation may be accompanied by a subtle positive expression, while a clarification request can use an attentive or uncertain appearance. Warning speech should be paired with a clearly distinguishable visual state rather than an ambiguous friendly animation. TTS timing information, dialogue state, and utterance boundaries can be used to schedule expression transitions so that verbal and nonverbal channels communicate compatible information.

Emotion rendering should be distinguished from emotion recognition. A robot may estimate whether a person appears confused, distressed, attentive, or satisfied, but such inference is inherently uncertain and context dependent. Rendering concerns what the robot communicates outwardly, whereas recognition concerns interpretation of the human. A robust HRI architecture should keep these functions conceptually separate so that uncertain estimates of human emotion do not automatically produce inappropriate robot behavior.

The term emotion should also be used carefully when describing robot behavior. A humanoid does not need to claim that it experiences human emotions in order to display socially meaningful cues. Expressions can instead represent functional interaction states such as confidence, uncertainty, availability, warning, acknowledgment, or task outcome. This framing improves transparency because the displayed behavior communicates information relevant to interaction without implying internal subjective experiences that the system cannot establish.

Expression intensity should reflect interaction context. Large exaggerated expressions may be useful at long distances or in demonstration environments, whereas subtle expressions may be more appropriate during close collaboration. Intensity can be adjusted according to user distance, ambient conditions, urgency, and communication channel availability. Safety-critical warnings should remain clearly distinguishable from ordinary social expressions even when the user is not closely observing the robot\'s face.

Facial behavior can also communicate task progress. During lengthy reasoning or planning, a processing state can indicate that the robot has received the command and is still working rather than ignoring the user. Successful completion may produce a brief acknowledgment, while failure can trigger a neutral explanatory state followed by verbal guidance. Such feedback reduces uncertainty about robot status and complements voice, lights, displays, and body gestures without requiring continuous spoken updates.

Expression selection should avoid conflicts with physical behavior. A humanoid approaching a person rapidly while displaying a friendly face may still appear threatening because motion dominates the safety interpretation. Likewise, a positive expression should not override warnings generated by proximity monitoring, collision avoidance, or task safety systems. Facial rendering is therefore subordinate to physical safety and should reinforce, rather than contradict, the robot\'s actual operational state.

Personalization can improve interaction, but adaptation should remain conservative. Different users may prefer different levels of expressiveness, eye contact, animation frequency, or conversational feedback. A robot can gradually adjust these parameters based on explicit preferences or reliable interaction history. It should not infer sensitive personal characteristics from facial appearance merely to select an expression style, and uncertainty about user preferences should result in neutral and broadly understandable behavior.

Cultural variation presents another challenge because facial expressions, gaze patterns, and acceptable levels of expressiveness can be interpreted differently across communities and situations. Some cues may be widely recognizable, while others depend strongly on social context. A production humanoid should therefore emphasize simple functional expressions, provide configurable behavior profiles where necessary, and validate interaction designs with representative users rather than assuming that one expressive vocabulary is universally understood.

Mechanical facial systems introduce additional control constraints. Actuator position, velocity, torque, temperature, backlash, and mechanical limits must be monitored to prevent damage and undesirable motion. Expressions should be generated within calibrated ranges and smoothly interpolated between configurations. If a facial actuator becomes unavailable, the rendering system should degrade gracefully, avoiding asymmetric or misleading expressions and relying more heavily on gaze, voice, displays, or other communication channels.

Rendering latency influences whether an expression appears connected to the event that caused it. If a confirmation expression occurs several seconds after the robot verbally acknowledges a command, users may interpret the two events separately. HRI software should therefore timestamp relevant dialogue, perception, and task events and schedule facial responses with predictable latency. Lightweight local rendering can continue operating even when higher-level reasoning requires remote or computationally expensive processing.

An expression manager can arbitrate between simultaneous requests from dialogue, navigation, manipulation, safety, and task-management modules. For example, a conversational acknowledgment may be active when a safety warning suddenly occurs. Priority rules should allow the warning state to preempt the social animation immediately. After the safety condition disappears, the system can return to an appropriate neutral or task-related state rather than blindly resuming an outdated expression sequence.

Evaluation should measure whether users correctly understand rendered states rather than only whether animations execute as designed. Relevant measures include recognition accuracy for intended expressions, response time, perceived clarity, appropriateness, distraction, trust calibration, and consistency with speech and robot motion. Testing should include different viewing distances, lighting conditions, user populations, simultaneous task activity, and transitions between normal, uncertain, successful, failure, and warning states.

A production facial expression and emotion-rendering system ultimately serves as a communication layer between internal HRI state and observable robot behavior. Semantic interaction states are transformed into coordinated facial motion, graphical animation, gaze, head movement, and speech while safety and hardware constraints remain authoritative. When expressions are timely, restrained, interpretable, and consistent with physical actions, they can make humanoid behavior easier to understand without requiring the robot to imitate every detail of human emotional behavior.

09.05. Physical Collaboration Handover Co Carry [w/Code]

Physical collaboration allows a humanoid robot and a person to perform tasks in which both participants interact with the same object, workspace, or physical process. Unlike interaction based only on speech or gesture, collaborative manipulation introduces direct forces between the human, robot, and environment. Object handover and co-carrying are representative tasks because they require perception, intention estimation, compliant control, whole-body coordination, and safety to operate together in real time.

Object handover can be viewed as a coordinated transfer of possession between two agents. The robot must identify the object, determine whether the human intends to give or receive it, select a suitable grasp or presentation pose, approach the exchange region, and recognize when transfer should occur. A successful handover therefore depends not only on manipulation accuracy but also on timing, accessibility, predictability, and the ability to interpret subtle human behaviors during the exchange.

The handover process begins with estimating the human interaction state. Visual perception can track body pose, arm motion, hand position, gaze, and object location, while speech or gesture may provide explicit intent. A person extending an object toward the robot may indicate a transfer request, but similar motion could also represent ordinary manipulation. The humanoid should combine multiple observations over time before committing to a physical response, especially when the interpretation would cause its arms to move close to the person.

When the robot gives an object to a human, the presentation pose should make the object easy and safe to grasp. The selected orientation should expose an appropriate grasp region while keeping dangerous, sharp, hot, fragile, or functionally important surfaces away from the recipient whenever possible. The robot should also consider human reachability, hand preference, body posture, and nearby obstacles rather than presenting every object at a fixed position relative to its own body.

Trajectory generation for handover should prioritize predictable motion. Rapid or unnecessarily complex arm trajectories can create discomfort even when they remain collision-free. The robot can move the object toward a visible exchange region using moderate velocity and acceleration, then reduce speed as the human hand approaches. Head orientation, gaze, or a short verbal cue can announce the intended transfer, helping the human understand where the object will be presented and when it is safe to take it.

Transfer detection determines when the robot should release an object. Visual evidence may indicate that the human hand has established contact, while tactile sensors, finger force sensors, joint torque measurements, or wrist force-torque sensors can detect grasp forces and external pulling. Relying on a single threshold can be unreliable, so multiple signals can be combined to estimate transfer confidence. The robot should release only when the human grasp appears sufficiently stable and the transfer does not create an unsafe object motion.

The opposite process occurs when the robot receives an object from a person. It must identify an accessible grasp region, approach without colliding with the person\'s fingers, establish contact, increase grasp force to a suitable level, and communicate that the object has been secured. The human should not be expected to release before the robot has achieved a stable grasp. A coordinated transition in which robot grasp confidence rises before human release reduces the probability of dropping the object.

Compliant control is fundamental during close physical collaboration. Purely rigid position control can generate large forces when human motion differs from the robot\'s planned trajectory. Impedance or admittance behavior allows the manipulator to respond to external forces while maintaining the task objective. Appropriate stiffness and damping should depend on the interaction phase: free-space motion can remain relatively precise, while the exchange or shared-contact phase generally requires greater compliance and conservative force limits.

Whole-body control becomes important because handover is not purely an arm task. A humanoid may need to shift its torso, bend its knees, adjust its stance, or move its center of mass to reach a comfortable exchange location while preserving balance. Arm motion can also create momentum that affects standing stability. The interaction controller should therefore coordinate posture, balance, manipulation, and collision constraints rather than solving the hand trajectory independently of the rest of the body.

Co-carrying extends physical collaboration from a short transfer event to sustained shared manipulation. The human and humanoid simultaneously support and move the same object, such as a box, panel, container, or long component. Neither participant has complete independent control because the object\'s motion depends on both. The robot must therefore infer shared motion intent continuously while regulating its own forces, maintaining balance, avoiding collisions, and preserving a stable grasp.

Force and torque measurements provide an important communication channel during co-carrying. When the human pushes, pulls, rotates, raises, or lowers the shared object, these forces can indicate the desired direction of motion. The robot can estimate an intended object velocity or pose change from measured interaction wrench and respond through compliant motion. Filtering is necessary because sensor noise, vibration, footsteps, object dynamics, and transient impacts should not be interpreted as deliberate human commands.

Human motion observation can complement force-based intent estimation. Body orientation, stepping direction, arm posture, gaze, and object motion can indicate where the person intends to move before substantial force develops. Combining visual prediction with force sensing allows the robot to respond earlier and more smoothly. For example, a human beginning to turn while carrying a long object can be detected visually, enabling the humanoid to adjust its footsteps before large torsional forces appear.

Shared object representation is essential for coordinated carrying. The robot should estimate the object\'s pose, dimensions, mass properties, grasp locations, and relationship to both participants. For large objects, collision checking must consider the complete object geometry rather than only the robot body. The planner should predict how object rotation or translation changes the swept volume and ensure that the shared load does not strike walls, furniture, equipment, or nearby people.

Load sharing determines how much of the object\'s weight and motion effort each participant supports. Equal force distribution is not always desirable because human capability, robot payload limits, grasp geometry, and task direction may differ. The humanoid can regulate vertical support force while allowing the human to guide horizontal motion, or assume a larger fraction of the load when appropriate. Load estimates should remain within actuator, joint, grasp, and balance limits throughout the task.

Locomotion and manipulation must be tightly integrated during mobile co-carrying. The robot cannot simply command its hands to follow the object while its feet execute an unrelated walking policy. Object motion influences desired body velocity, torso orientation, step placement, and balance. A coordinated controller can convert shared-object motion into whole-body objectives while maintaining feasible support contacts. This becomes especially important during turns, backward walking, narrow passages, ramps, or uneven terrain.

Communication remains useful even when physical forces provide continuous information. The robot can confirm readiness before lifting, request slower motion when approaching a limit, warn that an obstacle is nearby, or announce when it needs to reposition its feet. Short verbal cues, gaze, head motion, and body orientation can make internal constraints visible to the human. Effective collaboration avoids both extremes of silent unpredictability and excessive communication that interrupts the physical task.

Safety constraints must remain authoritative throughout handover and co-carrying. Joint torque, contact force, grasp force, velocity, acceleration, self-collision, human distance, balance margin, and actuator capability should be monitored continuously. If a limit is approached, the robot can reduce speed, increase compliance, request assistance, place the object safely, or stop. Sudden uncontrolled release should generally be avoided because dropping a shared load may create greater risk than temporarily maintaining contact.

Failure handling should account for changing human behavior and object conditions. The human may unexpectedly release the load, tighten the grasp, move in an unanticipated direction, stumble, or stop responding. The object may slip or collide with the environment. The humanoid should detect these changes quickly and transition to a stable response, such as increasing support, stopping motion, stabilizing posture, lowering the object, or requesting human intervention rather than continuing an obsolete trajectory.

Interaction roles can change during a collaborative task. At one moment the human may lead while the robot follows compliant forces; later the robot may lead because it knows the planned destination or obstacle-free path. A flexible controller should support leader-follower transitions without abrupt changes in impedance or motion. Shared autonomy can combine human physical guidance with robot perception and planning so that each participant contributes information or capability that the other lacks.

Personalization may improve collaboration because people differ in height, strength, reach, walking speed, preferred handover distance, and willingness to physically guide a robot. The humanoid can adapt presentation height, approach velocity, compliance, and load-sharing parameters using explicit settings or repeated interaction evidence. Adaptation should remain bounded by safety constraints, and uncertain estimates of human capability should lead to conservative behavior rather than aggressive assumptions.

Evaluation of handover should include more than whether the object reaches the other participant. Relevant measures include transfer success, handover duration, object drops, peak interaction force, grasp stability, human waiting time, robot waiting time, trajectory predictability, clarification frequency, and perceived comfort. Experiments should include different object sizes and masses, approach directions, user heights, intentional delays, failed grasps, and unexpected release conditions.

Co-carry evaluation should additionally measure shared-object trajectory error, load distribution, interaction wrench, walking stability, synchronization, obstacle clearance, task completion time, and recovery from disturbances. Tests should include straight transport, turns, narrow passages, changes in speed, temporary stops, asymmetric loads, and role transitions. Simulation and controlled laboratory testing can establish baseline behavior before progressively introducing realistic human variability and field conditions.

A production physical-collaboration system ultimately combines human perception, object understanding, intent estimation, force and tactile sensing, compliant manipulation, whole-body control, locomotion, communication, and safety supervision. Handover requires a reliable transition of possession, while co-carrying requires continuous negotiation of shared motion and load. When these mechanisms operate as a coordinated closed loop, a humanoid can collaborate physically with people while remaining predictable, responsive, and appropriately constrained by safety.

09.06. Proxemics and Social Navigation in HRI [w/Code]

Proxemics and social navigation address how a humanoid robot should position and move itself around people while respecting safety, comfort, and social expectations. Conventional navigation primarily seeks collision-free and efficient motion, whereas socially aware navigation must also consider interpersonal distance, body orientation, attention, group structure, and human activity. For humanoids operating in shared environments, navigation therefore becomes both a geometric planning problem and a Human--Robot Interaction problem.

Proxemics describes how people use physical distance during social interaction. The space surrounding a person is not simply an obstacle region with a fixed radius, because acceptable robot distance depends on whether the person is passing by, conversing, working, waiting, carrying an object, or intentionally interacting with the robot. A humanoid navigation system should therefore represent personal space as a dynamic, context-dependent region rather than as a constant collision margin.

Personal-space models can be constructed as cost fields around detected humans. The cost generally increases as the robot approaches a person, encouraging the planner to select trajectories with comfortable separation whenever alternatives exist. These fields do not need to be circular. A person may require more space in front than behind or more lateral clearance while walking. Orientation, velocity, gaze, activity, and environmental constraints can therefore shape anisotropic social cost distributions.

Human motion further changes the structure of personal space. A stationary person can be modeled using a relatively localized interaction region, whereas a walking person requires consideration of predicted future positions and motion direction. The robot should avoid cutting directly across the person\'s path when a small detour can produce more predictable behavior. Motion prediction can extend the social cost field forward, enabling the planner to anticipate rather than merely react to human movement.

Social navigation begins with reliable human detection and tracking. Cameras, depth sensors, LiDAR, or multimodal perception can estimate human position, velocity, body orientation, and pose. Tracking identities over time allows the robot to distinguish persistent people from transient detections and to predict motion more consistently. Uncertainty should remain part of the representation because inaccurate localization of a nearby person can directly affect both safety margins and socially appropriate path selection.

Human trajectory prediction can range from simple constant-velocity models to learned predictors that consider surrounding people and environmental geometry. In HRI, the purpose is not necessarily to forecast exact long-term behavior but to identify plausible short-term occupancy. Multiple possible trajectories can be represented probabilistically so that the humanoid avoids plans that are safe only under one prediction. Prediction uncertainty should generally increase navigation conservatism rather than encourage narrow passage.

Groups require a different representation from isolated individuals. Two or more people standing together may form a conversational group whose internal space should not be crossed by the robot even if a collision-free geometric path exists between them. Relative position, orientation, gaze, and interaction history can be used to infer group structure. The planner can then represent the group as a socially meaningful region and route around it instead of treating every person as an independent obstacle.

Approaching a person is also a social-navigation task. The shortest geometric path may place the robot behind the user or force the person to turn abruptly before interaction begins. A socially appropriate approach selects a visible and accessible interaction position based on the person\'s orientation, surrounding obstacles, and expected task. The robot can reduce speed during the final approach and use gaze, head orientation, gesture, or voice to indicate its intention before entering closer interpersonal space.

Passing behavior should consider relative motion and available space. In a wide corridor, the humanoid can maintain generous clearance and follow locally expected traffic conventions. In a narrow passage, closer proximity may be unavoidable, requiring lower speed and increased predictability. The robot should avoid oscillating repeatedly from one side to another when negotiating with an approaching person, because such behavior makes the robot\'s intended passing direction difficult to understand.

Yielding is an important component of socially understandable navigation. A robot should recognize situations in which allowing a person to pass first is preferable to aggressively optimizing its own travel time. Doorways, corridor intersections, elevators, narrow passages, and shared workspaces often require implicit negotiation. The humanoid can slow, stop, move slightly aside, or orient its body to communicate yielding, allowing its physical trajectory itself to function as an interaction signal.

Navigation near stationary people requires awareness of their activity. A person waiting beside a wall may tolerate a different passing distance from someone operating machinery, examining a shelf, assisting another person, or performing a delicate manual task. Social cost can therefore incorporate activity recognition and semantic environmental information. The goal is to prevent the planner from treating every open geometric region as equally suitable simply because no physical obstacle occupies it.

Robot body geometry is especially important for humanoid social navigation. Unlike a compact mobile base, a humanoid has swinging arms, moving legs, a torso, and potentially carried objects. The effective occupied region changes during walking and manipulation. Navigation safety should therefore consider the predicted whole-body swept volume rather than only the footprint between the feet, particularly when passing close to people or moving through constrained environments.

Carried objects further modify social space and collision risk. A humanoid transporting a long box, tray, tool, or panel may occupy significantly more space than its body footprint suggests. The planner should incorporate object geometry, orientation, and motion into collision and social-distance calculations. It may need to rotate the object, change walking orientation, wait for a person to pass, or choose a wider route even when a shorter path remains geometrically possible.

Social navigation should integrate naturally with gaze and communication. Before crossing near a person, the humanoid can orient its head toward the person to indicate awareness. When entering a shared workspace, it may use a short verbal cue if visual signaling is insufficient. Conversely, the robot should not require constant explicit conversation for ordinary passing behavior. Motion, speed, body orientation, and gaze can communicate many intentions implicitly when they are consistent and predictable.

Crowded environments create a more difficult planning problem because maintaining preferred personal distances from everyone may be impossible. The planner must distinguish hard safety constraints from softer social preferences. Collision avoidance, balance, and protective limits remain mandatory, while preferred interpersonal distance can be relaxed when necessary. Such relaxation should occur gradually and with reduced speed, preventing the robot from becoming immobilized while still avoiding unnecessarily intrusive behavior.

Navigation goals themselves may need social interpretation. A command such as "go to that person" does not specify an exact Cartesian pose. The robot must infer a suitable interaction position based on the person\'s orientation, activity, available free space, and purpose of the encounter. Similarly, "follow me" requires maintaining an appropriate relative distance and visibility rather than simply minimizing distance to the user\'s current position. HRI therefore influences both goal generation and trajectory planning.

Following behavior requires continuous adaptation to human speed and direction. The humanoid should avoid approaching so closely that normal changes in walking speed create repeated emergency braking, while excessive separation can make the interaction feel disconnected. A preferred following region can be maintained relative to the user, with velocity adjusted smoothly as the person accelerates, stops, turns, or passes through doors. Temporary occlusion should be handled through tracking and prediction rather than abrupt pursuit.

Social-navigation planning can combine global and local reasoning. A global planner identifies feasible routes through the environment, while a local planner adapts motion around people and dynamic obstacles. Social costs can influence both levels: a global route may avoid a crowded waiting area, while local control handles individual encounters. This separation allows longer-term efficiency and short-term social responsiveness to be optimized without treating every human interaction as a complete replanning problem.

Learning-based methods can complement rule-based social navigation by capturing interaction patterns that are difficult to encode manually. Demonstrations, pedestrian trajectories, simulation, or reinforcement learning can provide policies for passing, following, yielding, and approaching. However, learned behavior should operate within explicit safety constraints. A policy that appears socially natural in training data must not be permitted to violate minimum separation, velocity, stability, or collision-avoidance requirements.

Cultural and environmental conventions influence navigation behavior. Preferred passing sides, interpersonal distances, queue behavior, doorway etiquette, and acceptable robot approach patterns can vary between locations and user groups. A humanoid deployed across factories, hospitals, offices, retail environments, and homes may therefore require configurable social-navigation parameters. Adaptation should preserve predictable baseline behavior when local conventions or individual preferences remain uncertain.

Failure handling is necessary when human behavior differs from prediction. A person may suddenly stop, reverse direction, step into the robot\'s path, or intentionally approach the humanoid. The navigation system should update predictions rapidly and transition smoothly to slowing, stopping, yielding, or replanning. Repeated abrupt emergency responses indicate that prediction or planning margins may be inadequate, so operational monitoring should distinguish occasional unavoidable events from systematic navigation weaknesses.

Evaluation of social navigation should extend beyond collision rate and travel time. Relevant measures include minimum human distance, path efficiency, velocity smoothness, prediction error, yielding behavior, personal-space violations, unnecessary stops, interaction duration, perceived comfort, and predictability. Testing should include corridors, intersections, doorways, groups, stationary people, crossing pedestrians, following tasks, crowded areas, carried objects, and situations in which social preferences conflict with geometric efficiency.

A production social-navigation system ultimately combines human perception, trajectory prediction, proxemic modeling, semantic context, whole-body geometry, motion planning, locomotion control, communication, and safety supervision. Proxemics defines socially meaningful spatial relationships, while navigation converts those relationships into feasible robot motion. When these functions operate continuously as a closed loop, a humanoid can move efficiently through human environments while remaining physically safe, socially understandable, and appropriately respectful of shared space.

09.07. LLM Based Conversational HRI Agent [w/Code]

An LLM-based conversational HRI agent enables a humanoid robot to interact with people through flexible natural language while maintaining awareness of physical tasks, environmental context, and ongoing dialogue. Unlike conventional command interfaces based on fixed intents and predefined phrases, a Large Language Model can interpret varied expressions, maintain conversational context, explain robot behavior, and transform high-level requests into structured task representations suitable for embodied execution.

The conversational agent should be positioned above deterministic perception, planning, control, and safety layers rather than directly controlling robot actuators. Its primary role is to interpret human intent, organize dialogue context, reason about available information, and propose actions or task goals. Physical execution remains the responsibility of validated robot capabilities. This separation preserves the flexibility of language reasoning without allowing generated text to bypass constraints required for safe humanoid operation.

Human input may arrive through speech, text, gesture, gaze, or combinations of these channels. Automatic Speech Recognition converts spoken language into text, while multimodal perception provides information about people, objects, pointing directions, and current scene state. The conversational agent can combine these observations into a grounded interaction context. A request such as "bring me that box" becomes meaningful only when the linguistic reference can be associated with an actual object detected in the environment.

The LLM requires a structured representation of robot state to reason effectively about embodied tasks. Relevant context may include current location, visible objects, human participants, active task, grasp state, battery status, available skills, navigation constraints, and recent execution results. Rather than exposing unrestricted raw sensor streams, an HRI architecture can provide compact semantic observations through defined interfaces. This reduces unnecessary context while improving consistency between language reasoning and physical state.

Conversation memory allows instructions to extend across multiple turns. A user may first ask the robot to locate a container and later say, "take it to the table near the door." Correct interpretation requires retaining the referenced object and previous task context. The agent can maintain short-term dialogue state containing entities, goals, confirmations, unresolved references, and recent actions. Long-term information should be managed separately so that temporary conversational details do not become permanent assumptions about users or environments.

Grounding is essential because fluent language generation does not guarantee correspondence with the physical world. If the LLM refers to an object, location, person, or robot capability, that entity should be verified against perception, maps, semantic memory, or capability registries. When grounding fails, the agent should request clarification or report that the required information is unavailable. It should not invent a physical object or claim that a task has been completed merely because the language model can describe it plausibly.

A capability registry can define what the humanoid is actually able to do. Skills such as navigate, detect, grasp, place, follow, hand over, inspect, or speak can expose explicit parameters, preconditions, outputs, and failure states. The conversational agent maps natural-language requests onto these controlled interfaces instead of generating arbitrary low-level commands. This allows new language expressions to access existing robot functionality while preserving predictable execution contracts between the HRI agent and robot software.

Complex requests may require task decomposition. An instruction such as "clean up the items on the table and bring me the red bottle" contains multiple goals, ordering relationships, object references, and possible constraints. The LLM can propose a sequence of semantic subgoals, but a task planner should validate dependencies and feasibility. Decomposition becomes especially useful when tasks involve navigation, perception, manipulation, human interaction, and recovery steps that must be coordinated over an extended period.

Clarification dialogue is required whenever several interpretations remain plausible. If the user requests "the tool" while multiple tools are visible, the agent can ask which one is intended rather than selecting arbitrarily. Clarification can use available scene information to make questions concise, such as asking whether the user means the tool on the left. This converts uncertainty into an explicit interaction step and prevents linguistic ambiguity from propagating into unnecessary or potentially unsafe physical behavior.

Confidence should be treated as a system-level concept rather than relying only on the apparent certainty of generated language. ASR confidence, object detection reliability, reference-grounding ambiguity, planner feasibility, skill status, and safety conditions may all affect whether an instruction can be executed. The conversational agent should communicate uncertainty when relevant and avoid presenting tentative interpretations as established facts. Physical action should proceed only after the required evidence and validation thresholds have been satisfied.

Tool use allows the LLM to obtain information and invoke robot functions through controlled interfaces. Instead of attempting to answer every question from its internal language representation, the agent can query maps, object databases, task state, perception results, or external information services when permitted. Tool outputs should be treated as authoritative for the state they represent, while the LLM provides interpretation and conversational organization. This architecture reduces hallucination and improves traceability of decisions.

The agent should distinguish conversational reasoning from execution authorization. It may propose that the robot move closer, pick up an object, or follow a person, but a separate execution layer should verify collision constraints, reachability, payload limits, balance, human proximity, and operational permissions. Safety supervisors must retain the ability to reject, modify, pause, or terminate proposed actions regardless of the LLM\'s conversational conclusion.

Safety-critical commands require especially direct handling. Requests such as stop, release, move away, or cancel may need to bypass lengthy language reasoning and activate predefined high-priority behaviors. An LLM can still explain the resulting state afterward, but it should not become a mandatory computational dependency for emergency response. Deterministic stop channels, protective control, collision avoidance, and hardware safety mechanisms remain independent of conversational intelligence.

The conversational agent can also explain robot behavior. If a task cannot be completed, the robot can report whether the object was not detected, the path was blocked, the grasp failed, or a safety condition prevented execution. Explanations should be derived from actual subsystem status rather than generated speculatively. Clear causal feedback helps users decide whether to modify the instruction, change the environment, provide additional information, or abandon the task.

Failure recovery can combine dialogue with replanning. When a manipulation attempt fails, the agent may obtain the failure reason, ask the user for assistance if necessary, and request an alternative plan. If a destination becomes inaccessible, it can propose another location or wait for the obstruction to clear. The LLM is useful for organizing recovery options in natural language, while perception and planning components determine whether proposed alternatives are physically feasible.

Multimodal interaction extends conversational reasoning beyond words. A user may say "put this over there" while holding an object and pointing toward a surface. Gesture recognition, gaze estimation, object tracking, and scene understanding can supply candidate referents to the LLM or a grounding module. Temporal association is important because speech and gestures may occur at slightly different times. Maintaining a short multimodal context enables natural deictic expressions to be interpreted without requiring explicit object names.

The robot\'s conversational output should coordinate with gaze, facial expression, gesture, and body motion. When acknowledging a person, the humanoid can orient toward the speaker; when referring to an object, it can look or gesture toward that object. Before beginning a physical task, a short verbal acknowledgment can indicate the interpreted goal. Consistency across communication channels improves predictability because users can compare what the robot says with what it visibly attends to and prepares to do.

Latency must be managed carefully because conversational HRI occurs in real time. Large models may require more computation than conventional intent classifiers, particularly when processing long context or complex reasoning. Streaming ASR, compact prompts, structured state summaries, local inference, caching, and selective model invocation can reduce response time. Immediate acknowledgment can be separated from slower task reasoning so that the user knows the instruction was received while planning continues.

Edge and cloud deployment introduce different tradeoffs. Onboard inference can reduce communication latency, preserve operation during network loss, and keep sensitive sensor information local, but computational resources may constrain model size. Cloud inference can provide larger models and broader services but introduces network dependency and variable latency. Hybrid architectures can keep safety, grounding, essential dialogue, and execution interfaces local while selectively using remote reasoning for noncritical complex requests.

Privacy is particularly important because conversational humanoids may continuously encounter voices, faces, locations, and contextual information. Systems should minimize unnecessary recording, define when interaction sensing is active, and limit what information is retained or transmitted. Users should be able to understand when the robot is listening and what information is required for the task. Data retention and remote processing policies should be separated from the conversational convenience provided by the LLM.

Prompt and context management affect reliability. The agent should receive explicit descriptions of its role, available capabilities, current state, prohibited actions, and required output formats. Structured schemas can constrain task proposals and tool calls so that downstream software does not need to interpret arbitrary prose. Context should also be refreshed from current robot state because outdated observations can cause language reasoning to remain internally coherent while becoming physically incorrect.

Evaluation of an LLM-based HRI agent should extend beyond conversational fluency. Relevant measures include intent interpretation, reference-grounding accuracy, clarification success, task decomposition quality, tool-selection accuracy, hallucination rate, execution success, recovery performance, response latency, safety-rule compliance, and consistency between spoken claims and actual robot state. Tests should deliberately include ambiguous commands, unavailable objects, impossible tasks, changing environments, interrupted dialogue, and subsystem failures.

A production conversational HRI agent ultimately functions as a semantic coordination layer between human communication and embodied robot intelligence. The LLM provides flexible language understanding, contextual reasoning, clarification, task decomposition, and explanation, while perception grounds language in the environment and planners, skills, controllers, and safety systems govern physical execution. When these responsibilities remain clearly separated but tightly connected, conversational intelligence can make humanoid robots substantially easier to instruct without sacrificing the determinism required for dependable physical interaction.

09.08. User Intent Estimation and Proactive Assist [w/Code]

User intent estimation and proactive assistance enable a humanoid robot to anticipate what a person is likely trying to accomplish and provide useful support before receiving a complete explicit command. In Human--Robot Interaction, this capability moves the robot from purely reactive command execution toward context-aware collaboration. The objective is not to predict private thoughts, but to infer task-relevant intent from observable behavior, dialogue, environmental state, and interaction history.

Intent estimation begins with multimodal observation of the user and surrounding environment. Human pose, hand motion, gaze, walking direction, object interaction, speech, facial orientation, and proximity can provide evidence about an ongoing activity. The robot can combine these signals with object locations, workspace state, current task progress, and recent dialogue. No single cue should normally determine intent because similar human movements can correspond to very different goals depending on context.

Temporal information is especially important because intent develops through sequences of actions. A person looking toward a shelf, walking toward it, reaching for a container, and then turning toward a workbench provides stronger evidence than any individual observation. The robot can maintain a short history of human and environmental states and estimate how the observed sequence relates to known tasks. Temporal models can therefore distinguish meaningful action progression from incidental movements that should not trigger assistance.

Intent can be represented at several levels of abstraction. Low-level intent may describe an immediate movement such as reaching, walking, or handing over an object. Task-level intent may represent goals such as assembling a component, retrieving a tool, clearing a workspace, or carrying a container. Higher-level intent may describe broader objectives composed of several tasks. A practical HRI architecture should maintain these levels separately because confidence and appropriate robot responses differ substantially across them.

Activity recognition provides one source of evidence for task-level intent. Human actions can be identified from body pose, hand-object interaction, object state changes, and environmental context. However, recognizing what a person is doing does not always reveal what the person intends to do next. Intent estimation therefore extends beyond activity classification by predicting plausible future goals or transitions. The system should preserve multiple hypotheses when observations support more than one likely continuation.

Probabilistic reasoning is useful because human behavior is inherently uncertain. Instead of assigning one definitive intention, the robot can maintain a probability distribution over candidate goals. New observations continuously update these probabilities as the interaction progresses. When one hypothesis becomes sufficiently supported, the robot may prepare assistance, but preparation does not necessarily imply immediate execution. Separating prediction from action prevents uncertain inference from producing premature physical behavior.

Context significantly changes the interpretation of human actions. Reaching toward a bottle may indicate an intention to pick it up, move it, inspect it, or simply clear access to another object. Information about the current task, previous dialogue, object affordances, workspace configuration, and recent actions can narrow these possibilities. Semantic scene understanding therefore provides an important foundation for intent estimation by connecting human behavior to meaningful objects, locations, and task relationships.

Gaze can provide early evidence of attention before physical movement begins. A person may look repeatedly toward an object or destination before reaching or walking toward it. Gaze alone, however, is not a reliable declaration of intent because people frequently observe objects without acting on them. Combining gaze with body orientation, hand preparation, task context, and dialogue can produce stronger evidence while avoiding assistance triggered merely because the user briefly looked at something.

Speech provides explicit and implicit information about future goals. A direct statement such as "I need the wrench" strongly constrains the likely intent, while conversational phrases such as "this box is too heavy" may indirectly suggest a need for assistance. An LLM-based conversational agent can interpret such language within dialogue context, but its inferred intent should still be grounded against available objects, robot capabilities, and current task state before being converted into proactive physical action.

Object interaction is another powerful source of evidence. The robot can observe which objects the user touches, moves, opens, carries, or repeatedly attempts to manipulate. Changes in object state can indicate progress through a known procedure. For example, completing one assembly step may make a particular tool or component likely to be needed next. Task models can exploit these dependencies to predict useful assistance while remaining sensitive to deviations from the expected workflow.

Proactive assistance should be graded according to confidence and consequence. At low confidence, the robot may simply monitor the situation or prepare internal resources. At moderate confidence, it may offer assistance verbally or move to a nonintrusive standby position. At high confidence, it may perform a reversible preparatory action, such as retrieving a likely required object. Actions involving close contact, irreversible changes, significant force, or safety consequences should generally require explicit confirmation regardless of prediction confidence.

Offering assistance is often safer than automatically acting. If the robot infers that a person is struggling to carry an object, it can ask whether help is wanted rather than immediately grasping the load. If a frequently used tool appears likely to be needed next, the robot can indicate that it is available. This preserves human agency and allows incorrect predictions to be corrected with little cost. Proactivity should reduce user effort without making the robot intrusive or difficult to control.

The robot should also estimate whether assistance is actually desirable. A user may intentionally perform a task manually for inspection, training, preference, or safety reasons even when the robot could complete it faster. Repeated unsolicited intervention can reduce trust and disrupt workflow. Engagement state, previous acceptance or rejection of offers, explicit preferences, and current task roles can therefore influence the assistance policy independently of the predicted task intent.

Proactive behavior can include informational as well as physical assistance. The humanoid may warn that a required object is missing, indicate that a path is blocked, remind the user of an unfinished step, or report that a requested resource is available. Informational assistance generally carries less physical risk and can therefore be offered at lower confidence than manipulation. Nevertheless, excessive notifications can become distracting, so relevance and timing remain important.

Physical proactive assistance requires capability and safety validation before execution. Even when the inferred intent is correct, the proposed action may be infeasible because an object is too heavy, a grasp is unavailable, a person is too close, or the route is obstructed. The assistance manager should query perception, planning, manipulation, locomotion, and safety systems before committing to action. Intent confidence cannot substitute for physical feasibility or safety validation.

Shared workspaces require coordination between prediction and motion planning. If the robot anticipates that a person will move toward a particular location, it can avoid occupying that region or position itself where assistance can be provided without blocking the user. Predicted human motion can influence navigation, handover location, and object placement. Such anticipatory behavior can improve collaboration even when the robot performs no explicit task on the person\'s behalf.

Intent estimation should support continuous revision. A person may change goals, interrupt a task, respond to another event, or reject the robot\'s proposed assistance. The system should therefore avoid committing too strongly to earlier predictions. New observations must be able to lower the probability of an old hypothesis and activate alternatives. Proactive actions should preferably be reversible so that the robot can recover gracefully when the user\'s actual intention differs from its prediction.

Failure recovery includes recognizing incorrect assistance. If the user rejects an offered object or corrects the robot\'s interpretation, the system should update the current interaction state rather than repeatedly proposing the same action. The robot can acknowledge the correction, withdraw safely, and continue observing. Such corrections provide useful interaction evidence, but adaptation should remain bounded so that one unusual event does not permanently redefine the user\'s normal behavior.

Personalization can improve intent prediction over repeated interactions. Different users may follow different task sequences, prefer different tools, accept different levels of proactive help, or communicate intent through different combinations of speech, gaze, and gesture. The robot can adapt interaction parameters based on explicit preferences and reliable repeated patterns. Personalization should remain transparent and should not require unnecessary inference of sensitive personal characteristics.

Proactive assistance also requires arbitration when several possible helpful actions compete. The robot may predict that the user will need a tool, require space to move, and soon request transport assistance at the same time. An assistance policy can rank candidate actions according to confidence, urgency, usefulness, reversibility, execution cost, and risk. Safety-critical warnings should take priority over convenience, while low-value proactive actions can be postponed when they would interfere with the primary task.

Learning-based models can improve prediction by capturing complex relationships between human actions, objects, and task progression. Demonstration data, recorded interaction trajectories, simulation, sequence models, transformers, or probabilistic task models can be used to learn intent patterns. However, learned prediction should remain separated from safety authority. High statistical confidence from a model does not justify bypassing physical constraints, explicit permissions, or required confirmation.

Evaluation should measure both prediction quality and the usefulness of resulting assistance. Relevant metrics include intent classification accuracy, time-to-correct prediction, calibration of confidence, false proactive activation, missed assistance opportunities, acceptance rate, task completion time, user workload, interruption frequency, recovery after incorrect prediction, and safety compliance. Testing should include ambiguous behavior, changing goals, incomplete tasks, unexpected actions, rejected assistance, and multiple plausible future intentions.

A production intent-estimation and proactive-assistance system ultimately combines multimodal human perception, temporal reasoning, task context, probabilistic prediction, dialogue, robot capability models, planning, and safety supervision. Intent estimation proposes what the user may need, while the assistance policy determines whether, when, and how the robot should respond. When uncertainty and human agency remain explicit, a humanoid can anticipate useful actions without confusing prediction with permission or sacrificing predictable and safe collaboration.

09.09. HRI Safety ISO 13482 Service Robot Standard

HRI safety defines how a service or personal-care robot can interact physically and socially with people while keeping foreseeable hazards within acceptable limits. ISO 13482:2014 addresses safety requirements for personal care robots, making it an important reference for robots operating near ordinary users rather than inside fully isolated industrial cells. For humanoid HRI, safety must therefore be treated as a system property spanning mechanics, sensing, control, software, interaction, and operation.

ISO 13482 is particularly relevant because personal care robots may share spaces directly with people and perform mobility or assistance functions. The standard is referenced specifically as "Robots and robotic devices---Safety requirements for personal care robots." Its scope provides a foundation for considering hazards created by robot movement, contact, control failures, and human interaction, although technical compliance alone should not be interpreted as covering every legal or societal obligation.

Safety engineering begins with identifying hazards across the complete robot lifecycle and operating context. A humanoid can create risks through unexpected locomotion, arm motion, crushing or trapping, instability, falling, excessive contact force, dropped objects, electrical faults, thermal conditions, or incorrect autonomous decisions. HRI introduces additional uncertainty because humans can approach unexpectedly, misunderstand robot intentions, enter planned trajectories, or physically interact with the robot.

Risk assessment connects each identified hazard with foreseeable exposure, possible severity, and the effectiveness of available protective measures. The objective is not to assume that sophisticated AI will avoid every hazardous situation, but to design multiple layers that reduce risk even when perception, prediction, planning, or interaction reasoning is imperfect. Safety requirements should therefore be traceable from identified hazards to protective functions, implementation, verification, and validation.

Human detection and distance monitoring provide an important protective layer during interaction. Cameras, depth sensors, LiDAR, tactile sensing, or other safety-related sensing can estimate whether a person is entering a region affected by robot motion. The resulting information can modify velocity, restrict trajectories, increase separation, or trigger stopping behavior. The HRI architecture should not depend on conversational understanding alone to determine whether close physical operation remains safe.

Speed monitoring is closely related to separation management. A humanoid moving quickly through free space may need to reduce velocity when approaching a person, entering a shared workspace, or beginning physical collaboration. Speed limits can depend on operating mode, body region, task, environment, and proximity. Importantly, the system must consider moving arms, legs, torso, end effectors, and carried objects rather than treating the robot as a single point or static footprint.

Force monitoring becomes critical when physical contact is possible or intentional. Joint torque sensing, wrist force-torque sensors, tactile sensors, motor-current estimation, or other measurements can identify unexpected interaction forces. Handover, guidance, and co-carrying require controlled contact, whereas collision may require immediate mitigation. The controller must therefore distinguish expected task forces from hazardous contact while enforcing conservative force and torque limits independently of high-level HRI reasoning.

Collision detection should operate as part of an always-active safety architecture. A robot may collide because a person moves unexpectedly, an object is missed by perception, localization becomes inaccurate, or a planned trajectory no longer matches reality. Detection should trigger an appropriate protective response such as reducing torque, stopping motion, increasing compliance, retreating when safe, or entering a controlled fault state. Recovery should occur only after the relevant safety conditions have been reassessed.

Emergency stopping provides a separate mechanism for terminating hazardous motion when normal interaction or autonomous control cannot be trusted. Emergency-stop functionality should not depend on speech recognition, an LLM, network communication, or complex task reasoning. A spoken "stop" or gesture can provide an additional interaction channel, but it should complement rather than replace dedicated safety mechanisms. The safety path must remain available even when higher-level AI components fail.

Safe zones and virtual boundaries can restrict where particular robot behaviors are permitted. A humanoid may be allowed to walk autonomously in one region while manipulation, high-speed motion, or certain tools are prohibited in another. Dynamic zones can also be constructed around detected people. These mechanisms allow the operational environment to participate in risk reduction instead of expecting the robot controller to solve every safety problem through reactive collision avoidance.

Functional safety behavior must remain independent from ordinary task success. A perception model may correctly recognize an object while failing to detect a nearby hand, or an LLM may correctly interpret an instruction while proposing an action that is physically unsafe. Task confidence and safety confidence are therefore different quantities. Safety supervisors should retain authority to constrain, reject, pause, or terminate commands produced by planners, learned policies, conversational agents, or proactive-assistance modules.

Human-aware motion planning provides another preventive layer. Instead of waiting for a protective stop, the robot can plan trajectories that maintain separation, avoid sensitive body regions, limit approach speed, and remain predictable to nearby people. Whole-body geometry is important for humanoids because an apparently safe end-effector path may still cause the elbow, torso, knee, or carried object to pass dangerously close to a person. Safety assessment must consider the complete motion envelope.

Physical collaboration requires special attention because separation cannot always be maintained. During handover or co-carrying, humans and robots intentionally occupy overlapping workspaces and exchange forces. Compliant control, grasp monitoring, load estimation, stable whole-body posture, controlled velocity, and reliable release logic become central protective measures. If interaction becomes uncertain, the system should transition toward a safer state rather than increasing force to preserve the original trajectory.

Robot stability is itself an HRI safety concern. A humanoid that loses balance can create hazards even when its arms satisfy force limits. Whole-body controllers should monitor support conditions, center-of-mass behavior, contact state, disturbance recovery, and feasible joint limits. When stability margins deteriorate, maintaining balance or reaching a controlled safe posture should take priority over conversational goals, manipulation objectives, navigation efficiency, or the desire to complete a requested task.

Fault handling should include sensing, computation, communication, actuation, and power failures. Loss of a camera, stale human tracking, actuator overheating, communication interruption, inconsistent joint feedback, or depleted energy can invalidate assumptions used by normal operation. The system should detect such conditions and move toward a defined degraded or safe state. Redundancy, plausibility checking, watchdogs, diagnostic monitoring, and fault reporting can reduce dependence on any single component.

Safety-related HRI also depends on understandable robot behavior. Humans should be able to recognize whether the robot is active, waiting, approaching, yielding, stopping, or unable to perform a task. Gaze, body orientation, lights, displays, speech, and motion can make robot intentions more predictable, but these communication channels are not substitutes for protective functions. A friendly expression or verbal warning cannot compensate for unsafe speed, inadequate separation, or uncontrolled contact force.

AI-based perception and decision making introduce additional uncertainty because model behavior may vary with environments and inputs not represented during development. Learned human detection, intent estimation, trajectory prediction, or language reasoning should therefore be evaluated under degraded sensing, ambiguous situations, and out-of-distribution conditions. When confidence becomes insufficient, the robot should adopt conservative behavior rather than assuming that the most probable model output is correct.

Verification and validation should demonstrate that protective functions operate under both nominal and abnormal conditions. Testing can include human approach, unexpected path crossing, sensor loss, obstacle appearance, excessive force, failed grasp, unstable posture, communication failure, emergency-stop activation, and recovery behavior. Simulation, software-in-the-loop, hardware-in-the-loop, controlled laboratory tests, and progressively realistic field trials can provide complementary evidence before unrestricted deployment.

Safety evaluation should measure more than task completion or collision frequency. Relevant metrics include minimum separation, stopping response, residual motion, peak contact force, speed-limit compliance, false and missed human detection, safety-function latency, fault-detection time, stability margin, emergency-stop response, and successful transition to safe states. Test conditions should cover representative users, operating environments, foreseeable misuse, and combinations of failures rather than isolated nominal scenarios.

Technical safety standards do not by themselves resolve every issue associated with human-facing robots. The available reference notes that compliance with technical safety requirements does not necessarily establish compliance with the complete legal framework and highlights additional concerns involving data protection, responsibility, transparency, autonomy, and dignity. Human-facing robotic systems therefore require broader governance alongside engineering safety and product compliance.

Privacy is particularly relevant when HRI safety functions use cameras and continuous user monitoring. The available source notes that care robots may record and wirelessly communicate information and that these capabilities can create privacy concerns without appropriate regulation, policies, and protocols. Safety-oriented perception should therefore collect and retain only information justified by the application while protecting data used for human monitoring and interaction.

A production HRI safety architecture ultimately combines risk assessment, distance and speed monitoring, force limitation, collision detection, safe zones, emergency stopping, fault management, human-aware planning, compliant control, and systematic verification. ISO 13482 provides an important personal-care-robot safety reference, but dependable humanoid deployment requires these principles to be implemented throughout the complete system lifecycle and coordinated with applicable legal, privacy, and operational requirements.

09.10. HRI User Study and Evaluation Methodology

User studies provide a systematic method for evaluating how people understand, experience, and collaborate with humanoid robots. Technical benchmarks alone cannot determine whether an HRI system is intuitive, comfortable, trustworthy, predictable, or socially acceptable. Evaluation therefore combines objective robot performance with human-centered measures, allowing researchers to determine not only whether an interaction succeeds but also how users perceive and respond to the robot.

An HRI evaluation begins by defining a clear research question. A study may examine whether gaze improves handover understanding, whether proactive assistance reduces workload, or whether socially aware navigation increases comfort. The research question should be translated into measurable hypotheses and dependent variables. Clearly defined objectives prevent the experiment from collecting many unrelated metrics without establishing what aspect of interaction quality is actually being tested.

Independent variables represent the interaction conditions intentionally changed during an experiment. Examples include robot speed, interpersonal distance, communication modality, level of autonomy, explanation style, gaze behavior, or proactive assistance strategy. Dependent variables capture the resulting effects on users or task performance. Experimental design should isolate important factors sufficiently to support meaningful interpretation while preserving enough realism to represent the intended HRI application.

Within-subject and between-subject designs offer different advantages. In a within-subject study, each participant experiences multiple robot conditions, allowing direct comparison while reducing variability between individuals. However, learning, fatigue, and ordering effects can influence later trials. Between-subject studies expose different participant groups to different conditions, reducing carryover effects but generally requiring more participants to achieve comparable statistical sensitivity.

Counterbalancing can reduce ordering effects when participants experience several conditions. If every participant encounters the same interaction sequence, improvements may result from familiarity rather than the experimental manipulation. Conditions can therefore be presented in different orders across participants. Randomization is also useful for reducing systematic bias in participant assignment, task order, object selection, or scenario presentation when experimental constraints permit it.

Participant selection should reflect the population for which the humanoid system is intended. Age, experience with robots, technical background, physical capability, language proficiency, and familiarity with the task may influence interaction behavior. A laboratory study with robotics researchers may reveal technical problems but may not represent first-time users. Recruitment criteria and participant characteristics should therefore be documented so that study results can be interpreted within an appropriate population.

Sample size should be considered before data collection rather than determined only by convenience. Larger samples generally provide more reliable estimates of differences and variability, but HRI experiments involving physical robots can be expensive and time-consuming. Pilot studies can estimate variance, reveal procedural problems, and support later power analysis. The study design should distinguish exploratory evaluation from confirmatory experiments intended to support stronger statistical conclusions.

Ethical review and informed consent are essential when experiments involve human participants. Participants should understand the study procedure, expected interaction, potential risks, collected data, recording methods, and their ability to withdraw. Physical HRI requires additional attention because robots can move, make contact, or carry objects near people. Safety procedures should be established before recruitment and should remain independent of whether the experimental hypothesis concerns safety itself.

A standardized experimental protocol improves repeatability. Robot initial position, task instructions, environmental layout, object placement, sensor configuration, interaction timing, and failure-handling procedures should be controlled where appropriate. Researchers should also define what happens when the robot fails during a trial. Without consistent procedures, differences between participants may reflect uncontrolled experimental conditions rather than the HRI behavior being evaluated.

Objective task-performance metrics provide one dimension of evaluation. Measures may include task completion time, success rate, number of errors, navigation efficiency, handover duration, intervention frequency, clarification requests, response latency, collision or near-collision events, and recovery time. These metrics reveal whether an HRI design improves operational performance, but they do not necessarily indicate whether users found the interaction comfortable, understandable, or desirable.

Behavioral measures capture how participants physically respond to the robot. Researchers can analyze interpersonal distance, approach or avoidance behavior, gaze direction, hesitation, response time, body orientation, gesture frequency, intervention, and task strategy. Such observations are particularly valuable because users may report that an interaction was acceptable while their movement reveals discomfort or uncertainty. Behavioral data should therefore complement rather than automatically replace subjective reports.

Subjective questionnaires measure perceptions that cannot be obtained directly from robot logs. Participants can rate trust, comfort, perceived safety, usability, predictability, workload, social presence, intelligence, responsiveness, and willingness to interact again. Validated questionnaires should be preferred when suitable instruments exist because established scales improve comparability between studies. Custom questions may still be necessary for application-specific properties but should be written carefully to avoid leading participants.

Workload evaluation is important when interaction requires users to monitor, instruct, correct, or coordinate with the robot. A system may complete tasks successfully while imposing excessive cognitive demand on the human. Measures of mental effort, attention, frustration, temporal pressure, and physical effort can reveal this problem. Proactive assistance or natural-language interfaces should therefore be evaluated not only by task success but also by whether they genuinely reduce user effort.

Trust should be evaluated as a calibrated relationship rather than simply maximized. Excessive trust can be dangerous if users assume that the humanoid can perceive or execute more reliably than it actually can. Insufficient trust can cause unnecessary intervention and prevent useful autonomy. Experiments can compare user expectations with actual robot capability, examine reactions to failures, and measure whether explanations or recovery behavior help users form an appropriately accurate understanding of system reliability.

Perceived safety and physical safety are related but distinct measures. Sensor logs may show that all minimum-distance and force constraints were satisfied while users still report discomfort because motion was sudden or difficult to predict. Conversely, a participant may feel safe even when objective margins are inadequate. A complete HRI evaluation should therefore measure both engineering safety indicators and human perception rather than treating either one as a substitute for the other.

Qualitative interviews can explain why quantitative results occur. After an interaction, participants may describe which robot behaviors were confusing, helpful, intrusive, slow, or unexpectedly natural. Open-ended questions can reveal design issues that predefined questionnaires fail to capture. Interview analysis should follow a consistent coding process so that recurring themes can be distinguished from isolated comments and connected systematically to observed interaction behavior.

Think-aloud methods can provide detailed information about user reasoning during an interaction, particularly when evaluating interfaces or task instructions. However, speaking continuously can alter natural behavior and increase workload, making the method less appropriate for some physical collaboration experiments. Researchers should therefore decide whether concurrent verbalization, post-task interviews, or retrospective video review best preserves the interaction characteristics relevant to the research question.

Multimodal data collection can synchronize robot logs with video, audio, human pose, gaze, force measurements, task events, and questionnaire responses. Accurate timestamps allow researchers to reconstruct critical moments, such as when the robot moved, when the user hesitated, and when a safety intervention occurred. Synchronization becomes especially important for studying cause and effect in dynamic HRI because small timing differences may substantially change how a robot action is perceived.

Robot failures should be incorporated into evaluation when robustness and trust are important. A study that includes only successful interactions may overestimate usability because real deployments inevitably encounter perception errors, failed grasps, misunderstood commands, blocked paths, and communication delays. Controlled failure scenarios allow researchers to evaluate whether users recognize the problem, understand the robot\'s explanation, provide assistance when appropriate, and regain confidence after successful recovery.

Wizard-of-Oz experimentation can evaluate interaction concepts before every autonomous capability is fully implemented. A hidden human operator may control selected robot functions while participants believe they are interacting with an autonomous system. This approach can rapidly test dialogue, gesture, timing, or assistance strategies, but the operator\'s role must be carefully standardized. Results should not be interpreted as evidence that the corresponding autonomous technology has already achieved the demonstrated performance.

Ecological validity describes how well experimental conditions represent real use. Highly controlled laboratory studies improve repeatability but may not capture noise, interruptions, crowded environments, unfamiliar users, or long-term adaptation. Field studies provide greater realism but introduce uncontrolled variables. HRI development can therefore progress from controlled experiments toward increasingly realistic deployments, using each stage to answer different questions about interaction quality and system robustness.

Longitudinal evaluation becomes important when novelty effects may influence results. Users often behave differently during their first encounter with a humanoid than after days or weeks of repeated interaction. Initial curiosity may increase engagement, while repeated exposure may reveal annoyance, workload, or adaptation that short studies cannot detect. Long-term studies can examine how trust, preferred interaction style, reliance, proactive assistance acceptance, and task efficiency change with experience.

Statistical analysis should match the experimental design and characteristics of the collected data. Researchers should examine effect size and uncertainty in addition to whether a statistical significance threshold is reached. Missing trials, robot failures, outliers, repeated measures, and multiple comparisons should be handled transparently. Predefined analysis plans can reduce the temptation to select only metrics or comparisons that produce favorable conclusions after data collection.

Reproducibility requires detailed documentation of robot hardware, software version, control parameters, model configuration, environment, participant instructions, experimental conditions, and analysis methods. HRI results can depend strongly on seemingly minor differences in motion speed, timing, voice, gaze, or distance. Recording these details allows future experiments to distinguish genuine scientific differences from changes caused by implementation or procedural variation.

A comprehensive HRI evaluation ultimately combines technical performance, behavioral observation, subjective assessment, qualitative feedback, safety evidence, and statistical analysis. No single metric can determine whether a humanoid interaction is successful. By designing studies around explicit hypotheses, representative users, controlled procedures, realistic scenarios, and multiple complementary measures, researchers can transform human experience into systematic evidence for improving safe, understandable, efficient, and socially appropriate humanoid behavior.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Top comments (0)