DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Humanoid Robot Software — Ch.07: Humanoid Manipulation [EN]

STATERA Library · Series 08 · Volume 22 · Chapter 07 · EN
Source: https://github.com/Statera-Library/08_Vol_22_Humanoid_Robot_Software/blob/main/Chapter_07_Humanoid_Manipulation/Chapter_07_Humanoid_Manipulation_EN.md

Volume 22. Humanoid Robot Software

Chapter 07. Humanoid Manipulation

07.01. Humanoid Manipulation Architecture Hand Arm Body

Humanoid manipulation is fundamentally different from fixed-base industrial manipulation because the robot must coordinate its hands, arms, torso, and supporting body while maintaining balance. The manipulation architecture therefore cannot treat the arm as an isolated kinematic chain. Instead, it must organize manipulation as a whole-body process in which grasping, reaching, posture regulation, contact management, and balance constraints are continuously coupled. Volume_22_Humanoid_Robot_Softwa...

At the highest level, a humanoid manipulation system can be organized into task interpretation, perception, manipulation planning, whole-body motion generation, feedback control, and actuator execution. A requested task such as picking up an object is transformed into geometric and physical objectives, including target pose, grasp configuration, contact sequence, allowable forces, body posture, and completion conditions. These objectives are then distributed across the hand, arm, torso, and lower body.

The hand forms the terminal interaction layer between the humanoid and the physical environment. Unlike simple parallel grippers, dexterous hands may contain multiple articulated fingers whose coordinated motion determines grasp stability, contact geometry, and manipulation capability. Hand control therefore combines finger configuration, contact force, tactile feedback, slip detection, and grasp adaptation so that the robot can accommodate objects with different shapes, sizes, materials, and mechanical properties.

The arm provides the principal workspace for positioning and orienting the hand. Shoulder, elbow, and wrist joints collectively generate the end-effector pose required by the manipulation task, while redundant degrees of freedom permit multiple joint configurations for the same hand pose. This redundancy is valuable because the controller can simultaneously optimize reachability, joint limits, collision avoidance, manipulability, energy consumption, and compatibility with the posture of the remaining body.

The humanoid torso extends the effective manipulation workspace beyond the nominal reach of the arms. Waist rotation, trunk bending, and upper-body posture can reposition the shoulders relative to an object, allowing the robot to reach farther or manipulate large objects without excessive arm extension. Torso motion can also improve manipulability and reduce joint loading, but aggressive upper-body movement changes the robot\'s center of mass and momentum and therefore directly interacts with balance control.

The lower body provides the dynamic support structure for manipulation. During standing manipulation, the feet establish contact constraints and define the available support region, while the legs regulate body height, center-of-mass position, and reaction forces. When an object is heavy or an external contact produces substantial forces, the robot may need to alter its stance, bend its knees, move its pelvis, or take a step rather than attempting to compensate exclusively through arm motion.

A practical architecture therefore represents manipulation objectives as coordinated tasks rather than independent joint commands. The hand may receive a grasp objective, the wrist an end-effector pose objective, the elbow a preferred configuration, the torso a posture objective, and the pelvis and feet balance-related constraints. A whole-body controller resolves these simultaneous requirements into dynamically consistent joint torques, positions, or velocities while respecting physical and safety constraints.

Manipulation planning operates at several temporal and spatial scales. A task planner determines actions such as approach, reach, grasp, lift, transport, place, and release, while a motion planner determines feasible trajectories between configurations. A whole-body planner may additionally determine torso movement, stance adjustment, or stepping. The resulting hierarchy allows relatively slow semantic decisions to coexist with fast stabilization and contact-control loops required for physical interaction.

Perception supplies the manipulation architecture with continuously updated estimates of the robot, object, and environment states. Head cameras provide broader scene context, wrist cameras provide close-range observations near the hands, and force or tactile sensors reveal interaction states that vision alone cannot reliably determine. This relationship is consistent with the volume structure, which places head, wrist, 3D scene, object-pose, and tactile perception immediately before humanoid manipulation. Volume_22_Humanoid_Robot_Softwa...

Coordinate-frame management is particularly important because manipulation commands originate from different reference systems. Object poses may be estimated in camera coordinates, grasp targets expressed in object coordinates, hand trajectories generated relative to the world, and control errors evaluated in body or joint coordinates. Accurate transformations among world, pelvis, torso, shoulder, wrist, hand, finger, sensor, and object frames are therefore essential for maintaining geometric consistency throughout the manipulation pipeline.

Contact introduces another architectural boundary between free-space motion and physical interaction. Before contact, the system primarily regulates position and orientation while avoiding collisions. After contact, force, torque, compliance, friction, and object dynamics become central variables. The controller must consequently support transitions between position-dominant motion and compliant interaction, enabling tasks such as pushing, opening doors, inserting components, turning handles, carrying loads, and operating tools without generating unsafe contact forces.

Impedance control provides a useful interface between motion planning and physical interaction because it allows the hand or arm to behave like a programmable mechanical system rather than an infinitely rigid position source. Desired stiffness and damping can be adjusted according to task phase, uncertainty, and contact state. This capability becomes especially important for humanoids operating in human environments, where geometric models are imperfect and unexpected contact must be absorbed safely.

Bimanual manipulation adds coordination between two independently articulated arms and hands. The two end effectors may perform separate actions, cooperate on a shared object, or establish closed kinematic chains through simultaneous contact. In cooperative carrying, for example, the controller must regulate relative hand poses and internal forces while the whole body compensates for the object\'s mass and inertia. The source structure explicitly progresses from dexterous hand and arm control to bimanual coordination and object handover. Volume_22_Humanoid_Robot_Softwa...

Whole-body manipulation becomes necessary when the manipulation objective exceeds the comfortable workspace or force capability of the upper limbs. Instead of planning locomotion and manipulation independently, the robot can coordinate stepping, pelvis motion, torso orientation, arm trajectories, and hand contacts within a unified task. This enables the humanoid to approach objects, reposition while carrying them, push large structures, or continue manipulation across workspace boundaries without artificial separation between walking and working.

Safety constraints must remain active throughout every architectural layer. Joint position and velocity limits, torque limits, self-collision constraints, environmental collision constraints, contact-force limits, balance margins, actuator thermal limits, and emergency conditions constrain otherwise valid manipulation commands. Because a humanoid possesses substantial mass and many powered joints, a locally correct hand trajectory cannot be considered acceptable if it destabilizes the body or creates hazardous motion elsewhere.

The architecture should also distinguish nominal execution from recovery behavior. Object localization may change during reaching, grasp contact may occur earlier than predicted, fingers may slip, a tool may become obstructed, or an external disturbance may alter body posture. Feedback must therefore propagate from perception and low-level control toward manipulation planning so trajectories, grasp parameters, contact modes, or whole-body configurations can be modified rather than blindly executing a predetermined sequence.

Modern learning-based policies can be integrated above or within this architecture without eliminating its physical structure. A learned policy may generate grasp configurations, hand actions, arm trajectories, or full-body action targets, while conventional control layers enforce dynamics, contact, balance, and safety constraints. This separation also prepares the architecture for later imitation-learning, reinforcement-learning, and generalist manipulation policies while retaining deterministic mechanisms for critical low-level execution.

The resulting hand-arm-body architecture is therefore best understood as a hierarchy of tightly coupled physical functions rather than three independent subsystems. The hand establishes and regulates interaction, the arm positions and transports the interaction interface, and the body supplies workspace, momentum regulation, support, and mobility. Perception, planning, learning, whole-body control, and safety supervision connect these functions into one closed-loop manipulation system.

This architectural viewpoint provides the foundation for the remainder of humanoid manipulation development. Dexterous multi-finger control refines hand behavior, impedance control governs contact, bimanual coordination expands cooperative capability, handover and in-hand manipulation increase dexterity, and tool use extends functional reach. These capabilities ultimately converge in unified loco-manipulation and learning-based policies, matching the chapter\'s progression toward whole-body manipulation, hybrid IL/RL policies, and benchmark evaluation.

07.02. Dexterous Hand Control Multi Finger Policy [w/Code]

Dexterous hand control enables a humanoid robot to transform arm-level positioning into precise physical interaction with objects. Unlike a simple gripper that primarily opens and closes, a multi-finger hand contains many coupled degrees of freedom that must coordinate contact location, finger posture, force, and object motion. The control problem therefore combines high-dimensional motion generation with contact-rich feedback and task-dependent adaptation.

A dexterous hand typically consists of several articulated fingers with multiple joints per finger, including an opposable thumb that greatly expands the available grasp set. Each joint contributes to a large configuration space in which many different hand postures can achieve similar object contacts. Effective control must exploit this redundancy while avoiding joint limits, self-collision, excessive actuator loading, and mechanically weak configurations during interaction.

The fundamental objective of a multi-finger controller is not merely to command finger joint angles but to establish and maintain useful contacts between the hand and an object. Each fingertip contact introduces geometric and mechanical relationships involving position, surface normal, friction, and contact force. Stable manipulation emerges when these contacts collectively constrain unwanted object motion while preserving the freedom required by the intended task.

Grasp representation provides an important interface between perception, planning, and control. A grasp can be described through hand configuration, wrist pose, fingertip contact locations, contact normals, and expected force distribution. For unfamiliar objects, perception may estimate object geometry and pose before a grasp planner generates candidate configurations. The hand controller then converts the selected grasp into coordinated finger trajectories and continuously adjusts them as physical contact develops.

Multi-finger coordination is challenging because individual finger motions are strongly coupled through the object. Moving one finger can change object pose, redistribute forces, or alter contact conditions for every other finger. Independent position controllers can therefore produce excessive internal forces or destabilize the grasp. Coordinated control instead considers the hand-object system as a coupled mechanism and regulates finger motions according to their combined effect on object stability and manipulation.

Contact force control becomes essential once the fingers touch the object. Forces must be sufficiently large to resist gravity, inertia, and external disturbances but should not be unnecessarily high. Excessive force wastes energy, increases actuator temperature, risks damaging fragile objects, and reduces compliant behavior. A practical controller therefore regulates normal and tangential contact forces while respecting friction constraints and adapting grip strength to task requirements.

The friction cone provides a useful physical model for evaluating whether a contact can support a desired force without slipping. Tangential force must remain compatible with the available friction relative to the normal contact force. Because real friction coefficients are uncertain and object surfaces vary, practical policies should maintain appropriate margins rather than operating directly at theoretical boundaries. Tactile observations can further reveal when these assumptions no longer match actual contact behavior.

Tactile sensing gives dexterous manipulation information that visual perception cannot provide reliably. Distributed tactile sensors may estimate contact location, pressure, shear, vibration, and local deformation across fingertips or finger surfaces. These measurements allow the controller to determine whether contact has occurred, whether pressure is uneven, and whether the object is beginning to move relative to the hand. Tactile feedback therefore closes the control loop directly at the physical interaction surface.

Slip detection is particularly important for maintaining stable grasps with minimum necessary force. Small relative motions, changes in shear force, or characteristic tactile vibrations can indicate incipient slip before the object is visibly displaced. When slip is detected, the policy can increase grip force, reposition a finger, change wrist orientation, or modify the planned object acceleration. This feedback mechanism makes grasping more robust to uncertain mass, friction, and surface properties.

Multi-finger control can be organized hierarchically to reduce complexity. A higher-level policy determines desired grasp type, contact configuration, or object motion, while an intermediate controller converts these objectives into fingertip targets. Lower-level joint controllers then track position, velocity, torque, or impedance references at high frequency. Such decomposition separates task reasoning from fast physical stabilization while preserving feedback paths that allow contact events to influence higher-level decisions.

Hand synergies offer another mechanism for reducing the dimensionality of dexterous control. Human-like grasp postures often exhibit correlated finger motions rather than arbitrary independent movement of every joint. A synergy representation captures these correlations using a smaller set of coordinated variables. Policies operating in this reduced action space can learn useful grasp patterns more efficiently, while residual joint-level control can preserve the fine adjustments needed for object-specific contact.

Impedance control is valuable when exact contact geometry cannot be predicted. Instead of forcing each fingertip to follow a rigid trajectory, the controller specifies a compliant relationship between displacement and interaction force. Fingers can then conform naturally to object surfaces while maintaining desired grasp behavior. Variable impedance can make the hand relatively soft during initial contact, firmer during lifting, and compliant again when performing delicate manipulation or interacting with humans.

A multi-finger policy can also be learned from demonstrations, reinforcement learning, or combinations of model-based and data-driven control. Demonstrations provide examples of useful hand configurations and manipulation strategies, while reinforcement learning can optimize behaviors through repeated interaction. The policy may map proprioception, tactile signals, visual features, object state, and task commands to finger positions, torques, or lower-dimensional synergy actions.

Reinforcement learning for dexterous hands is difficult because the action space is large and contact dynamics are discontinuous. Small changes in finger position can create or break contact, dramatically changing subsequent object motion. Rewards must therefore represent task progress, grasp stability, control effort, collision avoidance, and successful completion without encouraging unstable shortcuts. Parallel simulation is often useful for generating the large volume of interaction experience required for such policies.

Simulation alone cannot perfectly reproduce real finger-object interaction. Friction, compliance, actuator backlash, tendon behavior, tactile response, and object surface characteristics can differ substantially between simulated and physical systems. Domain randomization can expose a learned policy to variations in these parameters during training, while real-world fine-tuning can adapt the policy after deployment. Robust control should therefore rely on feedback rather than assuming that simulated contact parameters remain exact.

In-hand manipulation extends dexterous control beyond maintaining a static grasp. The robot may rotate, translate, pivot, roll, or reorient an object using coordinated changes in finger contacts while the wrist remains relatively stationary. Such behavior requires deliberate contact transitions in which some fingers stabilize the object while others release and establish new contacts. The controller must preserve object security throughout these transitions while driving it toward the desired pose.

Finger gaiting provides one strategy for changing grasp configuration without releasing the object completely. A subset of fingers maintains force closure while another finger moves to a new contact location. The process is repeated until the desired hand-object configuration is reached. Planning these transitions requires reasoning about temporary grasp stability, reachable contact regions, collision constraints, and the mechanical consequences of each finger movement.

The hand policy must remain integrated with arm and wrist control because dexterity is distributed across the entire upper limb. Some object rotations are more efficiently produced by wrist motion, while others require finger-level manipulation. Similarly, an unfavorable hand configuration can often be corrected by repositioning the arm instead of forcing the fingers toward their limits. Coordinated optimization therefore determines which motion should be assigned to the fingers, wrist, arm, or larger body.

Safety and hardware constraints are especially important because compact hand actuators operate close to objects and humans. Joint limits, motor current, tendon tension, temperature, fingertip force, collision, and communication faults should be monitored continuously. When tactile or actuator measurements exceed safe thresholds, the controller can reduce force, stop motion, maintain a safe grasp, or release the object according to the risk associated with the current task.

A production-quality dexterous hand controller should also expose measurable indicators of performance. Grasp success rate, slip frequency, object pose error, contact-force stability, manipulation completion time, energy consumption, recovery success, and robustness across object variations provide complementary views of capability. Evaluation should include not only successful demonstrations but also systematic disturbances, uncertain object properties, perception errors, and repeated trials.

Ultimately, a multi-finger policy converts the humanoid hand from a collection of individually actuated joints into an adaptive physical interaction system. Reliable dexterity emerges from the combination of coordinated finger motion, contact mechanics, tactile feedback, force regulation, compliant control, perception, and learned behavior. When integrated with wrist, arm, and whole-body control, this capability provides the foundation for handover, tool use, in-hand manipulation, bimanual work, and general-purpose humanoid manipulation.

07.03. Arm Impedance Control for Contact Tasks [w/Code]

Arm impedance control allows a humanoid robot to interact safely and effectively with the physical environment by regulating the dynamic relationship between motion and force. Instead of commanding the arm to follow a rigid position trajectory regardless of external contact, impedance control shapes how the end effector responds when displaced by contact forces. This makes the arm behave like a programmable mechanical system with selectable stiffness, damping, and equilibrium behavior.

The central concept is to define a desired relationship between end-effector displacement, velocity, and interaction force. When the hand encounters an object, the controller does not simply continue pushing toward the commanded pose. It generates a restoring force according to the configured impedance. High stiffness produces accurate position regulation, while lower stiffness permits compliant motion that absorbs uncertainty and reduces impact during physical contact.

A common operational-space formulation represents the desired contact behavior using a virtual mass, damping, and stiffness. The equilibrium pose specifies where the end effector would move in the absence of disturbance, while the impedance parameters determine how strongly it reacts when contact moves it away from that pose. This formulation provides an intuitive bridge between geometric motion commands and physical interaction behavior.

For contact tasks, stiffness selection is critical. Excessively high stiffness can generate large impact forces when the environment position differs from the model, while stiffness that is too low may prevent the robot from applying sufficient force or maintaining accurate alignment. Practical controllers therefore select stiffness according to task direction, expected uncertainty, contact geometry, object fragility, and the level of force required to complete the operation.

Cartesian impedance control is especially useful because contact objectives are often naturally expressed at the hand rather than individual joints. Desired translational and rotational stiffness can be specified along task-space axes, allowing the controller to remain firm in some directions and compliant in others. During surface following, for example, the arm can be compliant normal to the surface while remaining comparatively stiff along directions required to maintain orientation.

Anisotropic impedance extends this principle by assigning different mechanical behavior to different task directions. An insertion task may require high rotational stiffness to preserve alignment but relatively low translational stiffness in directions where geometric uncertainty exists. A wiping task may require controlled normal compliance while permitting smooth tangential motion. Direction-dependent impedance therefore allows the controller to encode the physical structure of the task directly.

Joint-space impedance can also be useful, particularly when actuator-level compliance must be controlled directly. Each joint can be assigned desired stiffness and damping around a reference angle, creating compliant behavior throughout the kinematic chain. However, joint-space parameters do not directly correspond to end-effector mechanical properties because the mapping depends on robot configuration. Task-space impedance is therefore often combined with lower-level joint torque control.

Torque-controlled humanoid arms provide a strong foundation for impedance regulation because the controller can directly command joint torques required to realize desired end-effector forces. Robot dynamics, including inertia, gravity, Coriolis effects, and external forces, must be considered when converting task-space commands into actuator torques. Accurate dynamic compensation helps the apparent arm behavior match the impedance specified by the higher-level controller.

Force and torque sensing provides important feedback for contact tasks. A wrist force-torque sensor can measure interaction loads transmitted through the hand, while joint torque sensors can estimate distributed external forces along the arm. These measurements allow the controller to detect initial contact, regulate sustained force, identify unexpected collisions, and modify impedance when the physical interaction differs from the planned contact condition.

Contact detection is typically the transition point between free-space motion and compliant interaction. Before contact, the arm may track a trajectory using relatively firm position regulation. Once force measurements, tactile signals, or model residuals indicate contact, the controller can reduce stiffness, activate force objectives, or switch to a task-specific impedance profile. Smooth transitions are important because abrupt gain changes can create undesirable torque or velocity discontinuities.

Hybrid position-force behavior is often required when the task imposes different constraints along different directions. A robot polishing a surface may regulate normal contact force while controlling tangential position or velocity. Similarly, a door-opening task may require force accommodation along the constrained direction while tracking a curved hand trajectory. Impedance control provides a natural foundation for this mixed interaction because compliant directions can coexist with accurately controlled motion directions.

Variable impedance control changes stiffness and damping dynamically rather than maintaining fixed parameters throughout the task. During approach, relatively low stiffness may reduce collision severity. After stable contact is established, stiffness can increase to improve accuracy or load transmission. When uncertainty or unexpected resistance is detected, the controller can soften again. This adaptive behavior allows one arm controller to support multiple phases of contact-rich manipulation.

Damping is as important as stiffness because insufficient damping can cause oscillation after contact. A highly compliant arm with poorly selected damping may bounce against a rigid environment, producing unstable or inefficient interaction. Excessive damping, however, can make movement sluggish and increase tracking error. Appropriate damping should therefore be selected together with stiffness and adjusted according to arm inertia, contact rigidity, controller frequency, and expected task dynamics.

Passivity and stability become central concerns when a controlled robot interacts with an unknown environment. Even if the arm is stable in free space, interaction with a stiff external object can create coupled dynamics that lead to oscillation or instability. Sampling delay, sensor noise, actuator bandwidth, communication latency, and model errors can further reduce stability margins. Impedance design must therefore consider the combined robot-controller-environment system.

Null-space control allows the humanoid to regulate arm posture while maintaining the desired task-space impedance. Because humanoid arms often possess redundant degrees of freedom, multiple joint configurations can produce the same hand pose. Secondary objectives can move the elbow away from obstacles, maintain comfortable joint configurations, avoid singularities, or reduce actuator loading without significantly disturbing the primary contact task.

The torso and whole body may also contribute to impedance behavior when contact forces become significant. A contact task performed only with the arm may approach joint torque or workspace limits, while coordinated torso motion can redistribute the load across a larger portion of the robot. In whole-body manipulation, hand impedance must therefore remain compatible with balance constraints, foot contacts, center-of-mass regulation, and overall momentum control.

Contact tasks frequently involve geometric uncertainty. The actual surface may be several millimeters away from its estimated position, an object may be slightly rotated, or a fixture may deform under load. Rigid trajectory tracking converts these small errors directly into potentially large forces. Impedance control instead transforms part of the error into controlled displacement, allowing the robot to accommodate uncertainty while continuing to pursue the physical objective.

Assembly and insertion tasks are representative examples. Peg-in-hole insertion, connector mating, component placement, and tool engagement require precise alignment but often contain small pose errors. Compliant motion allows the contacting geometry to guide the end effector toward the correct configuration. Properly designed impedance can therefore exploit environmental constraints rather than fighting them, reducing the force peaks associated with imperfect perception and calibration.

Tool use introduces additional challenges because the effective interaction point may lie beyond the hand. The controller must account for tool geometry, mass, inertia, and contact location when determining the mechanical behavior at the task point. A screwdriver, scraper, brush, or lever can significantly change the dynamics experienced by the arm. Impedance parameters should therefore be defined relative to the functional tool frame rather than only the wrist.

Human-robot physical interaction also benefits from compliant arm behavior. During handover, co-carrying, guidance, or collaborative assembly, the human may intentionally move the robot arm or generate forces that should not be rejected aggressively. Lower stiffness and appropriate damping allow the robot to yield naturally while still maintaining task objectives. Force thresholds and safety supervision can further distinguish intended collaboration from potentially hazardous contact.

Learning-based methods can extend impedance control by selecting or adapting impedance parameters from experience. A policy can observe object state, contact forces, task phase, tactile information, and execution history and then output desired stiffness, damping, equilibrium pose, or force targets. This separates learned adaptation from the lower-level physical controller, allowing data-driven behavior while retaining explicit control over stability and actuator constraints.

Reinforcement learning can optimize impedance schedules for tasks where suitable compliance is difficult to design manually. The policy can be rewarded for successful completion, low contact force, short execution time, energy efficiency, or robustness to uncertainty. Training in simulation can explore many contact situations, but the resulting policy should be transferred carefully because friction, stiffness, latency, and actuator behavior often differ between simulation and real hardware.

Safety supervision remains active regardless of the impedance policy. Maximum joint torque, end-effector force, collision force, velocity, power, temperature, and workspace limits should constrain commanded behavior. If contact forces exceed safe thresholds or the environment becomes unexpectedly rigid, the robot may reduce stiffness, retreat, freeze the arm, or transition to a predefined recovery mode rather than allowing the impedance controller to continue indefinitely.

Performance evaluation should measure both task completion and interaction quality. Useful indicators include contact-force peak, steady-state force error, position error, settling time, oscillation amplitude, insertion success, disturbance recovery, energy consumption, and sensitivity to model uncertainty. Repeated testing across different environmental stiffness values and contact geometries provides a more meaningful assessment than evaluating only nominal trajectories.

Ultimately, arm impedance control provides the mechanical interface that connects humanoid motion planning with real physical interaction. It enables the robot to remain accurate when accuracy is required, compliant when uncertainty or human contact is present, and adaptive when interaction conditions change. Combined with perception, force sensing, dexterous hand control, whole-body coordination, and safety supervision, it forms a core capability for reliable contact-rich humanoid manipulation.

07.04. Bimanual Coordination for Humanoid [w/Code]

Bimanual coordination enables a humanoid to use both arms and hands as a unified manipulation system rather than as two independent manipulators. Many physical tasks require one hand to stabilize an object while the other performs an operation, or require both hands to jointly support, orient, and transport the same object. Effective coordination therefore combines perception, relative hand control, force regulation, task planning, and whole-body stabilization.

Bimanual tasks can be classified according to the relationship between the two hands. In symmetric manipulation, both hands perform similar motions, such as lifting a large box or carrying a tray. In asymmetric manipulation, the hands assume complementary roles, such as holding a container while opening its lid. The controller must represent these roles explicitly because each pattern imposes different requirements on motion synchronization and force distribution.

A useful representation defines task objectives relative to the manipulated object rather than controlling each hand independently in the world frame. The object pose can serve as a common reference, while desired left-hand and right-hand contact transforms describe how each hand should remain attached. This object-centric formulation preserves coordinated geometry when the object translates or rotates and simplifies planning for rigid bimanual grasps.

Relative hand pose is especially important when both hands contact the same rigid object. If each arm tracks an independent trajectory with small geometric errors, the hands can unintentionally pull against one another and generate large internal forces. A coordinated controller therefore regulates both the global object motion and the relative transformation between the hands, allowing small compliance where necessary to prevent excessive mechanical stress.

Grasp selection should consider the pair of grasps as a coupled configuration. Two individually stable grasps may still produce poor bimanual manipulation if they restrict object rotation, create uncomfortable arm postures, or place the hands too close together. The planner should evaluate grasp separation, force transmission, arm reachability, collision clearance, manipulability, and the expected object trajectory before selecting the final contact pair.

Approach planning must coordinate both arms before contact occurs. Simultaneous approach can reduce execution time but increases the possibility of arm-arm or hand-object interference. Sequential approach may allow one hand to establish a stable reference contact before the second hand joins. The selected strategy depends on object size, uncertainty, available workspace, and whether temporary one-handed support is mechanically feasible.

Contact establishment should be verified independently at each hand while preserving the shared task objective. Tactile sensing, finger force, joint torque, and wrist force-torque measurements can confirm whether both grasps have formed correctly. If one hand establishes contact earlier, the controller should avoid allowing that arm to move the object unexpectedly while the other hand is still approaching. Temporary compliance can absorb small synchronization errors.

Once both hands are attached, load sharing becomes a central control problem. The object weight and externally applied forces should be distributed between the arms according to their posture, strength, joint limits, and contact quality. Equal load sharing is not always optimal. One arm may be better positioned to support vertical weight while the other provides orientation control or resists lateral forces required by the task.

Internal force is a distinctive property of closed-chain bimanual manipulation. Forces applied by the two hands can cancel at the object level while still producing significant stress inside the object-hand-arm loop. Some internal force is useful because it maintains secure contact, but excessive internal force wastes actuator capacity and can damage fragile objects. The controller should regulate it independently from the net wrench that moves the object.

Hybrid position-force control can separate object-motion objectives from contact-force objectives. The robot may control the object\'s position and orientation along unconstrained directions while regulating forces along constrained contact directions. Impedance control provides additional tolerance to calibration and geometric errors. This combination is valuable when manipulating rigid objects whose exact dimensions or grasp locations are not perfectly known.

Asymmetric tasks require explicit role allocation between the hands. A stabilizing hand may maintain object pose and resist reaction forces while the active hand performs rotation, insertion, cutting, fastening, or another functional action. The roles may also change during execution. A task planner should therefore represent hand roles dynamically instead of permanently assigning one hand as dominant and the other as supportive.

Opening a container illustrates asymmetric coordination. One hand establishes a stable grasp on the container body while the other approaches and grasps the lid. The supporting hand resists the torque generated by lid rotation, while the active hand follows the required rotational trajectory. Force and motion feedback can indicate whether the lid is turning, jammed, or completely released, allowing both arms to adapt their behavior.

Large-object manipulation emphasizes symmetric cooperation. When lifting a box, both hands must establish compatible contacts, increase supporting force, and raise the object without generating excessive tilt. During transport, the hands maintain relative geometry while the torso and legs compensate for payload motion. Before placement, both arms must lower the object consistently so that one side does not contact the surface significantly earlier than the other.

Bimanual object reorientation may require coordinated motion beyond simple rigid transport. The robot can rotate an object between the hands, shift grasp locations, or temporarily allow one hand to become the primary support while the other regrasps. These transitions change the contact structure and available force distribution. Planning must ensure that the remaining contacts maintain sufficient stability throughout every regrasping phase.

Hand-to-hand transfer is another important coordination skill. An object may need to move from one hand to the other because the current grasp is unsuitable for a subsequent task or because the opposite arm provides better reachability. During transfer, the receiving hand establishes contact before the original hand gradually reduces grip force. Tactile and force feedback verify that load ownership has changed safely before release.

Perception for bimanual manipulation must estimate more than object pose. The system should track both hand configurations, contact regions, relative hand-object transforms, surrounding obstacles, and task-relevant functional features. Visual occlusion becomes more severe when two arms surround the same object, making wrist cameras, tactile sensing, proprioception, and force measurements important complements to external or head-mounted vision.

Bimanual coordination is strongly coupled with torso configuration. Rotating or leaning the torso can increase the common reachable workspace of both arms and improve manipulability. Poor torso posture may force one arm toward a joint limit even when the other remains comfortable. Whole-body planning should therefore select torso and pelvis configurations that maintain useful joint margins and balanced capability across both manipulators.

The combined object and arm dynamics become increasingly important for heavy payloads. The controller must account for object mass, inertia, center of mass, and acceleration when computing arm forces and torques. Rapid object rotation can generate substantial dynamic loads that differ between the two arms. Coordinated acceleration limits and feedforward compensation help prevent one manipulator from unexpectedly receiving most of the transient load.

Bimanual manipulation during locomotion adds another level of coupling. Both hands may remain attached to an object while the humanoid walks, changes stance, or steps around obstacles. Arm motion must compensate for torso oscillation while the legs manage the altered center of mass caused by the payload. The object should remain stable even as the support polygon and ground contact configuration change during locomotion.

Collision avoidance must consider interactions between the two arms as well as the environment. Elbows, forearms, wrists, and hands can collide even when both end-effector trajectories appear individually feasible. The manipulated object can also collide with the torso or legs during reorientation. Whole-body collision checking should therefore be integrated into trajectory generation rather than applied only to each arm separately.

Learning can improve bimanual coordination when analytical task decomposition becomes difficult. Demonstrations provide examples of synchronization, role allocation, object stabilization, regrasping, and coordinated force application. Imitation learning can reproduce these structures, while reinforcement learning can improve robustness to variations in object geometry, mass, friction, and contact timing. Learned policies can also provide residual corrections around model-based controllers.

Failure detection should identify both local and coordinated errors. One hand may slip while the other remains secure, the object may rotate unexpectedly, internal force may rise, or the two arms may become kinematically incompatible. The system should monitor relative pose, contact state, force balance, and object motion. Detecting these conditions early allows the robot to reduce motion and prevent a local error from destabilizing the complete manipulation.

Recovery should exploit the redundancy provided by two hands. If one grasp weakens, the other hand may temporarily support more load while the unstable hand regrasps. If object orientation becomes uncertain, both hands can increase compliance and return the object to a stable configuration. When recovery is impossible, the robot should place the object on a nearby support surface rather than continuing an increasingly unstable closed-chain interaction.

Safety constraints are especially important because two arms can generate large opposing forces without producing obvious object motion. Joint torque, internal force, fingertip pressure, object stress, arm-arm distance, collision risk, and whole-body balance should therefore be monitored continuously. Safety supervision must be able to reduce force, increase compliance, stop coordinated motion, or release one contact when the closed-chain configuration becomes unsafe.

Performance evaluation should distinguish individual-arm accuracy from true coordination quality. Useful metrics include bimanual task success, relative hand-pose error, object-pose accuracy, internal-force magnitude, load-sharing error, synchronization delay, contact loss, regrasp success, collision rate, and completion time. Tests should vary object dimensions, mass, grasp locations, friction, initial pose, and disturbances to determine whether coordination remains robust.

Ultimately, bimanual coordination transforms two humanoid arms into a cooperative physical system capable of manipulating objects that exceed the practical abilities of either arm alone. By combining object-centric planning, synchronized contact, internal-force regulation, dynamic role allocation, compliant control, perception, and whole-body stabilization, humanoids can perform stable carrying, assembly, tool use, hand-to-hand transfer, and complex collaborative manipulation.

07.05. Object Handover and In Hand Manipulation [w/Code]

Object handover and in-hand manipulation represent two closely related capabilities that require a humanoid to manage objects through continuous changes in contact, force, pose, and ownership. Handover transfers an object between the robot and another agent, while in-hand manipulation changes the object\'s configuration within the grasp. Both require coordination among perception, dexterous hand control, arm motion, tactile feedback, and contact-aware planning.

A successful handover begins before physical contact occurs. The robot must identify the object, estimate its pose, determine an appropriate grasp, recognize the intended receiver, and select a transfer configuration that is reachable and safe for both participants. The chosen handover pose should expose an accessible portion of the object while allowing the robot to maintain sufficient grasp stability until the receiving person or robot establishes secure contact.

Human-oriented handover additionally requires estimation of readiness and intent. Visual observations of hand position, body posture, gaze, and approach motion can indicate whether a person is prepared to receive an object. The robot should avoid releasing solely because another hand enters the workspace. Instead, perception and physical feedback should jointly determine whether the receiver has intentionally established a stable grasp and is prepared to accept the object\'s load.

Trajectory generation for handover should produce predictable and comfortable motion. Rapid acceleration, unnecessary path curvature, or unexpected wrist rotation can make interaction difficult and reduce perceived safety. The arm should approach the transfer region with controlled velocity while maintaining an object orientation that supports easy acquisition. Near the receiver, lower speed and compliant behavior are generally desirable because positional uncertainty and physical interaction become more significant.

The handover pose is constrained by object geometry, robot kinematics, receiver accessibility, and task context. A cup may be presented by exposing its handle, while a tool should normally be offered so that the receiver can immediately grasp its functional handle. Sharp, hot, fragile, or heavy objects introduce additional constraints. Handover planning therefore considers not only geometric reachability but also semantic information about how the object should be safely transferred.

Force and tactile sensing become critical when the receiver contacts the object. Changes in fingertip pressure, wrist force, object load, or tangential force can reveal that another agent has begun supporting or pulling the object. The robot can use these signals to distinguish incidental contact from a meaningful transfer event. Multimodal confirmation is especially valuable because visual occlusion commonly increases when two hands simultaneously interact with the same object.

Release should be treated as a controlled transition rather than an instantaneous command. As the receiver assumes the object\'s weight, the robot can gradually reduce grip force while monitoring whether stable external support remains present. If the receiving grasp weakens or disappears, the robot may restore its own grasp instead of releasing completely. This progressive transfer strategy reduces the probability of dropping objects when communication or timing is imperfect.

Robot-to-human handover also benefits from arm impedance control. A compliant arm can accommodate small differences between predicted and actual receiver motion rather than resisting them with high stiffness. If the person gently pulls the object, the robot can permit limited displacement while evaluating transfer intent. Appropriate compliance makes the interaction feel less rigid and also reduces forces generated by uncertainty in hand position or timing.

Receiving an object reverses many of these requirements but introduces its own challenges. The humanoid must identify an available grasp region without interfering with the giver\'s fingers, move its hand into position, establish sufficient contact, and gradually accept the object\'s load. Only after grasp stability is confirmed should the robot signal or physically permit the giver to release. The transition therefore requires coordination of approach, contact establishment, load transfer, and retreat.

In-hand manipulation begins when the desired object pose cannot be achieved efficiently through wrist or arm motion alone. The hand may need to rotate, translate, roll, pivot, or shift an object relative to the palm while maintaining control. Examples include turning a tool into an operating orientation, rotating a component before insertion, adjusting a bottle for pouring, or repositioning an object so that another person can grasp it conveniently.

The hand-object system during in-hand manipulation is a contact-rich dynamic mechanism. Finger motions change both the object\'s pose and the distribution of forces among contacts. Some contacts may remain fixed, others may roll or slide, and selected fingers may temporarily detach before establishing new contacts. The controller must reason about these transitions because an object can become unstable even when each individual finger trajectory appears locally reasonable.

Rolling contact allows the object and fingertip surfaces to move relative to their coordinate frames without substantial sliding at the contact interface. This can produce precise object rotation while maintaining contact stability. Sliding contact deliberately permits tangential motion when friction conditions allow it. The manipulation policy must determine which contact mode is appropriate and regulate normal force so that unintended slip does not replace the intended rolling or sliding behavior.

Finger gaiting extends the reachable object motion beyond what can be achieved while maintaining all contacts continuously. One or more fingers stabilize the object while another finger releases, moves to a new location, and reestablishes contact. Repeating this sequence changes the overall grasp configuration without dropping the object. Effective finger gaiting requires planning temporary support conditions and ensuring that the remaining contacts can safely maintain the object during each transition.

Object pose estimation is difficult during in-hand manipulation because the hand itself frequently occludes the object from external cameras. Wrist cameras may provide partial observations, but tactile sensing and proprioception become increasingly important as visual visibility decreases. Contact locations, finger joint states, measured forces, and object motion models can be fused with vision to maintain an estimate of the object\'s six-degree-of-freedom pose throughout manipulation.

Tactile feedback provides information about local interaction that object pose alone cannot capture. Pressure distributions can indicate whether a fingertip is centered on a surface, shear measurements can reveal tangential loading, and high-frequency signals can indicate incipient slip. A controller can respond by increasing normal force, modifying the planned motion, repositioning a finger, or changing the wrist pose before the grasp becomes unstable.

In-hand manipulation can be formulated as an object-centric control problem. Instead of independently specifying every finger trajectory, the higher-level controller defines a desired object motion and determines how available contacts should generate that motion. Contact forces and finger movements are then coordinated to produce the desired object wrench while maintaining friction, joint, collision, and actuator constraints. This representation directly connects manipulation objectives with physical object behavior.

Hand synergies and learned latent action spaces can reduce the dimensionality of multi-finger manipulation. Rather than controlling every finger joint independently, a policy can generate coordinated patterns representing common grasp adjustments or object rotations. Residual joint-level actions can then refine these patterns using tactile feedback. This combination preserves efficient learning while retaining the fine control required for contact-sensitive manipulation.

Learning-based policies are particularly useful when analytical contact planning becomes too complex. Demonstrations can provide examples of handover timing, grasp transitions, and object reorientation, while reinforcement learning can improve robustness through repeated interaction. Policy inputs may include vision, tactile signals, joint states, object pose, force measurements, and task commands, with outputs defining finger actions, wrist motion, impedance, or desired object displacement.

Simulation can generate large amounts of training experience, but handover and in-hand manipulation are sensitive to simulation-to-reality differences. Friction, fingertip compliance, actuator backlash, tactile response, object mass, and contact geometry may differ from physical hardware. Domain randomization and parameter variation can improve robustness, but real feedback remains necessary because small contact-model errors can accumulate rapidly during dexterous manipulation.

Recovery behavior is essential because contact transitions frequently fail. A receiver may withdraw during handover, an object may begin slipping, a finger may miss its intended contact, or object pose estimation may become uncertain. The robot should detect these conditions and choose an appropriate response, such as strengthening the current grasp, returning to a stable configuration, repositioning the arm, requesting another transfer attempt, or safely placing the object down.

Safety constraints apply throughout both handover and in-hand manipulation. Grip force, fingertip pressure, arm velocity, joint torque, hand closing speed, collision distance, and workspace boundaries should remain monitored. Human hand proximity requires particular care because fingers may enter regions where the robot originally planned to close its grasp. Perception and tactile safety mechanisms should therefore override nominal manipulation actions whenever hazardous contact becomes possible.

Evaluation should measure more than whether the object eventually reaches its destination. Handover performance can include transfer success, release timing, peak interaction force, completion time, drop rate, and recovery success. In-hand manipulation can additionally measure final object pose error, unintended slip, contact stability, number of regrasp events, energy consumption, and robustness across variations in object geometry, mass, friction, and initial pose.

Together, object handover and in-hand manipulation transform grasping from a static holding capability into a dynamic process of managing contact. Handover coordinates contact between agents, while in-hand manipulation coordinates changing contacts within the robot hand itself. Integrated with perception, tactile sensing, compliant arm control, multi-finger policies, and whole-body coordination, these capabilities allow humanoids to transfer, reorient, prepare, and use objects naturally across complex manipulation tasks.

07.06. Table Top Manipulation Pick Place Pour [w/Code]

Table-top manipulation is a fundamental humanoid capability that combines perception, grasp planning, arm control, dexterous hand behavior, and contact reasoning within a structured but physically demanding workspace. Tasks such as picking, placing, rearranging, and pouring require the robot to understand object geometry and state while coordinating the hand, wrist, arm, torso, and balance system throughout the manipulation sequence.

A typical table-top task begins with scene perception. Head and wrist cameras observe the work surface, detect relevant objects, estimate their poses, identify free space, and distinguish supporting surfaces from obstacles. Depth sensing or multi-view geometry can provide three-dimensional structure, while semantic perception identifies object categories and functional regions such as handles, openings, lids, containers, and target placement areas.

The manipulation system must represent the table as more than a flat geometric plane. Objects may overlap, partially occlude one another, rest near edges, or occupy regions that restrict approach directions. The robot therefore constructs a task-relevant scene model containing object poses, approximate geometry, support relationships, collision regions, grasp candidates, and available placement locations. This representation provides the common state used by planning and control.

Pick operations require selection of both a grasp and an approach trajectory. The chosen grasp should provide sufficient stability while remaining reachable from the current arm and body configuration. The approach direction should avoid collisions with neighboring objects and the table surface. For dexterous hands, the planner may additionally determine finger preshaping so that the hand reaches the object with an appropriate configuration before contact occurs.

The approach phase usually transitions from relatively fast free-space motion to slower contact-sensitive motion near the object. Wrist-mounted perception can refine object pose as the hand approaches, reducing errors caused by calibration uncertainty or earlier observations. Visual servoing can continuously update the hand trajectory according to the measured object location, allowing the system to compensate for moderate discrepancies between the original scene estimate and the actual configuration.

Grasp establishment requires confirmation that physical contact matches the planned grasp. Finger position alone cannot guarantee that the object has been acquired successfully. Tactile sensors, motor current, joint torque, or fingertip force measurements can determine whether contact exists and whether the load is distributed appropriately. If contact is asymmetric or unstable, the hand can adjust finger positions or grip force before attempting to lift.

Lifting converts the grasp from a supported contact state into a load-bearing state. The controller should increase vertical motion gradually while monitoring slip, contact force, and object pose. Unexpected changes can indicate that the object remains constrained by surrounding geometry or that the grasp is insufficient. A safe policy can stop the lift, lower the object, modify the grasp, and retry rather than continuing with an unstable load.

Place operations require more than reversing the pick trajectory. The target location must provide sufficient free space and a stable support configuration for the object. Desired placement orientation may also depend on task semantics. A bottle should generally remain upright, a tool may require a specific orientation for subsequent use, and an assembly component may need precise alignment with a fixture or another object.

During placement, the arm moves the object toward the target while maintaining collision clearance for both the object and the robot. Near the supporting surface, motion should become slower and more compliant. Force or tactile feedback can detect initial contact with the table, after which the controller gradually transfers the object\'s weight from the hand to the surface. Only after stable support is confirmed should the fingers release the object.

Placement uncertainty can be managed through compliant control rather than requiring perfect geometric accuracy. If the estimated table height differs slightly from reality, a rigid downward trajectory can generate excessive force. Impedance control allows the hand to yield after contact while maintaining a desired downward interaction. This behavior is particularly useful when placing fragile objects or interacting with surfaces whose geometry is not known precisely.

Rearrangement tasks combine multiple pick-and-place operations with additional reasoning about intermediate states. An object blocking access to another may need to be moved temporarily, or a crowded table may require free regions to be created before the final arrangement is possible. Planning therefore extends beyond individual trajectories to determine an ordering of actions that maintains reachability, collision-free space, and stable object configurations throughout the task.

Pouring introduces continuous object motion and dynamic reasoning beyond conventional pick-and-place. The robot must grasp a source container, identify the receiving container, move the source above an appropriate location, and rotate it around a suitable pouring axis. The required wrist and arm motion depends on container geometry, fill level, desired flow, and the location of the receiving opening.

The pouring trajectory should maintain spatial alignment between the source outlet and receiving container while gradually changing orientation. Rotating too quickly can generate excessive flow or dynamic disturbances, whereas insufficient tilt may produce no flow. The controller therefore coordinates translation and rotation so the source container remains positioned over the target while its orientation changes smoothly through the pouring sequence.

Liquid manipulation introduces state variables that may not be directly observable from standard robot proprioception. The remaining amount of liquid, flow rate, and distribution inside the container affect both task progress and dynamics. Vision can observe the receiving container or liquid stream when visible, while weight, force, or learned perception can provide indirect information. The manipulation system must therefore estimate task state continuously rather than relying only on predetermined motion.

The changing liquid distribution also alters the effective mass and center of mass of the held container. These changes influence wrist torque and arm loading, particularly when the container is large or held far from the body. Force-torque sensing can reveal these variations, allowing the controller to adapt grip force and arm behavior. Whole-body posture may also be adjusted to preserve comfortable joint configurations and balance.

Pouring should include explicit termination logic. The robot may stop after a requested quantity has been transferred, when a visual fill level reaches a threshold, when the source becomes empty, or when abnormal flow is detected. The container is then rotated back toward an upright orientation before being transported away. Returning too abruptly can cause residual liquid to spill, so the recovery motion should remain smooth and controlled.

Table-top manipulation benefits strongly from task-space and impedance control because many actions transition repeatedly between free motion and contact. Picking requires contact establishment, placing requires controlled support transfer, and pouring requires stable grasping under changing loads. A unified controller can modify stiffness, damping, force targets, and trajectory speed according to the current manipulation phase instead of treating every phase as rigid position tracking.

Collision avoidance must consider the complete robot-object system. A hand trajectory may appear collision-free while the elbow intersects the table, the forearm contacts another object, or the carried object strikes an obstacle. Planning should therefore evaluate hand, arm, torso, and manipulated-object geometry simultaneously. Redundant humanoid arm configurations can be exploited to reposition the elbow or torso while preserving the desired hand pose.

Bimanual manipulation can extend table-top capability when objects are large, unstable, or require simultaneous operations. One hand may stabilize a container while the other removes a lid, hold a bowl while another hand pours, or reposition an object while the second hand performs a tool operation. Such tasks require coordination of relative hand poses, contact forces, and object constraints while maintaining a feasible whole-body posture.

Failure detection is necessary because apparently simple table-top tasks contain many uncertain events. A grasp may miss, an object may slip during lifting, the placement region may become occupied, or pouring may deviate from the intended target. The system should monitor expected state transitions and compare them with perception and contact feedback. A mismatch should trigger replanning or recovery rather than blindly continuing the original sequence.

Recovery strategies should return the task to a known stable state whenever possible. A failed grasp can lead to hand withdrawal and re-perception, unstable lifting can return the object to the table, and uncertain placement can maintain the grasp until support is verified. During pouring, abnormal flow or unexpected motion can trigger rotation toward upright and stabilization. These behaviors prevent local errors from developing into larger task failures.

Learning-based policies can complement geometric planning for table-top manipulation. Imitation learning can capture demonstrations of reaching, grasping, placing, and pouring, while reinforcement learning can improve robustness to variations in pose, friction, object geometry, and contact. Learned policies can operate at the level of grasp selection, residual motion correction, impedance adaptation, or complete manipulation skills while conventional safety constraints remain active.

Performance evaluation should separate perception, planning, control, and task-level outcomes. Useful measures include pick success rate, grasp stability, placement position and orientation error, collision rate, execution time, recovery success, spill rate, pouring accuracy, and robustness to scene variation. Repeated evaluation across clutter, different object shapes, uncertain poses, changing surface heights, and varying container states provides a realistic measure of manipulation capability.

Table-top pick, place, and pour tasks provide a compact but comprehensive benchmark for humanoid manipulation. They combine visual understanding, geometric planning, dexterous grasping, compliant contact, dynamic load handling, failure recovery, and task sequencing within everyday physical environments. Mastering these operations establishes a practical foundation for more complex humanoid activities including assembly, food handling, logistics, laboratory work, and general service tasks.

07.07. Tool Use and Instrument Manipulation [w/Code]

Tool use extends humanoid manipulation beyond direct hand-object interaction by allowing the robot to modify the environment through an intermediate physical instrument. A tool effectively changes the geometry, reach, force transmission, and functional capability of the hand. Reliable tool use therefore requires coordinated perception, grasping, tool-frame estimation, contact control, motion planning, and whole-body stabilization rather than treating the tool as an ordinary passive object.

The manipulation structure places tool and instrument manipulation after dexterous hand control, impedance-based contact tasks, object handover, and table-top manipulation, and before unified whole-body manipulation. This progression reflects the fact that tool use depends on stable grasping and contact interaction while often requiring larger coordinated motions of the arm and body. Volume_22_Humanoid_Robot_Softwa...

A tool should be represented according to its functional structure rather than only its external geometry. The system may distinguish a grasp region, functional working region, interaction axis, effective tip, and reference frame associated with the intended task. For a screwdriver, for example, the handle supports grasping while the shaft and tip determine the functional interaction. This decomposition allows planning to reason directly about how the tool changes the robot\'s action space.

Tool recognition begins with visual and geometric perception, but category recognition alone is insufficient. The robot must estimate the tool\'s six-degree-of-freedom pose and identify functional regions that determine how it should be grasped and used. Orientation can be especially important because two visually similar grasps may place the working end in very different configurations relative to the wrist, environment, and target.

Affordance reasoning connects perceived geometry with possible actions. A hammer supports striking, a screwdriver supports rotational engagement, a brush supports distributed surface contact, and a lever supports force amplification around a constraint. The manipulation system must therefore infer not only what an object is but what physical transformations it can produce. Tool selection can then be based on the compatibility between the required task effect and available affordances.

Grasp planning for tools differs from generic object grasping because grasp stability is only one objective. The selected grasp must also provide useful orientation, sufficient torque transmission, comfortable joint configuration, and an adequate manipulation range. A mechanically stable grasp near the wrong portion of a tool may make the intended operation impossible. Functional grasp quality must therefore be evaluated relative to the subsequent task trajectory.

Once grasped, the tool becomes an extension of the robot\'s kinematic structure. The controller must establish a transformation between the hand or wrist frame and a tool coordinate frame, particularly the tool center point or functional interaction point. Motion commands can then be expressed relative to the actual working end instead of the palm. Calibration error in this transformation directly becomes positioning error at the environment contact.

Long tools amplify both useful reach and geometric error. A small angular error at the wrist can produce a significant displacement at a distant tool tip, making orientation accuracy particularly important. The robot may refine tool pose after grasping using wrist cameras, external vision, known geometry, or contact observations. Online correction can compensate for small variations in grasp position that cannot be eliminated by initial planning alone.

Instrument manipulation often contains an engagement phase before the primary action begins. A screwdriver must enter a screw head, a key must align with a slot, and a probe may need to touch a precise surface location. These operations combine fine positioning with contact detection. Rigid trajectory tracking can create excessive force when alignment is imperfect, so compliant motion and impedance control are valuable for allowing environmental geometry to guide final alignment.

Contact forces should be regulated according to tool function. A writing instrument requires relatively small normal force, while scraping or tightening may require substantially greater force or torque. The controller should distinguish forces needed to maintain tool engagement from forces that actually perform the task. Wrist force-torque sensing, joint torque estimation, and tactile feedback can be combined to estimate the interaction transmitted through the tool.

Many tools impose constrained motion after engagement. A screwdriver rotates primarily around its shaft, a door handle follows a mechanical rotational constraint, and a cutting or wiping instrument may move along a surface. Instead of commanding unconstrained Cartesian trajectories, the planner can represent these operations using task coordinates that correspond to allowed and constrained directions. This reduces unnecessary forces and improves physical consistency.

Hybrid position-force and impedance control are particularly suitable for instrument manipulation. Position can regulate motion along directions where accurate trajectory tracking is required, while force control maintains contact along constrained directions. Impedance can absorb residual geometric uncertainty between these objectives. The resulting controller allows the tool to follow physical structures rather than attempting to overpower small discrepancies between the world model and reality.

Tool dynamics must also be incorporated into manipulation. The mass and inertia of the instrument alter the effective dynamics of the arm, while an offset center of mass can generate substantial wrist torque. Heavy or long tools can change the robot\'s overall momentum and balance requirements. Dynamic compensation should therefore account for the grasped payload, and the whole-body controller may reposition the torso or stance to maintain feasible joint loading.

Bimanual tool use is required when one hand cannot simultaneously stabilize the environment and operate the instrument. One hand may hold a component while the other tightens a fastener, stabilize a container while the other manipulates its lid, or guide a long tool using two separated contact points. Such tasks require relative hand coordination and internal-force management while preserving the functional trajectory of the working tool.

Tool exchange introduces another manipulation problem. A robot may need to retrieve an instrument from storage, receive it from a human, transfer it between its own hands, or return it after use. Each transition changes the active grasp and tool reference frame. The system must preserve knowledge of tool identity, orientation, and functional end throughout these transitions so subsequent planning remains geometrically consistent.

Some instrument tasks require discrete operating states in addition to mechanical motion. A trigger, button, switch, latch, or adjustable mechanism may alter how the tool behaves. The manipulation planner should represent these states explicitly because identical arm trajectories can have different effects depending on instrument configuration. Perception or force feedback can verify whether the expected state transition actually occurred before execution continues.

Task execution should therefore be organized around observable physical effects rather than only commanded trajectories. When tightening a fastener, completion may be indicated by increasing resistance or reaching a torque threshold. During wiping, coverage of the desired region may define completion. During probing, contact at specified locations may be the relevant result. Effect-based monitoring enables the robot to determine whether tool motion actually accomplished its intended purpose.

Tool use can involve significant uncertainty because environmental geometry, friction, mechanical resistance, and tool condition may not be known accurately. A screw may resist unexpectedly, a tool may slip from its target, or a surface may deform under force. The controller should continuously compare expected and measured motion, force, and contact state. Large residuals can indicate that the assumed interaction model is no longer valid.

Failure recovery should preserve both the tool and surrounding environment. If engagement is lost, the robot can reduce applied force, withdraw slightly, re-perceive the target, and attempt alignment again. If torque rises unexpectedly, continuing the motion may damage the tool or target, so execution should stop or reverse. Recovery policies should prefer returning to a known stable contact state rather than continuing uncertain interaction.

Learning-based manipulation can complement analytical models when tool contact is complex or difficult to describe precisely. Demonstrations can provide examples of approach direction, engagement, force application, and completion behavior. Reinforcement learning can improve robustness to pose and contact variation, while residual policies can correct model-based trajectories using visual, tactile, and force observations without replacing the underlying safety controller.

Generalization requires the robot to separate task principles from individual tool geometry. A policy that only memorizes one screwdriver or one brush has limited practical value. Functional representations based on grasp regions, action axes, contact surfaces, and tool center points can support transfer across instruments with different dimensions or appearances. Semantic perception can further associate novel tools with previously learned physical affordances.

Whole-body coordination becomes increasingly important as tool forces increase. Pushing, drilling, levering, or operating long instruments can generate reaction forces that propagate through the arm into the torso and legs. The robot may need to adjust its center of mass, widen its stance, rotate the torso, or reposition its feet. Tool manipulation is therefore naturally connected to the subsequent unified loco-manipulation capability in the chapter structure. Volume_22_Humanoid_Robot_Softwa...

Safety supervision must constrain both robot motion and the physical effect of the instrument. Limits on tool-tip velocity, contact force, applied torque, joint loading, collision distance, and workspace should remain active throughout execution. The controller should also distinguish intentional tool contact from unintended contact involving the hand, arm, environment, or nearby people and immediately reduce energy when unsafe interaction is detected.

Performance evaluation should measure functional task success as well as motion accuracy. Useful indicators include tool-grasp success, tool-tip pose error, engagement success, contact-force stability, applied torque accuracy, task completion time, unintended slip, recovery success, and robustness across tool and target variations. Evaluation should also test errors in initial pose, friction, target alignment, payload, and environmental stiffness.

Ultimately, tool and instrument manipulation allows a humanoid to convert dexterous grasping into functional interaction with a much broader physical world. The tool becomes a temporary extension of the hand whose geometry, dynamics, affordances, and contact state must be integrated into the robot\'s control model. Combined with perception, compliant control, bimanual coordination, learning, and whole-body stabilization, this capability enables increasingly general physical task execution.

07.08. Whole Body Manipulation Loco Manip Unified [w/Code]

Whole-body manipulation extends humanoid manipulation from isolated arm and hand motion to coordinated use of the entire body. The hands create task contacts, the arms regulate interaction, the torso expands reach and redistributes momentum, and the legs maintain support or reposition the robot. Unified loco-manipulation treats these functions as one coupled physical behavior rather than separating locomotion and manipulation into independent stages.

Traditional manipulation often assumes that the robot base remains fixed while the arm performs the task. This assumption becomes restrictive for humanoids operating in large or changing workspaces. An object may lie outside comfortable reach, a carried load may require body repositioning, or interaction forces may exceed what the upper body can safely support. Whole-body manipulation resolves these situations by allowing the support configuration itself to participate in the task.

The central architectural principle is that manipulation objectives and locomotion objectives share the same body and therefore the same dynamic constraints. Moving an arm changes the center of mass and angular momentum, while stepping changes the reference frame and reachable workspace of the hands. A unified controller must therefore coordinate hand pose, contact force, torso posture, center-of-mass motion, foot placement, and momentum instead of optimizing each subsystem independently.

Task representation usually begins at the interaction points. Desired hand poses, object trajectories, contact forces, tool motions, or bimanual constraints define the primary manipulation objectives. Additional objectives regulate torso orientation, pelvis height, gaze, joint posture, and foot contacts. The controller combines these goals according to task priority while enforcing physical constraints such as joint limits, friction, collision avoidance, and actuator capability.

Whole-body kinematics provides the geometric relationship between these task objectives and the robot\'s many degrees of freedom. Because humanoids are highly redundant, the same hand pose can often be achieved using different combinations of shoulder, elbow, torso, pelvis, and leg configurations. This redundancy can be exploited to increase manipulability, avoid singularities, maintain joint margins, reduce energy consumption, or prepare the body for subsequent motion.

Dynamic consistency becomes essential when manipulation produces significant forces. Pushing a heavy object, pulling a door, carrying a large payload, or operating a long tool generates reaction forces that propagate through the arms into the torso and legs. The controller must distribute these loads across available contacts while maintaining feasible ground reaction forces. Manipulation is therefore constrained by the entire robot-environment contact system rather than only by arm torque limits.

Foot contacts form the primary support structure during standing manipulation. The available center-of-pressure region, friction limits, and foot geometry determine how much external force the robot can resist without stepping. As manipulation forces increase, the controller can shift the center of mass, modify torso orientation, bend the knees, or redistribute load between the feet. These adjustments increase the range of interactions that can be performed while maintaining stable support.

Static balance criteria alone are insufficient for dynamic manipulation. Rapid arm motion, object acceleration, and external impacts generate momentum that must be managed throughout the body. Centroidal momentum regulation provides a useful representation for coordinating these effects. The controller can use torso, arm, and leg motion to regulate linear and angular momentum while preserving the hand trajectory and required interaction forces.

Stepping becomes part of manipulation when the desired task cannot be completed within the current support configuration. Rather than treating a step as an interruption, unified loco-manipulation considers foot placement as another decision variable. The robot may step closer to an object, move laterally while carrying it, rotate its stance during a tool operation, or reposition its feet to generate a more favorable direction for applying force.

Reachability should therefore be evaluated over both arm workspace and potential base motion. A target that is unreachable from the current stance may become easily manipulable after a small step or torso displacement. Planning can compare alternatives such as extending the arm, leaning, rotating the torso, or moving the feet. The selected strategy should balance task efficiency against stability, joint loading, collision risk, and future workspace requirements.

Locomotion and manipulation may occur sequentially or simultaneously. In a sequential strategy, the humanoid walks to a favorable stance, stabilizes, and then manipulates. In a simultaneous strategy, the robot continues moving while carrying, pushing, pulling, or repositioning an object. Simultaneous loco-manipulation is more demanding because hand-object constraints must remain satisfied while foot contacts and support conditions change continuously.

Object carrying is a representative unified task. A humanoid may lift an object from a table, bring it close to the body, begin walking, navigate around obstacles, and place it at another location. The payload changes the combined center of mass and inertia of the system, affecting balance and gait. The controller must account for these changes while preserving grasp stability and preventing excessive arm loading throughout locomotion.

Large or heavy objects often require bimanual whole-body coordination. Both hands may establish contact with the object while the torso and legs provide additional motion and force capacity. Relative hand pose must remain compatible with object rigidity, while internal forces between the hands should be regulated. The robot may also change stance or walk while maintaining both contacts, creating a closed-chain interaction involving the complete body.

Pushing and pulling illustrate the direct coupling between manipulation force and locomotion. When pushing a cart or large object, the robot may lean forward and generate ground reaction forces that support the desired hand force. During pulling, the stance and center of mass may shift in the opposite direction. If the required interaction exceeds the current support margin, stepping can restore a mechanically favorable configuration instead of simply increasing arm torque.

Whole-body impedance control can provide compliant behavior across multiple body segments. The hands may be assigned task-specific impedance while the torso and legs regulate posture and balance. External forces at the hand can then produce coordinated motion throughout the body rather than being absorbed only by the arm. This distributes interaction loads and enables more natural adaptation when manipulating uncertain environments or collaborating with humans.

Task prioritization is important because multiple objectives may compete for the same degrees of freedom. Maintaining balance and safe contact usually has higher priority than exact hand tracking, while posture optimization or visual alignment may have lower priority. Hierarchical optimization can enforce critical constraints first and use remaining redundancy for secondary objectives. This prevents lower-level preferences from compromising physical stability or safety.

Optimization-based whole-body control commonly solves constrained problems at high frequency. Decision variables may include joint acceleration, torque, contact force, or generalized velocity. Constraints represent rigid-body dynamics, foot contacts, friction cones, torque limits, joint limits, and collision conditions. Task objectives then specify desired hand motion, center-of-mass behavior, momentum regulation, posture, and interaction force within this feasible region.

Planning operates over a longer horizon than the low-level whole-body controller. A planner may determine object motion, hand contact sequence, stance transitions, footholds, and intermediate body configurations. The controller then realizes these decisions while reacting to disturbances and modeling errors. Separating predictive planning from fast feedback allows complex manipulation sequences to remain adaptable during real physical execution.

Contact transitions are particularly important in unified loco-manipulation because both hands and feet can establish or break contact. A foot may lift while both hands remain attached to an object, or one hand may release while the legs maintain support. Each transition changes the robot\'s constraint structure and available force distribution. Planning must ensure that the remaining contacts provide sufficient stability before another contact is removed.

Perception must track both manipulation targets and locomotion-relevant geometry. The robot needs object pose, grasp state, tool configuration, obstacles, support surfaces, foothold regions, and human motion when applicable. Because the body itself can occlude sensors during manipulation, information from head cameras, wrist cameras, proprioception, tactile sensing, and force sensing should be combined into a continuously updated world and body-state estimate.

Failure recovery should use the entire body rather than relying only on local arm corrections. If an object begins slipping, the robot can reduce walking speed, bring the object closer to its torso, strengthen the grasp, or stop and establish a wider stance. If interaction forces become excessive, it may step in the force direction, modify posture, or release the task safely. Whole-body recovery therefore expands the available response space.

Learning-based policies can complement model-based whole-body control by selecting high-level behaviors, contact sequences, footholds, or residual actions. Reinforcement learning can discover coordinated patterns that are difficult to specify manually, especially for dynamic carrying or contact-rich locomotion. A practical architecture can combine learned decision making with explicit dynamics, balance constraints, torque limits, and safety supervision at lower control levels.

Training unified policies requires exposure to variations in payload, friction, object geometry, terrain, disturbances, and contact timing. Simulation provides scalable experience, while domain randomization can improve robustness to uncertain dynamics. However, real-world feedback remains essential because errors in contact, actuator behavior, and payload estimation directly influence balance. Sim-to-real transfer should therefore preserve conservative physical constraints during deployment.

Safety supervision must operate across manipulation and locomotion simultaneously. Joint torque, hand force, foot friction, center-of-mass state, collision distance, gait stability, payload motion, and actuator temperature should remain within defined limits. When safety margins decrease, the system can slow manipulation, reduce interaction force, stop walking, widen the stance, place the object down, or transition to another stable configuration.

Evaluation should measure integrated performance rather than treating walking and manipulation as unrelated benchmarks. Useful measures include task completion, hand tracking error, interaction-force accuracy, grasp stability, foot-placement error, balance margin, energy consumption, recovery success, payload variation tolerance, and execution time. Testing should include standing manipulation, stepping during contact, object carrying, pushing, pulling, and bimanual transport.

Ultimately, unified loco-manipulation transforms the humanoid from a mobile platform carrying manipulators into a single dynamically coordinated physical system. Hands, arms, torso, pelvis, and legs jointly contribute to task execution, while contact forces and motion are distributed across the body according to changing physical requirements. This integration enables humanoids to reach farther, manipulate heavier objects, move while interacting, and operate continuously across large real-world workspaces.

07.09. IL RL Hybrid Policy for Dexterous Tasks [w/Code]

Dexterous humanoid manipulation requires policies that can reproduce purposeful human-like behavior while remaining robust to contact uncertainty, object variation, and unexpected disturbances. Imitation learning provides structured behavior from demonstrations, whereas reinforcement learning improves performance through interaction and reward-driven exploration. An IL-RL hybrid policy combines these strengths to create manipulation skills that are both data-efficient and adaptable.

Pure imitation learning can acquire useful manipulation behavior without manually specifying every contact transition. Human demonstrations provide examples of reaching, grasp formation, finger coordination, object reorientation, tool interaction, and recovery. The policy learns a mapping from observations to actions that reflects successful task structure. This is especially valuable for dexterous tasks whose multi-finger trajectories are difficult to derive analytically.

Demonstration data can be collected through teleoperation, motion capture, kinesthetic teaching, virtual reality interfaces, or autonomous expert controllers. For humanoid hands, demonstrations should ideally include finger joint states, wrist and arm motion, object pose, tactile observations, force measurements, and task context. Synchronizing these modalities allows the learned policy to associate visible task progress with the contact events that actually produce manipulation.

Behavior cloning is a direct starting point for imitation learning. A neural policy is trained to predict demonstrated actions from corresponding observations, effectively performing supervised learning over expert trajectories. This approach can efficiently reproduce common behaviors when deployment states remain close to the demonstration distribution. However, small execution errors can move the robot into states that were rarely or never observed during training.

This distribution shift becomes particularly problematic in dexterous manipulation because contact errors accumulate quickly. A slight fingertip displacement can alter friction, object pose, and subsequent contact geometry, causing later actions to become inappropriate. The policy must therefore recover from off-demonstration states rather than merely replay nominal trajectories. Additional demonstrations, corrective data collection, and interactive imitation can broaden the state distribution represented during training.

Imitation learning can also operate on higher-level representations instead of raw joint commands. Demonstrations may define desired object motion, end-effector pose, grasp transitions, contact modes, or latent manipulation skills. Lower-level controllers then translate these commands into joint actions and forces. This hierarchical representation can improve transfer across robot configurations and allows model-based control to preserve stability while the learned policy captures task strategy.

Reinforcement learning complements imitation by optimizing behavior through interaction. A dexterous policy receives observations, selects actions, and receives rewards describing task progress, stability, efficiency, or safety. Repeated experience allows the policy to discover corrections and contact strategies that may not appear in demonstrations. RL is therefore particularly useful for improving robustness around uncertain transitions and recovering from execution errors.

Reward design strongly influences learned manipulation behavior. A reward can include object pose accuracy, grasp stability, contact maintenance, task completion, motion efficiency, and penalties for excessive force, collision, dropping, or unsafe joint configurations. Dense shaping rewards can accelerate early learning, while terminal success rewards preserve the actual task objective. Poorly designed rewards may produce behaviors that maximize numerical return without achieving physically meaningful manipulation.

The hybrid approach commonly begins by pretraining the policy using imitation learning and then refining it with reinforcement learning. IL provides a useful initial policy that already reaches relevant task states, reducing the exploration burden of RL. Reinforcement learning subsequently explores variations around demonstrated behavior and optimizes performance under disturbances. This sequence is particularly effective when random exploration would rarely discover successful dexterous contact sequences.

Demonstrations can remain useful during RL rather than being discarded after initialization. Training batches may combine expert transitions with newly collected policy experience, while an imitation loss encourages the policy to remain near useful demonstrated behavior. The relative weight between imitation and reinforcement objectives can change during training. Early stages can emphasize demonstrations, while later stages permit greater exploration and task-specific optimization.

Another strategy uses reinforcement learning as a residual correction around an imitation or model-based policy. The nominal controller generates a safe and meaningful action, while the learned residual modifies finger position, wrist motion, force, or impedance according to observed errors. Residual learning reduces the action space that RL must explore and can preserve the interpretable structure of an existing controller while improving robustness to unmodeled contact effects.

Dexterous tasks require rich policy observations. Vision describes object geometry and scene context, proprioception provides hand and arm configuration, tactile sensing identifies contact distribution and slip, and force-torque sensing reveals interaction loads. Object pose estimates and task commands provide additional structure. A multimodal policy must fuse these signals despite differences in sampling rate, noise, dimensionality, and temporary sensor occlusion.

Temporal information is equally important because instantaneous observations may not reveal the current contact state. A short history of tactile changes can indicate slip, while force evolution can reveal whether insertion or tightening is progressing. Recurrent networks, temporal encoders, or state estimators can maintain information across time. This internal context helps distinguish visually similar states that require different manipulation actions.

The action representation determines how directly the policy interacts with robot dynamics. Policies may output joint positions, joint velocities, torques, fingertip targets, end-effector poses, desired object motion, or impedance parameters. High-level actions simplify learning and allow conventional controllers to enforce dynamics, while low-level actions provide greater flexibility. Hybrid systems often combine learned task-space commands with reliable lower-level force and joint control.

Multi-finger manipulation introduces a very large action space, making direct independent control of every joint difficult to learn. Hand synergies, latent actions, or coordinated finger primitives can reduce dimensionality by representing common patterns of finger motion. The policy can operate in this compact space and generate residual corrections when necessary. Such representations preserve useful coordination while still allowing adaptation to individual object geometry and contact conditions.

Curriculum learning can progressively increase dexterous task difficulty. Training may begin with fixed object poses and reliable grasps, then introduce variations in position, orientation, friction, mass, geometry, and disturbances. More difficult contact transitions or longer task sequences can be added after simpler behaviors become stable. This progression reduces early exploration complexity while gradually building the robustness required for real-world operation.

Simulation is particularly important because reinforcement learning may require far more interactions than can safely be collected on physical humanoids. Parallel simulation can generate experience across thousands of object and environment variations. Domain randomization modifies friction, mass, compliance, actuator response, sensor noise, and contact parameters so the policy cannot depend on a single simulated model and is better prepared for physical deployment.

Sim-to-real transfer remains challenging for dexterous manipulation because contact dynamics are sensitive to small modeling errors. Fingertip compliance, friction transitions, actuator backlash, tactile response, and object surface properties may differ substantially from simulation. Policies should therefore avoid exploiting unrealistic simulator behavior. Real-world fine-tuning, conservative action limits, system identification, and residual adaptation can reduce the remaining transfer gap.

Offline reinforcement learning provides another path when large demonstration or robot-operation datasets already exist. Instead of requiring continuous online exploration, the policy learns from previously collected transitions containing successful and unsuccessful behavior. This can improve safety and data reuse, but the algorithm must avoid selecting actions that lie far outside the available data distribution. Demonstration quality and dataset coverage therefore remain important.

Failure data can be as valuable as successful demonstrations. Dropped objects, unstable grasps, missed contacts, excessive force, and unsuccessful insertions reveal regions of the state space that should trigger corrective behavior. Hybrid training can use these examples to learn failure predictors, value estimates, or recovery policies. The objective is not simply to reproduce success but to recognize when execution is diverging and return to a recoverable state.

Recovery can be represented as part of the same policy or as a separate skill layer. When tactile feedback indicates slip, the robot may increase grip force, reposition a finger, support the object with the palm, or move the arm to reduce load. If uncertainty becomes too high, the system can terminate the current attempt and return the object to a stable surface. Explicit recovery behavior greatly improves practical task reliability.

Safety constraints should remain active independently of the learned policy. Joint limits, torque limits, fingertip force, collision distance, hand closing speed, arm velocity, and whole-body stability can be enforced by lower-level controllers or safety filters. Reinforcement learning should optimize behavior inside this feasible region rather than discovering safety boundaries through physical failure. This separation is especially important when policies are transferred from simulation to real humanoids.

Evaluation must measure generalization rather than success on memorized training conditions. Tests should vary object pose, geometry, mass, friction, initial grasp, sensor noise, and external disturbances. Useful metrics include task success, completion time, object pose error, slip frequency, peak contact force, recovery success, demonstration efficiency, and performance degradation under distribution shift. Comparisons between IL-only, RL-only, and hybrid policies reveal the contribution of each component.

An effective IL-RL hybrid architecture therefore treats demonstrations as structured prior knowledge and reinforcement learning as a mechanism for adaptation and optimization. Imitation places the policy near meaningful manipulation behavior, while RL expands robustness beyond the demonstrated trajectories. Multimodal feedback, hierarchical actions, simulation, recovery learning, and explicit safety constraints together transform this combination into a practical framework for dexterous humanoid manipulation.

Ultimately, IL-RL hybrid policies provide a bridge between human knowledge and autonomous physical learning. They allow humanoids to acquire sophisticated finger and arm behaviors without discovering every manipulation sequence from random exploration, while still adapting beyond the limitations of demonstrations. This combination supports increasingly general grasping, in-hand manipulation, tool use, contact-rich interaction, and whole-body dexterous tasks in variable real-world environments.

07.10. Humanoid Manipulation Benchmark Protocol

A humanoid manipulation benchmark protocol provides a systematic framework for measuring whether manipulation capabilities remain accurate, robust, safe, and repeatable across representative physical tasks. The benchmark should evaluate more than isolated grasp success. It must examine perception, reaching, contact establishment, dexterous control, object handling, recovery, and whole-body coordination under controlled variations that expose practical system limitations.

A useful benchmark begins with standardized test conditions. Robot hardware configuration, software version, controller parameters, sensor calibration, object properties, workspace geometry, and initial posture should be recorded before each experiment. Environmental variables such as illumination, surface friction, object placement tolerance, and external disturbances should also be specified so results can be reproduced and compared across different systems or development iterations.

Manipulation evaluation should be organized around task families that progressively increase physical and cognitive difficulty. Basic tests can measure reaching, grasp acquisition, lifting, and stable holding. Intermediate tests can include pick-and-place, object reorientation, handover, pouring, and tool use. Advanced tests should introduce bimanual coordination, contact-rich operations, payload handling, and unified whole-body manipulation involving posture adjustment or stepping.

Object sets should represent meaningful variations rather than a single collection of convenient laboratory items. Test objects can vary in size, shape, mass, surface friction, compliance, fragility, and grasp affordance. Cylindrical, box-like, irregular, articulated, deformable, and handled objects expose different limitations. The benchmark should distinguish objects seen during development from previously unseen objects used specifically to evaluate generalization.

Initial object pose should also be systematically varied. Objects can be placed at different positions and orientations within the reachable workspace, near workspace boundaries, close to obstacles, or in partially occluded configurations. A manipulation system that succeeds only from a carefully prepared canonical pose provides limited evidence of practical robustness. Pose randomization therefore measures sensitivity to perception error, reachability, grasp selection, and planning.

Pick performance should evaluate the complete transition from perception to stable lifting. A successful trial requires detecting the target, selecting a feasible grasp, approaching without unintended collision, establishing contact, and lifting the object to a defined height while maintaining control. Metrics can include acquisition success rate, grasp stability, approach time, number of retries, unintended contact, object displacement before grasp, and slip during lifting.

Placement evaluation should measure both task completion and final configuration accuracy. The robot must transport the object to a target region, establish stable support, transfer the load to the surface, and release without disturbing the desired pose. Position error, orientation error, placement success, object tipping, residual motion, release quality, and completion time provide complementary information about planning accuracy and compliant contact control.

Dexterous hand evaluation should examine capabilities that cannot be represented by simple parallel-jaw grasping. Tests can require finger coordination, controlled regrasping, rolling, sliding, finger gaiting, or object rotation within the hand. Performance should measure final object pose, unintended slip, contact loss, number of regrasp events, manipulation time, and recovery success while varying object geometry and friction conditions.

Handover benchmarks should evaluate both robot-to-human and human-to-robot transfer when human interaction is within the intended application scope. Trials should measure whether the robot presents an accessible grasp region, detects receiver readiness, regulates interaction force, and releases at an appropriate time. Transfer success, peak force, release delay, object drop rate, receiver effort, and recovery from interrupted handover can characterize interaction quality.

Pouring tests introduce continuous manipulation and changing payload dynamics. The robot should grasp a source container, align it with a receiving container, generate a controlled pouring trajectory, terminate at the intended quantity or state, and return the container safely. Pouring accuracy, spill amount, trajectory smoothness, residual liquid loss, completion time, grasp stability, and robustness to different fill levels provide useful benchmark measures.

Tool-use benchmarks should evaluate whether the robot understands and controls the functional relationship between hand, tool, and target. Tasks can include grasping a tool by an appropriate region, aligning its working end, establishing contact, and performing a constrained operation. Tool-tip pose error, engagement success, applied force or torque, unintended slip, completion quality, and recovery from misalignment can quantify functional manipulation performance.

Contact-rich manipulation requires dedicated evaluation because geometric trajectory accuracy alone does not describe physical interaction quality. Tests can involve pressing, inserting, wiping, turning, pulling, or constrained sliding. The protocol should record interaction force, force tracking error, contact stability, excessive-force events, completion success, and response to small geometric offsets. Such tests directly evaluate impedance and force-control behavior.

Bimanual manipulation should test whether both hands can operate as a coordinated system rather than as two independent manipulators. Representative tasks can require carrying a large object, stabilizing with one hand while manipulating with the other, or maintaining two contacts on a rigid object. Relative hand-pose error, internal force, object stability, synchronization, completion time, and contact loss can reveal deficiencies in multi-arm coordination.

Whole-body manipulation benchmarks should evaluate how manipulation interacts with posture, balance, and locomotion. Tasks can include reaching beyond the nominal arm workspace, carrying a payload while walking, pushing or pulling an object, manipulating while stepping, and transporting large objects bimanually. Measurements should include balance margin, foot placement, hand tracking, interaction force, payload stability, recovery behavior, and overall task completion.

Robustness testing should intentionally perturb conditions rather than relying only on nominal trials. Object pose estimates can contain controlled errors, friction can change, payload mass can vary, obstacles can move, and external forces can disturb the robot or object. The objective is to determine how performance degrades as uncertainty increases. Robust systems should maintain acceptable behavior across a measurable operating envelope instead of failing abruptly.

Generalization tests should separate memorized task execution from reusable manipulation capability. Previously unseen objects, target locations, object orientations, tool dimensions, payloads, and environmental layouts can be introduced without task-specific retraining. The benchmark should report the performance gap between familiar and novel conditions. A small degradation indicates that the policy has learned transferable physical structure rather than narrow task-specific trajectories.

Failure detection should be evaluated independently from nominal task success. The benchmark can intentionally create missed grasps, unstable contacts, blocked trajectories, object slip, unexpected resistance, or unavailable placement regions. The system should recognize that execution has diverged from expectation before damage or loss of control occurs. Detection latency, false alarms, missed failures, and state-estimation confidence can be recorded.

Recovery performance should then measure whether the robot can restore a useful state after detecting failure. A failed grasp may trigger withdrawal and regrasping, while slip may trigger force adjustment or object support. Excessive interaction force may require retreat, and uncertain placement may require maintaining the grasp until support is verified. Recovery success rate and additional completion time reveal practical autonomy beyond first-attempt performance.

Safety evaluation should operate throughout every benchmark category. Relevant quantities include joint torque, actuator load, fingertip force, hand closing velocity, tool-tip velocity, collision distance, external interaction force, balance state, and emergency-stop activation. A task should not be considered successful if completion requires violating defined safety limits. Safety events should therefore be reported separately from conventional task-success metrics.

Repeatability requires multiple trials under identical nominal conditions. Reporting only the best execution can hide stochastic failures, perception instability, or sensitivity to initialization. Success rate, mean performance, variance, percentile statistics, and failure distributions should be calculated over repeated trials. The number of trials should be stated explicitly so confidence in reported performance can be interpreted appropriately.

Benchmark logging should preserve enough information for post-test diagnosis. Sensor observations, estimated object states, commanded and measured joint motion, contact forces, tactile signals, controller states, policy outputs, failure flags, and timing information should be synchronized whenever possible. Video from external and onboard cameras provides additional context. Structured logs allow failures to be traced to perception, planning, control, or hardware limitations.

Timing measurements should separate perception latency, planning latency, control execution, physical task duration, and recovery overhead. A system may achieve high success but remain impractically slow, while another may execute quickly at the cost of unstable contact. Reporting both success and timing reveals these tradeoffs. Real-time deadline violations and control-loop irregularities should also be recorded for physically demanding tasks.

Comparative evaluation should use identical task definitions and scoring rules whenever different algorithms or policy versions are tested. Changes in perception, grasp planning, impedance control, imitation learning, reinforcement learning, or whole-body control can then be evaluated through controlled ablation. Each comparison should modify as few factors as possible, allowing performance improvements to be attributed to specific architectural components.

A composite score can summarize benchmark performance, but individual metrics should remain available. Combining success, accuracy, speed, force quality, robustness, recovery, and safety into a single number can simplify system-level comparison while hiding important weaknesses. The protocol should therefore preserve category-level results and failure distributions alongside any aggregate score, allowing engineering teams to identify the actual limiting subsystem.

The final benchmark report should document test configuration, task definitions, object sets, environmental conditions, trial counts, metrics, statistical results, observed failure modes, and recovery behavior. Results should clearly distinguish nominal performance from robustness and generalization tests. This documentation turns benchmarking into a repeatable validation process rather than a demonstration of selected successful examples.

Ultimately, a humanoid manipulation benchmark protocol should determine whether manipulation skills remain dependable when perception, contact, objects, and body configuration vary. By combining standardized pick, place, dexterous manipulation, handover, pouring, tool use, bimanual interaction, and whole-body tasks with robustness, recovery, and safety testing, the benchmark provides an integrated measure of readiness for real-world humanoid manipulation.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Top comments (0)