DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Manipulation and Grasping AI — Ch.06: Force and Tactile Sensing [EN]

STATERA Library · Series 08 · Volume 17 · Chapter 06 · EN
Source: https://github.com/Statera-Library/08_Vol_17_Manipulation_and_Grasping_AI/blob/main/Chapter_06_Force_and_Tactile_Sensing/Chapter_06_Force_and_Tactile_Sensing_EN.md

Volume 17 Manipulation and Grasping AI

Chapter 06. Force and Tactile Sensing

06.01. Force Torque Sensor Integration for Manipulation [w/Code]

Force--torque sensing gives a robotic manipulator direct information about the mechanical interaction between the robot, the manipulated object, and the surrounding environment. Unlike vision or joint position sensing, which mainly describe geometric state, a force--torque sensor measures the physical consequences of contact. This capability is essential when manipulation requires compliant motion, controlled insertion, surface following, delicate grasping, or reliable detection of unexpected collisions.

A typical manipulation system uses a six-axis force--torque sensor capable of measuring three orthogonal force components and three torque components. The measured wrench is commonly represented as \(W=[F_x,F_y,F_z,\tau_x,\tau_y,\tau_z]^T\). Sensors are frequently installed between the robot wrist and the end effector, although they may also be integrated into joints, grippers, fixtures, or tools depending on the required observability and mechanical architecture.

Wrist-mounted sensing is particularly useful because it captures the combined interaction wrench acting through the end effector. However, the raw measurement does not represent external contact alone. It also contains the effects of tool weight, payload mass, center-of-mass offset, acceleration, vibration, cable forces, and sensor bias. Reliable manipulation therefore requires calibration and compensation before the measured wrench can be interpreted as meaningful environmental interaction.

Sensor integration begins with accurate definition of coordinate frames. Measurements generated in the sensor frame often need to be transformed into the tool, end-effector, robot base, object, or task frame. Because forces and torques depend on both orientation and reference point, a wrench transformation must account for rotational transformation and the moment produced by positional offsets. Incorrect frame handling can make an otherwise accurate sensor produce misleading control behavior.

Gravity compensation is one of the most important processing steps. The manipulator must estimate the gravitational wrench generated by the tool and payload for the current robot orientation and subtract it from the sensor measurement. This requires estimates of payload mass and center of mass. When tools or objects change during operation, these parameters may need to be identified automatically or updated through the manipulation task state.

Bias and zero-offset compensation are equally important. Force--torque sensors can exhibit offsets caused by temperature, mounting stress, electronics, or mechanical preload. A zeroing procedure is normally performed under a known unloaded condition, but repeated zeroing alone is insufficient for high-precision systems. Practical implementations may include temperature compensation, drift monitoring, periodic recalibration, and plausibility checks to prevent slowly changing bias from being interpreted as persistent contact.

Raw force measurements also contain noise and structural vibration. Filtering is therefore required, but aggressive filtering introduces latency and can destabilize high-bandwidth interaction control. Low-pass filters, moving averages, notch filters, and model-based estimators can be selected according to the mechanical system and control frequency. The filtering strategy should preserve meaningful contact transients while suppressing frequencies associated with motors, gearboxes, structural resonance, and electrical noise.

The processed wrench can support several levels of manipulation intelligence. At the simplest level, force thresholds provide contact detection, collision detection, overload protection, and task-state transitions. More advanced systems analyze the direction, magnitude, temporal pattern, and relationship between force and motion. This allows the robot to distinguish free motion, initial contact, sliding contact, obstruction, successful seating, jamming, and other mechanically meaningful states.

Force control directly regulates one or more interaction-force components rather than commanding position alone. For example, a robot polishing a surface may regulate normal force while controlling tangential motion through position or velocity commands. Hybrid position--force control formalizes this separation by assigning different task-space directions to motion and force objectives. The approach is particularly valuable when environmental constraints determine some degrees of freedom.

Impedance control provides another major integration strategy. Instead of commanding a fixed force, the controller establishes a desired dynamic relationship among position error, velocity, and interaction force. The manipulator can consequently behave as if it possesses programmable stiffness and damping. Low stiffness enables compliant interaction with uncertain environments, while higher stiffness provides precise tracking when geometric accuracy is required. Variable impedance can adapt these properties during different task phases.

Admittance control is often preferred when the underlying industrial robot exposes a stable position or velocity interface rather than direct torque control. The measured external wrench is converted through a virtual dynamic model into commanded motion. When the robot experiences contact force, it yields according to configured virtual mass, damping, and stiffness. This architecture allows force-guided behavior to be implemented around position-controlled manipulators without modifying their internal servo loops.

Assembly tasks demonstrate the practical importance of force--torque integration. During peg-in-hole insertion, connector mating, bearing placement, or mechanical fastening, small geometric errors can generate rapidly increasing forces. A force-aware controller can detect contact, search for alignment, regulate insertion force, recognize jamming, and verify final seating. Combining force measurements with position and orientation information makes the task more robust to tolerances that cannot be eliminated through calibration alone.

Force--torque information is also valuable for grasping and object manipulation. A wrist sensor can detect whether an object has been lifted, whether it contacts an obstacle, or whether unexpected loads occur during transport. When combined with gripper sensing, joint torque estimation, vision, and tactile sensing, the system can separate global interaction forces from local contact phenomena and construct a richer representation of the physical state of the manipulation task.

Safety functions can use force information to identify abnormal mechanical interaction, but force--torque sensing should not automatically be treated as a substitute for certified robot safety systems. Operational thresholds can trigger motion reduction, withdrawal, task recovery, or emergency logic, while safety-rated functions remain responsible for regulatory protection where required. The architecture should distinguish task-level contact intelligence from formally validated safety mechanisms.

Real-time implementation strongly affects performance. Sensor acquisition, timestamping, coordinate transformation, filtering, compensation, contact estimation, and control should operate with predictable latency. Communication through EtherCAT, CAN, Ethernet, or other interfaces must be evaluated not only by average bandwidth but also by update rate, jitter, synchronization accuracy, and failure behavior. Fast force feedback becomes ineffective when unpredictable communication delays dominate the control loop.

Time synchronization becomes especially important when force sensing is fused with cameras, tactile arrays, joint encoders, IMUs, and external perception systems. A contact event observed by the force sensor must correspond to the correct robot pose and visual frame. Hardware timestamping or synchronized clocks can reduce temporal ambiguity, enabling reliable multimodal learning and post-task analysis. This is increasingly important for data-driven manipulation and Physical AI systems.

Software integration should separate sensor drivers, calibration parameters, wrench processing, contact-state estimation, control logic, and task-level decision making. In ROS 2 or comparable middleware, the raw sensor stream can be preserved while compensated and transformed wrench topics are provided to higher layers. Maintaining this separation improves debugging because developers can determine whether failures originate from sensing, calibration, transformation, control, or task interpretation.

Fault handling must also be designed explicitly. Saturation, communication loss, discontinuous readings, excessive drift, mechanical overload, or implausible wrench changes should not silently propagate into the controller. Health monitoring can evaluate sensor status and measurement validity before force data are accepted for control. When confidence becomes insufficient, the manipulator should transition to an appropriate degraded, stopped, or recovery state rather than continuing force-dependent motion.

Validation should cover the complete sensing chain rather than only the sensor specification. Static weights can verify force magnitude and direction, known lever arms can validate torque measurements, and multiple robot orientations can test gravity compensation. Controlled contact experiments can evaluate threshold detection, while representative assembly or surface-interaction tasks reveal whether filtering, latency, compliance, and controller gains remain effective under realistic dynamic conditions.

For learning-based manipulation, force--torque signals provide a compact representation of interaction dynamics that complements visual observations. Demonstrations can associate characteristic wrench trajectories with successful insertion, grasp stabilization, tool use, or failure. Policies can use force histories as observations, while learned models can predict contact transitions or estimate latent task states. Careful normalization and synchronization are essential because wrench distributions vary greatly across tools and tasks.

A mature force--torque integration architecture therefore treats the sensor as more than an additional peripheral. It establishes a physical feedback channel connecting perception, control, task reasoning, and safety supervision. When calibration, frame transformation, compensation, filtering, synchronization, and control are designed as one coherent pipeline, the manipulator can move beyond purely geometric automation toward contact-aware, compliant, and increasingly adaptive physical interaction.

06.02. Impedance and Admittance Control for Contact [w/Code]

Contact-rich manipulation requires a robot to regulate not only where its end effector moves but also how mechanically it interacts with the environment. Pure position control can generate excessive forces when geometric errors or environmental uncertainty exist. Impedance and admittance control address this problem by defining a controlled relationship among motion, velocity, acceleration, and interaction force rather than treating position and force as completely independent variables.

Impedance control shapes the apparent mechanical behavior of a manipulator during contact. Instead of directly commanding a specific contact force, the controller defines how the robot should respond when its actual motion differs from a desired trajectory. A common Cartesian formulation is \(F_{ext}=M_d\ddot{e}+D_d\dot{e}+K_de\), where \(M_d\), \(D_d\), and \(K_d\) represent desired inertia, damping, and stiffness, and \(e\) represents task-space displacement error.

The physical interpretation of impedance control resembles a virtual mass--spring--damper system connected between the desired and actual robot poses. Desired stiffness determines how strongly the manipulator resists displacement, damping determines how motion energy is dissipated, and virtual inertia influences transient acceleration. By selecting these parameters, the same robot can behave rigidly for precise positioning or softly for compliant interaction without changing its physical mechanical structure.

Cartesian impedance control is particularly useful because contact tasks are usually described relative to an object, surface, or task coordinate system rather than individual robot joints. Translational and rotational stiffness can be specified independently along different axes. A robot can therefore remain stiff along a machining direction while remaining compliant normal to an uncertain surface, or maintain orientation while allowing lateral motion during an insertion operation.

Joint-space impedance control applies a similar principle directly to robot joints. Desired joint positions and velocities generate restoring torques according to virtual stiffness and damping. This approach can exploit torque-controlled actuators and provide natural whole-arm compliance. However, the relationship between joint impedance and Cartesian contact behavior depends on robot configuration and Jacobian geometry, making task-space design important when precise environmental interaction characteristics are required.

Admittance control reverses the input-output interpretation. Instead of receiving motion error and producing a force or torque response, an admittance controller measures external force and converts it into desired motion. A typical virtual dynamics model can be expressed as \(M_d\ddot{x}+D_d\dot{x}+K_d(x-x_d)=F_{ext}\). The resulting displacement, velocity, or acceleration command is then sent to the underlying robot motion controller.

This architecture is especially practical for industrial manipulators that provide high-performance position or velocity control but do not expose low-level motor torque interfaces. A wrist force--torque sensor measures environmental interaction, and an outer admittance loop calculates how the commanded robot trajectory should yield. The existing inner servo controller then tracks this modified trajectory, allowing compliant behavior to be added without replacing the manufacturer\'s internal control architecture.

The distinction between impedance and admittance is therefore closely related to the robot\'s native actuation and control interface. Torque-controlled robots naturally support impedance behavior because commanded actuator torques can directly shape mechanical response. Position-controlled robots often favor admittance control because measured force can be translated into motion references. In practical systems, however, implementations may combine elements of both approaches and include additional force-feedback loops.

Controller tuning determines whether contact feels stable, responsive, and safe. High stiffness improves geometric tracking but can amplify force peaks caused by positioning errors. Low stiffness increases compliance but may reduce task accuracy. Insufficient damping can produce oscillation, while excessive damping makes interaction sluggish. Virtual inertia influences transient response and can become especially important when the commanded motion changes rapidly or when the robot interacts with stiff environments.

Contact stability depends on more than controller gains. Sensor delay, digital sampling, communication jitter, structural flexibility, actuator dynamics, friction, gearbox compliance, and environmental stiffness all affect the closed-loop system. A controller that behaves well against a soft surface may oscillate when contacting rigid metal. Consequently, impedance or admittance parameters should be evaluated with the expected range of environmental mechanical properties rather than tuned only in free space.

Force--torque sensing is central to high-quality admittance control and can also enhance impedance implementations. Raw measurements generally require bias removal, gravity compensation, coordinate transformation, and filtering before they enter the control loop. Excessive filtering reduces noise but introduces phase delay, while insufficient filtering allows vibration to influence commanded motion. The sensing and control pipelines must therefore be designed together rather than optimized independently.

The coordinate frame used for compliance strongly affects task behavior. In peg-in-hole insertion, lateral directions may require low stiffness so that the peg can self-align, while the insertion axis requires controlled downward motion and bounded force. During surface polishing, normal compliance regulates contact pressure while tangential directions follow the planned path. Task-frame impedance enables these different mechanical behaviors to coexist within a single controller.

Rotational compliance is equally important in many assembly tasks. Small angular misalignment between a connector and socket can produce large contact moments even when translational position error is small. By configuring rotational stiffness and damping independently from translational parameters, the robot can passively or actively accommodate orientation errors. This capability is valuable for connectors, gears, shafts, covers, fixtures, and other tightly constrained mechanical assemblies.

Variable impedance control changes mechanical parameters during task execution instead of using one fixed setting. Free-space motion may use relatively high stiffness for accurate and rapid trajectory tracking, initial contact may reduce stiffness to limit impact, and insertion may apply direction-dependent compliance. After successful seating, stiffness can increase again to stabilize the final pose. Task-state-dependent impedance therefore combines precision and compliance within one manipulation sequence.

Adaptive impedance extends this idea by modifying stiffness, damping, or reference trajectories according to measured interaction. Adaptation can use force error, estimated environmental stiffness, contact state, task progress, or learned models. Such methods are useful when material properties or geometry vary between operations. However, parameter adaptation must preserve stability because rapid or poorly constrained changes in controller dynamics can create unintended energy and unstable contact.

Passivity provides an important framework for analyzing stable physical interaction. A passive system does not generate unlimited net energy through its interaction port, making it naturally compatible with many physical environments. Passivity observers, passivity controllers, energy tanks, and related techniques can monitor or restrict energy exchange. These approaches become valuable when delays, variable impedance, teleoperation, or uncertain environmental dynamics make conventional stability analysis difficult.

Impedance and admittance control also support human--robot interaction. When an operator pushes or guides a robot, measured forces can be converted into compliant motion so that the robot feels lightweight and responsive. The controller may constrain motion to permitted directions, provide virtual fixtures, or increase resistance near restricted regions. This allows human intention and robotic precision to be combined while maintaining explicit limits on velocity, workspace, and interaction force.

In grasping, compliance helps distribute contact forces and tolerate uncertainty in object pose or geometry. A rigidly controlled manipulator may push an object away when the estimated grasp pose is slightly inaccurate, whereas a compliant approach can accommodate contact and settle into a stable configuration. Whole-arm impedance can also reduce impact when the forearm or wrist unexpectedly touches the environment during manipulation in cluttered spaces.

Insertion and fastening applications often combine impedance or admittance control with contact-state logic. The robot first approaches under position control, detects contact through force or torque, transitions to compliant alignment, searches for geometric constraints, and then executes controlled insertion. Changes in axial force, lateral force, or torque can indicate successful engagement, cross-threading, jamming, bottoming-out, or completion and trigger the next task state.

Safety-related limits should surround the compliant controller even when the nominal behavior is soft. Maximum force, torque, velocity, displacement, power, and workspace limits prevent abnormal sensor signals or unstable control from producing dangerous motion. Watchdogs should detect sensor communication loss, saturation, timing faults, and implausible wrench changes. Compliance improves interaction behavior, but it does not eliminate the need for independent safety supervision and certified protection where required.

Real-time execution is critical because mechanical interaction evolves faster than many perception tasks. Sensor acquisition, compensation, controller calculation, robot-state feedback, and command transmission should operate at predictable rates with bounded latency and jitter. High-frequency inner loops are generally desirable for stiff contact, while supervisory task logic can operate more slowly. Timestamp synchronization helps ensure that force measurements correspond to the robot state used by the controller.

Simulation can accelerate controller development by allowing stiffness, damping, inertia, friction, contact geometry, and sensor noise to be varied systematically. Nevertheless, contact simulation rarely reproduces every property of the physical system, especially backlash, structural resonance, surface friction, and communication delay. Parameters that appear stable in simulation should therefore be transferred conservatively and validated progressively from soft contacts toward the intended real-world environment.

Learning-based methods can complement classical impedance and admittance control by predicting suitable controller parameters or reference trajectories from demonstrations and sensor observations. A policy may select stiffness according to contact phase or infer the direction in which compliance should be increased. Keeping a conventional compliant controller beneath the learned policy can provide a structured physical interface while machine learning handles higher-level adaptation and task variability.

A robust contact-control architecture ultimately integrates mechanical modeling, force--torque sensing, coordinate transformations, real-time control, task-state estimation, and safety supervision. Impedance control determines how strongly the robot mechanically resists motion disturbances, while admittance control determines how measured interaction forces generate motion. Properly designed, both approaches transform rigid trajectory execution into controlled physical interaction capable of tolerating uncertainty while preserving manipulation accuracy.

06.03. Force Guided Assembly Peg in Hole [w/Code]

Peg-in-hole assembly is a representative contact-rich manipulation problem in which a robot must insert a peg into a mating hole despite uncertainty in position, orientation, geometry, and mechanical properties. Pure position control can fail even when perception and calibration errors are small because tight clearances convert minor misalignment into large contact forces. Force-guided assembly uses measured interaction forces and torques to recognize contact, correct alignment, regulate insertion, and determine whether assembly has succeeded.

The task can be interpreted as a sequence of mechanical states rather than a single Cartesian motion. The robot typically moves through free-space approach, initial contact, surface localization, alignment, hole search, insertion, and final seating. Each state has different sensing and control requirements. Position control is effective during approach, while compliant force-aware behavior becomes increasingly important after contact because environmental constraints begin to determine allowable motion.

A six-axis force--torque sensor mounted near the robot wrist provides three force components and three torque components describing the interaction wrench. Before control, raw measurements normally undergo bias compensation, gravity and payload compensation, coordinate transformation, and filtering. The resulting wrench should be represented in a task frame aligned with the insertion geometry so that axial force, lateral forces, and alignment moments have direct physical meaning.

Geometric uncertainty may originate from camera localization error, robot calibration, fixture tolerance, thermal deformation, manufacturing variation, or movement of the workpiece. Even submillimeter translational error can become significant when the clearance between peg and hole is small. Angular misalignment is often more problematic because it produces asymmetric contact and moments that increase rapidly as insertion depth grows, potentially causing wedging or jamming.

Initial contact detection provides the transition from free-space motion to contact-aware control. The robot may detect contact when axial force exceeds a threshold, but a robust detector should account for sensor noise, acceleration-induced loads, and transient vibration. Threshold hysteresis, persistence conditions, filtered derivatives, or model-based residuals can prevent false transitions. The approach velocity should also be limited so that detection latency does not allow excessive impact force to develop.

After contact with the surrounding surface, the robot must determine the relative location of the hole. When visual localization is sufficiently accurate, only a small local correction may be necessary. With greater uncertainty, the end effector can execute a structured search while maintaining a bounded normal force. Spiral, raster, circular, or directional search patterns allow the peg tip to explore the surface until a characteristic change in displacement, force, or moment indicates engagement with the hole entrance.

Search motion must be designed carefully because larger trajectories increase the probability of locating the hole but also increase cycle time and the risk of damaging the surface. Search radius should therefore reflect the expected localization uncertainty rather than being chosen arbitrarily. Velocity, normal force, and lateral force limits should be coordinated so that the peg remains in controlled contact while avoiding excessive friction, scratching, or edge loading.

Compliance is essential during alignment because a perfectly rigid robot attempts to preserve the commanded pose even when the environment indicates that the pose is incorrect. Cartesian impedance control can assign low lateral and rotational stiffness while maintaining controlled behavior along the insertion axis. Alternatively, admittance control can convert measured lateral forces and moments into corrective displacement or orientation commands, particularly when the robot provides a position-controlled interface.

The contact wrench contains information about the direction of misalignment. A lateral force may indicate that the peg is pressing against one side of the hole, while moments about transverse axes can reveal angular error. Controllers can exploit these signals to generate corrective translations and rotations. However, the relationship between wrench and pose error depends on contact geometry, friction, and insertion depth, so simple proportional correction is not always sufficient.

Remote center compliance concepts are useful for understanding passive and active alignment. By shaping translational and rotational compliance around an appropriate virtual center, contact forces can naturally generate motions that reduce misalignment rather than amplify it. Mechanical remote center compliance devices achieve this through physical elasticity, while robotic impedance or admittance controllers can approximate similar behavior through software and adjustable task-space compliance.

Once the peg enters the hole, the control objective changes from search to stable insertion. The robot generally commands motion along the insertion axis while limiting axial force and allowing smaller lateral or angular corrections. Increasing insertion depth strengthens geometric constraints, so compliance parameters may need to change accordingly. Excessive lateral freedom can reduce precision, whereas excessive stiffness can convert small remaining alignment errors into rapidly growing contact loads.

Jamming and wedging are major failure modes. Jamming occurs when contact forces and friction prevent further insertion, while wedging can arise when multiple contacts create a mechanically locked configuration. Both may appear as increasing axial force with little insertion progress, often accompanied by characteristic lateral forces or moments. Detection should therefore combine wrench measurements with displacement, velocity, and task progress rather than relying on a single force threshold.

A recovery strategy should respond to abnormal contact before hardware limits are reached. The robot may stop axial motion, retract slightly, reduce stiffness, adjust orientation, repeat local search, or return to a previous task state. Recovery parameters should be bounded so that repeated attempts do not progressively damage components. Logging the wrench trajectory and pose history also enables engineers to distinguish localization errors from unsuitable compliance settings or mechanical tolerance problems.

Successful seating must be detected reliably because reaching the commanded position does not necessarily mean that the assembly is complete. Completion may be identified through insertion depth, a characteristic rise in axial force, reduced motion under commanded displacement, geometric constraints, or a combination of these signals. For connectors and snap-fit components, a distinctive transient force pattern may provide stronger evidence of successful engagement than position alone.

Friction significantly influences peg-in-hole behavior. Surface material, lubrication, edge geometry, roughness, and contamination alter the relationship between contact force and corrective motion. High friction can prevent passive self-alignment and make a valid insertion appear jammed, while very low friction changes the wrench signatures used for state detection. Validation should therefore include realistic component surfaces rather than relying only on idealized simulation or low-friction laboratory fixtures.

Chamfers and lead-in geometry can substantially improve assembly robustness. A chamfer converts some axial insertion force into lateral alignment force, allowing moderate position errors to self-correct. However, the effectiveness of this mechanism depends on chamfer angle, clearance, friction, and compliance. Force-guided control should exploit helpful mechanical geometry rather than attempt to compensate entirely through software for an assembly interface that is inherently difficult to mate.

Orientation control becomes especially important for long pegs, small clearances, and deep insertion. A small angular error at the wrist can produce a large lateral displacement at the peg tip. Rotational compliance around appropriate task axes allows contact moments to correct this error. For asymmetric connectors or keyed components, orientation may also include rotation around the insertion axis, requiring perception and force information to cooperate in identifying the correct mating configuration.

Hybrid position--force control provides another framework for organizing the task. Motion can be commanded along unconstrained directions while force is regulated along directions constrained by contact. Before insertion, the robot may regulate normal force against the surface while searching laterally. During insertion, axial motion may be position- or velocity-driven with an explicit force limit, while lateral directions remain compliant. The selection matrix can change as the assembly state evolves.

Task-state estimation is therefore central to reliable automation. Force-guided assembly should not use one controller configuration for the entire operation. A finite-state machine, behavior tree, or equivalent task executive can coordinate approach, contact detection, search, alignment, insertion, verification, and recovery. Each state defines expected wrench ranges, motion constraints, timeout conditions, and valid transitions, making abnormal behavior easier to recognize and diagnose.

Vision and force sensing provide complementary information. Vision offers global geometric localization before contact, whereas force--torque sensing provides precise local information after physical interaction begins. Combining them reduces the burden on either modality. Vision can place the peg near the expected hole, after which force-guided search compensates for residual error. Updated visual observations can also verify object motion or detect gross failures that force signals alone cannot distinguish.

Tactile sensing can further improve the process by providing spatially distributed contact information. A wrist force--torque sensor measures the net interaction wrench but cannot directly reveal where contact occurs. Tactile sensors on the gripper or peg fixture can identify local pressure patterns and contact locations. Fusion of wrench, tactile, joint, and visual observations can therefore provide a richer estimate of alignment and contact state, particularly for complex connectors.

Real-time performance strongly affects insertion quality. Sensor acquisition, compensation, filtering, state estimation, controller computation, and robot commands must execute with predictable latency and limited jitter. Excessive delay can cause the robot to continue pushing after force rises or make corrective motion arrive too late. Control frequency requirements depend on robot dynamics and contact stiffness, but deterministic timing is generally more important than high average computation throughput.

Simulation is valuable for exploring search patterns, controller gains, friction ranges, clearances, and contact geometry before hardware testing. Nevertheless, contact models can differ significantly from reality, especially for friction, compliance, backlash, and microscopic surface effects. Simulation should therefore be used to narrow the design space rather than establish final parameters. Hardware validation should progress from generous clearances and low forces toward the actual production tolerance.

Learning-based approaches can improve assembly when contact relationships are difficult to model explicitly. Demonstrations can teach search strategies, wrench-to-correction mappings, or task-state classifiers, while reinforcement learning can optimize compliant behaviors under randomized pose and friction conditions. A practical architecture can place learned adaptation above deterministic force, motion, and safety controllers so that exploration or prediction does not bypass physical operating limits.

Performance should be evaluated using more than successful insertion rate. Relevant measures include cycle time, maximum contact force and torque, search distance, insertion depth before failure, number of recovery attempts, sensitivity to pose error, and component damage. Testing across systematic position and orientation perturbations produces an assembly tolerance map that reveals where the controller succeeds reliably and where mechanical or sensing improvements are required.

Force-guided peg-in-hole assembly ultimately demonstrates how perception, sensing, compliant control, task reasoning, and mechanical design must operate together. Reliable insertion is not achieved by force control alone, but by recognizing changing contact states and applying appropriate behavior at each stage. When vision provides coarse localization and force--torque feedback supplies local physical evidence, the robot can transform uncertain geometric alignment into a controlled and verifiable assembly process.

06.04. Contact State Estimation from F T Signals [w/Code]

Contact state estimation from force--torque signals enables a robot to infer what is physically happening at the interface between its end effector, manipulated object, and environment. Rather than treating force measurements only as control feedback, the robot interprets temporal wrench patterns as evidence of free motion, initial contact, sliding, insertion, jamming, seating, or loss of contact. This converts mechanical interaction into an observable state that can support manipulation decisions.

A six-axis force--torque sensor typically produces a wrench \(W=[F_x,F_y,F_z,\tau_x,\tau_y,\tau_z]^T\). Individual components provide useful directional information, but contact states are rarely identifiable from one channel alone. The relationship among forces, moments, robot motion, and time is more informative. A lateral force combined with a transverse moment, for example, can indicate edge contact or angular misalignment that an axial force threshold cannot distinguish.

Reliable estimation begins with signal conditioning. Raw sensor outputs contain bias, gravitational loading, payload effects, acceleration forces, structural vibration, electrical noise, and thermal drift. Bias removal, gravity compensation, coordinate transformation, and filtering are therefore required before interpreting the wrench as external interaction. Poor compensation can create apparent contact even during free motion or hide weak physical contact beneath systematic measurement errors.

Coordinate frames are especially important because a contact pattern should be interpreted relative to the task rather than the sensor housing. Transforming the wrench into a task frame allows force components to represent meaningful directions such as surface normal, insertion axis, lateral alignment axes, or tool tangent. The same physical contact may appear very different in sensor coordinates as the robot changes orientation, whereas task-frame signals retain consistent semantic meaning.

Temporal information distinguishes transient events from persistent mechanical states. Initial impact may produce a short force spike, while sustained contact produces a nonzero wrench over a longer interval. Sliding can create relatively stable normal force with changing tangential components, and jamming can produce increasing force without corresponding motion. Estimation algorithms should therefore analyze signal history rather than classify every sample independently.

Simple threshold methods remain useful because they are deterministic, computationally inexpensive, and easy to validate. Contact can be declared when a compensated force magnitude or task-axis component exceeds a defined threshold for a minimum duration. Separate entry and exit thresholds provide hysteresis and prevent rapid state switching near a boundary. Thresholds should reflect sensor noise, expected dynamic loads, tool properties, and the minimum interaction force that must be detected.

Force derivatives provide additional evidence about contact transitions. A rapid increase in force can indicate impact even before the absolute force becomes large, while a sudden decrease may indicate release or loss of contact. Derivative-based features are sensitive to noise, so filtering and appropriate time windows are necessary. Combining force magnitude, rate of change, and persistence often provides more robust transition detection than any individual feature.

Motion information makes wrench interpretation substantially stronger. High force with commanded motion but little measured displacement can indicate obstruction or jamming, whereas similar force accompanied by steady displacement may represent normal sliding or insertion. Robot position, velocity, acceleration, and joint state should therefore be synchronized with force--torque measurements. Contact estimation becomes a multimodal state inference problem rather than a force classification problem alone.

In surface interaction, the normal and tangential wrench components provide characteristic signatures. Stable normal force with small tangential force can represent stationary contact, while increasing tangential force followed by motion may indicate the transition from static to kinetic friction. Repeated oscillations can reveal stick--slip behavior. These patterns are useful for polishing, wiping, sanding, inspection, and other tasks where maintaining the correct contact mode is essential.

Assembly produces more complex contact states because geometric constraints evolve during the task. A peg may transition from free motion to surface contact, edge contact, partial insertion, two-point contact, stable insertion, jamming, and final seating. Each configuration produces a different combination of axial force, lateral forces, and moments. Estimating these states allows the controller to change compliance, search behavior, insertion velocity, or recovery logic at appropriate times.

Contact geometry can sometimes be inferred from wrench relationships. If a force acts at an offset from the sensor origin, the resulting moment contains information about the effective line of action. Under suitable assumptions, force and torque can therefore estimate a contact point or contact direction. Accuracy depends on tool rigidity, contact model, friction, sensor noise, and whether one or multiple contacts are present, so geometric inference should include validity checks.

Feature extraction converts raw time-series data into quantities that better separate contact states. Useful features include mean force, peak force, variance, root-mean-square amplitude, force magnitude, torque magnitude, derivatives, impulse, spectral energy, cross-axis correlation, and ratios between normal and tangential components. Features may be calculated over sliding windows so that classification reflects recent interaction history while retaining sufficiently fast response.

Frequency-domain information can reveal contact phenomena that are difficult to distinguish from average force alone. Sliding across textured surfaces, chatter during machining, stick--slip friction, and mechanical vibration can generate characteristic frequency components. Fourier or wavelet representations can capture these signatures. However, frequency analysis requires sufficient observation time, creating a tradeoff between classification confidence and detection latency in real-time manipulation.

Model-based estimation compares measured wrench signals with predictions from robot and contact models. Expected inertial, gravitational, and frictional effects can be removed to obtain an external-force residual. Changes in this residual provide evidence of contact or collision. More advanced observers can estimate external wrench from joint torques and robot dynamics, allowing physical sensors and model-based estimates to cross-check each other and improve fault tolerance.

Probabilistic methods are useful when contact states overlap and sensor measurements are uncertain. Hidden Markov models, Bayesian filters, or related state estimators can combine observation likelihoods with transition probabilities. For example, final seating is unlikely to occur directly from free motion without preceding insertion. Encoding such temporal structure prevents physically implausible state jumps and makes estimates more robust when individual measurements are ambiguous.

Machine-learning classifiers can learn nonlinear relationships among wrench channels and motion features. Support vector machines, decision trees, random forests, neural networks, and temporal architectures can distinguish interaction states from labeled demonstrations. Temporal convolutional networks, recurrent models, or transformer-based sequence models can exploit longer histories. Their benefit is greatest when analytical boundaries between contact modes are difficult to define.

Training data quality strongly determines learned estimator performance. Datasets should cover variations in object pose, robot speed, payload, friction, stiffness, geometry, sensor noise, and failure conditions. If training contains only nominal successful executions, the estimator may perform poorly precisely when abnormal contact occurs. Synthetic or simulated interaction data can broaden coverage, but real sensor recordings remain important because contact dynamics are difficult to reproduce perfectly.

Contact-state labels also require careful definition. Labels should correspond to mechanically meaningful conditions that lead to different robot actions rather than arbitrary divisions of a signal trace. Ambiguous transition regions may require probabilistic labels or short uncertainty intervals. Consistent labeling across demonstrations is particularly important because two operators may otherwise describe the same edge-contact or partial-insertion event differently.

Confidence estimation helps prevent uncertain classifications from immediately changing robot behavior. An estimator can output both the most likely contact state and a confidence score or probability distribution. Low-confidence observations may cause the controller to maintain its current state, slow down, gather additional measurements, or transition to a safe diagnostic behavior. This is preferable to forcing every noisy measurement into a definite class.

State transitions should be constrained by task logic. A finite-state machine or behavior tree can define which transitions are physically valid and what evidence is required. During peg insertion, for example, the sequence may progress from approach to contact, search, alignment, insertion, and seating, with alternative transitions to jamming or recovery. Combining signal-based estimation with task structure produces more reliable behavior than using either mechanism independently.

Sensor fusion extends estimation beyond wrist wrench measurements. Joint torque, motor current, tactile arrays, vision, acoustic signals, and IMU data can provide complementary evidence. Tactile sensing reveals spatial contact distribution, vision provides geometric context, and motor currents can indicate loading throughout the mechanism. Multimodal fusion becomes particularly valuable when several different physical situations generate similar net force--torque signatures.

Real-time implementation requires predictable timing throughout acquisition, preprocessing, feature extraction, inference, and state publication. Contact transitions can occur within milliseconds, so excessive filtering or large inference windows may delay corrective action. The estimator should therefore balance robustness against latency. Timestamp synchronization is essential when combining wrench data with robot motion because even modest temporal misalignment can corrupt the relationship between force and displacement.

Fault detection should remain distinct from ordinary contact classification. Sensor saturation, communication loss, excessive drift, discontinuities, or implausible wrench combinations may indicate sensing failure rather than a valid mechanical state. Health monitoring should identify these conditions before the data reach the state estimator. When measurement integrity is uncertain, the robot should enter a predefined degraded or safe behavior instead of interpreting corrupted data as physical contact.

Validation should test both classification accuracy and operational consequences. Confusion matrices, precision, recall, transition latency, and false-alarm rates describe estimator performance, but manipulation-level metrics are equally important. A small classification delay may be acceptable during slow surface following but unacceptable during high-stiffness impact. Evaluation should therefore connect estimation errors to force peaks, task failures, recovery frequency, cycle time, and component damage.

Systematic perturbation testing can reveal the estimator\'s operating envelope. Position error, orientation error, speed, payload, friction, environmental stiffness, sensor noise, and contact geometry can be varied across controlled experiments. The resulting state-estimation map shows where classifications remain reliable and where uncertainty increases. Such testing is especially valuable before deploying the same estimator across different tools, products, or manufacturing stations.

Contact state estimation ultimately transforms force--torque sensing from a reactive measurement channel into a source of physical perception. By combining compensated wrench signals, robot motion, temporal features, contact models, probabilistic reasoning, and task constraints, a manipulator can infer not merely how much force is present but what interaction is occurring. This capability forms a foundation for adaptive, contact-aware Physical AI systems that reason about the mechanical consequences of their actions.

06.05. Tactile Based Slip Prevention in Grasping [w/Code]

Tactile-based slip prevention enables a robotic gripper to maintain stable contact with an object by detecting changes at the finger--object interface before the object is dropped or significantly displaced. Vision can estimate object pose and grasp geometry, but it cannot directly observe local frictional conditions after contact. Tactile sensing provides this missing physical information by measuring pressure, deformation, shear, vibration, and contact distribution during grasping.

Slip occurs when the tangential load at a contact exceeds the frictional capability of the interface. Under a simplified Coulomb friction model, stable contact requires the tangential force to remain within a friction cone determined by normal force and the coefficient of friction. In real manipulation, however, friction varies with material, surface contamination, texture, contact area, temperature, and deformation, making fixed grasp-force settings inefficient or unreliable.

Applying a very large normal force can prevent many slip events, but this is not an appropriate general solution. Excessive grip force can deform fragile objects, damage packaging, crush food, accelerate gripper wear, or disturb the object\'s pose. An effective tactile controller instead seeks the minimum force margin necessary for stability and increases gripping force only when measured contact conditions indicate that the current grasp is approaching instability.

Tactile sensors can use several physical principles. Piezoresistive and capacitive arrays estimate pressure distribution, optical tactile sensors observe deformation of an elastomeric surface, magnetic sensors measure changes in embedded magnetic fields, and piezoelectric elements respond strongly to dynamic events. Different technologies provide different combinations of spatial resolution, bandwidth, force range, durability, sensitivity, and integration complexity.

A tactile array produces spatial information that cannot be obtained from a single wrist force--torque sensor. Individual sensing elements, often called taxels, form a pressure or deformation map across the fingertip. From this map, the robot can estimate contact area, pressure centroid, normal-force distribution, edge loading, and changes in contact geometry. These quantities provide evidence about whether an object is centered, rotating, rolling, or beginning to escape from the grasp.

Slip is not always a single instantaneous event. Incipient slip begins locally when parts of the contact patch start moving while other regions remain stuck. If detected early, the robot can increase normal force or modify its motion before gross slip develops. Detecting incipient slip is therefore more valuable than recognizing an object only after substantial displacement has occurred, because corrective action can preserve both grasp stability and task accuracy.

Shear information is particularly important because slip is fundamentally associated with tangential loading. Sensors capable of measuring tangential deformation or shear force can directly observe increasing lateral stress across the contact patch. The ratio between tangential and normal loading provides an estimate of proximity to the friction limit. A rapidly changing shear pattern may also indicate object rotation, external disturbance, or redistribution of load between fingers.

Vibration provides another strong slip signature. When a surface begins sliding across a fingertip, microscopic texture and stick--slip interactions generate high-frequency vibrations. Piezoelectric sensors, accelerometers, or high-bandwidth tactile measurements can detect these transients. Frequency-domain features may distinguish stable contact from sliding, although sensitivity depends on surface texture and relative velocity, so vibration should often be combined with slower pressure and shear information.

Tactile signal processing begins with calibration, offset compensation, normalization, and noise reduction. Individual taxels may have different sensitivities, hysteresis, drift, or nonlinear response. Flexible sensor skins can also exhibit mechanical coupling between neighboring elements. Calibration should therefore characterize not only the nominal sensor output but also repeatability, spatial uniformity, temperature dependence, and the effects of repeated loading over the expected operating range.

Contact segmentation identifies which portions of the tactile surface are actively interacting with the object. A thresholded pressure map can provide a simple contact region, while more advanced processing can track connected regions, boundaries, pressure gradients, and centroid motion. Changes in contact shape over time are often more informative than absolute pressure because object rolling or partial slip may move the pressure distribution before the total gripping force changes substantially.

Temporal features are essential for distinguishing stable grasping from developing instability. Useful quantities include pressure derivative, centroid velocity, contact-area variation, shear-rate change, vibration energy, cross-correlation between taxels, and differences between opposing fingers. Sliding windows can summarize recent history while maintaining real-time response. Multi-scale windows are useful because rapid slip transients and slow load redistribution occur at different time scales.

Rule-based slip detectors can operate using explicit thresholds on shear, vibration, pressure redistribution, or contact motion. Their behavior is interpretable and can be validated systematically, making them suitable for well-defined industrial tasks. However, fixed thresholds may not generalize across objects with different masses, materials, geometries, or surface conditions. Adaptive thresholds based on estimated contact force or object properties can improve robustness.

Model-based approaches relate tactile measurements to contact mechanics and friction. The controller can estimate whether measured tangential forces remain inside an approximate friction cone or whether pressure distribution indicates loss of contact at the patch boundary. These methods provide physically meaningful criteria, but their accuracy depends on assumptions about friction, deformation, and contact geometry that may not hold for soft, irregular, or unknown objects.

Machine-learning methods can learn slip signatures directly from tactile time series. Classifiers may use engineered features or raw sensor arrays, while convolutional networks can exploit spatial pressure patterns and recurrent or temporal models can capture dynamic evolution. Training examples should include stable grasps, incipient slip, gross slip, rolling, intentional repositioning, and external disturbances so that the model learns physically distinct interaction modes rather than a simple binary boundary.

Training data should cover variation in object mass, center of gravity, material, surface texture, shape, grasp location, finger speed, and environmental conditions. A detector trained only on rigid laboratory objects may fail on deformable packaging or smooth consumer products. Domain randomization and simulated tactile data can broaden coverage, but physical experiments remain important because detailed frictional and deformation behavior is difficult to reproduce accurately in simulation.

Once slip risk is detected, the controller must choose an appropriate corrective action. The simplest response is to increase normal grip force by a small increment. More sophisticated strategies can slow manipulator acceleration, change wrist orientation, redistribute force among fingers, reposition contacts, or modify the planned trajectory. Corrective behavior should address the cause of instability rather than continuously increasing force without limit.

Closed-loop grip-force regulation can maintain a stability margin instead of using a predetermined force. The tactile estimator produces a slip probability or friction-margin estimate, and the controller adjusts finger force accordingly. When contact is stable, force may gradually decrease to avoid unnecessary compression. When incipient slip appears, force increases until the estimated margin returns to a safe region, creating an adaptive grasp that responds to changing loads.

Grip-force adaptation should include upper and lower limits. The lower bound prevents accidental release, while the upper bound protects the object, sensor, and gripper. Rate limits prevent abrupt squeezing that could excite robot dynamics or damage fragile materials. For delicate manipulation, object-specific constraints can define maximum pressure or allowable deformation, making tactile control responsible for balancing slip prevention against damage avoidance.

Object lifting provides a useful example of tactile feedback. Before lifting, the gripper establishes initial contact and estimates whether both fingers have adequate contact area. As vertical acceleration increases, tangential load rises and the tactile system monitors shear and micro-slip. If instability appears, grip force is increased before the object moves visibly. Once acceleration decreases, the required normal force can be reduced while preserving an appropriate safety margin.

Manipulation introduces disturbances beyond gravity. Translational acceleration, wrist rotation, tool interaction, collisions, and object inertia can redistribute loads between fingers. A grasp that is stable while stationary may become unstable during rapid motion. Tactile slip prevention should therefore consider robot dynamics and planned acceleration, allowing grip force to increase proactively before high-load trajectory segments rather than responding only after slip begins.

Sensor fusion can improve reliability when tactile evidence is ambiguous. Wrist force--torque sensing provides the net external wrench, joint torque estimates describe loads transmitted through the arm, and vision can detect larger object motion. Tactile sensing supplies local contact information. Combining these sources helps distinguish true slip from sensor artifacts, intentional finger motion, environmental contact, or changes in manipulator acceleration.

Multi-finger hands require coordinated interpretation of tactile information. One finger may begin slipping while others remain stable, causing object rotation rather than immediate loss of grasp. The controller should estimate load distribution across contacts and determine whether to strengthen one finger, redistribute forces, or alter hand posture. This turns slip prevention into a grasp-level control problem rather than a collection of independent fingertip reactions.

Contact-state estimation can organize tactile control into meaningful phases such as approach, first contact, grasp establishment, stable hold, incipient slip, recovery, manipulation, and release. Different phases require different thresholds and controller behavior. During grasp establishment, contact area may still be increasing, whereas during stable holding the same change could indicate object motion. Task context therefore improves interpretation of tactile signals.

Real-time performance is essential because the interval between incipient and gross slip may be short. Sensor sampling, preprocessing, feature extraction, inference, and actuator response must operate with sufficiently low and predictable latency. High sensor bandwidth alone does not guarantee effective prevention if communication or gripper actuation is slow. The complete perception--control loop must therefore be evaluated as one dynamic system.

Failure monitoring should distinguish slip from tactile sensor faults. Dead taxels, saturated measurements, cable failures, thermal drift, delamination, or unexpected baseline changes can produce misleading patterns. Sensor-health logic can monitor spatial consistency, range limits, signal continuity, and agreement between fingers. When tactile confidence is low, the system can adopt a conservative grip strategy or transition to a controlled stop rather than trusting corrupted feedback.

Validation should measure more than slip-classification accuracy. Important metrics include detection latency, false alarms, maximum object displacement before correction, minimum stable grip force, peak pressure, object damage rate, recovery success, and robustness across surface conditions. Tests should systematically vary mass, friction, acceleration, grasp location, and disturbances to establish the operating envelope of the tactile control system.

Tactile-based slip prevention transforms robotic grasping from an open-loop force-setting problem into continuous physical interaction regulation. By observing pressure distribution, shear, vibration, contact geometry, and temporal change, the robot can detect instability before visual displacement becomes significant. Combined with adaptive grip control, robot dynamics, and multimodal sensing, tactile feedback enables stable yet gentle manipulation across objects whose friction and mechanical properties cannot be known perfectly in advance.

06.06. GelSight DIGIT Tactile Sensor Integration [w/Code]

GelSight and DIGIT tactile sensors provide robots with high-resolution information about physical contact by converting deformation of a compliant surface into visual observations. Instead of measuring only a scalar force, vision-based tactile sensing captures spatial patterns associated with contact geometry, pressure, shear, texture, and motion. Integration into a robotic gripper therefore creates a local physical perception channel that complements cameras, force--torque sensors, and joint feedback.

The fundamental sensing principle uses a compliant elastomer positioned above an internal imaging system. When an object presses against the elastomer, the surface deforms according to local contact geometry and applied loading. Internal illumination makes these deformations visible to a camera. Image-processing or learned models can then convert the observed pattern into representations such as contact masks, surface geometry, displacement fields, shear estimates, or task-specific tactile features.

GelSight represents a broader family of optical tactile sensing techniques based on imaging deformation of an elastomeric contact surface. Different implementations may use reflective membranes, controlled illumination, multiple light directions, and photometric reconstruction to estimate detailed surface shape. The approach can reveal small geometric features that conventional robot vision cannot observe once the object is hidden inside a gripper or pressed against another surface.

DIGIT is a compact vision-based tactile sensor designed for integration with robotic fingers and manipulation research. Its small form factor combines an elastomeric sensing surface, internal illumination, and a camera in a fingertip-scale package. For robotic integration, the sensor behaves partly like a camera because it produces image streams, but the images describe local mechanical interaction rather than the external visual scene.

Mechanical integration strongly affects tactile performance. The sensor should be mounted so that expected grasp forces pass through the active elastomer region without introducing unwanted housing contact or structural interference. Finger geometry should expose sufficient sensing area while protecting the optical module from impacts beyond its intended operating range. Mount stiffness, sensor orientation, cable routing, and replaceability should be considered during gripper design.

The elastomer is both the physical contact interface and a critical sensing component. Its thickness, stiffness, surface coating, friction, durability, and optical properties influence the resulting tactile image. A softer layer can produce larger deformation and greater sensitivity but may saturate under high load or reduce spatial fidelity. A harder layer can support greater forces but may provide weaker signals for small contact variations.

Sensor placement should reflect the expected manipulation task. Flat fingertip mounting is suitable for parallel-jaw grasping, while angled or curved arrangements may provide better coverage for object edges or multi-finger hands. Two opposing tactile sensors allow comparison of contact patterns on both sides of an object. Additional sensors can be placed on fingertips, inner finger surfaces, or palm regions when manipulation involves rolling, regrasping, or complex contact transitions.

The raw tactile image normally requires calibration before it becomes useful for physical inference. Reference images acquired without contact can characterize illumination and membrane appearance. Background subtraction, normalization, geometric correction, and masking can remove fixed optical structure. Calibration should also account for sensor-to-sensor variation because nominally identical units may differ in illumination intensity, camera response, elastomer properties, or mounting conditions.

Contact detection is one of the simplest functions. Differences between the current image and the no-contact reference reveal regions where the elastomer has deformed. A contact mask can be generated using intensity differences, optical flow, learned segmentation, or reconstructed surface depth. The contact area and its centroid can then support grasp verification, object centering, and detection of asymmetric loading between opposing fingers.

Surface geometry reconstruction is a major advantage of GelSight-style sensing. Changes in reflected illumination can encode local surface orientation, enabling photometric methods or neural models to estimate a height map of the contacted object. This allows the robot to perceive fine edges, ridges, embossed features, texture, and local curvature through touch. Such information can support object recognition, pose refinement, insertion, and inspection.

Shear and tangential motion can be estimated by tracking features within the tactile image. Natural texture, artificial markers, or learned image correspondences can reveal how the elastomer stretches as the object applies lateral load. Spatial displacement fields provide richer information than a single tangential-force value because they show where the contact patch is sticking, deforming, rotating, or beginning to slip.

Slip detection is therefore an important integration target. During stable grasping, the contact pattern remains relatively consistent except for elastic deformation. Incipient slip may produce local motion at the boundary of the contact patch, while gross slip causes larger coherent displacement or vibration. Optical flow, marker motion, temporal image differences, or learned temporal models can classify these transitions and trigger corrective grip-force control.

Tactile images can also estimate object pose locally. Contact edges and geometric features provide constraints on the relative position and orientation between the finger and object. When an external camera provides an approximate pose, tactile sensing can refine the estimate after contact. This is particularly valuable when visual observations become occluded by the gripper or when assembly tolerances are smaller than the residual error of external perception.

For grasp control, tactile feedback can regulate both force and geometry. The gripper can initially close until sufficient contact area is detected rather than stopping at a predetermined finger position. If pressure-related deformation becomes excessive, closing can stop to protect the object. If contact is asymmetric, finger position or wrist pose can be adjusted. During lifting, detected shear or slip can trigger an incremental increase in gripping force.

Force estimation from vision-based tactile sensing is possible but requires calibration or learned mappings between image deformation and applied load. Normal force may correlate with contact area, deformation magnitude, or reconstructed indentation, while shear force relates to tangential displacement. Because elastomer mechanics are nonlinear and history dependent, force estimation should be validated over the expected range of objects, loading rates, and contact geometries.

Image-stream processing introduces computational requirements different from conventional scalar tactile sensors. Camera acquisition, image rectification, contact segmentation, optical flow, neural inference, and temporal filtering may need to execute at interactive or control rates. GPU acceleration can support dense vision algorithms, but latency must remain bounded. A highly accurate tactile model is of limited value if its output arrives too late for slip prevention or contact control.

Data transport should preserve timestamps because tactile images must often be synchronized with robot joint states, gripper position, wrist force--torque measurements, and external cameras. USB or other camera interfaces can introduce buffering and variable latency. The software architecture should record acquisition timestamps as close to the sensor as possible and monitor dropped frames. Synchronization becomes especially important when tactile data are used for learning dynamic manipulation policies.

In ROS 2 or a similar middleware architecture, the tactile sensor can publish raw images while separate processing nodes generate contact masks, deformation fields, slip probabilities, geometric features, or estimated forces. Camera calibration and sensor metadata should be maintained independently from task logic. This modular structure allows tactile algorithms to be replaced or upgraded without redesigning the complete manipulation controller.

Multi-sensor fusion provides a broader understanding of physical interaction. External vision supplies global object pose and scene context, wrist force--torque sensing measures the net interaction wrench, and GelSight or DIGIT provides high-resolution local contact information. Joint states describe robot configuration, while gripper encoders provide finger displacement. Together, these signals can distinguish object motion, finger deformation, environmental contact, and grasp instability.

Assembly tasks can benefit from tactile geometry and shear information. During connector mating or peg insertion, fingertip sensors can detect edge contact, local rotation, asymmetric pressure, or small relative motion that is difficult to infer from wrist force alone. Tactile feedback can therefore complement compliant control by indicating where and how contact occurs, enabling more targeted alignment corrections and reducing unnecessary search motion.

Learning-based manipulation is a natural application because tactile sensors produce information-rich images compatible with modern representation-learning methods. Convolutional encoders or vision transformers can map tactile images into compact latent representations. These features can be fused with visual and proprioceptive observations for imitation learning, reinforcement learning, or multimodal policies that learn grasp stabilization, insertion, reorientation, and material-sensitive manipulation.

Training tactile models requires diverse data. Object geometry, material, texture, grasp force, contact location, illumination variation, elastomer aging, and sensor replacement can all alter image appearance. Data augmentation can address some optical variation, while domain randomization may broaden model robustness. Nevertheless, real contact datasets remain important because detailed elastomer deformation, friction, wear, and surface interaction are difficult to simulate faithfully.

Sensor aging must be considered in long-term deployment. Repeated contact can scratch, contaminate, polish, stretch, or permanently deform the elastomer surface. These changes alter the reference image and may degrade models trained on new sensors. Periodic reference updates, recalibration, replaceable sensing skins, health monitoring, and maintenance schedules help preserve measurement consistency across extended operation.

Contamination is particularly important in industrial environments. Dust, oil, metal particles, adhesive residue, and moisture can modify both optical appearance and contact friction. Protective design should avoid obstructing the active tactile surface while making cleaning practical. Algorithms can monitor background-image changes or unusual spatial patterns to detect contamination and request maintenance before sensing quality deteriorates significantly.

Fault detection should distinguish real contact changes from optical or electronic problems. Frozen frames, dropped images, illumination failure, camera exposure changes, damaged elastomer, and communication faults can produce outputs that resemble contact events. Sensor-health monitoring can evaluate frame timing, image statistics, reference consistency, and agreement with gripper or force--torque measurements before tactile estimates are accepted for control.

Validation should cover sensing accuracy and manipulation performance. Contact localization error, geometry reconstruction accuracy, slip-detection latency, force-estimation error, frame rate, processing latency, and repeatability provide sensor-level metrics. Task-level tests should evaluate grasp success, minimum stable force, object damage, insertion success, recovery behavior, and robustness across objects. Testing multiple sensor units also reveals manufacturing and calibration variability.

A robust GelSight or DIGIT integration therefore treats vision-based tactile sensing as part of the robot\'s physical perception architecture rather than as an isolated camera mounted on a finger. Mechanical design, elastomer behavior, optical calibration, image processing, synchronization, state estimation, control, learning, and maintenance must be considered together. When properly integrated, high-resolution tactile images allow robots to perceive contact geometry and dynamics that remain invisible to conventional external perception.

06.07. Learning Force Policies from Human Demonstration [w/Code]

Learning force policies from human demonstration provides a practical way to teach robots contact-rich manipulation behaviors that are difficult to specify using geometric trajectories alone. A human demonstrator naturally adjusts force, motion, orientation, and compliance while interacting with an uncertain environment. By recording these coordinated behaviors, a robot can learn not only where to move but also how physical interaction should evolve during task execution.

A demonstration should capture both kinematic and interaction information. Typical observations include end-effector pose, linear and angular velocity, joint configuration, gripper state, force--torque measurements, and task context. Vision or tactile sensing can additionally describe objects and local contact geometry. Time synchronization is essential because a force response has meaning only when associated with the robot motion and contact condition that produced it.

Demonstrations can be collected through kinesthetic teaching, teleoperation, motion capture, instrumented tools, or bilateral control devices. In kinesthetic teaching, an operator physically guides a compliant robot through the task while force and motion are recorded. Teleoperation allows demonstrations in environments where direct guidance is inconvenient, while force-feedback interfaces can preserve information about the interaction experienced by the human operator.

Raw human demonstrations should not be treated as perfect control references. Human motion contains hesitation, corrections, tremor, variable speed, and inconsistent force application. Sensor measurements also contain bias, noise, gravity effects, and dynamic loads. Preprocessing therefore includes coordinate transformation, force--torque compensation, filtering, temporal alignment, trajectory segmentation, and normalization so that the learning system observes meaningful task behavior rather than recording artifacts.

Task segmentation separates a demonstration into interaction phases with different physical objectives. An assembly example may contain approach, first contact, search, alignment, insertion, seating, and release. The desired force behavior during each phase differs substantially. Learning one undifferentiated mapping across the complete trajectory can obscure this structure, whereas phase-aware learning allows policies to associate particular force patterns with specific mechanical states.

Force demonstrations should normally be represented in a task-relevant coordinate frame. For surface following, the normal direction determines contact force while tangential directions describe motion. For insertion, the insertion axis and lateral alignment directions are more meaningful than fixed world coordinates. Task-frame representations improve transfer because the learned relationship can remain valid when the object or fixture changes position and orientation.

Imitation learning can formulate the demonstrated behavior as a mapping from observations to robot actions. The observation may contain pose error, velocity, measured wrench, tactile features, visual information, and estimated contact state. Actions may be Cartesian velocity, pose increment, force target, joint torque, impedance parameters, or combinations of these quantities. The choice of action representation strongly influences stability, generalization, and compatibility with the robot controller.

Behavior cloning is the simplest learning approach. A supervised model is trained to reproduce demonstrated actions from corresponding observations. It is straightforward and can work well when demonstrations densely cover the states encountered during execution. However, small prediction errors can move the robot into states absent from the training data, causing errors to accumulate. This distribution shift is particularly dangerous during contact because incorrect motion can rapidly generate large forces.

Corrective demonstration methods can reduce distribution shift. After an initial policy is trained, the robot executes the task while the human provides corrections or additional demonstrations in states where the policy performs poorly. The expanded dataset contains recovery behavior and difficult contact configurations rather than only ideal executions. Iterative data collection therefore teaches the robot how to return toward successful interaction when small deviations occur.

Probabilistic movement representations provide another approach for learning variable demonstrations. Instead of reproducing one exact trajectory, the model captures a distribution over demonstrated motion and force profiles. Regions with low demonstration variance can be interpreted as strongly constrained, while regions with larger variance allow flexibility. Such representations are useful for contact tasks because humans often demonstrate precise force behavior in constrained directions while permitting substantial variation elsewhere.

Dynamic movement primitives can encode demonstrated trajectories using stable dynamical systems whose parameters are learned from examples. Force or coupling terms can be added so that execution adapts when measured interaction differs from the nominal demonstration. This provides a structured alternative to unrestricted neural policies. The resulting behavior retains convergence properties while allowing contact feedback to modify the trajectory during execution.

Force policy learning is closely related to impedance learning because humans rarely regulate force independently of motion. A demonstrator may become compliant while searching for a hole, increase directional stiffness during insertion, and stiffen again after seating. Instead of predicting raw force commands, a policy can learn desired stiffness, damping, equilibrium pose, or compliance direction as functions of task state, creating behavior that more closely reflects human physical interaction.

Variable impedance policies are particularly useful when environmental uncertainty changes during a task. Free-space motion can use higher stiffness for accurate tracking, initial contact can reduce stiffness to limit impact, and constrained manipulation can assign different stiffness values to different axes. Human demonstrations provide evidence about when these transitions should occur, while the low-level impedance controller preserves a structured mechanical response.

Learning residual force behavior can improve safety and data efficiency. A conventional controller provides nominal motion, force limits, and stable compliance, while the learned policy predicts only a correction to the reference trajectory, target force, or impedance parameters. Because the learning system does not control the complete robot dynamics directly, its authority can be bounded. This architecture combines known control principles with demonstration-derived adaptation.

Contact-state estimation can serve as an intermediate representation between sensing and policy execution. Rather than requiring a neural policy to infer every interaction condition implicitly, a dedicated estimator can identify free motion, contact, sliding, alignment, insertion, jamming, or seating. The learned policy then selects appropriate force and motion behavior according to the estimated state, improving interpretability and simplifying validation.

Tactile sensing enriches demonstration learning by capturing local interaction information unavailable from a wrist force--torque sensor. Pressure distribution, shear, contact area, deformation, and slip can reveal how the demonstrator maintains stable grasping or aligns components. Vision-based tactile sensors can provide dense representations that are fused with wrench and proprioceptive observations, enabling policies to associate local contact geometry with corrective action.

Multimodal learning is important because force signals alone are often ambiguous. The same axial force can correspond to normal insertion, collision, or final seating depending on object pose and task progress. Vision provides global geometry, tactile sensing provides local contact, force--torque sensing provides the net interaction wrench, and proprioception describes robot configuration. Combining these modalities allows the policy to infer the physical context behind similar force measurements.

Sequence models are well suited to demonstrated contact behavior because the correct action often depends on recent interaction history. A single force measurement may not reveal whether force is increasing, stable, or recovering from impact. Recurrent neural networks, temporal convolutional networks, and transformer-based architectures can process observation histories and learn temporal relationships between contact transitions, human corrections, and resulting robot actions.

Dataset diversity determines how well the learned policy generalizes. Demonstrations should include variation in object pose, geometry, friction, stiffness, payload, approach error, insertion tolerance, and disturbance. Repeating only one successful nominal trajectory encourages memorization rather than interaction understanding. Purposefully demonstrating recoverable errors can be especially valuable because deployment failures frequently occur outside the narrow distribution of ideal executions.

Demonstration quality can be weighted rather than treating every trial equally. Successful, smooth, low-force demonstrations may receive greater importance, while inefficient or unsafe examples can be rejected or assigned lower weight. However, difficult recovery demonstrations should not automatically be considered poor quality. Dataset curation should distinguish undesirable behavior from valuable examples of how an expert responds to unexpected contact conditions.

Learning from multiple demonstrators introduces both diversity and inconsistency. Different operators may solve the same task using different force levels, speeds, or search strategies. This variation can improve robustness when it reflects valid alternatives, but it can also confuse the policy if demonstrations represent incompatible behaviors. Conditioning on strategy, task parameters, or operator style can help separate meaningful modes instead of averaging them into an ineffective compromise.

Force normalization may be necessary when transferring demonstrations between robots or tools. Differences in payload, gripper geometry, sensor placement, actuator capability, and mechanical stiffness mean that identical numerical force values may not represent equivalent behavior. Representations based on task geometry, normalized load, force ratios, or estimated contact conditions can improve portability compared with directly replaying recorded wrench trajectories.

Sim-to-real methods can extend demonstration datasets by generating controlled variations around human examples. Simulation can perturb object pose, friction, stiffness, sensor noise, latency, and geometric tolerance while preserving the basic demonstrated strategy. The resulting data help expose the policy to conditions not manually demonstrated. Nevertheless, real contact data remain necessary because friction, compliance, backlash, and surface effects are difficult to simulate accurately.

Reinforcement learning can refine a policy initialized from demonstrations. Human examples provide a useful starting distribution, reducing the amount of unsafe or inefficient exploration required. Reinforcement learning can then optimize success rate, cycle time, force peaks, energy, or recovery behavior. Reward design should penalize excessive contact forces and unsafe states so that improved task performance does not emerge at the expense of hardware or object safety.

Offline reinforcement learning offers an alternative when physical exploration is undesirable. The policy is optimized from a fixed dataset of demonstrations and previously collected robot interactions without repeatedly experimenting on hardware. This can be attractive for industrial contact tasks, but the algorithm must avoid assigning excessive value to actions poorly represented in the dataset. Conservative methods are therefore important when extrapolation could produce unsafe forces.

Policy uncertainty should influence execution. When observations lie far from demonstrated conditions or the model produces low-confidence actions, the robot can reduce speed, increase compliance, return to a known state, request human assistance, or terminate the attempt. Uncertainty-aware control prevents the system from treating unfamiliar contact situations with the same authority as well-supported behavior and provides a practical boundary around learned autonomy.

Safety supervision should remain independent of the learned policy. Hard limits on force, torque, velocity, workspace, joint motion, and gripper pressure should be enforced by deterministic mechanisms whenever possible. Watchdogs should detect sensor failure, communication loss, abnormal timing, and persistent high force. A learned policy may optimize interaction behavior, but it should not be responsible for deciding whether fundamental hardware safety limits can be exceeded.

Real-time deployment requires predictable inference and communication latency. Force-sensitive manipulation can become unstable if observations are delayed or actions arrive irregularly. Sensor acquisition, preprocessing, multimodal fusion, policy inference, and command transmission should therefore be measured as an end-to-end loop. Computationally heavy models may require GPU acceleration, model compression, or separation between fast low-level control and slower learned adaptation.

Validation should evaluate more than imitation accuracy against recorded human actions. A policy can reproduce demonstrations numerically yet fail under small physical disturbances. Evaluation should measure task success, peak force and torque, cycle time, contact stability, recovery rate, object damage, sensitivity to pose error, and performance on previously unseen conditions. Comparison with conventional controllers also reveals whether learning provides a meaningful operational advantage.

Systematic perturbation testing is particularly important for learned force policies. Object position, orientation, friction, stiffness, mass, clearance, sensor noise, and external disturbances can be varied across controlled trials. The resulting performance envelope identifies where the policy remains reliable and where uncertainty or failure increases. This provides stronger evidence of robustness than demonstrating repeated success under a single nominal condition.

Learning force policies from human demonstration ultimately transfers physical interaction strategy rather than merely copying motion. Human examples reveal when to establish contact, how strongly to push, when to become compliant, how to react to resistance, and how to recover from imperfect alignment. When demonstration learning is combined with multimodal sensing, structured compliant control, uncertainty estimation, and deterministic safety supervision, robots can acquire adaptable contact skills while retaining a controlled physical interaction framework.

06.08. Whole Arm Contact Awareness for Safety [w/Code]

Whole-arm contact awareness enables a manipulator to detect and interpret physical interaction occurring anywhere along its structure rather than only at the end effector. This capability is important when robots operate near people, equipment, fixtures, or cluttered environments where unexpected contact may occur at the links, joints, elbow, wrist, or tool. Contact awareness transforms collision detection from a simple emergency trigger into a distributed physical perception function.

Traditional manipulation systems often concentrate force sensing at the wrist because the primary interaction is expected to occur through the tool. A wrist force--torque sensor provides accurate information about tool contact but may not reliably detect forces applied upstream of the sensor. Whole-arm awareness therefore requires additional information from joint torque sensors, motor currents, robot dynamics, tactile skins, link sensors, or combinations of these sources.

Joint torque sensing provides a useful foundation for detecting external disturbances throughout the robot. Measured joint torques contain contributions from gravity, inertia, Coriolis effects, friction, commanded actuation, and external contact. If the robot model predicts the internal contributions, the difference between measured and predicted torque can be interpreted as a residual associated with unmodeled interaction. Significant residuals can indicate that an external force has acted on the mechanism.

Model accuracy strongly influences residual-based contact detection. Incorrect payload parameters, center-of-mass estimates, friction models, actuator dynamics, or joint calibration can create residual torques even when no external contact exists. Dynamic motion increases this challenge because acceleration and transmission effects become more important. Reliable systems therefore combine accurate identification, state estimation, filtering, and adaptive thresholds rather than assuming every torque-model mismatch represents collision.

Motor current can provide an indirect estimate of joint torque when dedicated torque sensors are unavailable. Current-based detection can make whole-arm awareness feasible on conventional industrial manipulators, but gearbox friction, temperature, motor efficiency, backlash, and transmission elasticity reduce accuracy. Such signals are often most useful for detecting relatively large disturbances or supporting other sensors rather than measuring precise contact forces independently.

Momentum observers offer another model-based approach to external interaction estimation. Instead of directly differentiating noisy measurements, an observer uses robot dynamics and momentum relationships to estimate generalized external forces. The resulting residual can respond to collisions while remaining computationally suitable for real-time control. Observer bandwidth must balance rapid detection against sensitivity to model error, sensor noise, and normal dynamic motion.

Detecting that contact occurred is only the first level of awareness. For useful physical interaction, the robot should also estimate which link was contacted, where the contact occurred, what direction the force acted, and how severe the interaction is. Contact localization allows the controller to generate a response appropriate to the actual collision geometry rather than treating every external disturbance as if it occurred at the end effector.

The relationship between a Cartesian contact force and measured joint torque is described through the contact Jacobian. A force applied to a point on a robot link produces generalized torque according to the local kinematic configuration. If the contacted link and approximate contact point are known, the external force can be estimated from torque residuals. Conversely, identifying an unknown contact location from joint torques is more difficult because several contact configurations can produce similar generalized effects.

Geometric reasoning can narrow the localization problem. Candidate contact points can be distributed over the surfaces of robot links, and predicted torque signatures can be compared with observed residuals. Optimization or probabilistic inference can identify locations most consistent with the measurements. Collision geometry, surface normals, environmental maps, and known obstacles provide additional constraints that improve localization and reject physically implausible solutions.

Tactile skin provides direct distributed contact sensing across robot surfaces. Flexible arrays containing pressure, capacitive, piezoresistive, optical, or other sensing elements can be attached to links and joints. Unlike model-based observers, tactile skin measures local physical interaction directly and can indicate the region, magnitude, and spatial distribution of contact. This is valuable for detecting light touches that may be difficult to distinguish from model uncertainty.

Covering an entire manipulator with tactile sensors introduces significant engineering challenges. Sensors must tolerate repeated impact, bending, abrasion, cleaning, cable motion, electromagnetic interference, and mechanical servicing. Joint regions require flexible routing and must not restrict motion. Large numbers of sensing elements also increase wiring, communication bandwidth, calibration effort, and failure points, making modular skin architectures attractive for maintainability.

Sensor fusion can combine the strengths of distributed tactile sensing and model-based estimation. Joint torque residuals provide coverage even where tactile skin is absent, while skin sensors provide direct localization and sensitivity to low-force contact. Wrist force--torque sensing describes tool interaction, and vision supplies environmental context. Agreement among these channels can increase confidence that an observed disturbance represents real physical contact.

Contact classification should distinguish expected interaction from unexpected collision. A manipulator performing insertion, polishing, or force-guided assembly intentionally experiences external forces, while contact on the elbow may be abnormal. The system therefore needs task context and expected-contact models. Contact location, direction, magnitude, timing, and current task phase can determine whether the event should be accepted, regulated, investigated, or treated as a safety-related condition.

Threshold selection is critical for reliable detection. A very low threshold detects gentle contact quickly but can generate false alarms from model error, vibration, or rapid acceleration. A high threshold reduces nuisance triggers but may allow excessive collision force before detection. Adaptive thresholds can vary with robot speed, configuration, payload, task state, expected contact, and model uncertainty, providing greater sensitivity when the robot is moving slowly near people.

Temporal reasoning improves robustness because physical collision produces characteristic changes over time. Short impulses, sustained loads, gradually increasing constraints, and oscillatory contacts should not necessarily produce identical responses. Persistence tests, derivative features, impulse estimates, and temporal classifiers can distinguish isolated sensor noise from meaningful interaction. The duration and evolution of contact can also help determine whether an obstacle remains trapped against the robot.

Contact severity can be evaluated using more than peak force. Estimated impulse, energy transfer, relative velocity, effective robot mass, contact duration, and body location all influence the consequences of collision. A low force sustained for a long period may indicate pinching, while a larger short transient may represent a brief impact. Safety-aware logic should therefore interpret multiple physical quantities rather than relying on one universal threshold.

Effective mass is configuration and direction dependent. The same joint motion can produce different apparent inertia at different points on the robot, and a collision near the base differs mechanically from one near the wrist. Whole-arm safety analysis should account for these variations when setting velocity or force limits. Dynamic models can estimate directional effective mass and support configuration-aware motion planning or collision response.

Once unexpected contact is detected, the controller must choose an appropriate reaction. Possible responses include stopping commanded motion, reducing torque, entering gravity compensation, increasing compliance, retracting along the estimated contact normal, or moving away from the contact location. The safest behavior depends on whether contact is transient, sustained, trapped, intentional, or associated with a person or fixed structure.

Immediate stopping is not always the optimal response. If the robot rigidly freezes while pressing against a person or object, contact force may remain high. A controlled compliant response can reduce stored mechanical energy and unload the contact. However, uncontrolled retreat may create a secondary collision. Reaction trajectories should therefore be constrained by the environment, joint limits, nearby obstacles, and confidence in the estimated contact direction.

Whole-arm impedance control can make the manipulator mechanically responsive to distributed disturbances. Joint-level torque control or Cartesian objectives can be combined with null-space behavior so that external forces applied to the links produce bounded motion. Low stiffness improves compliance, while sufficient damping prevents oscillation. Task performance and collision response must be balanced because an excessively soft arm may be unable to execute precise manipulation.

Null-space control is particularly useful when the end effector should continue satisfying a task while the rest of the arm avoids contact. For redundant manipulators, link-level collision information can generate a secondary objective that moves the elbow or intermediate links away from an obstacle without significantly disturbing the tool pose. This creates a form of contact-aware redundancy resolution that extends avoidance behavior beyond purely visual proximity sensing.

Pre-contact sensing complements physical contact awareness. Cameras, depth sensors, proximity sensors, and environmental models can predict potential collisions before they occur. Contact sensing remains necessary because perception can be occluded, calibration can be imperfect, and objects can move unexpectedly. Combining proximity information with contact detection creates layered protection in which the robot first avoids, then compliantly responds if physical interaction still occurs.

Human--robot collaboration makes whole-arm awareness especially important because a person may touch any part of the manipulator. Light intentional touch can also be used as an interaction command, such as guiding the elbow or requesting motion adjustment. The system must therefore distinguish cooperative physical guidance from hazardous collision. Contact pattern, location, duration, task mode, and additional human-state information can help determine the intended meaning.

Whole-arm contact awareness contributes to safety but does not by itself constitute a complete safety system. Safety-rated functions may require certified hardware, monitored speed, separation constraints, emergency stopping, power and force limiting, or other measures defined by the applicable robot and application standards. Research-level collision estimators and tactile sensors should not automatically be assumed to satisfy functional-safety requirements.

Redundant safety channels are valuable because contact sensors can fail. Torque sensors may drift, tactile elements may become damaged, motor-current estimates may change with temperature, and communication can be interrupted. Health monitoring should continuously check range, bias, continuity, timing, and agreement between sensing modalities. Loss of confidence should trigger a predefined degraded mode rather than silently removing collision awareness.

Real-time performance determines how much collision force develops before the robot responds. Sensor acquisition, dynamics computation, residual estimation, contact classification, decision logic, and actuator commands all contribute to detection and reaction latency. Fast algorithms are useful, but deterministic timing is equally important. A system with occasional long delays can be less safe than one with slightly slower but tightly bounded response.

Logging high-rate contact data supports both validation and future improvement. Torque residuals, tactile maps, robot states, commands, contact classifications, and response actions should be recorded with synchronized timestamps. These logs allow engineers to reconstruct collision events, identify false detections, refine models, and build datasets for learned contact estimators. They also provide evidence for evaluating changes to control or hardware.

Learning-based estimators can improve contact localization and classification when analytical models are insufficient. Neural networks can map joint torque residuals, tactile measurements, robot configuration, and temporal features to contact location or event type. Synthetic collisions in simulation can provide broad initial coverage, while controlled hardware experiments refine the model. Learned estimators should nevertheless expose uncertainty because rare collision configurations may lie outside training data.

Validation requires systematic contact testing across the robot body. Controlled forces can be applied at different links, positions, directions, speeds, and robot configurations while measuring detection probability, localization error, force-estimation error, and latency. Tests should also include normal robot motion without collision to measure false-positive rates. A detector that responds correctly to impacts but repeatedly interrupts ordinary operation is not practically robust.

Operational evaluation should measure the complete response rather than only detection performance. Important quantities include maximum contact force, impulse, time to detection, time to actuator response, unloading time, residual force after reaction, unintended end-effector displacement, and recovery success. Testing should cover stationary contact, moving impact, sustained pushing, pinching scenarios, intentional tool contact, and unexpected link contact.

Whole-arm contact awareness ultimately gives a manipulator a distributed sense of its own physical interaction with the world. By combining joint torque residuals, dynamics observers, motor information, tactile skin, wrist force sensing, geometric reasoning, and task context, the robot can determine when and where contact occurs and select an appropriate response. Integrated with compliant control and independent safety mechanisms, this capability supports robots that interact more safely and intelligently in shared, uncertain physical environments.

06.09. Force Tactile Data Logging and Analysis Pipeline [w/Code]

A force--tactile data logging and analysis pipeline provides the infrastructure required to transform raw physical interaction measurements into synchronized, traceable, and reusable engineering data. Manipulation experiments may produce wrist force--torque signals, joint torques, tactile arrays, tactile images, gripper states, robot poses, and task events simultaneously. A structured pipeline preserves these heterogeneous streams so contact behavior can be reconstructed and analyzed after execution.

Pipeline design begins by defining the data sources and their physical meaning. Force--torque sensors typically provide six-axis wrench measurements, while tactile devices may produce pressure arrays, shear estimates, deformation fields, vibration signals, or high-resolution images. Robot controllers contribute joint positions, velocities, torques, Cartesian poses, and command references. Recording the relevant control and perception states is essential because isolated force measurements provide limited information about why interaction occurred.

Each stream should contain explicit metadata describing the sensor, coordinate frame, units, sampling frequency, calibration state, and acquisition configuration. A numerical value is difficult to interpret reliably if engineers cannot determine whether force is expressed in the sensor frame, tool frame, or task frame. Metadata should also identify sensor serial numbers, firmware versions, tactile skin configuration, robot model, tool geometry, payload parameters, and experiment software versions when reproducibility is important.

Time synchronization is one of the most critical requirements. Force sensors may operate at hundreds or thousands of samples per second, tactile cameras at tens or hundreds of frames per second, and robot state feedback at another frequency. Without a common temporal reference, contact events cannot be aligned accurately. Hardware timestamps, synchronized clocks, or carefully managed middleware timestamps should therefore be used to establish a consistent experiment timeline.

Clock offset and drift must be considered during long experiments. Two devices may appear synchronized initially but gradually diverge because their local oscillators run at slightly different rates. Periodic synchronization, Precision Time Protocol, hardware triggering, or post-processing alignment can reduce this problem. Synchronization accuracy should be treated as a measurable system property because temporal error directly affects force--motion correlation and dynamic contact analysis.

The logging architecture should separate acquisition from downstream analysis. Sensor interfaces acquire raw measurements and attach timestamps, while a dedicated recorder stores the streams without requiring complex interpretation during the experiment. This reduces the risk that computationally expensive visualization or inference causes dropped samples. Online monitoring can still display key signals, but archival logging should prioritize deterministic acquisition and data integrity.

Raw data should normally be preserved whenever storage capacity permits. Filtering, coordinate transformation, contact segmentation, or feature extraction can be repeated later, but information removed before recording cannot be recovered. Maintaining an immutable raw layer also allows improved algorithms to be applied to historical experiments. Processed datasets should therefore reference the raw source rather than replacing it, creating a traceable relationship between acquisition and analysis products.

High-rate force and tactile data can generate substantial storage requirements. Tactile images are particularly demanding because each finger may produce continuous video-like streams, while dense tactile arrays can contain hundreds or thousands of channels. Compression can reduce storage, but the method should preserve the information required for later analysis. Lossless compression is preferable for quantitative measurements when subtle contact patterns may be important.

Data should be organized into clearly identifiable experiment sessions. Each session can contain sensor streams, robot states, commands, calibration records, task parameters, environmental conditions, and operator annotations. Unique experiment identifiers prevent ambiguity when multiple trials use similar filenames. A manifest can summarize the contents and provide links between raw files, processed results, configuration information, and evaluation metrics.

Event markers greatly improve later interpretation. The system can record events such as task start, first contact, grasp closure, lift, insertion, detected slip, collision, recovery, seating, release, success, and failure. These markers may originate from the controller, contact estimator, human operator, or post-processing. Event timelines allow engineers to navigate large datasets without manually searching every force trace or tactile recording.

Calibration information should be versioned alongside experimental data. Force--torque bias, payload compensation parameters, tactile reference images, taxel sensitivity maps, camera calibration, coordinate transformations, and temperature corrections may change over time. If calibration files are stored separately without version references, historical measurements may become impossible to reproduce accurately. Every dataset should therefore identify the calibration state used during acquisition and processing.

Data integrity checks should occur during and after logging. Sequence numbers can reveal missing packets, timestamp analysis can identify timing gaps, and expected sample counts can expose dropped data. Range checks can detect sensor saturation or invalid values, while file hashes can verify that archived datasets have not changed. Logging software should report these problems explicitly instead of silently producing incomplete experimental records.

Force--torque preprocessing generally begins with bias correction and compensation for known mechanical effects. Tool weight, payload center of mass, gravity orientation, inertial loads, and sensor-frame transformations may need to be considered before interpreting measured wrench as environmental interaction. Both raw and compensated wrench signals should be retained when possible so that compensation algorithms can later be audited or improved.

Tactile preprocessing depends on sensor type. Pressure arrays may require baseline subtraction, gain normalization, dead-taxel detection, and spatial filtering. Vision-based tactile sensors may require reference-image subtraction, geometric rectification, illumination normalization, contact segmentation, or optical-flow computation. Processing parameters should be stored with the output because changes in thresholds or filters can significantly alter derived contact features.

Coordinate normalization enables meaningful comparison between trials. Forces expressed in a moving sensor frame may appear different even when the physical interaction relative to the task is identical. Transforming wrench, contact direction, and robot motion into a task frame can align demonstrations across object poses. Original coordinates should still be preserved so transformations remain reversible and errors in frame definitions can be diagnosed.

Resampling is often required when combining sensors operating at different frequencies. A common timeline can be constructed and continuous signals interpolated where appropriate, while image or event data may use nearest-neighbor association or explicit timestamp lookup. Resampling should not create false precision. The original timestamps and native sample rates should remain available because high-frequency impact analysis may require access to the unmodified data.

Feature extraction converts synchronized signals into representations suitable for engineering analysis or machine learning. Force features can include magnitude, direction, derivative, impulse, spectral energy, and force--torque ratios. Tactile features may include contact area, pressure centroid, peak pressure, shear, deformation, optical flow, slip indicators, and spatial moments. Robot features can describe velocity, acceleration, pose error, joint configuration, or commanded impedance.

Sliding-window analysis captures temporal structure around contact events. Short windows can characterize impact transients and vibration, while longer windows reveal sustained loading, gradual slip, or force buildup during jamming. Multi-scale features are useful because physical interaction occurs across different time ranges. Every feature should retain its source interval and timestamp so that derived values can be traced back to the underlying measurements.

Frequency-domain analysis is valuable for vibration-sensitive contact phenomena. Fourier transforms, power spectral density, or wavelet methods can reveal chatter, stick--slip behavior, surface texture interactions, or mechanical resonance. Care is required when selecting window size, overlap, and sampling frequency because these parameters determine frequency resolution and detection latency. Frequency features should be interpreted together with task state rather than as isolated signatures.

Visualization provides the first practical layer of analysis. Time-aligned plots of force, torque, tactile pressure, gripper position, and robot motion can reveal relationships that are difficult to detect in individual streams. Tactile heat maps or image sequences can be displayed beside wrench traces and robot state. Interactive event navigation is especially useful for examining short intervals around slip, collision, insertion failure, or successful seating.

Contact-state labeling transforms continuous sensor data into semantically meaningful intervals. Labels may include free motion, first contact, stable contact, sliding, rolling, alignment, insertion, jamming, seating, release, or abnormal collision. Labels can be generated manually, automatically, or through a hybrid workflow in which an estimator proposes states and a human reviewer corrects them. Label provenance should be recorded so analysts know how ground truth was established.

Dataset quality depends on consistent labeling rules. Mechanical definitions should specify what constitutes initial contact, slip, jamming, successful insertion, or other states. Without clear criteria, different operators may label identical events differently. Ambiguous transition periods can be represented with uncertainty intervals instead of forcing an artificial exact boundary. This improves the reliability of datasets used for training and benchmarking contact estimators.

Statistical analysis can compare repeated trials and quantify interaction variability. Mean force profiles, confidence intervals, peak loads, contact durations, slip frequency, insertion force, and recovery time can be evaluated across objects or task conditions. Grouping results by material, payload, speed, controller parameters, or environmental conditions helps identify which variables influence performance and where the manipulation system becomes sensitive to uncertainty.

Failure analysis should connect sensor evidence with task outcomes. Excessive lateral force may precede insertion failure, asymmetric tactile pressure may indicate poor grasp alignment, and increasing torque without displacement may indicate jamming. By examining these patterns across many trials, engineers can identify recurring failure precursors. Such findings can then inform controller thresholds, recovery logic, mechanical design, or new perception features.

Machine-learning datasets can be generated from the same logging pipeline. Synchronized observation windows can be paired with contact-state labels, slip events, force targets, corrective actions, or task outcomes. Dataset partitions should prevent nearly identical consecutive samples or repeated trials from leaking between training and test sets. Splitting by object, session, operator, or environmental condition provides a more meaningful measure of generalization.

Dataset versioning is necessary as recordings, labels, and processing algorithms evolve. A dataset release should identify which raw sessions are included, which preprocessing pipeline was used, how labels were produced, and which samples were excluded. Derived features and model-ready tensors should be reproducible from versioned raw data and configuration files. This converts experimental recordings into a maintainable engineering asset rather than a collection of disconnected files.

Automated analysis can produce standardized reports after each experiment. The pipeline may calculate synchronization quality, missing samples, force peaks, contact duration, tactile coverage, slip events, success status, and processing warnings. Consistent reports make abnormal trials easier to identify and reduce dependence on manual inspection. They also support large-scale experimentation where thousands of manipulation episodes must be evaluated.

Real-time analytics can operate alongside archival processing when immediate feedback is required. Lightweight modules may estimate contact state, slip probability, force limits, or sensor health during execution, while more computationally expensive analysis runs offline. Online and offline algorithms should share consistent definitions and calibration whenever possible so that a detected event during operation corresponds to the same physical interpretation used in later evaluation.

Long-term storage should preserve both accessibility and traceability. Frequently used datasets may remain on fast local or network storage, while completed experiments can be transferred to archival systems. Checksums, manifests, redundant copies, and documented directory structures reduce the risk of silent corruption or loss. Sensitive or proprietary experimental data may additionally require access control, encryption, and retention policies.

A well-designed force--tactile data pipeline ultimately creates a physical-interaction evidence base for manipulation development. By combining synchronized acquisition, immutable raw data, calibration tracking, event annotation, preprocessing, feature extraction, visualization, statistical analysis, and dataset versioning, engineers can reconstruct why a robot succeeded or failed. The same infrastructure supports controller tuning, safety validation, tactile learning, contact-state estimation, and scalable Physical AI dataset development.

06.10. Contact Rich Manipulation Case Polishing Wiping

Polishing and wiping are representative contact-rich manipulation tasks because successful execution depends on continuously coordinating motion with physical interaction. Unlike free-space trajectory tracking, the robot must maintain controlled contact while moving across a surface whose position, curvature, friction, and compliance may be uncertain. The objective is therefore not simply to follow a geometric path, but to regulate the evolving relationship among tool pose, contact force, surface condition, and motion.

A typical robotic polishing system consists of a manipulator, polishing tool, force--torque sensor, workpiece fixture, and perception system. Wiping may use a sponge, cloth, pad, brush, or specialized cleaning tool. The tool itself introduces compliance, friction, deformation, and wear. These properties become part of the manipulation dynamics and must be considered together with robot stiffness, controller bandwidth, surface geometry, and desired process force.

Before contact, the robot requires an estimate of the workpiece surface and target region. CAD geometry, calibrated fixtures, RGB-D cameras, 3D scanners, or previously generated maps can provide an initial surface model. This information defines the nominal tool path and surface normal. However, perception and calibration errors mean that the actual contact surface rarely matches the nominal model exactly, making force feedback essential during execution.

The approach phase should transition smoothly from free-space motion to physical contact. Moving rapidly toward a rigid surface can generate a large impact force before the controller reacts. A practical strategy reduces approach velocity near the expected surface and detects initial contact using force--torque measurements, joint torque residuals, tactile sensing, or tool displacement. Once contact is confirmed, the controller transitions from position-dominant motion to compliant contact control.

Surface-relative coordinates simplify the control problem. The local surface normal defines the direction in which contact force should be regulated, while tangent directions define polishing or wiping motion. Instead of controlling all Cartesian axes identically, the robot can regulate normal force while tracking tangential velocity or position. This hybrid decomposition reflects the physical structure of the task and improves robustness against moderate surface-position errors.

Force control can maintain a desired normal load during surface traversal. If measured force falls below the target, the controller moves slightly toward the surface; if force becomes excessive, it retreats. The force loop must be tuned carefully because high gain can cause oscillation when interacting with stiff surfaces, while low gain may produce large force errors. Sensor filtering, tool compliance, robot dynamics, and communication delay all influence achievable bandwidth.

Impedance control provides another effective strategy by defining a compliant relationship between position error and interaction force. The robot follows a virtual mechanical behavior characterized by stiffness and damping rather than rigidly enforcing the nominal trajectory. Lower normal stiffness allows the tool to accommodate surface variation, while tangential stiffness can remain higher for accurate path tracking. This anisotropic behavior is particularly useful on curved or imperfectly located workpieces.

Admittance control is attractive when the robot accepts position or velocity commands but provides accurate force measurements. The measured force error is converted into a corrective motion through a virtual mass--spring--damper model. The underlying position controller then executes this modified trajectory. Proper parameter selection is essential because excessive virtual compliance can reduce tracking quality, whereas insufficient compliance can produce high contact loads or unstable interaction.

Tool compliance can provide passive protection in addition to active control. Springs, elastomer pads, compliant polishing heads, flexible cloths, or remote-center-compliance mechanisms absorb small geometric errors and reduce impact sensitivity. Passive compliance responds without computational delay and can stabilize high-frequency interaction that the robot controller cannot regulate. Active and passive compliance are often most effective when designed as complementary layers.

Path generation should account for complete surface coverage. Common patterns include parallel raster passes, zigzag trajectories, spirals, contours, and geometry-following paths. Adjacent passes generally overlap so that no untreated strips remain between trajectories. Coverage spacing depends on tool footprint, desired process uniformity, surface curvature, material removal characteristics, and deformation of the pad under load.

Tool orientation should remain consistent with local surface geometry. For a flat pad, the tool axis is typically aligned with the surface normal so pressure is distributed predictably. On curved surfaces, orientation must evolve continuously as the robot moves. Abrupt orientation changes can produce edge loading, torque spikes, uneven wear, or loss of contact. Smooth interpolation of surface normals therefore becomes an important part of trajectory planning.

Polishing quality depends not only on force but also on tangential speed, tool rotation, dwell time, abrasive properties, and overlap. Excessive force or slow motion may remove too much material or overheat the surface, while insufficient force may produce little effect. Process planning should therefore consider contact pressure and exposure together. A constant force trajectory does not necessarily create a uniform result if speed or tool footprint varies substantially.

Wiping has related but different objectives. The robot may need to remove dust, liquid, contamination, adhesive residue, or biological material rather than remove workpiece material. Performance depends on contact pressure, wiping direction, cloth condition, friction, and whether contaminants are transported or redistributed. Repeated passes may be required, and the tool may need periodic replacement, folding, cleaning, or repositioning as its effective surface becomes saturated.

Force--torque sensing at the wrist provides the net interaction wrench between the robot tool and environment. Normal force supports pressure regulation, while tangential forces reveal friction and resistance to motion. Measured moments can indicate misalignment or uneven tool contact. However, tool weight, center of mass, acceleration, and orientation generate additional wrench components, so gravity and dynamic compensation should be applied before interpreting measurements as process forces.

Tactile sensing can provide spatial information unavailable from a wrist force--torque sensor. A distributed pressure array behind a pad can reveal whether contact is centered, concentrated near an edge, or uneven across the tool. Such information allows the robot to adjust orientation as well as force. Vision-based tactile or deformable sensing surfaces may additionally reveal local slip, deformation, contamination, or contact transitions during wiping.

Contact-state estimation helps organize the task into meaningful phases such as approach, first contact, stable contact, sliding, partial contact, contact loss, excessive load, and release. Different control actions are appropriate in each state. For example, contact loss may trigger a controlled search toward the surface, while excessive load should cause immediate unloading. Explicit state estimation prevents one controller from treating fundamentally different physical situations identically.

Surface discontinuities present a major challenge. Edges, holes, seams, steps, and sharp curvature changes can suddenly alter contact area and force. A polishing pad moving beyond an edge may lose support and then strike another surface during recovery. Geometry-aware planning should therefore identify risky regions and reduce speed or modify orientation near them. Force and tactile feedback can provide additional protection when the model is incomplete.

Friction variation also changes robot behavior. A contaminated region, different material, wet surface, or worn abrasive can alter tangential force even when normal force remains constant. Monitoring the ratio and direction of tangential to normal force can provide information about changing contact conditions. Unexpected increases may indicate sticking, tool damage, surface defects, or contamination and can trigger reduced speed or inspection.

Slip is not always undesirable in polishing and wiping because controlled sliding is fundamental to the task. The relevant distinction is between intended tangential motion and uncontrolled tool displacement. The controller should therefore evaluate slip relative to commanded motion, contact geometry, and force. Local tactile measurements can help determine whether the pad is sliding uniformly or whether part of the contact patch is sticking and generating uneven loading.

Trajectory adaptation can compensate for errors discovered during execution. If force feedback indicates that the real surface lies above or below the nominal model, the robot can update the normal offset of subsequent path points. Repeated observations can gradually refine a local surface representation. This converts contact sensing from a purely reactive mechanism into a source of geometric information that improves later portions of the manipulation trajectory.

Learning-based methods can complement conventional contact controllers. Demonstrations from skilled operators can provide examples of force, velocity, orientation, and path adaptation over difficult surfaces. A learned policy can predict corrections to nominal force or trajectory based on force--torque, tactile, vision, and robot-state observations. Restricting learning to bounded residual corrections preserves a stable conventional control layer while allowing data-driven adaptation.

Process monitoring should detect gradual changes as well as immediate faults. Abrasive wear, pad deformation, cloth saturation, tool heating, sensor drift, or accumulating debris can change interaction over many cycles. Trends in normal force error, tangential resistance, vibration, motor current, tactile distribution, or task duration can provide indicators of degradation. Maintenance can then be triggered by measured process condition rather than only fixed operating intervals.

Safety limits should remain independent of the nominal process controller. Maximum force, torque, speed, workspace, joint motion, and tool rotation should be bounded so that controller errors or unexpected geometry cannot generate dangerous interaction. Sudden force spikes, communication loss, sensor saturation, or abnormal contact should trigger a predefined response such as unloading, stopping, or retracting along a verified safe direction.

Data logging is important for understanding process quality. Synchronized records of robot pose, commanded trajectory, measured wrench, tactile signals, tool speed, controller state, and task events allow engineers to reconstruct each surface pass. Surface location can be associated with measured interaction so that force or pressure maps are generated over the workpiece. These maps reveal underprocessed regions, excessive loading, edge effects, and systematic calibration errors.

Quality evaluation should connect physical interaction measurements with the resulting surface condition. Polishing may be assessed using roughness, reflectivity, dimensional change, visual appearance, or material-removal uniformity. Wiping can be evaluated through contamination detection, image inspection, residual particle measurements, or coverage metrics. Correlating these results with recorded force and motion data enables process parameters to be optimized from measurable outcomes.

Validation should include geometric and physical variation rather than testing only one nominal surface. Workpiece position, orientation, curvature, stiffness, friction, tool wear, approach error, and contamination can be varied systematically. Metrics can include force tracking error, peak force, coverage, cycle time, contact-loss frequency, surface quality, recovery success, and repeatability. These tests define the operating envelope within which the contact controller remains reliable.

A robust polishing or wiping system ultimately combines geometric planning with continuous physical perception. Vision or CAD information provides global surface structure, force sensing regulates interaction magnitude, tactile sensing describes local contact distribution, and compliant control accommodates uncertainty. Contact-state estimation and adaptation connect these components during execution, while logging and quality inspection close the engineering feedback loop.

Polishing and wiping therefore illustrate a central principle of contact-rich manipulation: successful behavior emerges from controlling motion and physical interaction as one coupled process. The robot must approach safely, establish contact, regulate force, follow surface geometry, maintain controlled sliding, recognize abnormal interaction, and adapt to changing conditions. Integrating sensing, compliance, planning, learning, safety supervision, and process validation turns a simple surface motion into a reliable Physical AI manipulation capability.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Top comments (0)