Augmented feedback, Wearable, Multisensory stimuli, Sensorimotor synchronization
This page describes a research which is currently under review for publication in a journal - a preliminary study can be found here and a brief description is below.
In this study, we looked at the effect of augmented visuotactile feedback on people’s motor performance. We were interested in ways of getting people more engaged with motor training, particularly the skill of sensorimotor synchronisation. We investigated the unobtrusive way to display feedback without disturbing surrounding people, and analyse motion quality with a multi-layered computational framework. This project serves as a benchmark study for future investigations with minor stroke patients. We believe the multisensory feedback, with proper design, can benefit not only healthy people to gain motor skills but stroke patients to regain lost motor functions.
Sketch rhythmic patterns by hand or arm following a sample rhythm. Experiment setup is shown in the figure below.
The rhythm timing: Following figure explains auditory and vibrotactile signal onset
Following figure explains visual concomitant of the auditory-vibrotactile rhythm
We follow the multi-layered conceptual framewrok proposed by Camurri et al, 2016, to analysis motion qualities.
Multi-layered conceptual framewrok:
| LAYERS | PARAMETERS | EXAMPLES |
|---|---|---|
| Layer 4 | Qualities communication | Emotions, social signals |
| Layer 3 | Mid-level features | Amodel features like smoothness |
| Layer 2 | Low-level features | Speed, Accelerations |
| Layer 1 | Physical signals, virtual sensors | Time-series data |
Based on the first layer of computing, we analysied motion features in layer 2 and 3:
Layer 2: Rhythmic motion precision - measured by dynamic time warping (DTW) distances.
Layer 3: Sketch motion smoothness - measured by the number of velocity peaks.
Rhythm accuracy - measured by calculating the correlations between the sketched rhythm and the sample rhythm.
(Performance with multi-sensory feedback have the correlation all above 0.5, as shown in the figure above, which indicates a good accuracy of sketched rhythm.)
We learned that VA (Visual-auditory) feedback support the best rhythmic motion precision, but worst motion smoothness, and that VH (Visual-haptic) feedback support the best motion smoothness.
To support Sensorimotor Synchronization skill recovery, providing VA and VH feedback could improve motion precision and movement smoothness. However, these results need to be verified with real patients in the future.
Providing vibrotactile feedback is indeed facilitate rhythmic motion performance in an unobtrusive way.
Can augmening motion features (DTW distance or the level of motion smoothness ) as the real-time multimodal feedback further support motor skill recovery?