Projects /

Rhythmic Sketch

20 Apr 2019

Augmented Visuotactile Feedback Support Sensorimotor Synchronization


Augmented feedback, Wearable, Multisensory stimuli, Sensorimotor synchronization

This page describes a research which is currently under review for publication in a journal - a preliminary study can be found here and a brief description is below.

Research purpose

In this study, we looked at the effect of augmented visuotactile feedback on people’s motor performance. We were interested in ways of getting people more engaged with motor training, particularly the skill of sensorimotor synchronisation. We investigated the unobtrusive way to display feedback without disturbing surrounding people, and analyse motion quality with a multi-layered computational framework. This project serves as a benchmark study for future investigations with minor stroke patients. We believe the multisensory feedback, with proper design, can benefit not only healthy people to gain motor skills but stroke patients to regain lost motor functions.

Study requirements
  1. easy to pick up the practice
  2. real-time feedback of motion
  3. multisensory feedback
Practice task

Sketch rhythmic patterns by hand or arm following a sample rhythm. Experiment setup is shown in the figure below.

Stimuli for the training

The rhythm timing: Following figure explains auditory and vibrotactile signal onset

Following figure explains visual concomitant of the auditory-vibrotactile rhythm

Evaluation and results

We follow the multi-layered conceptual framewrok proposed by Camurri et al, 2016, to analysis motion qualities.

Multi-layered conceptual framewrok:

Layer 4 Qualities communication Emotions, social signals
Layer 3 Mid-level features Amodel features like smoothness
Layer 2 Low-level features Speed, Accelerations
Layer 1 Physical signals, virtual sensors Time-series data

Based on the first layer of computing, we analysied motion features in layer 2 and 3:

Rhythm accuracy - measured by calculating the correlations between the sketched rhythm and the sample rhythm.

Motion precision, smoothness and accuracy

(Performance with multi-sensory feedback have the correlation all above 0.5, as shown in the figure above, which indicates a good accuracy of sketched rhythm.)

The remaining question

Can augmening motion features (DTW distance or the level of motion smoothness ) as the real-time multimodal feedback further support motor skill recovery?