Dynamic Hand Gesture Recognition using Cyclical Patterns of Hand Movement and Its Applications
Main Article Content
Abstract
This paper tackles a new prototype of dynamic hand gestures and its advantages to apply to controlling smart home appliances. The proposed gestures convey cyclical patterns of hand shapes as well as hand movements. Thanks to the periodicity of defined gestures, on one hand, common technical issues that appear when deploying the application (e.g., spotting gestures from a video stream) are addressed. On the other hand, they are supportive features for deploying a robust recognition scheme. To this end, we propose a novel hand representation in a temporal-spatial space. Particularly, the phase continuity of the gesture's trajectory is taken into account underlying the temporal-spatial space. This scheme obtains very promising results with the best accuracy rate is 96%. The proposed techniques are deployed to control home appliances such as lamps, fans. These systems have been evaluated in both lab-based environment and real exhibitions. In the future, the proposed method will be evaluated in term of the naturalness of end-users and/or robustness of the systems.
Keywords
Human Computer Interaction, Dynamic Hand Gesture Recognition, Spatial-Temporal Features
Article Details
References
1. S. Marcel, O.Bernier, J.-E. Viallet, and D. Collobert, Hand gesture recognition using input-output hidden markov models, FG, 2000, pp. 456-461.
2. Z. Ren, J. Yuan, and Z. Zhang, Robust Hand Gesture Recognition Based on Finger-Earth Movers Distance with a Commodity Depth Camera, International Conference on Multimedia, 2011.
3. Y. Song, D. Demirdjian, and R. Davis, Tracking body and hands for gesture recognition: Natops aircraft handling signals database, FG, 2011, pp. 500-506.
4. A. I. Maqueda, C. del Blanco, and F. G. Jaureguizar, Human-computer interaction based on visual recognition using volumegrams of local binary patterns, ICCE, 2015, pp. 583-584.
5. A. Kurakin, Z. Zhang, and Z. Liu, A real time system for dynamic hand gesture recognition with a depth, EUSIPCO, 2012, pp. 1975-1979.
6. Y.-T. Li, and J. P. Wachs, Hierarchical elastic graph matching for hand gesture recognition, ICPR., 2012, pp. 308-315.
7. D. Kim, and J. Song, Simultaneous Gesture Segmentation and Recognition Based on Forward Spotting Accumlative HMMs. Journal of Pattern Recognition Society, vol. 40. pp. 1-4, 2007.
8. T.-K. Kim, and R. Cipolla, Canonical Correlation Analysis of Video Volume Tensors for Action Categorization and Detection, TPAMI, 2009, pp. 1415-1428.
9. I. Bayer, and T. Silbermann, A multi modal approach to gesture recognition from audio and video data, ICMI, pp. 461-466, 2013.
10. X. Chen, and M. Koskela, Online rgb-d gesture recognition with extreme learning machines, ICMI, 2013, pp. 467-474.
11. A. El-Sawah, C. Joslin, and N. Georganas, A dynamic gesture interface for virtual environments based on hidden markov models, HAVE, 2005, pp. 109-114.
12. S. Escalera, J. Gonzalez, X. Baro, M. Reyes. O. Lopes, I. Guyon, V. Athitsos, and H. Escalante, Multi-modal gesture recognition challenge 2013: Dataset and results, ICMI, pp. 445-452, 2013.
13. J. B. Tenenbaum, V. de Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction, Science, vol. 290, no. 5500, pp. 2319-2323, 2000.
14. J.Shi and C.Tomasi, Good features to track, IJCAI. 1994, pp. 593-600.
15. C. J. Burges. A tutorial on support vector machines for pattern recognition, Data mining and knowledge discovery, vol.2, по.2, pp. 121-167, 1998.
16. http://www.microsoft.com/en/us/kinectforwindows.
17. C. Stauffer, and W. E. L. Grimson, Adaptive background mixture models for real-time tracking, CVPR, vol. 2, 1999, pp. 246-252.
18. H.-G. Doan, H. Vu, T.-H. Tran, and E. Castelli, A combination of user-guide scheme and kernel descriptor on rgb-d data for robust and realtime hand posture recognition, EAAI, vol. 49, pp. 103-113, Mar. 2016.
19. B. D. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, IJCAI, 1981, pp. 674-679.
20. H.-G. Doan, H. Vu, and T.-H. Tran. Phase synchronization in a manifold space for recognizing dynamic hand gestures from periodic image sequence, RIVF, 2016, pp. 163-168.
21. K. McGuinness, and N. E. O Connor, A comparative evaluation of interactive segmentation algorithms, Pattern Recognition, vol. 43, по. 2, pp. 434-444, Feb. 2010.
22. http://research.microsoft.com/
23. P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz, Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network, CVPR, 2016, pp. 4207-4215.
24. O. Oreifej, and Z. Liu, Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences, CVPR, 2013, pp. 716-723.
25. X. Yang, and Y. Tian, Super normal vector for activity recognition using depth sequences, CVPR, 2014, pp. 804-811.
26. H. G. Doan, H. Vu, and T. H. Tran, Recognition of hand gestures from cyclic hand movements using spatial-temporal features, SoICT, 2015, pp. 260-267.
27. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, Learning spatial-temporal features with 3d convolutional networks, ICCV, 2015.
2. Z. Ren, J. Yuan, and Z. Zhang, Robust Hand Gesture Recognition Based on Finger-Earth Movers Distance with a Commodity Depth Camera, International Conference on Multimedia, 2011.
3. Y. Song, D. Demirdjian, and R. Davis, Tracking body and hands for gesture recognition: Natops aircraft handling signals database, FG, 2011, pp. 500-506.
4. A. I. Maqueda, C. del Blanco, and F. G. Jaureguizar, Human-computer interaction based on visual recognition using volumegrams of local binary patterns, ICCE, 2015, pp. 583-584.
5. A. Kurakin, Z. Zhang, and Z. Liu, A real time system for dynamic hand gesture recognition with a depth, EUSIPCO, 2012, pp. 1975-1979.
6. Y.-T. Li, and J. P. Wachs, Hierarchical elastic graph matching for hand gesture recognition, ICPR., 2012, pp. 308-315.
7. D. Kim, and J. Song, Simultaneous Gesture Segmentation and Recognition Based on Forward Spotting Accumlative HMMs. Journal of Pattern Recognition Society, vol. 40. pp. 1-4, 2007.
8. T.-K. Kim, and R. Cipolla, Canonical Correlation Analysis of Video Volume Tensors for Action Categorization and Detection, TPAMI, 2009, pp. 1415-1428.
9. I. Bayer, and T. Silbermann, A multi modal approach to gesture recognition from audio and video data, ICMI, pp. 461-466, 2013.
10. X. Chen, and M. Koskela, Online rgb-d gesture recognition with extreme learning machines, ICMI, 2013, pp. 467-474.
11. A. El-Sawah, C. Joslin, and N. Georganas, A dynamic gesture interface for virtual environments based on hidden markov models, HAVE, 2005, pp. 109-114.
12. S. Escalera, J. Gonzalez, X. Baro, M. Reyes. O. Lopes, I. Guyon, V. Athitsos, and H. Escalante, Multi-modal gesture recognition challenge 2013: Dataset and results, ICMI, pp. 445-452, 2013.
13. J. B. Tenenbaum, V. de Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction, Science, vol. 290, no. 5500, pp. 2319-2323, 2000.
14. J.Shi and C.Tomasi, Good features to track, IJCAI. 1994, pp. 593-600.
15. C. J. Burges. A tutorial on support vector machines for pattern recognition, Data mining and knowledge discovery, vol.2, по.2, pp. 121-167, 1998.
16. http://www.microsoft.com/en/us/kinectforwindows.
17. C. Stauffer, and W. E. L. Grimson, Adaptive background mixture models for real-time tracking, CVPR, vol. 2, 1999, pp. 246-252.
18. H.-G. Doan, H. Vu, T.-H. Tran, and E. Castelli, A combination of user-guide scheme and kernel descriptor on rgb-d data for robust and realtime hand posture recognition, EAAI, vol. 49, pp. 103-113, Mar. 2016.
19. B. D. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, IJCAI, 1981, pp. 674-679.
20. H.-G. Doan, H. Vu, and T.-H. Tran. Phase synchronization in a manifold space for recognizing dynamic hand gestures from periodic image sequence, RIVF, 2016, pp. 163-168.
21. K. McGuinness, and N. E. O Connor, A comparative evaluation of interactive segmentation algorithms, Pattern Recognition, vol. 43, по. 2, pp. 434-444, Feb. 2010.
22. http://research.microsoft.com/
23. P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz, Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network, CVPR, 2016, pp. 4207-4215.
24. O. Oreifej, and Z. Liu, Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences, CVPR, 2013, pp. 716-723.
25. X. Yang, and Y. Tian, Super normal vector for activity recognition using depth sequences, CVPR, 2014, pp. 804-811.
26. H. G. Doan, H. Vu, and T. H. Tran, Recognition of hand gestures from cyclic hand movements using spatial-temporal features, SoICT, 2015, pp. 260-267.
27. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, Learning spatial-temporal features with 3d convolutional networks, ICCV, 2015.