Article Sidebar

pdf
Date Published: 15/10/2026
Abstract Views: 26
Views pdf: 0
DOI: 10.51316/jst.194.dsai.2026.36.1.3
Issue
Vol. 36 No. 1 (2026): Journal of Science and Technology – Data Science and Artificial Intelligence
Section
Articles

HybridLVLM: Semantic Core-Frame Selection and Two-Stage Chain-of-Thought Reasoning for Continuous Driver Action Recognition

Hiep Bui Hoang1, Binh Nguyen Thuy2, Linh Vo Thien2, Quan Nguyen Hong 3, Frédéric Rayar4, Lan Le Thi5,
1 School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Ha Noi, Vietnam
2 Faculty of Electrical-Electronic Engineering, University of Transport and Communications, Ha Noi, Vietnam
3 Faculty of Information Technology, Viet-Hung Industrial University, Ha Noi, Vietnam
4 University of Tours, LIFAT (EA6300), 37200, Tours, France
5 Ha Noi University of Science and Technology, Ha Noi, Vietnam

Main Article Content

Abstract

Driver action recognition (DAR) is a core capability of advanced driver assistance systems, however, it remains difficult because distracted driver actions involve small interacting objects, subtle hand movements, and severe self-occlusion inside a confined cabin. Most existing methods are trained on pre-segmented video clips with predefined action labels requiring substantial annotation effort and additional labeled data when extending to new action classes, which limits their scalability to real-world applications. Moreover, they cannot localize actions in an untrimmed video stream. This paper presents HybridLVLM, a training-free framework for continuous driver action recognition in untrimmed videos that leverages large pretrained vision-language models. Firstly, the framework transforms every sampled frame into a natural-language caption, filters the captions with a text-only language model to extract distracted action evidence and core frames for video representation. Secondly, an open-vocabulary detector then spatially localizes the bounding boxes corresponding to the driver and several task-relevant objects on each core frame. Then, these bounding boxes together with the set of core frames are fed forward to a two-stage Chain-of-Thought (CoT) reasoning model to infer an action class for each core frame based on task-relevant visual evidences. Finally, a temporal post-processing is applied to refine and extend frame-level recognition to continuous recognition presented by a predicted action tube. Experiments on the UTCDriverAct dataset show that HybridLVLM attains a Video-mAP of 52.0% and a mean F1-score of 61.4% at an STT-IoU threshold of 0.3 without any training on driving data. These findings point to a promising research direction in video understanding for applications without real-time constraints. Additionally, an ablation study on the input sampling rate is also conducted to investigate the effect of the sampling rate on the overall recognition performance. The results indicate that the optimal sampling density depends on action duration, with higher rates for rapid actions and lower rates for gradual actions

Keywords

Chain-of-Thought reasoning, Continuous action recognition, Driver action recognition, Key frame selection, Large vision-language model.

Article Details

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

References

[1] M. Lu, Y. Hu, and X. Lu, Driver action recognition using deformable and dilated Faster R-CNN with optimized region proposals, Applied Intelligence, vol. 50, no. 4, pp. 1100–1111, Apr. 2020. https://doi.org/10.1007/s10489-019-01603-4
[2] M. T. Tran, M. Q. Vu, N. D. Hoang, and K.-H. N. Bui, An effective temporal localization method with multi-view 3D action recognition for untrimmed naturalistic driving videos, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Jun. 2022, pp. 3168–3173.
[3] H.-Q. Nguyen, T.-B. Nguyen, T. K. Tran, V.-N. Hoang, T.-L. Le, T.-H. Tran, and H. Vu, End-to-end deep learning-based framework for driver action recognition, in 2022 International Conference on Multimedia Analysis and Pattern Recognition (MAPR), 2022, pp. 1–6.
[4] H. I. Qamar, U. Saeed, and M. Hussain, Driver distraction detection using a multi-stream deep fusion network, in International Conference on Computing and Emerging Technologies, 2023, pp. 97–104.
[5] Y. Ma, L. Yuan, A. Abdelraouf, K. Han, R. Gupta, Z. Li, and Z. Wang, M2DAR: Multi-view multi-scale driver action recognition with vision transformer, pp. 5287–5294, Jun. 2023. https://doi.org/10.1109/CVPRW59228.2023.00557
[6] R. Pizarro, R. Valle, L. M. Bergasa, J. M. Buenaposada, and L. Baumela, Pose-guided multi-task video transformer for driver action recognition, arXiv preprint arXiv:2407.13750, Jul. 2024.
[7] H.-H. Bui, K.-H. Bui, T.-L. Vo, H.-Q. Nguyen, T.-B. Nguyen, T.-T. Dao, and T.-L. Le, HIT4DAR: Holistic interaction transformer for driver action recognition, Transport and Communications Science Journal, vol. 77, iss. 4, pp. 500–514, May 2026. https://doi.org/10.47869/tcsj.77.4.12
[8] Q.-T. Nguyen, T.-B. Nguyen, H.-Q. Nguyen, and T.-L. Le, MAE4DAR: Masked autoEncoder for driver action recognition, VNU Journal of Science: Computer Science and Communication Engineering, Jun. 2026. https://doi.org/10.25073/2588-1086/vnucsce.7153
[9] M. Z. Hasan, J. Chen, J. Wang, M. S. Rahman, A. Joshi, S. Velipasalar, C. Hegde, A. Sharma, and S. Sarkar, Vision-language models can identify distracted driver behavior from naturalistic videos, IEEE Transactions on Intelligent Transportation Systems, vol. 25, iss. 9, pp. 11602–11616, Sep. 2024. https://doi.org/10.1109/TITS.2024.3381175
[10] K. Zhang, S. Wang, N. Jia, L. Zhao, C. Han, and L. Li, Integrating visual large language model and reasoning chain for driver behavior analysis and risk assessment, Accident Analysis & Prevention, vol. 198, Feb. 2024, Art. no, 107497. https://doi.org/10.1016/j.aap.2024.107497
[11] D. Kondratyuk, L. Yuan, Y. Li, L. Zhang, M. Tan, M. Brown, and B. Gong, MoViNets: Mobile video networks for efficient video recognition, pp. 16015–16025, Mar. 2021. https://doi.org/10.48550/arXiv.2103.11511
[12] C. Feichtenhofer, X3D: Expanding Architectures for Efficient Video Recognition, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 200–210, 2020.
[13] S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, in Advances in Neural Information Processing Systems, pp. 91–99, Dec. 2015.
[14] Z. Xu and J. Xu, Spatiotemporal decoupling attention transformer for 3D skeleton-based driver action recognition, Complex & Intelligent Systems, vol. 11, pp. 1–12, Feb. 2025, Art. no. 195.
[15] G. J. Faure, M.-H. Chen, and S.-H. Lai, Holistic interaction transformer network for action detection, in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 3329–3339.
[16] P.Wang, S. Bai, S. Tan, S.Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al., Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution, arXiv preprint arXiv:2409.12191, 2024.
[17] S. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, Qwen2.5 technical report, 2025, [Online] URL: https://arxiv.org/abs/2502.13923.
[18] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, arXiv preprint arXiv:2402.03216, 2025.
[19] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al., Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection, in European Conference on Computer Vision, 2024, pp. 38–55.
[20] C. Feichtenhofer, A. Pinz, and R. P. Wildes, Spatiotemporal residual networks for video action recognition, in Advances in Neural Information Processing Systems, 2016, pp. 3468–3476.