本論文重點介紹即時疲勞檢測流程。該系統將在 Python 內部完成這一切，並逐步構建它，以便能夠檢測到不同的姿勢，特別是困倦的跡象。 為了做到這一點，我們使用一些關鍵模型並使用 MediaPipe Holistic 來提取關鍵點。 這將使我們能夠從臉部提取關鍵點。 該系統使用 Tensorflow 和 Keras，並建立了一個長短期記憶模型 long short-term memory（LSTM），能夠預測螢幕上顯示的動作。我們需要做的是收集關於我們所有不同關鍵點的一些數據，所以我們收集我們臉上的數據並將它們保存為 Numpy 數據，以便處理多維的陣列或矩陣。人臉檢測方法基於一個深度神經網絡，使用 Sklearn 進行評估和測試，並使用 Matplotlib 幫助進行圖像可視化。能夠從臉部檢測到 468個地標，提取臉部的重要特徵並對數據進行變換，以便將數據導入 LSTM 模型。使用 LSTM 層繼續並預測時間分量，它能夠從多個幀預測動作，而不僅僅是單個幀。使用 Opencv 進行集成，然後使用網路攝影機進行即時預測。本研究成功使用 MediaPipe 與 LSTM 模型相結合，提出一套疲勞檢測的系統。實驗結果顯示，經機器學習後其檢測平均準確率能達到 90%。

This paper focuses on the instant fatigue detection process. The system will do all this inside python and build it incrementally to be able to detect different poses, especially signs of drowsiness. To do this, we use some key models and use MediaPipe Holistic to extract keypoints. This will allow us to extract keypoints from the face. The system uses Tensorflow and Keras and builds a long short-term memory (LSTM) model that is able to predict actions displayed on the screen. What we need to do is collect some data about all our different keypoints, so we collect data on our faces and save them as Numpy data in order to work with multidimensional arrays or matrices. The face detection method is based on a deep neural network, evaluated and tested using Sklearn and aided in image visualization using Matplotlib. Able to detect 468 landmarks from faces, extract important features of faces and transform the data so that it can be imported into an LSTM model. Continuing and predicting the temporal component using an LSTM layer, it is able to predict action from multiple frames, not just a single frame. Integrate with Opencv, then use a webcam for instant prediction. This study successfully uses MediaPipe combined with LSTM model to propose a fatigue detection system. The experimental results show that the detection accuracy can reach an average of 90% after machine learning.

[1] Triyanti V. Iridiastadi H, "Challenges in detecting drowsiness based on driver's behavior, " Mater. Sci. Eng. Conf. Series, vol. 277, issue 1, p. 12-42, 2017.
[2] Yang Z. Ren H, "Feature Extraction and Simulation of EEG Signals During Exercise-Induced Fatigue, " in IEEE Access, vol. 7, pp. 46389-46398, 2019.
[3] Gasser T. Sroka L. Möcks J, "The transfer of EOG activity into the EEG for eyes open and closed, " Electroencephalography and clinical neurophysiology, vol. 61, issue 2, pp. 181–193, 1985.
[4] Mardi Z. Ashtiani S.N.M. Mikaili M, "EEG-based drowsiness detection for safe driving using chaotic features and statistical tests, " J. Med. Signals Sens, vol. 1, issue 2, pp. 130–137, 2011.
[5] 半導體投資聯盟, "浙江大華股份有限公司官方網站", 15 Apr.2022, www.dahuatech.com/about/company.html.
[6] Alioua N. Amine A. Rziza M, "Driver's fatigue detection based on yawning extraction, " Int. J. Vehicular Technology, vol. 1, pp. 47–75, 2014.
[7] Wang T. Shi P, "Yawning detection for determining driver drowsiness," in Proceedings of the IEEE International Workshop on VLSI Design and Video Technology, pp. 373-376, Suzhou, China, May 2005.
[8] Choi I.H. Kim Y.G, "Head pose and gaze direction tracking for detecting a drowsy driver," IEEE Int. Conf. on Big Data and Smart Computing, Bangkok, Thailand, vol. 9, no. 2, pp. 241–244, 2014.
[9] CLOUDMATRIX, "GFace面部表情檢測系統", 15 Apr.2022, www.cloudmatrix.com.tw/gfac e-facial-recognition-system/.
[10] Ray Lin, "駕駛員監控系統DMS介紹", 15 Apr.2022, medium.com/學以廣才/駕駛者監控系統-driver-monitoring-system-dms-2830f3008c81.
[11] Byoung Chul Ko, "A Brief Review of Facial Emotion Recognition Based on Visual Information," Sensors (Basel), January 2018.
[12] Kim D.H. Baddar W. Jang J. Ro Y.M, "Multi-objective based Spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition," IEEE Transactions on Affective Computing, pp.99, April 2017.
[13] Graves A. Mayer C. Wimmer M. Schmidhuber J. Radig B, "Facial expression recognition with recurrent neural networks," Proceedings of the International Workshop on Cognition for Technical Systems, Santorini, Greece, pp.1–6, October 2008.
[14] Jain D.K. Zhang Z. Huang K, "Multi angle optimal pattern-based deep learning for automatic facial expression recognition," Pattern Recognition Letters. Volume 139, pp.157-165, November 2020.
[15] Arqam M. Al-Nuimi, Ghassan J. Mohammed, "Face Direction Estimation based on Mediapipe Landmarks", pp.25-26, August 2021.
[16] Huang K. Liu X. Fu S. Guo D. & Xu M, "A Lightweight Privacy Preserving CNN Feature Extraction Framework for Mobile Sensing," IEEE Transactions on Dependable and Secure Computing, pp.1441–1455, 2019.
[17] Jogin M. Mohana M. Madhulika M. Divya G. Meghana R. & Apoorva S, "Feature Extraction using Convolution Neural Networks(CNN) and Deep Learning," 2018 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), 2018.
[18] JY, "時間序列分析與預測方法",15 Dec.2021, ithelp.ithome.co m.tw/article s/10272289.
[19] "梯度消失問題", 維基百科, 15 Apr.2022, zh.wikipedia.org/zh-tw/梯度消失問题。
[20] "Intel RealSense D435i", Intel® RealSense™ Depth Camera官方網站, 15 Apr.2022, www.intelrealsense.com/depth-camera-d435i/.
[21] "Intel NUC", Intel官方網站,15 Jun.2022, www.intel.com.tw/content/ww
w/tw/zh/products/details/nuc/boards.html.
[22] 計算機視覺, "雙目立體視覺",12 Jul.2022, www.getit01.com/p20180201.
[23] Omidyeganeh M.et al, "Yawning Detection Using Embedded Smart Cameras," in IEEE Transactions on Instrumentation and Measurement, vol. 65, no. 3, pp. 570-582, March 2016.
[24] Li X. Hong L. Wang J. Liu X, "Fatigue driving detection model based on multi-feature fusion and semi-supervised active learning," in IET Intelligent Transport Systems, vol. 13, no. 9, pp. 1401-1409, 2019.
[25] Kazemi V. Sullivan J, "One millisecond face alignment with an ensemble of regression trees," 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, pp. 1867-1874, 2014.
[26] Ji Y. Wang S. Zhao Y. Wei J. Lu Y, "Fatigue State Detection Based on Multi-Index Fusion and State Recognition Network," in IEEE Access, vol. 7, pp. 64136-64147, 2019.
[27] Mandal B. Li L. Wang G. S. Lin J, "Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State," in IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 3, pp. 545-557, March 2017.