Article Sidebar

pdf
Date Published: 15/10/2026
Abstract Views: 14
Views pdf: 17
DOI: 10.51316/jst.194.dsai.2026.36.1.2
Issue
Vol. 36 No. 1 (2026): Journal of Science and Technology – Data Science and Artificial Intelligence
Section
Articles

Post-Deployment Monitoring of Prediction-Error Deterioration in Machine Learning Models Using a Two-Channel Exponentially Weighted Moving Average Chart

Quynh Pham Ba1, Du Nguyen Huu2, Son Ta Anh2, Hoa Le Thanh3,
1 Department of Mathematics, Nguyen Truong To Secondary School, Ha Noi, Vietnam
2 Faculty of Mathematics and Informatics, Hanoi University of Science and Technology, Ha Noi, Vietnam
3 Department of Mathematics, Faculty of Basic Sciences, Vietnam Maritime University, Hai Phong, Vietnam

Main Article Content

Abstract

This study develops a forward-cross-fitted two-channel EWMA (FCF-DS-EWMA) chart for post-deployment monitoring of predictionerror deterioration in continuous-response machine-learning models. Temporally held-out Phase-I residuals are transformed into robust signed and magnitude channels, and an equal-share rule uses channel-specific finite-horizon limits whose common marginal exceedance level is selected jointly. In a unified Monte Carlo study with 𝑅 = 1000, the native false-alarm probability (PFA) is 0.176–0.196 for the in-sample-residual chart and 0.094–0.097 for FCF-DS across the three in-control environments. Six deterioration mechanisms are compared on a common ΔMSE scale. The signed and magnitude channels specialize to different changes, while equal-share calibration restores directional sensitivity relative to a common-limit maximum (abrupt-shift detection probability 𝑃𝐷 = 0.724 versus 0.582). One-factor analyses of native PFA identify serial dependence, inadequate block length, and monitoring horizon as the clearest sources of false-alarm inflation. An oracle diagnostic shows that the temporary-model/final-model reference mismatch mainly shifts the operating point by a small amount. The ethanol study illustrates channel attribution and reference-location sensitivity, and paired perturbations of actual post-Phase-I Metro traffic residual windows provide quantitative evidence in a second regression domain.

Keywords

prediction-error monitoring, cross-fitting, EWMA, control chart, residual monitoring, statistical process monitoring

Article Details

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

References

[1] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, Learning under concept drift: A review, IEEE Trans. Knowl. Data Eng. vol. 31, no. 12, pp. 2346–2363, 2019. https://doi.org/10.1109/TKDE.2018.2876857.
[2] J. M. Lucas and M. S. Saccucci, Exponentially weighted moving average control schemes: Properties and enhancements, Technometrics, vol. 32, no. 1, pp. 1–12, 1990. https://doi.org/10.1080/00401706.1990.10484583.
[3] G. Chen, S. W. Cheng, and H. Xie, Monitoring process mean and variability with one EWMA chart, J. Qual. Technol. vol. 33, no. 2, pp. 223–233, 2001. https://doi.org/10.1080/00224065.2001.11980069.
[4] A. Ostadsharif Memar and S. T. A. Niaki, The Max EWMAMS control chart for joint monitoring of process mean and variance with individual observations, Qual. Reliab. Eng. Int. vol. 27, no. 4, pp. 499–514, 2011. https://doi.org/10.1002/qre.1146.
[5] W. H. Woodall, D. J. Spitzner, D. C. Montgomery, and S. Gupta, Using control charts to monitor process and product quality profiles, J. Qual. Technol. vol. 36, no. 3, pp. 309–320, 2004. https://doi.org/10.1080/00224065.2004.11980276.
[6] W. A. Jensen, L. A. Jones-Farmer, C. W. Champ, and W. H. Woodall, Effects of parameter estimation on control chart properties: A literature review, J. Qual. Technol. vol. 38, no. 4, pp. 349–364, 2006. https://doi.org/10.1080/00224065.2006.11918623.
[7] A. Gandy and J. T. Kvaløy, Guaranteed conditional performance of control charts via bootstrap methods, Scand. J. Stat. vol. 40, no. 4, pp. 647–668, 2013. https://doi.org/10.1002/sjos.12006.
[8] G. J. Ross, N. M. Adams, D. K. Tasoulis, and D. J. Hand, Exponentially weighted moving average charts for detecting concept drift, Pattern Recognit. Lett. vol. 33, no. 2, pp. 191–198, 2012. https://doi.org/10.1016/j.patrec.2011.08.019.
[9] A. Bifet and R. Gavaldà, Learning from time-changing data with adaptive windowing, in Proc. 7th SIAM Int. Conf. Data Mining, 2007, pp. 443–448.
[10] K. Zhang, A. T. Bui, and D. W. Apley, Concept drift monitoring and diagnostics of supervised learning models via score vectors, Technometrics, vol. 65, no. 2, pp. 137–149, 2023. https://doi.org/10.1080/00401706.2022.2124310.
[11] J. Wu and D. W. Apley, Bootstrapped control limits for score-based concept drift control charts, Technometrics, 2026.
https : / / doi . org / 10 . 1080 / 00401706 . 2026 . 2676855, Published online.
[12] J. Pyeon, D. Cacciarelli, and K. Paynabar, An adaptive sampling framework for detecting localized concept drift under label scarcity, 2025. [Online] Available: https://doi.org/10.48550/arXiv.2511.02452.
[13] B. Wang, S. Sun, J. Wang, and Y. Tang, Robust outlier detection and low-latency concept drift detection for data stream regression: A dual-channel architecture, Concurrency Computat.: Pract. Exper. vol. 38, no. 9, pp. e70730, 2026. https://doi.org/10.1002/cpe.70730.
[14] C. Qian, G. Wang, Z. Wang, and C. Zou, Changepoint detection in complex models: Cross-fitting is needed, 2024. [Online] Available: https://doi. org/10.48550/arXiv.2411.07874.
[15] L. J. Tashman, Out-of-sample tests of forecasting accuracy: An analysis and review, Int. J. Forecast. vol. 16, no. 4, pp. 437–450, 2000. https://doi.org/10.1016/S0169-2070(00)00065-0.
[16] P. J. Rousseeuw and C. Croux, Alternatives to the median absolute deviation, J. Amer. Statist. Assoc. vol. 88, no. 424, pp. 1273–1283, 1993. https://doi.org/10.1080/01621459.1993.10476408.
[17] D. V. Hinkley, Inference about the change-point from cumulative sum tests, Biometrika, vol. 58, no. 3, pp. 509–523, 1971.
https://doi.org/10.1093/biomet/58.3.509.
[18] A. Vergara, Gas Sensor Array Drift at Different Concentrations, UCI Machine Learning Repository, 2012. [Online] Available: https://doi.org/10.24432/C5MK6M.
[19] A. Vergara, S. Vembu, T. Ayhan, M. A. Ryan, M. L. Homer, and R. Huerta, Chemical gas sensor drift compensation using classifier ensembles, Sens. Actuators B Chem. vol. 166–167, pp. 320–329, 2012. https://doi.org/10.1016/j.snb.2012.01.074.
[20] I. Rodríguez-Luján, J. Fonollosa, A. Vergara, M. Homer, and R. Huerta, On the calibration of sensor arrays for pattern recognition using the minimal number of experiments, Chemom. Intell. Lab. Syst. vol. 130, pp. 123–134, 2014. https://doi.org/10.1016/j.chemolab.2013.10.012.
[21] T. Chen and C. Guestrin, XGBoost: A scalable tree boosting system, in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 785–794.
[22] J. Hogue, Metro Interstate Traffic Volume, UCI Machine Learning Repository, 2019. [Online] Available: https://doi.org/10.24432/C5X60B.