ML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems

[Context] Systems that incorporate Machine Learning (ML) models, often referred to as ML-enabled software systems, have become commonplace. However, empirical evidence on how ML-enabled systems are engineered in practice is still limited; this is especially true for activities surrounding ML model dissemination.
[Goal] We investigate contemporary industrial practices and problems related to ML model dissemination, focusing on the model deployment and the monitoring ML lifecycle phases.
[Method] We conducted an international questionnaire-based online survey to gather practitioner insights on how ML-enabled systems are engineered. We gathered 188 complete responses from 25 countries. We analyze the status quo and problems reported for the model deployment and monitoring phases. We conducted statistical analyses on contemporary practices using bootstrapping with confidence intervals and qualitative analyses on the reported problems involving open and axial coding procedures.
[Results] Practitioners perceive the model deployment and monitoring phases as relevant but also as difficult. With respect to model deployment, models are typically deployed as separate services, with limited adoption of MLOps principles. Reported problems include difficulties in designing the architecture of the infrastructure for production deployment and legacy application integration. Concerning model monitoring, many of the models in production are not monitored. The main monitored aspects are inputs, outputs, and decisions. Reported problems involve the absence of monitoring practices, the need to create custom monitoring tools, and challenges in selecting suitable metrics.
[Conclusion] Our results help already providing a better understanding of the adopted practices and problems in practice and support guiding ML deployment and monitoring research in a problem-driven manner.

Authors: Eduardo Zimelewicz, Antonio Pedro Santos Alves, Marcos Kalinowski, Daniel Mendez, Görkem Giray, Kelly Azevedo, Hugo Villamizar, Niklas Lavesson, Tatiana Escovedo, Helio Lopes, Stefan Biffl, Juergen Musil, Michael Felderer, Stefan Wagner, Maria Teresa Baldassarre and Tony Gorschek

Talk language: English
Level: Expert
Target group:

Company:
Pontifical Catholic University of Rio de Janeiro

Prof. Dr. Marcos Kalinowski