Összefoglalás. A mély mesterséges neuronhálók elterjedése az ipari alkalmazásokban évekkel azok megbízhatóságával, értelmezhetőségével, és biztonságával kapcsolatos szakterületek fejlődését megelőzően történt. Az egyik, gyakorlatban is jelentős területen, a képfelismerésben például a megvalósult megoldások szinte már emberi teljesítményre képesek, de ezzel együtt célzott zajjal ezek a rendszerek félrevezethetők, megzavarhatók. Jelen kéziratban ismertetünk néhány tipikus biztonsági problémát, valamint rámutatunk arra, hogy a hagyományos szoftverfejlesztés területén alkalmazott minőségbiztosítási módszerekkel rokon megoldásokra szükség van az MI-re épülő rendszerek fejlesztésében, akár a mesterséges neuronhálók biztonságát, akár az MI rendszerek hagyományos komponenseinek fejlesztését tartjuk szem előtt.
Summary. Research on the trustworthiness, interpretability and security of deep neural networks lags behind the widespread application of the technology in industrial applications. For example, in image recognition, modern solutions are capable of nearly human performance. However, with targeted adversarial noise, these systems can be arbitrarily manipulated. Here, we discuss some of the security problems and point out that quality assurance methods used in traditional software development should also be adapted when developing AI-based systems, whether in the security of artificial neural networks or traditional components of AI systems. One of the main concerns about neural networks today that – to the best of our knowledge – affects all deep neural networks is the existence of adversarial examples. These examples are relatively easy to find and according to a recent experiment, a well-chosen input can attack more networks at the same time. In this paper we also present a wider perspective of security of neural architectures borrowed from the traditional software engineering discipline. While in traditional development several methods are widely applied for software testing and fault localization, there is a lack of similar well-established methods in the neural network context. In case of deep neural networks, systematic testing tools and methods are in the early stage, and a methodology to test and verify the proper behavior of the neural networks is highly desirable. Robustness testing of machine learning algorithms is a further issue. This requires the generation of large random input data using fuzz testing methods. The adaptation of automatic fault localization techniques has already started by defining notions like code coverage to neural networks. Lastly, we argue that the effective development of high quality AI-based systems need well suited frameworks that can facilitate the daily work of scientists and software developers – like the Deep-Water framework, presented in the closing part of the paper.
Beszédes Á. , Horváth F. , Di Penta M. & Gyimóthy T. , 'Leveraging Contextual Information from Function Call Chains to Improve Fault Localization ' 27th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2020) , , London, Ontario, Canada (accepted for presentation) .
Chen Tsong Yueh , Kuo Fei-Ching , Liu Huai , Poon Pak-Lok , Towey Dave , Tse T. H. & Zhou Zhi Quan , 'Metamorphic Testing ' (2018 ) 51 (1 ) ACM Computing Surveys , 04 .
https://www.comet.ml/ Comet website Comet
Csuvik Viktor , Horvath Deniel , Horvath Ferenc & Vidacs Laszlo , 'Utilizing Source Code Embeddings to Identify Correct Patches ' 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF) , , .
Databricks Inc. Mlflow, an open source platform for the machine learning lifecycle 2020 https://mlflow.org/
Eniser H.F. , Gerasimou S. & Sen A. R. Hähnle & W. van der Aalst (eds)DeepFault: Fault Localization for Deep Neural Networks Fundamental Approaches to Software Engineering. FASE 2019. Lecture Notes in Computer Science Springer, Cham. 11424 2019
European Commission (eds) On Artificial Intelligence – A European approach to excellence and trust. White paper COM(2020) 65, Brussels 2020.02.19. https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-feb2020_en.pdf 2020.03.04. 2020
Fischetti Matteo & Jo Jason , 'Deep neural networks and mixed integer linear optimization ' (2018 ) 23 (3 ) Constraints , 04 .
Forti M. , Nistri P. & Papini D. , 'Global Exponential Stability and Global Convergence in Finite Time of Delayed Neural Networks With Infinite Gain ' (2005 ) 16 (6 ) IEEE Transactions on Neural Networks , 11 .
Gandy A. & Scott J. , 'Unit Testing for MCMC and other Monte Carlo Methods ' (2020 ) arXiv: Methodology .
Greff Klaus , Klein Aaron , Chovanec Martin , Hutter Frank & Schmidhuber Jürgen , 'The Sacred Infrastructure for Computational Research ' (2018 ) figshare .
Gleave A. , Dennis M. , Wild C. , Kant N. , Levine S. & Russell S. , 'Adversarial Policies: Attacking Deep Reinforcement Learning ' International Conference on Learning Representations , , .
Hamm Lonnie , Brorsen B. Wade & Hagan Martin T. , 'Comparison of Stochastic Global Optimization Methods to Estimate Neural Network Weights ' (2007 ) 26 (3 ) Neural Processing Letters , 09 .
Harrold M. J. & Stasko J. , 'Visualization of test information to assist fault localization ' 24th International Conference on Software Engineering, ACM , , .
Hodovan Renata & Kiss Akos , 'Fuzzinator: An Open-Source Modular Random Testing Framework ' 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) , , .
Hodován R. , Vince D. & Kiss Á. , 'Fuzzing JavaScript Environment APIs with Interdependent Function Calls ' Integrated Formal Methods – 15th International Conference, IFM 2019 , , Bergen, Norway .
Islam J. , Pan R. , Nguyen G. & Rajan H. , 'Repairing Deep Neural Networks: Fix Patterns and Challenges ' ICSE’20: The 42nd International Conference on Software Engineering , , .
Megyeri I. , Hegedűs I. & Jelasity M. , 'Adversarial Robustness of Model Sets ' Proceedings of the International Joint Conference on Neural Networks (IJCNN) , , .
Azure Machine Learning Studio Microsoft 2018 https://studio.azureml.net/
Ehmer Mohd & Khan Farmeena , 'A Comparative Study of White Box, Black Box and Grey Box Testing Techniques ' (2012 ) 3 (6 ) International Journal of Advanced Computer Science and Applications .
Polyaxon an open source platform for reproducible machine learning at scale POLYAXON 2019 https://polyaxon.com/
Rauber Paulo E. , Fadel Samuel G. , Falcao Alexandre X. & Telea Alexandru C. , 'Visualizing the Hidden Activity of Artificial Neural Networks ' (2017 ) 23 (1 ) IEEE Transactions on Visualization and Computer Graphics , 01 .
Ferenc Rudolf , Viszkok Tamás , Aladics Tamás , Jász Judit & Hegedűs Péter , 'Deep-water framework: The Swiss army knife of humans working with machine learning models ' (2020 ) 12 SoftwareX , 07 .
STUDIO.ML COMMUNITY Studio.ml STUDIO.ML COMMUNITY 2017 https://studioml.readthedocs.io/en/latest/
Szegedy C. , Zaremba W. , Sutskever I. , Bruna J. , D. Erhan D. , Goodfellow I. J. & Fergus R. , 'Intriguing properties of neural networks ' 2nd International Conference on Learning Representations (ICLR) , , .
Tarlow D. , Moitra S. , Rice A. , Chen Z. , Manzagol P-A. , Sutton C. & Aftandilian E. , 'Learning to Fix Build Errors with Graph2Diff Neural Networks ' 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW) , , Seoul, Republic of Korea .
University of Szeged Department of Software Engineering Deep Water Framework University of Szeged Department Of Software Engineering 2019 https://github.com/sed-inf-u-szeged/DeepWaterFramework
Zhu Hong , Liu Dongmei , Bayley Ian , Harrison Rachel & Cuzzolin Fabio , 'Datamorphic Testing: A Method for Testing Intelligent Applications ' 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) , , .
Xie Xiaofei , See Simon , Ma Lei , Juefei-Xu Felix , Xue Minhui , Chen Hongxu , Liu Yang , Zhao Jianjun , Li Bo & Yin Jianxiong , 'DeepHunter: a coverage-guided fuzz testing framework for deep neural networks ' the 28th ACM SIGSOFT International Symposium , , .
Xu Han , Ma Yao , Liu Hao-Chen , Deb Debayan , Liu Hui , Tang Ji-Liang & Jain Anil K. , 'Adversarial Attacks and Defenses in Images, Graphs and Text: A Review ' (2020 ) 17 (2 ) International Journal of Automation and Computing , 03 .
Sun Youcheng , Huang Xiaowei , Kroening Daniel , Sharp James , Hill Matthew & Ashmore Rob , 'Structural Test Coverage Criteria for Deep Neural Networks ' (2019 ) 18 (5s ) ACM Transactions on Embedded Computing Systems , 10 .