View More View Less
  • 1 MTA-SZTE Mesterséges Intelligencia Kutatócsoport [University of Szeged and Hungarian Academy of Sciences, MTA-SZTE Research Group on Artificial Intelligence], Szeged, Hungary
  • | 2 Szegedi Tudományegyetem Számítógépes Algoritmusok és Mesterséges Intelligencia Tanszék [University of Szeged, Department of Algorithms and AI], Szeged, Hungary
  • | 3 Szegedi Tudományegyetem Szoftverfejlesztési Tanszék [University of Szeged, Department of Software Engineering], Szeged, Hungary
Open access

Összefoglalás. A mély mesterséges neuronhálók elterjedése az ipari alkalmazásokban évekkel azok megbízhatóságával, értelmezhetőségével, és biztonságával kapcsolatos szakterületek fejlődését megelőzően történt. Az egyik, gyakorlatban is jelentős területen, a képfelismerésben például a megvalósult megoldások szinte már emberi teljesítményre képesek, de ezzel együtt célzott zajjal ezek a rendszerek félrevezethetők, megzavarhatók. Jelen kéziratban ismertetünk néhány tipikus biztonsági problémát, valamint rámutatunk arra, hogy a hagyományos szoftverfejlesztés területén alkalmazott minőségbiztosítási módszerekkel rokon megoldásokra szükség van az MI-re épülő rendszerek fejlesztésében, akár a mesterséges neuronhálók biztonságát, akár az MI rendszerek hagyományos komponenseinek fejlesztését tartjuk szem előtt.

Summary. Research on the trustworthiness, interpretability and security of deep neural networks lags behind the widespread application of the technology in industrial applications. For example, in image recognition, modern solutions are capable of nearly human performance. However, with targeted adversarial noise, these systems can be arbitrarily manipulated. Here, we discuss some of the security problems and point out that quality assurance methods used in traditional software development should also be adapted when developing AI-based systems, whether in the security of artificial neural networks or traditional components of AI systems. One of the main concerns about neural networks today that – to the best of our knowledge – affects all deep neural networks is the existence of adversarial examples. These examples are relatively easy to find and according to a recent experiment, a well-chosen input can attack more networks at the same time. In this paper we also present a wider perspective of security of neural architectures borrowed from the traditional software engineering discipline. While in traditional development several methods are widely applied for software testing and fault localization, there is a lack of similar well-established methods in the neural network context. In case of deep neural networks, systematic testing tools and methods are in the early stage, and a methodology to test and verify the proper behavior of the neural networks is highly desirable. Robustness testing of machine learning algorithms is a further issue. This requires the generation of large random input data using fuzz testing methods. The adaptation of automatic fault localization techniques has already started by defining notions like code coverage to neural networks. Lastly, we argue that the effective development of high quality AI-based systems need well suited frameworks that can facilitate the daily work of scientists and software developers – like the Deep-Water framework, presented in the closing part of the paper.

  • Beszédes Á., Horváth F., Di Penta M. & Gyimóthy T. , 'Leveraging Contextual Information from Function Call Chains to Improve Fault Localization ' 27th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2020) , , London, Ontario, Canada (accepted for presentation).

    • Search Google Scholar
    • Export Citation
  • Chen Tsong Yueh, Kuo Fei-Ching, Liu Huai, Poon Pak-Lok, Towey Dave, Tse T. H. & Zhou Zhi Quan , 'Metamorphic Testing ' (2018 ) 51 (1 ) ACM Computing Surveys , 04 .

  • https://www.comet.ml/ Comet website Comet

  • Csuvik Viktor, Horvath Deniel, Horvath Ferenc & Vidacs Laszlo , 'Utilizing Source Code Embeddings to Identify Correct Patches ' 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF) , , .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Databricks Inc. Mlflow, an open source platform for the machine learning lifecycle 2020 https://mlflow.org/

  • Eniser H.F., Gerasimou S. & Sen A. R. Hähnle & W. van der Aalst (eds)DeepFault: Fault Localization for Deep Neural Networks Fundamental Approaches to Software Engineering. FASE 2019. Lecture Notes in Computer Science Springer, Cham. 11424 2019

    • Crossref
    • Search Google Scholar
    • Export Citation
  • European Commission (eds) On Artificial Intelligence – A European approach to excellence and trust. White paper COM(2020) 65, Brussels 2020.02.19. https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-feb2020_en.pdf 2020.03.04. 2020

    • Search Google Scholar
    • Export Citation
  • Fischetti Matteo & Jo Jason , 'Deep neural networks and mixed integer linear optimization ' (2018 ) 23 (3 ) Constraints , 04 .

  • Forti M., Nistri P. & Papini D. , 'Global Exponential Stability and Global Convergence in Finite Time of Delayed Neural Networks With Infinite Gain ' (2005 ) 16 (6 ) IEEE Transactions on Neural Networks , 11 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gandy A. & Scott J. , 'Unit Testing for MCMC and other Monte Carlo Methods ' (2020 ) arXiv: Methodology .

  • Greff Klaus, Klein Aaron, Chovanec Martin, Hutter Frank & Schmidhuber Jürgen , 'The Sacred Infrastructure for Computational Research ' (2018 ) figshare .

  • Gleave A., Dennis M., Wild C., Kant N., Levine S. & Russell S. , 'Adversarial Policies: Attacking Deep Reinforcement Learning ' International Conference on Learning Representations , , .

    • Search Google Scholar
    • Export Citation
  • Hamm Lonnie, Brorsen B. Wade & Hagan Martin T. , 'Comparison of Stochastic Global Optimization Methods to Estimate Neural Network Weights ' (2007 ) 26 (3 ) Neural Processing Letters , 09 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harrold M. J. & Stasko J. , 'Visualization of test information to assist fault localization ' 24th International Conference on Software Engineering, ACM , , .

    • Search Google Scholar
    • Export Citation
  • Hodovan Renata & Kiss Akos , 'Fuzzinator: An Open-Source Modular Random Testing Framework ' 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) , , .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hodován R., Vince D. & Kiss Á. , 'Fuzzing JavaScript Environment APIs with Interdependent Function Calls ' Integrated Formal Methods – 15th International Conference, IFM 2019 , , Bergen, Norway .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Islam J., Pan R., Nguyen G. & Rajan H. , 'Repairing Deep Neural Networks: Fix Patterns and Challenges ' ICSE’20: The 42nd International Conference on Software Engineering , , .

    • Search Google Scholar
    • Export Citation
  • Megyeri I., Hegedűs I. & Jelasity M. , 'Adversarial Robustness of Model Sets ' Proceedings of the International Joint Conference on Neural Networks (IJCNN), , .

    • Search Google Scholar
    • Export Citation
  • Azure Machine Learning Studio Microsoft 2018 https://studio.azureml.net/

  • Ehmer Mohd & Khan Farmeena , 'A Comparative Study of White Box, Black Box and Grey Box Testing Techniques ' (2012 ) 3 (6 ) International Journal of Advanced Computer Science and Applications .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Polyaxon an open source platform for reproducible machine learning at scale POLYAXON 2019 https://polyaxon.com/

  • Rauber Paulo E., Fadel Samuel G., Falcao Alexandre X. & Telea Alexandru C. , 'Visualizing the Hidden Activity of Artificial Neural Networks ' (2017 ) 23 (1 ) IEEE Transactions on Visualization and Computer Graphics , 01 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ferenc Rudolf, Viszkok Tamás, Aladics Tamás, Jász Judit & Hegedűs Péter , 'Deep-water framework: The Swiss army knife of humans working with machine learning models ' (2020 ) 12 SoftwareX , 07 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • STUDIO.ML COMMUNITY Studio.ml STUDIO.ML COMMUNITY 2017 https://studioml.readthedocs.io/en/latest/

  • Szegedy C., Zaremba W., Sutskever I., Bruna J., D. Erhan D., Goodfellow I. J. & Fergus R. , 'Intriguing properties of neural networks ' 2nd International Conference on Learning Representations (ICLR) , , .

    • Search Google Scholar
    • Export Citation
  • Tarlow D., Moitra S., Rice A., Chen Z., Manzagol P-A., Sutton C. & Aftandilian E. , 'Learning to Fix Build Errors with Graph2Diff Neural Networks ' 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW) , , Seoul, Republic of Korea .

    • Search Google Scholar
    • Export Citation
  • University of Szeged Department of Software Engineering Deep Water Framework University of Szeged Department Of Software Engineering 2019 https://github.com/sed-inf-u-szeged/DeepWaterFramework

    • Search Google Scholar
    • Export Citation
  • Zhu Hong, Liu Dongmei, Bayley Ian, Harrison Rachel & Cuzzolin Fabio , 'Datamorphic Testing: A Method for Testing Intelligent Applications ' 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) , , .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Xie Xiaofei, See Simon, Ma Lei, Juefei-Xu Felix, Xue Minhui, Chen Hongxu, Liu Yang, Zhao Jianjun, Li Bo & Yin Jianxiong , 'DeepHunter: a coverage-guided fuzz testing framework for deep neural networks ' the 28th ACM SIGSOFT International Symposium , , .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Xu Han, Ma Yao, Liu Hao-Chen, Deb Debayan, Liu Hui, Tang Ji-Liang & Jain Anil K. , 'Adversarial Attacks and Defenses in Images, Graphs and Text: A Review ' (2020 ) 17 (2 ) International Journal of Automation and Computing , 03 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sun Youcheng, Huang Xiaowei, Kroening Daniel, Sharp James, Hill Matthew & Ashmore Rob , 'Structural Test Coverage Criteria for Deep Neural Networks ' (2019 ) 18 (5s ) ACM Transactions on Embedded Computing Systems , 10 .

    • Crossref
    • Search Google Scholar
    • Export Citation
The author instructions are available in separate PDFs.
Please, download the Hungarian version from HERE, the English version from HERE.
The Submissions template is available in MS Word.
Please, download the file from HERE.
 

Ministry of Interior
Science Strategy and Coordination Department
Address: H-2090 Remeteszőlős, Nagykovácsi út 3.
Phone: (+36 26) 795 906
E-mail: scietsec@bm.gov.hu

Publication Model Gold Open Access
Submission Fee none
Article Processing Charge none

Scientia et Securitas
Language Hungarian
English
Size A4
Year of
Foundation
2020
Publication
Programme
2020 Volume 1
Volumes
per Year
1
Issues
per Year
4
Founder Academic Council of Home Affairs and
Association of Hungarian PhD and DLA Candidates
Founder's
Address
H-2090 Remeteszőlős, Hungary, Nagykovácsi út 3.
H-1055 Budapest, Hungary Falk Miksa utca 1.
Publisher Akadémiai Kiadó
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN ISSN 2732-2688

Editor-in-Chief:

  • Tamás NÉMETH 
    (Institute for Soil Sciences and Agricultural Chemistry, Centre for Agricultural Research
    Budapest, Hungary)

Managing Editor:

  • István SABJANICS (Ministry of Interior, Budapest, Hungary)

Editorial Board:

  • Melinda KOVÁCS (Szent István University Kaposvár Campus)Á
  • Miklós MARÓTH (Eötvös Loránd Research Network)
  • Charaf HASSAN (Budapest University of Technology and Economics)
  • Zoltán GYŐRI (Hungaricum Committee)
  • József HALLER (University of Public Service)
  • Attila ASZÓDI (Budapest University of Technology and Economics)
  • Zoltán BIRKNER (National Research, Development and Innovation Office)
  • Tamás DEZSŐ (Migration Research Institute)
  • Imre DOBÁK (University of Public Service)
  • András KOLTAY (University of Public Service)
  • Gábor KOVÁCS (University of Public Service)
  • József PALLO (University of Public Service)
  • Marcell Gyula GÁSPÁR (University of Miskolc)
  • Judit MÓGOR (Ministry of Interior National Directorate General for Disaster Management)
  • István SABJANICS (Ministry of Interior)
  • Péter SZABÓ (Hungarian University of Agriculture and Life Sciences (MATE))
  • Miklós SZÓCSKA (Semmelweis University)
  • János JÓZSA (Budapest University of Technology and Economics)
  • Valéria CSÉPE (Research Centre for Natural Sciences, Brain Imaging Centre)

Monthly Content Usage

Abstract Views Full Text Views PDF Downloads
Jan 2021 0 42 49
Feb 2021 0 34 36
Mar 2021 0 28 36
Apr 2021 0 32 33
May 2021 0 19 20
Jun 2021 0 11 16
Jul 2021 0 0 0