Scientia et Securitas
Volume/Issue:
Volume 4: Issue 4
Egzakt, véges mintás, sztochasztikus garanciák gépi tanulási módszerekhez
Author:
Ambrus Tamás Magyar Kutatási Hálózat (HUN-REN), Számítástechnikai és Automatizálási Kutatóintézet (SZTAKI) Budapest Magyarország; Hungarian Research Network (HUN-REN), Institute for Computer Science and Control (SZTAKI) Budapest Hungary
Eötvös Loránd Tudományegyetem (ELTE), Matematikai Intézet Budapest Magyarország; Eötvös Loránd University (ELTE), Institute of Mathematics Budapest Hungary

Search for other papers by Ambrus Tamás in
Current site
Google Scholar
PubMed Close
https://orcid.org/0000-0002-6049-4626
Pages:
302–311
Online Publication Date:
05 Aug 2024
Publication Date:
05 Aug 2024
Article Category:
Research Article
DOI:
https://doi.org/10.1556/112.2023.00176
Keywords:
statistical learning theory; uncertainty quantification; confidence regions;
Keywords (Hungarian):
statisztikus tanuláselmélet; bizonytalanság vizsgálat; konfidenciahalmazok;
Open access

Összefoglalás.

A gépi tanulás a mesterségesintelligencia-kutatások egyik fő pillére, melynek matematikai hátterét a statisztikus tanuláselmélet biztosítja. A gépi tanulási módszerek bizonytalanságának meghatározása esszenciálissá vált számos alkalmazás esetében, többek között a biztonság, a stabilitás és a minőség garantálása érdekében. Ebben a tanulmányban újra-mintavételező eljáráson alapuló konfidenciahalmaz-becsléseket mutatunk be, amelyek eloszlásfüggetlen és véges mintás korlátokat biztosítanak a becslések bizonytalanságára. A konfidenciahalmazokat egzakt és konzisztens rangtesztek segítségével konstruáljuk meg, és számos példán szemléltetjük. A várható értékre és a feltételes várhatóérték-függvényre adunk halmazbecslést felügyelt tanulási feladatok (regresszió és osztályozás) esetében.

Summary.

Machine learning (ML) is one of the fundamental methodologies within artificial intelligence (AI), which is widely used in several fields, such as finance, health care, manufacturing and control. A classical problem in ML is to estimate the unknown parameters of an environment based on noisy observations. In many real-world applications involving safety and stability criterions, it is essential to quantify the uncertainty of these estimates.

Confidence regions are classical tools in uncertainty quantification, often used to assess the quality of an estimate. In this paper we investigate a resampling framework to construct confidence regions and hypothesis tests for several supervised learning problems. First, we present a resampling scheme to build confidence intervals for the expected value of an unknown symmetric distribution. We generate alternative observations based on a candidate parameter, then compare the original sample to the alternative dataset. If the candidate parameter is equal to the true expected value, then the alternative samples have the same distribution as the original sample, otherwise the alternative samples follow a different distribution. This distributional difference can be detected with a rank test. The main ideas of this procedure can be generalized to regression and binary classification problems, in which cases we build confidence regions for the true regression function.

The presented methods are endowed with strong theoretical guarantees. The user-chosen confidence levels of the confidence regions are non-asymptotically exact. The confidence sets are defined via flexible rank tests, while our distributional assumptions are very mild. We usually assume symmetry or exchangeability and therefore the presented algorithms are distribution-free, thus superior to many other methods in practice. We also investigate the asymptotic behaviors of the presented methods and establish several theoretical results regarding the consistency of the confidence regions.

  • 1

    Bishop, C. M. (2006) Pattern Recognition and Machine Learning. New York, Springer

  • 2

    Carè, A. (2022) A Simple Condition for the Boundedness of Sign-Perturbed-Sums (SPS) confidence regions. Automatica, Vol. 139. 110150.

  • 3

    Carè, A., Campi, M. C., Csáji, B. Cs., & Weyer, E. (2021) Facing Undermodelling in Sign-Perturbed Sums System Identification. Systems & Control Letters, Elsevier, Vol. 153. 104936.

  • 4

    Csáji, B. Cs., Campi, M. C., & Weyer, E. (2014) Sign-Perturbed Sums: A New System Identification Approach for Constructing Exact Non-Asymptotic Confidence Regions in Linear Regression Models. IEEE Transactions on Signal Processing, Vol. 63. No. 1. pp. 169–181.

  • 5

    Csáji, B. Cs., & Kis, K. B. (2019) Distribution-Free Uncertainty Quantification for Kernel Methods by Gradient Perturbations. Machine Learning, Springer, Special Issue of the European Conference on Machine Learning (ECML PKDD Journal Track), Vol. 108. pp. 1677–1699.

  • 6

    Csáji, B. Cs., & Tamás, A. (2019) Semi-Parametric Uncertainty Bounds for Binary Classification, 58th IEEE Conference on Decision and Control (CDC), Nice, France, pp. 4427–4432.

  • 7

    Csáji, B. Cs., & Weyer, E. (2015) Closed-Loop Applicability of the Sign-Perturbed Sums Method. 54th IEEE Conference on Decision and Control (CDC), Osaka, Japan, pp. 1441–1446.

  • 8

    Efron, B., & Tibshirani, R. J. (1994) An Introduction to the Bootstrap. New York, CRC Press

  • 9

    Haykin, S. (1998) Neural Networks: A Comprehensive Foundation. Upper Saddle River, Prentice Hall PTR

  • 10

    McCulloch, W. S., & Pitts, W. (1943) A Logical Calculus of the Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biophysics, Vol. 5. No. 4. pp. 115–133.

  • 11

    Misuraca, G., & Van Noordt, C. (2020) AI Watch: Artificial Intelligence in public services: Overview of the use and impact of AI in public services in the EU. JRC Research Reports of the European Commission, JRC118163

  • 12

    Pitman, E. J. G. (1937) Significance Tests Which May Be Applied to Samples from any Populations. II. The Correlation Coefficient Test. Supplement to the Journal of the Royal Statistical Society, Vol. 4. No. 2. pp. 225–232.

  • 13

    Quenouille, M. H. (1956) Notes on Bias in Estimation. Biometrika, Vol. 43. No. 3–4. pp. 353–360.

  • 14

    Rosenblatt, F. (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, Vol. 65. No. 6. pp. 386–408.

  • 15

    Russell, S. J., & Norvig, P. (2020) Artificial Intelligence: A Modern Approach. 4th ed., New Jersey, Prentice Hall Series in Artificial Intelligence

  • 16

    Schölkopf, B., & Smola, A. J. (2002) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, USA, MIT Press

  • 17

    Student (1908) The Probable Error of a Mean. Biometrika, Vol. 6. No. 1. pp. 1–25.

  • 18

    Szentpéteri, Sz., & Csáji, B. Cs. (2023) Sample Complexity of the Sign-Perturbed Sums Identification Method: Scalar Case, 22nd IFAC World Congress (World Congress of the International Federation of Automatic Control). Yokohama, Japan, July 9–14. pp. 10363–10370.

  • 19

    Tamás, A., Bálint, D. Á., & Csáji, B. Cs. (2023) Robust Independence Tests with Finite Sample Guarantees for Synchronous Stochastic Linear Systems. IEEE Control Systems Letters (L-CSS), IEEE Press, Vol. 7. pp. 2701–2706.

  • 20

    Tamás, A., & Csáji, B. Cs. (2020) Sztochasztikus garanciák bináris klasszifikációhoz. Alkalmazott Matematikai Lapok, Vol. 37. No. 2. pp. 1–15.

  • 21

    Tamás, A., & Csáji, B. Cs. (2022) Exact Distribution-Free Hypothesis Tests for the Regression Function of Binary Classification via Conditional Kernel Mean Embeddings. IEEE Control Systems Letters (L-CSS), IEEE Press, Vol. 6. pp. 860–865.

  • 22

    Tamás, A., & Csáji, B. Cs. (2023) Distribution-Free Inference for the Regression Function of Binary Classification. arXiv preprint, arXiv:2308.01835

  • 23

    Vovk, V., Gammerman, A., & Shafer, G. (2005) Algorithmic Learning in a Random World. New York, Springer

  • 24

    Wald, A. (1943) Tests of Statistical Hypotheses Concerning Several Parameters When the Number of Observations is Large. Transactions of the American Mathematical Society, Vol. 54. No. 3. pp. 426–482.

  • 25

    Weyer, E., Campi, M. C., & Csáji, B. Cs. (2017) Asymptotic Properties of SPS Confidence Regions. Automatica, Vol. 82. pp. 287–294.

  • 26

    Zhu, L. (2005) Nonparametric Monte Carlo Tests and Their Applications. New York, Springer Science & Business Media

  • Collapse
  • Expand
Cover Scientia et Securitas
Scientia et Securitas
Online ISSN:
2732-2688
Submit Your Manuscript

 

The author instructions are available in PDF.
Please, download the Hungarian version from HERE, the English version from HERE.
The Submissions templates are available in MS Word.
For articles in Hungarian, please download it from HERE and for articles in English from HERE.

 

Editor-in-Chief:

Founding Editor-in-Chief:

  • Tamás NÉMETH

Managing Editor:

  • István SABJANICS (Ministry of Interior, Budapest, Hungary)

Editorial Board:

  • Attila ASZÓDI (Budapest University of Technology and Economics)
  • Zoltán BIRKNER (University of Pannonia)
  • Valéria CSÉPE (Research Centre for Natural Sciences, Brain Imaging Centre)
  • Gergely DELI (University of Public Service)
  • Tamás DEZSŐ (Migration Research Institute)
  • Imre DOBÁK (University of Public Service)
  • Marcell Gyula GÁSPÁR (University of Miskolc)
  • József HALLER (University of Public Service)
  • Charaf HASSAN (Budapest University of Technology and Economics)
  • Zoltán GYŐRI (Hungaricum Committee)
  • János JÓZSA (Budapest University of Technology and Economics)
  • András KOLTAY (National Media and Infocommunications Authority)
  • Gábor KOVÁCS (University of Public Service)
  • Levente KOVÁCS (Óbuda University)
  • Melinda KOVÁCS (Hungarian University of Agriculture and Life Sciences (MATE))
  • Miklós MARÓTH (Avicenna Institue of Middle Eastern Studies )
  • Judit MÓGOR (Ministry of Interior National Directorate General for Disaster Management)
  • József PALLO (University of Public Service)
  • István SABJANICS (Ministry of Interior)
  • Péter SZABÓ (Hungarian University of Agriculture and Life Sciences (MATE))
  • Miklós SZÓCSKA (Semmelweis University)

Ministry of Interior
Science Strategy and Coordination Department
Address: H-2090 Remeteszőlős, Nagykovácsi út 3.
Phone: (+36 26) 795 906
E-mail: scietsec@bm.gov.hu

DOAJ

2023  
CrossRef Documents 32
CrossRef Cites 15
Days from submission to acceptance 59
Days from acceptance to publication 104
Acceptance Rate 81%

2022  
CrossRef Documents 38
CrossRef Cites 10
Days from submission to acceptance 54
Days from acceptance to publication 78
Acceptance Rate 84%

2021  
CrossRef Documents 46
CrossRef Cites 0
Days from submission to acceptance 33
Days from acceptance to publication 85
Acceptance Rate 93%

2020  
CrossRef Documents 13
CrossRef Cites 0
Days from submission to acceptance 30
Days from acceptance to publication 62
Acceptance Rate 93%

Publication Model Gold Open Access
Submission Fee none
Article Processing Charge none

Scientia et Securitas
Language Hungarian
English
Size A4
Year of
Foundation		 2020
Volumes
per Year		 1
Issues
per Year		 4
Founder Academic Council of Home Affairs and
Association of Hungarian PhD and DLA Candidates
Founder's
Address		 H-2090 Remeteszőlős, Hungary, Nagykovácsi út 3.
H-1055 Budapest, Hungary Falk Miksa utca 1.
Publisher Akadémiai Kiadó
Publisher's
Address		 H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Responsible
Publisher		 Chief Executive Officer, Akadémiai Kiadó
Applied
Licenses		 CC-BY 4.0
CC-BY-NC 4.0
ISSN ISSN 2732-2688

Monthly Content Usage

Abstract Views Full Text Views PDF Downloads
Apr 2024 0 0 0
May 2024 0 0 0
Jun 2024 0 0 0
Jul 2024 0 0 0
Aug 2024 0 229 35
Sep 2024 0 33 8
Oct 2024 0 0 0