Conventional machine learning algorithms

Unsupervised learning is used when the available training data consists of data points neither classified nor labelled. The algorithm uses different methods – such as k-means clustering, principal component analysis, Gaussian mixture model, or hidden Markov model – to identify previously unknown patterns [3]. As an example, images of cats and dogs can be fed to the algorithm without labels (so we do not tell the algorithm which is a cat and which is a dog). If the algorithm can distinguish cats from dogs with high accuracy, it means that the algorithm was able to identify the dominant pattern that is unique to cats and was able to successfully distinguish between the two groups without human assistance. An example in radiology may be a classification of kidney tumors. In this case, the images of the tumors will be grouped, but we can not know how (based on which property) the groups are formed. We can not choose the desired separation criteria (for example we can not separate based on histology), so the algoritm may ditinguish based on the contrast agent enhancement, size of the necrotic part or any other property.

Supervised learning means that categorized input and output data are provided for the algorithm in the learning phase. For classification problems (when we want to predict qualitative outputs), support vector machines, discriminant analysis, naive Bayesian classifiers, or nearest neighbour methods can be used. In supervised learning of conventional machine learning methods, the algorithms create a mathematical model based on the training data points that gives the least number of wrong classifications, in case of a classification problem. For regression problems (when we want to predict quantitative output), linear regression, support vector regression, ensemble methods, or decision trees can be used [3]. If we want to solve a regression problem, the mathematical model is created based on algorithm-specific regularization parameters. An error term can be defined (for example mean squared error) and the algorithm chooses its model by minimizing this error term [4]. As an example, we can feed the algorithm with a large number of images of cats and dogs with labels (so we tell the algorithm whether the image is of a cat or a dog), and then give input images of cats and dogs mixed. The algorithm will be able to determine if a cat is present in the given image or if that is not a cat, so it will separate the data into two classes, with one group of cats and another group of not cats. In radiology, supervised learning makes it possible to train an algorithm for the classification of tumorous tissues and healthy tissues. It required to feed the algorithm with a large number of images of tumorous organs and a large number of healthy organs. Based on the learned pattern, the algorithm will be able to determine if a tumor is present in a new image.

Reinforcement learning is mathematically an optimal sequential decision-making algorithm [5]. The basic concept is to learn by trial and the subsequent error based on the interaction with the environment. After each decision, the environment changes, which makes this method appropriate for dynamic environments. A reinforcement learning method does not know what the right output is, as it has unlabelled data. Thus, it cannot adapt itself to the right output and cannot know how accurate it is. There is no binary feedback after each decision, it knows the goal and chooses the decision path that gives the highest accumulated reward over time. Reinforcement learning uses reward and penalty to compute the loss value as feedback information, and the algorithm learns as maximizing the reward while minimizing the penalty [6]. In medical imaging, reinforcement learning framework can be used for image segmentation. With the controlling of local threshold and post-processing parameters using a reinforcement learning agent, the segmentation of the prostate in ultrasound images can be achieved [7].

Neural networks

Unsupervised learning aims to find hidden patterns or to cluster data. In the case of unsupervised learning of neural networks, the input data is unlabelled and uncategorized, and the algorithm learns by itself and uses more complex processing methods and operations compared to supervised learning. That is why it requires orders of magnitude more data [8].

The latest and most complex form of supervised learning is the convolutional neural network, which is able to learn from the accuracy of the predicted output. If its actual output is wrong, it learns and improves its efficiency by minimizing the so-called loss value as feedback information.

Deep reinforcement learning stands for cases when a deep neural network is used to learn representations to solve reinforcement learning problems. It performs sequential decisions, and the success or failure is determined after multiple steps. Deep learning methods for reinforcement learning problems make it possible to scale up to previously unmanageable problems [6].

Conventional machine learning algorithms

Support vector machines (SVMs) are one of the most widespread examples of classical ML algorithms. SVMs are designed for both classification and regression problems [8]. In the case of a support vector classifier (SVC), the algorithm finds the optimal hyperplane that separates the predefined groups of the training cases [9]. In the case of a two-dimensional dataset with two variables, the algorithm defines the x and y axis along the first and second variable and aims to find a line with the largest possible margin from the support vector data points which separate the cases into two segments, so to classify our data into two separate groups (Fig. 2). The algorithm can also work with a larger number of variables, e.g. in 3D, it computes a plane instead of a line, moreover by increasing the number of variables, it is capable of computing the optimal hyperplane in even higher dimensions that cannot be further visualized. Although the algorithm was originally designed to separate linearly separable classes, one of its biggest advantages that makes it the most widely used ML algorithm is its flexibility. With the so-called “kernel trick”, the usage of the algorithm can be expanded to classify linearly non-separable data. It is achieved by mapping the original data to higher dimensional space and finding a linear separating hyperplane there [10].

Support vector machines. Support vector machines (SVMs) are one of the most widespread examples of classical ML algorithms. The algorithm finds an optimal hyperplane (in 2D a hyperplane is a straight line), which separates the data points into classes in a way that the distance to the support vectors is minimal. The data points which support the hyperplane are called support vectors

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Support vector machines. Support vector machines (SVMs) are one of the most widespread examples of classical ML algorithms. The algorithm finds an optimal hyperplane (in 2D a hyperplane is a straight line), which separates the data points into classes in a way that the distance to the support vectors is minimal. The data points which support the hyperplane are called support vectors

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Support vector machines. Support vector machines (SVMs) are one of the most widespread examples of classical ML algorithms. The algorithm finds an optimal hyperplane (in 2D a hyperplane is a straight line), which separates the data points into classes in a way that the distance to the support vectors is minimal. The data points which support the hyperplane are called support vectors

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Artificial neural networks and deep learning

Artificial Neural Networks (ANNs) are networks consisting of artificial neurons, which imitate the human nervous system. In the brain, the neurons fire depending on the sum of the excitatory and inhibitory input signals in many consecutive layers of the grey matter. Similarly, an ANN has nodes that sum the inputs (multiplied by the corresponding weight of the input) and turn on or off depending on the result of the activation function (Fig. 3). The weights of the inputs can be interpreted as the strength of the corresponding incoming connection. These weights are the parameters which have to be trained to configure the network. An ANN can be divided into multiple layers: an input layer, at least one hidden layer, and an output layer. The output of the previous layer functions as the input of the consecutive layer, and the output layer linearly combines the outputs of the hidden units, so each patch is mapped to any desired output value. ANNs that have only one hidden layer are also known as shallow neural networks, while an ANN is called a deep neural network if it has more than one hidden layer. The advantage of deep architectures is that the consecutive hidden layer of a deep neural network can re-use the features which were computed in the previous hidden layer, which can improve the generalization and precision [11].

The nodes of the artificial neural networks. They are similar to the neurons of the brain. The neurons get inputs from multiple sources and the neuron activates depending on the sum of all input signals. Mathematically, activation is decided depending on the result of the activation function

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The nodes of the artificial neural networks. They are similar to the neurons of the brain. The neurons get inputs from multiple sources and the neuron activates depending on the sum of all input signals. Mathematically, activation is decided depending on the result of the activation function

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

The nodes of the artificial neural networks. They are similar to the neurons of the brain. The neurons get inputs from multiple sources and the neuron activates depending on the sum of all input signals. Mathematically, activation is decided depending on the result of the activation function

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Along with the development of information technology, the number of types of neural networks is constantly increasing. In the field of medical imaging, convolutional neural networks (CNNs) are considered as the most popular type of ANNs, but Recurrent Neural Networks (RNNs) and Generative Adversarial Networks (GANs) are becoming more and more popular and widespread.

Convolutional neural networks can perform convolutions, also known as template matching. Convolution is a mathematical operation: $\underset{-\infty }{\overset{\infty }{{\displaystyle \int }}}f(\tau )g\left(t-\tau \right)d\tau$. What it means in other words is the amount of overlap as one of the functions is shifted over the other function. The structure of a CNN includes multiple neurons that share weights, which enables to apply filters (or convolutional kernels) to the input images. This means that a subset of pixels of the input image is multiplied by the filter matrix, and the result matrix is summed up, so that for example from a 2 × 2 matrix, the result will be one pixel value (Figs 4 and 5). The CNN layer which performs this operation multiple times at different locations of the image is called a convolutional layer. The advantage of using convolutional layers is extracting different sharp and smooth image features while also compressing the data to save computation capacity and reduce memory usage. To further compress the data during processing while also reducing overfitting, convolutional layers are usually combined with max-pooling layers in the network, which execute similar filtering processes, but choose the maximum value of the input pixels and add this maximal value to the result matrix [12].

Application of the kernel matrix. The pixel values are multiplied by the kernel matrix. The kernel filter is applied to consecutive parts of the original pixel data

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Application of the kernel matrix. The pixel values are multiplied by the kernel matrix. The kernel filter is applied to consecutive parts of the original pixel data

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Application of the kernel matrix. The pixel values are multiplied by the kernel matrix. The kernel filter is applied to consecutive parts of the original pixel data

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Results of the filters. The red line shows the chosen part of the image. The representation of this particular filter is a curved line (see the values with 30 surrounded by 0 values), the corresponding matrix representing the given selected part of the analysed image is shown on the left side of the Figure. During applying the kernel, each value of the image matrix is multiplied by the corresponding kernel unit value in the same location, then the entire matrix is summed up. If we apply the filter to the given matrix, the result of the multiplication and summation will be a large number, as the kernel units with large values overlap with large pixel values in the same location

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Results of the filters. The red line shows the chosen part of the image. The representation of this particular filter is a curved line (see the values with 30 surrounded by 0 values), the corresponding matrix representing the given selected part of the analysed image is shown on the left side of the Figure. During applying the kernel, each value of the image matrix is multiplied by the corresponding kernel unit value in the same location, then the entire matrix is summed up. If we apply the filter to the given matrix, the result of the multiplication and summation will be a large number, as the kernel units with large values overlap with large pixel values in the same location

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Results of the filters. The red line shows the chosen part of the image. The representation of this particular filter is a curved line (see the values with 30 surrounded by 0 values), the corresponding matrix representing the given selected part of the analysed image is shown on the left side of the Figure. During applying the kernel, each value of the image matrix is multiplied by the corresponding kernel unit value in the same location, then the entire matrix is summed up. If we apply the filter to the given matrix, the result of the multiplication and summation will be a large number, as the kernel units with large values overlap with large pixel values in the same location

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

In recurrent neural networks, the connections are not simply feedforward, but they can form cycles. Consequently, this kind of network can execute more complex computations and gives its best performance in tasks with dependent observations. RNNs can preserve and maintain an internal memory over time, they are able to recognize or generate temporal patterns. Therefore, RNNs can be used for the perception of video or speech, linguistic tasks, translation, and motoric planning and control [11]. However, there have been examples in medical imaging as well [13].

There are some additional methods to fine-tune and improve the performance of the algorithms. Generative adversarial networks (GANs) have gained attention recently as the latest breakthrough in the field of deep learning. The basic idea behind this technique is that two neural networks are being simultaneously trained: one for image generation and one for discrimination. The first network transforms random patterns into images, while the second network separates the synthetic images and the authentic examples of the desired category. That way it is possible to integrate unlabelled samples into training and generate synthetic data. In medical imaging, GANs are used either to generate new images as a form of data augmentation (after the exploration of the underlying structure of the training data) or to discriminate between normal and abnormal images [14].

Defining ground truth

The evaluation of the data from clinical studies has many crucial steps. Firstly, suitable cases must be selected, and the data is needed to be anonymized. Secondly, the proper labelling of the images (or defining the ground truth) should be done by experts. For some diagnoses, the image itself contains enough information (e.g., intracranial haemorrhage), but in other cases, clinical information (e.g., pathology) or additional imaging (e.g., lung cancer on X-ray and subsequent CT) examinations may be required [17].

Another important potential bias is population variability, local disease prevalence, and different imaging protocols [17]. To avoid overfitting, the development of an AI algorithm needs diverse but balanced training data, which should be from different counties and various geographic regions. Unfortunately, limited access to medical data makes it difficult to obtain a diverse enough collection of images.

Curation, augmentation

Data from the real world is usually diverse. If we want to analyse them with neural networks, there are parameters that have to be the same in the case of all the data. For example, analysing volumetric CT or MRI images is challenging due to the huge variety in voxel size, the number of slices, slice thickness, inter-slice gap, etc. The curation of medical data consists of quality checking, noise filtering, metadata creation, and annotation of the data. To create more data and make the algorithms more generalizable, some forms of data augmentation can also be applied such as flipping or rotating the image, zooming, changing the contrast or resolution, or altering the location of the lesion [2].

Training, validation, testing

With the help of the training data, the algorithm gets trained, and the parameters are optimized. The validation during training is performed on the validation set to investigate the performance of the model and find the best-fitting one. Finally, the performance of the trained model is evaluated on the independent test dataset. All these steps again require high-quality, but independent datasets.

Image classification

Classification means assigning the data points or images to discrete (specified number of) categories. Image classification is one of the first problems for which AI algorithms have been used in radiology. A typical image classification problem can be explained as the following: one or multiple images are the input, and the algorithm decides whether the specified disease is present or not, so the output is a single decision (Fig. 6A). One strategy is to use our own data for the training phase to train the algorithm. The other strategy is to use pre-trained networks by transfer learning which is preferable in that case if only a limited number of training cases are available. Then, we can either use the original weights of a previously published, trained neural network, or we can finetune its weights by continuing its training with cases of our own training dataset [18]. Convolutional neural networks have shown excellent results in connection with image classification. The first layer has an input at the level of pixels, and each operation is a small convolution, which then spread forward to the network. In that way, CNNs can reduce the number of hyperparameters needed while also effectively extracting the features.

Comparison of image classification, object detection, instance and semantic segmentation. The classification decides if a dog is present in the image or not. Object detection localizes the dogs. The instance segmentation draws the outlines of the dogs (each of the dogs separately). The semantic segmentation categorizes each pixel: dog, floor, or rug

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Comparison of image classification, object detection, instance and semantic segmentation. The classification decides if a dog is present in the image or not. Object detection localizes the dogs. The instance segmentation draws the outlines of the dogs (each of the dogs separately). The semantic segmentation categorizes each pixel: dog, floor, or rug

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Comparison of image classification, object detection, instance and semantic segmentation. The classification decides if a dog is present in the image or not. Object detection localizes the dogs. The instance segmentation draws the outlines of the dogs (each of the dogs separately). The semantic segmentation categorizes each pixel: dog, floor, or rug

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Object detection

The main goal of object detection is to decide whether objects of a specific category is present in the image or not (Fig. 6B). There are two types of object detection methods. One of them is the proposal-based framework, which means that the algorithm selects regions (region proposals) and then classifies these into categories. The widely used networks R-CNN [19], Fast R-CNN [20] and Mask R-CNN [21] are using this approach. The other method is regression-based, so it uses a unified framework and directly executes categories and localizations. MultiBox [22], You only look once (YOLO) [23] and RetinaNet [24] networks belong to this category [25].

Segmentation

Segmentation stands for the partition of the image pixels/voxels into subgroups called image segments. Practically it means to delineate the boundaries of the desired area or volume (i.e., the boundaries of an organ, tumour, or lesion). Currently, the gold standard is manual segmentation. There are several segmentation programs, such as 3D Slicer [26], which make it possible to draw the boundary of the object manually in each of the slices. In the case of instance segmentation, each of the objects is drawn independently (Fig. 6C). During semantic segmentation, the AI algorithm segments the images based on the pixels, so a semantic label is assigned to each of the pixels (Fig. 6D). Thus, this method does not only identify the object itself but marks its boundaries as well. A potential application area is the delineation of tumour boundaries (Fig. 7), in that case, precise delineation is crucial for clinical decision-making or surgical resection [27].

* Segmentation of the liver. Based on the original CT scan (A), a trained CNN can propose a segmentation heatmap (B). A probability value is assigned to each pixel. Warmer colour means higher possibility that the current pixel belongs to the liver. A threshold can be set, only the pixels with probability value over this threshold will be assigned as part of the segmentation. The gold standard is manual segmentation (C). The segmented area can be reconstructed in 3 dimensions too (D). * Central illustration

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

* Segmentation of the liver. Based on the original CT scan (A), a trained CNN can propose a segmentation heatmap (B). A probability value is assigned to each pixel. Warmer colour means higher possibility that the current pixel belongs to the liver. A threshold can be set, only the pixels with probability value over this threshold will be assigned as part of the segmentation. The gold standard is manual segmentation (C). The segmented area can be reconstructed in 3 dimensions too (D). * Central illustration

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

* Segmentation of the liver. Based on the original CT scan (A), a trained CNN can propose a segmentation heatmap (B). A probability value is assigned to each pixel. Warmer colour means higher possibility that the current pixel belongs to the liver. A threshold can be set, only the pixels with probability value over this threshold will be assigned as part of the segmentation. The gold standard is manual segmentation (C). The segmented area can be reconstructed in 3 dimensions too (D). * Central illustration

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Data availability

In the area of digital healthcare, it is still difficult to collect enough data as the medical data is not as clean, structured, and complete as in other fields. Unfortunately, medical images are often noisy and indistinct. Neural networks usually have hundred-thousands or millions of hyperparameters; for the effective training of machine learning algorithms, large amount of data is required which we call ‘data-hungry’. In clinical practice, collecting such amount of information is often not feasible as the patients may not be willing to allow their private data for research purposes [8]. Even the distribution and prevalence of the given disease could make it challenging to assemble such a dataset. The dataset should be representative of the population and must be precisely annotated. It should be balanced in patient age, ethnicity and imaging protocols among other properties. In the case of more expensive examinations, such as MRI, a relatively small number of patients are screened, which may also depend on the examination protocol of the different health systems [28].

Overfitting, overtraining

Overfitting can be a large problem in the application of machine learning algorithms. When the proposed model has near zero error – i.e., the constructed hyperplane of the support vector classifier fits all the data without any error – it is possible that our model captures the noise of our data rather than the real pattern we are looking for. In such case, after a random train/test split of the dataset, the shape of the hyperplane or the mathematical equation of it can show huge differences depending on the given training cases. In the case of neural networks, the phenomenon of overfitting is often caused by excessively maximized training accuracy (Fig. 8) [29].

Overfitting. In case of overfitting, the model may find trivial patterns – noise – in the dataset, so the proposed curve has zero error on the training dataset (A). The overfitted model can not be generalized, because it produces large error in external datasets. The optimal classification works with relatively small error in both the training and the external testing datasets, which makes it generalizable (B)

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Overfitting. In case of overfitting, the model may find trivial patterns – noise – in the dataset, so the proposed curve has zero error on the training dataset (A). The overfitted model can not be generalized, because it produces large error in external datasets. The optimal classification works with relatively small error in both the training and the external testing datasets, which makes it generalizable (B)

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Overfitting. In case of overfitting, the model may find trivial patterns – noise – in the dataset, so the proposed curve has zero error on the training dataset (A). The overfitted model can not be generalized, because it produces large error in external datasets. The optimal classification works with relatively small error in both the training and the external testing datasets, which makes it generalizable (B)

Citation: Imaging 14, 2; 10.1556/1647.2022.00104

• Download Figure
• Download figure as PowerPoint slide

Legal and ethical challenges, patient acceptance

AI may be used in clinical practice in the future, but acceptance by the patients and by the medical professionals may be controversial. The case is similar to many new technologies (e.g., self-driving cars), when the legal regulations are far behind the technology. The legal system does not define who would be accountable for possible mistakes that AI makes. Another question is whether patient will accept and trust the clinical workflow without human involvement [12].