Deep Learning on Algoritma Technical Blog

Lung-Based COVID-19 Detection

Thu, 16 May 2024 00:00:00 +0000

Public Health

Lung-Based COVID-19 Detection

Background

“It is better to light a candle than curse the darkness."

Nowadays, every country in a corner of the world is still surviving about COVID-19. COVID-19, the worldwide-pandemic, is an issue that need to be highly concerned by everyone. Based on WHO’s data, ’till March 2021, the number of COVID-19’s survivor in the world is around 127 million people. Wow, it’s a very high number, right?!

Like the quote above, we better to light a candle than curse the darkness. Using the power of technology and especially artificial intelligence, we can light the candle in this darkness because of COVID-19 pandemic. Yay!

Because of this virus is a new virus and didn’t exist before, the researchers is still investigate the symptomps and the affect of COVID-19 to help the health-sector provide the best action to handle this virus. Based on theconversation.com, lungs are the organ most commonly affected by COVID-19, with a spectrum of severe effects. Because of COVID-19, people can have pneumonia, acute respiratory distress syndrome (ARDS), and other respiratory syndrome. Compared to other respiratory viruses, it causes marked clotting in the small blood vessels of the lungs.

So, in this case we want to detect COVID-19 on the human’s body by using Lung CT Scan-Images.

Data Preparation

Data that used on this case is images from https://www.kaggle.com/luisblanche/covidct. The images are collected from COVID19-related papers from medRxiv, bioRxiv, NEJM, JAMA, Lancet, etc. CTs containing COVID-19 abnormalities are selected by reading the figure captions in the papers.

We will use Python to processing our data and build the model

Import Library

First of all, we need to import library and package that needed on our next modeling

Import Data

And then, we import the data that we’ve already downloaded before. Because of our data is images, we use some library like os and glob to connecting and joining our images data.

We have 2 categories of images, lung of image from COVID-19 survivor and not.

1
2
3

data_root='dataset'
path_positive_covid = os.path.join(data_root, 'CT_COVID')
path_negative_covid = os.path.join(data_root, 'CT_NonCOVID')

1
2

# Print the paths to verify them
print(f"Path to positive COVID images: {path_positive_covid}")

`1`	`#> Path to positive COVID images: dataset\CT_COVID`

`1`	`print(f"Path to negative COVID images: {path_negative_covid}")`

`1`	`#> Path to negative COVID images: dataset\CT_NonCOVID`

# Collecting image file paths
positive_images = glob(os.path.join(path_positive_covid, "*.png"))
negative_images = glob(os.path.join(path_negative_covid, "*.png"))
negative_images.extend(glob(os.path.join(path_negative_covid, "*.jpg")))

covid = {
    'class': 'CT_COVID',
    'path': path_positive_covid,
    'images': positive_images
}

non_covid = {
    'class': 'CT_NonCOVID',
    'path': path_negative_covid,
    'images': negative_images
}

Exploratory Data Analysis

We can check our data detail to make sure that we already import the data properly.

if positive_images and negative_images:
    try:
        # Read the images
        ex_positive = cv2.imread(positive_images[0])
        ex_negative = cv2.imread(negative_images[0])

        # Check if the images are loaded correctly
        if ex_positive is not None and ex_negative is not None:
            # Convert images from BGR to RGB (OpenCV loads images in BGR format)
            ex_positive = cv2.cvtColor(ex_positive, cv2.COLOR_BGR2RGB)
            ex_negative = cv2.cvtColor(ex_negative, cv2.COLOR_BGR2RGB)

            # Create the plot
            fig = plt.figure(figsize=(10, 10))
            ax1 = fig.add_subplot(1, 2, 1)
            ax1.imshow(ex_positive)
            ax1.set_title('Positive Image')
            ax1.axis('off')

            ax2 = fig.add_subplot(1, 2, 2)
            ax2.imshow(ex_negative)
            ax2.set_title('Negative Image')
            ax2.axis('off')

            plt.show()  # Display the plot
        else:
            print("Error: One of the images couldn't be loaded. Check the file paths.")
    except Exception as e:
        print(f"An error occurred: {e}")
else:
    print("Error: Image lists are empty. Please provide valid image paths.")

#> <matplotlib.image.AxesImage object at 0x000001E0DAAA5ED0>
#> Text(0.5, 1.0, 'Positive Image')
#> (-0.5, 579.5, 334.5, -0.5)
#> <matplotlib.image.AxesImage object at 0x000001E0DAADEE90>
#> Text(0.5, 1.0, 'Negative Image')
#> (-0.5, 370.5, 217.5, -0.5)

1
2

#Check the number of Positive and Negative Cases
print("Total Positive Cases Covid19 images: {}".format(len(positive_images)))

`1`	`#> Total Positive Cases Covid19 images: 349`

`1`	`print("Total Negative Cases Covid19 images: {}".format(len(negative_images)))`

`1`	`#> Total Negative Cases Covid19 images: 397`

Modeling

The image classification is a classical problem of image processing, computer vision and machine learning fields. With image classification, machine can classifying an image from a fixed set of categories. One of the techniques that can use for image classification is Convolutional Neural Network (CNN) model.

Convolutional Neural Network

Is a neural network in which at least one layer is a convolutional layer. A typical convolutional neural network consists of some combination of the following layers:

Convolutional layers
Pooling layers
Dense layers

To build this model in this project, the workflow that can be used is :

Data Preparation
Modeling
Evaluation

To building the model, there are several step that will be used :

Data Splitting
Determine The Parameters
Building CNN Model

Data Splitting

In this part, we will separate our data to training data and testing data.

Before that, we need to make our directory or the place that we will use on the splitting data.

After that, we copy the images to test set and train set.

# Combine All Images
total_train_covid = len(os.listdir('dataset/train/CT_COVID'))
total_train_noncovid = len(os.listdir('dataset/train/CT_NonCOVID'))
total_test_covid = len(os.listdir('dataset/test/CT_COVID'))
total_test_noncovid = len(os.listdir('dataset/test/CT_NonCOVID'))

print("Train sets images COVID: {}".format(total_train_covid))

`1`	`#> Train sets images COVID: 313`

`1`	`print("Train sets images Non COVID: {}".format(total_train_noncovid))`

`1`	`#> Train sets images Non COVID: 357`

`1`	`print("Test sets images COVID: {}".format(total_test_covid))`

`1`	`#> Test sets images COVID: 36`

`1`	`print("Test sets images Non COVID: {}".format(total_test_noncovid))`

`1`	`#> Test sets images Non COVID: 40`

Core Model

To building CNN Model, we need to determine the parameters first.

batch_size = 128
epochs = 15
IMG_HEIGHT = 150
IMG_WIDTH = 150

#Generator Scale for Our Data

train_image_generator = ImageDataGenerator(rescale=1./255)
test_image_generator = ImageDataGenerator(rescale=1./255)

train_dir = os.path.join('dataset/train')
test_dir = os.path.join('dataset/test')

total_train = total_train_covid + total_train_noncovid
total_test = total_test_covid + total_test_noncovid

#Collecting Training Data

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
                                                           directory=train_dir,
                                                           shuffle=True,
                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                           class_mode='binary')

`1`	`#> Found 670 images belonging to 2 classes.`

#Collecting Testing Data

test_data_gen = test_image_generator.flow_from_directory(batch_size=batch_size,
                                                              directory=test_dir,
                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                              class_mode='binary')

`1`	`#> Found 76 images belonging to 2 classes.`

#Building CNN Model

model = Sequential([
    Conv2D(16, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(),
    Conv2D(32, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Conv2D(64, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Flatten(),
    Dense(512, activation='relu'),
    Dense(1)
])

1
2
3

model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])

In this section, we evaluate the model based on parameter and metrics.

`1`	`model.summary()`

#> Model: "sequential"
#> _________________________________________________________________
#>  Layer (type)                Output Shape              Param #   
#> =================================================================
#>  conv2d (Conv2D)             (None, 150, 150, 16)      448       
#>                                                                  
#>  max_pooling2d (MaxPooling2D  (None, 75, 75, 16)       0         
#>  )                                                               
#>                                                                  
#>  conv2d_1 (Conv2D)           (None, 75, 75, 32)        4640      
#>                                                                  
#>  max_pooling2d_1 (MaxPooling  (None, 37, 37, 32)       0         
#>  2D)                                                             
#>                                                                  
#>  conv2d_2 (Conv2D)           (None, 37, 37, 64)        18496     
#>                                                                  
#>  max_pooling2d_2 (MaxPooling  (None, 18, 18, 64)       0         
#>  2D)                                                             
#>                                                                  
#>  flatten (Flatten)           (None, 20736)             0         
#>                                                                  
#>  dense (Dense)               (None, 512)               10617344  
#>                                                                  
#>  dense_1 (Dense)             (None, 1)                 513       
#>                                                                  
#> =================================================================
#> Total params: 10,641,441
#> Trainable params: 10,641,441
#> Non-trainable params: 0
#> _________________________________________________________________

Conclusion

history = model.fit(
    train_data_gen,
    steps_per_epoch=total_train // batch_size,
    epochs=epochs,
    validation_data=test_data_gen,
    validation_steps=total_test // batch_size
)

#> Epoch 1/15
#> 
#> 1/5 [=====>........................] - ETA: 12s - loss: 0.6901 - accuracy: 0.4297
#> 2/5 [===========>..................] - ETA: 3s - loss: 3.2448 - accuracy: 0.4570 
#> 3/5 [=================>............] - ETA: 2s - loss: 2.5687 - accuracy: 0.4583
#> 4/5 [=======================>......] - ETA: 0s - loss: 2.4764 - accuracy: 0.4638
#> 5/5 [==============================] - ETA: 0s - loss: 2.0964 - accuracy: 0.4594
#> 5/5 [==============================] - 8s 1s/step - loss: 2.0964 - accuracy: 0.4594
#> Epoch 2/15
#> 
#> 1/5 [=====>........................] - ETA: 5s - loss: 0.6351 - accuracy: 0.3906
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.6804 - accuracy: 0.5000
#> 3/5 [=================>............] - ETA: 2s - loss: 0.7045 - accuracy: 0.5104
#> 4/5 [=======================>......] - ETA: 0s - loss: 0.7119 - accuracy: 0.5072
#> 5/5 [==============================] - ETA: 0s - loss: 0.7085 - accuracy: 0.5295
#> 5/5 [==============================] - 6s 1s/step - loss: 0.7085 - accuracy: 0.5295
#> Epoch 3/15
#> 
#> 1/5 [=====>........................] - ETA: 5s - loss: 0.6529 - accuracy: 0.5781
#> 2/5 [===========>..................] - ETA: 0s - loss: 0.6519 - accuracy: 0.5633
#> 3/5 [=================>............] - ETA: 1s - loss: 0.6491 - accuracy: 0.5315
#> 4/5 [=======================>......] - ETA: 0s - loss: 0.6367 - accuracy: 0.5870
#> 5/5 [==============================] - ETA: 0s - loss: 0.6350 - accuracy: 0.5886
#> 5/5 [==============================] - 5s 1s/step - loss: 0.6350 - accuracy: 0.5886
#> Epoch 4/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.5970 - accuracy: 0.6250
#> 2/5 [===========>..................] - ETA: 4s - loss: 0.5825 - accuracy: 0.6289
#> 3/5 [=================>............] - ETA: 2s - loss: 0.5911 - accuracy: 0.6458
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.5939 - accuracy: 0.6406
#> 5/5 [==============================] - ETA: 0s - loss: 0.5999 - accuracy: 0.6453
#> 5/5 [==============================] - 7s 1s/step - loss: 0.5999 - accuracy: 0.6453
#> Epoch 5/15
#> 
#> 1/5 [=====>........................] - ETA: 5s - loss: 0.5128 - accuracy: 0.7656
#> 2/5 [===========>..................] - ETA: 0s - loss: 0.5391 - accuracy: 0.7532
#> 3/5 [=================>............] - ETA: 1s - loss: 0.5404 - accuracy: 0.7203
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.5537 - accuracy: 0.6763
#> 5/5 [==============================] - ETA: 0s - loss: 0.5631 - accuracy: 0.6531
#> 5/5 [==============================] - 6s 1s/step - loss: 0.5631 - accuracy: 0.6531
#> Epoch 6/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.5437 - accuracy: 0.6094
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.5263 - accuracy: 0.6836
#> 3/5 [=================>............] - ETA: 2s - loss: 0.5257 - accuracy: 0.7109
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.5441 - accuracy: 0.7285
#> 5/5 [==============================] - ETA: 0s - loss: 0.5440 - accuracy: 0.7251
#> 5/5 [==============================] - 6s 986ms/step - loss: 0.5440 - accuracy: 0.7251
#> Epoch 7/15
#> 
#> 1/5 [=====>........................] - ETA: 1s - loss: 0.5883 - accuracy: 0.5333
#> 2/5 [===========>..................] - ETA: 4s - loss: 0.4944 - accuracy: 0.7152
#> 3/5 [=================>............] - ETA: 2s - loss: 0.4920 - accuracy: 0.7063
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.4757 - accuracy: 0.7246
#> 5/5 [==============================] - ETA: 0s - loss: 0.4750 - accuracy: 0.7343
#> 5/5 [==============================] - 6s 1s/step - loss: 0.4750 - accuracy: 0.7343
#> Epoch 8/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.4733 - accuracy: 0.7812
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.4687 - accuracy: 0.7812
#> 3/5 [=================>............] - ETA: 2s - loss: 0.4585 - accuracy: 0.7786
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.4424 - accuracy: 0.7891
#> 5/5 [==============================] - ETA: 0s - loss: 0.4335 - accuracy: 0.7875
#> 5/5 [==============================] - 6s 1s/step - loss: 0.4335 - accuracy: 0.7875
#> Epoch 9/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.4186 - accuracy: 0.8359
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.4214 - accuracy: 0.7930
#> 3/5 [=================>............] - ETA: 2s - loss: 0.3908 - accuracy: 0.8099
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.3910 - accuracy: 0.8066
#> 5/5 [==============================] - ETA: 0s - loss: 0.3917 - accuracy: 0.8044
#> 5/5 [==============================] - 6s 1s/step - loss: 0.3917 - accuracy: 0.8044
#> Epoch 10/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.3538 - accuracy: 0.8281
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.3355 - accuracy: 0.8477
#> 3/5 [=================>............] - ETA: 2s - loss: 0.3546 - accuracy: 0.8385
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.3563 - accuracy: 0.8438
#> 5/5 [==============================] - ETA: 0s - loss: 0.3604 - accuracy: 0.8359
#> 5/5 [==============================] - 6s 1s/step - loss: 0.3604 - accuracy: 0.8359
#> Epoch 11/15
#> 
#> 1/5 [=====>........................] - ETA: 1s - loss: 0.4912 - accuracy: 0.6667
#> 2/5 [===========>..................] - ETA: 4s - loss: 0.3186 - accuracy: 0.8354
#> 3/5 [=================>............] - ETA: 2s - loss: 0.3876 - accuracy: 0.8322
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.3677 - accuracy: 0.8430
#> 5/5 [==============================] - ETA: 0s - loss: 0.3661 - accuracy: 0.8395
#> 5/5 [==============================] - 6s 1s/step - loss: 0.3661 - accuracy: 0.8395
#> Epoch 12/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.3390 - accuracy: 0.7578
#> 2/5 [===========>..................] - ETA: 0s - loss: 0.3429 - accuracy: 0.7595
#> 3/5 [=================>............] - ETA: 1s - loss: 0.3544 - accuracy: 0.7657
#> 4/5 [=======================>......] - ETA: 0s - loss: 0.3444 - accuracy: 0.7874
#> 5/5 [==============================] - ETA: 0s - loss: 0.3114 - accuracy: 0.8210
#> 5/5 [==============================] - 6s 1s/step - loss: 0.3114 - accuracy: 0.8210
#> Epoch 13/15
#> 
#> 1/5 [=====>........................] - ETA: 1s - loss: 0.4676 - accuracy: 0.7667
#> 2/5 [===========>..................] - ETA: 4s - loss: 0.3209 - accuracy: 0.8544
#> 3/5 [=================>............] - ETA: 2s - loss: 0.2974 - accuracy: 0.8811
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.3004 - accuracy: 0.8744
#> 5/5 [==============================] - ETA: 0s - loss: 0.2880 - accuracy: 0.8708
#> 5/5 [==============================] - 5s 1s/step - loss: 0.2880 - accuracy: 0.8708
#> Epoch 14/15
#> 
#> 1/5 [=====>........................] - ETA: 6s - loss: 0.2019 - accuracy: 0.9375
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.2004 - accuracy: 0.9297
#> 3/5 [=================>............] - ETA: 2s - loss: 0.2138 - accuracy: 0.9167
#> 4/5 [=======================>......] - ETA: 1s - loss: 0.2454 - accuracy: 0.9062
#> 5/5 [==============================] - ETA: 0s - loss: 0.2546 - accuracy: 0.8969
#> 5/5 [==============================] - 6s 1s/step - loss: 0.2546 - accuracy: 0.8969
#> Epoch 15/15
#> 
#> 1/5 [=====>........................] - ETA: 5s - loss: 0.2089 - accuracy: 0.9062
#> 2/5 [===========>..................] - ETA: 3s - loss: 0.2323 - accuracy: 0.8867
#> 3/5 [=================>............] - ETA: 1s - loss: 0.2352 - accuracy: 0.8846
#> 4/5 [=======================>......] - ETA: 0s - loss: 0.2373 - accuracy: 0.8961
#> 5/5 [==============================] - ETA: 0s - loss: 0.2275 - accuracy: 0.9077
#> 5/5 [==============================] - 6s 1s/step - loss: 0.2275 - accuracy: 0.9077

Yay, we’ve already build a model with the high accuracy. This model can help healthcare industry to contribute on COVID-19 issue, to detect COVID-19 patient with lung images

IMAGE CLASSIFICATION WITH CONVOLUTIONAL NETWORK

Tue, 06 Apr 2021 00:00:00 +0000

Introduction

Deep learning is a great approach to deal with unstructured data such as text, sound, video and image. There are a lot of implementation of deep learning in image classification and image detection, such as classifying image of dog or cats, detecting different objects in an image or do facial recognition.

On this article, we will try to build a simple image classification that will classify whether the presented image is a cat, dog, or a panda.

Data

You can download the data and the source code for practice here. The data is a modified version the Animal Image Dataset(DOG, CAT and PANDA) on Kaggle.

Library And Setup

You need to install the pillow package in your conda environment to manipulate image data. Here is the short instruction on how to create a new conda environment with tensorflow and pillow inside it.

Open the terminal, either in anaconda command prompt or directly in RStudio.

Create new conda environment by running the following command.

conda create -n tf_image python=3.7

Active the conda environment by running the following command.

conda activate tf_image

Install the tensorflow package into the environment.

conda install -c conda-forge tensorflow=2

Install the pillow package.

pip install pillow

The next step is just call your conda environment using reticulate::use_python() and insert the location of the python from the tf_image environment. You can locate the path or the location of the environment by typing conda env list in the terminal.

1
2
3

library(reticulate)
# Use python in your anaconda3 environment folder
reticulate::use_python("C:/Users/Lenovo/anaconda3/envs/algolearn", required = T)

The following is the list of required packages to build and evaluate our image classification.

# Data wrangling
library(tidyverse)

# Image manipulation
library(imager)

# Deep learning
library(keras)

# Model Evaluation
library(caret)

options(scipen = 999)

Exploratory Data Analysis

Let’s explore the data first before building the model. In image classification problem, it is a common practice to put each image on separate folders based on the target class/labels. For example, inside the train folder in our data, you can that we have 3 different folders, respectively for cats, dogs, and panda.

If you open the cat folder, you can see that we have no table or any kind of structured data format, we only have the image for the cat. We will directly extract information from the images instead of using a structured dataset.

Let’s try to get the file name of each image. First, we need to locate the folder of each target class. The following code will give you the folder name inside the train folder.

1
2
3

folder_list <- list.files("image_class/train/")

folder_list

`1`	`#> [1] "cats" "dogs" "panda"`

We combine the folder name with the path or directory of the train folder in order to access the content inside each folder.

1
2
3

folder_path <- paste0("image_class/train/", folder_list, "/")

folder_path

1
2

#> [1] "image_class/train/cats/"  "image_class/train/dogs/" 
#> [3] "image_class/train/panda/"

We will use the map() function to loop or iterate and collect the file name for each folder (cat, dog, panda). The map() will return a list so if we want to combine the file name from 3 different folders we simply use the unlist() function.

# Get file name
file_name <- map(folder_path, 
                 function(x) paste0(x, list.files(x))
                 ) %>% 
  unlist()

# first 6 file name
head(file_name)

1
2
3

#> [1] "image_class/train/cats/cats_001.jpg" "image_class/train/cats/cats_002.jpg"
#> [3] "image_class/train/cats/cats_003.jpg" "image_class/train/cats/cats_004.jpg"
#> [5] "image_class/train/cats/cats_005.jpg" "image_class/train/cats/cats_006.jpg"

You can also check the last 6 images.

1
2

# last 6 file name
tail(file_name)

#> [1] "image_class/train/panda/panda_00994.jpg"
#> [2] "image_class/train/panda/panda_00995.jpg"
#> [3] "image_class/train/panda/panda_00996.jpg"
#> [4] "image_class/train/panda/panda_00997.jpg"
#> [5] "image_class/train/panda/panda_00998.jpg"
#> [6] "image_class/train/panda/panda_00999.jpg"

Let’s check how many images we have.

`1`	`length(file_name)`

`1`	`#> [1] 2659`

To check the content of the file, we can use the load.image() function from the imager package. For example, let’s randomly visualize 6 images from the data.

# Randomly select image
set.seed(99)
sample_image <- sample(file_name, 6)

# Load image into R
img <- map(sample_image, load.image)

# Plot image
par(mfrow = c(2, 3)) # Create 2 x 3 image grid
map(img, plot)

#> [[1]]
#> Image. Width: 184 pix Height: 149 pix Depth: 1 Colour channels: 3 
#> 
#> [[2]]
#> Image. Width: 500 pix Height: 375 pix Depth: 1 Colour channels: 3 
#> 
#> [[3]]
#> Image. Width: 500 pix Height: 375 pix Depth: 1 Colour channels: 3 
#> 
#> [[4]]
#> Image. Width: 500 pix Height: 466 pix Depth: 1 Colour channels: 3 
#> 
#> [[5]]
#> Image. Width: 499 pix Height: 375 pix Depth: 1 Colour channels: 3 
#> 
#> [[6]]
#> Image. Width: 500 pix Height: 397 pix Depth: 1 Colour channels: 3

Check Image Dimension

One of important aspects of image classification is understand the dimension of the input images. You need to know the distribution of the image dimension to create a proper input dimension for building the deep learning model. Let’s check the properties of the first image.

1
2
3

# Full Image Description
img <- load.image(file_name[1])
img

`1`	`#> Image. Width: 499 pix Height: 375 pix Depth: 1 Colour channels: 3`

You can get the information about the dimension of the image. The height and width represent the height and width of the image in pixels. The color channel represent if the color is in grayscale format (color channels = 1) or is in RGB format (color channels = 3). To get the value of each dimension, we can use the dim() function. It will return the height, width, depth, and the channels.

1
2

# Image Dimension
dim(img)

`1`	`#> [1] 499 375 1 3`

So we have successfully insert an image and get the image dimensions. On the following code, we will create a function that will instantly get the height and width of an image and convert it into a data.frame.

# Function for acquiring width and height of an image
get_dim <- function(x){
  img <- load.image(x) 
  
  df_img <- data.frame(height = height(img),
                       width = width(img),
                       filename = x
                       )
  
  return(df_img)
}

get_dim(file_name[1])

1
2

#>   height width                            filename
#> 1    375   499 image_class/train/cats/cats_001.jpg

Now we will sampling 1,000 images from the file name and get the height and width of the image. We use sampling here because it will take a quite long time to load all images.

# Randomly get 1000 sample images
set.seed(123)
sample_file <- sample(file_name, 1000)

# Run the get_dim() function for each image
file_dim <- map_df(sample_file, get_dim)

head(file_dim, 10)

#>    height width                                filename
#> 1     375   500 image_class/train/panda/panda_00780.jpg
#> 2     375   500 image_class/train/panda/panda_00836.jpg
#> 3     375   500 image_class/train/panda/panda_00521.jpg
#> 4     294   363     image_class/train/cats/cats_526.jpg
#> 5     269   172     image_class/train/cats/cats_195.jpg
#> 6     375   500 image_class/train/panda/panda_00072.jpg
#> 7     255   234   image_class/train/dogs/dogs_00313.jpg
#> 8     499   375   image_class/train/dogs/dogs_00450.jpg
#> 9     374   500   image_class/train/dogs/dogs_00466.jpg
#> 10    500   440   image_class/train/dogs/dogs_00187.jpg

Now let’s get the statistics for the image dimensions.

`1`	`summary(file_dim)`

#>      height           width          filename        
#>  Min.   :  50.0   Min.   :  59.0   Length:1000       
#>  1st Qu.: 331.0   1st Qu.: 350.0   Class :character  
#>  Median : 375.0   Median : 499.0   Mode  :character  
#>  Mean   : 368.2   Mean   : 425.4                     
#>  3rd Qu.: 411.2   3rd Qu.: 500.0                     
#>  Max.   :1023.0   Max.   :1024.0

The image data has a great variation in the dimension. Some images has less than 60 pixels in height and width while others has up to 1024 pixels. Understanding the dimension of the image will help us on the next part of the process: data preprocessing.

Data Preprocessing

Data preprocessing for image is pretty simple and can be done in a single step in the following section.

Data Augmentation

Based on our previous summary of the image dimensions, we can determine the input dimension for the deep learning model. All input images should have the same dimensions. Here, we can determine the input size for the image, for example transform all image into 64 x 64 pixels. This process will be similar to us resizing the image. You can use other choice of image dimensions, such as 125 x 125 pixels or even 200 x 200 pixels. Bigger dimensions will have more features but will also take longer time to train. However, if the image size is too small, we will lose a lot of information from the data. So balancing this trade-off is the art of data preprocessing in image classification.

We also set the batch size for the data so the model will be updated every time it finished training on a single batch. Here, we set the batch size to 32.

# Desired height and width of images
target_size <- c(64, 64)

# Batch size for training the model
batch_size <- 32

Since we have a little amount of training set, we will build artificial data using method called Image Augmentation. Image augmentation is one useful technique in building models that can increase the size of the training set without acquiring new images. The goal is that to teach the model not only with the original image but also the modification of the image, such as flipping the image, rotate it, zooming, crop the image, etc. This will create more robust model. We can do data augmentation by using the image data generator from keras.

To do image augmentation, we can fit the data into a generator. Here, we will create the image generator for keras with the following properties:

Scaling the pixel value by dividing the pixel value by 255
Flip the image horizontally
Flip the image vertically
Rotate the image from 0 to 45 degrees
Zoom in or zoom out by 25% (zoom 75% or 125%)
Use 20% of the data as validation dataset

You can explore more features about the image generator on this link.

# Image Generator
train_data_gen <- image_data_generator(rescale = 1/255, # Scaling pixel value
                                       horizontal_flip = T, # Flip image horizontally
                                       vertical_flip = T, # Flip image vertically 
                                       rotation_range = 45, # Rotate image from 0 to 45 degrees
                                       zoom_range = 0.25, # Zoom in or zoom out range
                                       validation_split = 0.2 # 20% data as validation data
                                       )
# Generator for data validation
test_data_gen <- image_data_generator(rescale = 1/255, # Rescale
                                      validation_split = 0.2) # Reassign 20% as validation data

Now we can insert our image data into the generator using the flow_images_from_directory(). The data is located inside the data folder and inside the train folder, so the directory will be data/train. From this process, we will get the augmented image both for training data and the validation data.

# Training Dataset
train_image_array_gen <- flow_images_from_directory(directory = "image_class/train/", # Folder of the data
                                                    target_size = target_size, # target of the image dimension (64 x 64)  
                                                    color_mode = "rgb", # use RGB color
                                                    batch_size = batch_size , 
                                                    seed = 123,  # set random seed
                                                    subset = "training", # declare that this is for training data
                                                    generator = train_data_gen
                                                    )

# Validation Dataset
val_image_array_gen <- flow_images_from_directory(directory = "image_class/train/",
                                                  target_size = target_size, 
                                                  color_mode = "rgb", 
                                                  batch_size = batch_size,
                                                  seed = 123,
                                                  subset = "validation", # declare that this is the validation data
                                                  generator = test_data_gen
                                                  )

Here we will collect some information from the generator and check the class proportion of the train dataset. The index correspond to each labels of the target variable and ordered alphabetically (cat, dog, panda).

# Number of training samples
train_samples <- train_image_array_gen$n

# Number of validation samples
valid_samples <- val_image_array_gen$n

# Number of target classes/categories
output_n <- n_distinct(train_image_array_gen$classes)

# Get the class proportion
table("\nFrequency" = factor(train_image_array_gen$classes)
      ) %>% 
  prop.table()

#> 
#> Frequency
#>         0         1         2 
#> 0.3344293 0.3363081 0.3292626

Convolutional Neural Network

The Convolutional Neural Network or Convolutional Layer is a popular layer for image classification. If you remember, an image is just a 2 dimensional array with certain height and width. For example, an image with 64 x 64 pixels means that it has 4096 pixels that is distributed in a 64 x 64 array instead of a single dimensional vector. The benefit of using image as a 2D array is that we can extract certain features from the image such as the shape of nose, the shape of eyes, hand, etc.

Take the following amazing example from setosa.io. We have an image and its 2D array representation. The value on the array is the pixel value from the image, higher value means brighter pixel.

To extract features from the image, we create something called a filter kernel. A filter is an array with certain size, for example 3 x 3 array to capture features from image. In the following figure, the rectangle illustrated a single filter kernel.

For example, the left side of the following picture is a 5 x 5 images. The kernel has a weight that will capture certain features, which in this example is an X features that indicated by the value of 1 create an X shape. The new convoluted feature is the product of the image section with the kernel feature. The more similar image section with the kernel, the higher the score of the convoluted feature.

The kernel wil move sideway to the right to capture each section of the image to create new convoluted feature. If the kerel has reach the edge, it will go down 1 row below and continue the process. The process is illustrated as follows. Visit stanford course on Convolutional Neural Network for more info about the figure and other info.

To highglight the most important feature and also downsize the dimension of the convoluted feature, we can use a method called the Max pooling which take only the maximum value from certain window. For example, in the top left array that contains the value of 1, 1, 5, and 6, it only take the max value, which is 6.

Below is the illustration of doing max pooling of 2 x 2 pooling area along the sections. Max pooling only take the maximum value of each pooling area.

To make the extracted 2D array into a 1D array, we use the flattening layer so we can continue using the fully-connected dense layer and to the output layer.

The following figure illustrate the full deep learning model with CNN, max pooling and fully connected dense layer.

You can see the explanation from MIT for introduction to Deep Learning and Convolutional Neural Network on the deeper level.

MODEL ARCHITECTURE

We can start building the model architecture for the deep learning. We will build a simple model first with the following layer:

Convolutional layer to extract features from 2D image with relu activation function
Max Pooling layer to downsample the image features
Flattening layer to flatten data from 2D array to 1D array
Dense layer to capture more information
Dense layer for output with softmax activation function

Don’t forget to set the input size in the first layer. If the input image is in RGB, set the final number to 3, which is the number of color channels. If the input image is in grayscale, set the final number to 1.

1
2

# input shape of the image
c(target_size, 3)

`1`	`#> [1] 64 64 3`

# Set Initial Random Weight
tensorflow::tf$random$set_seed(123)

model <- keras_model_sequential(name = "simple_model") %>% 
  
  # Convolution Layer
  layer_conv_2d(filters = 16,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu",
                input_shape = c(target_size, 3) 
                ) %>% 

  # Max Pooling Layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Flattening Layer
  layer_flatten() %>% 
  
  # Dense Layer
  layer_dense(units = 16,
              activation = "relu") %>% 
  
  # Output Layer
  layer_dense(units = output_n,
              activation = "softmax",
              name = "Output")
  
model

#> Model: "simple_model"
#> ________________________________________________________________________________
#>  Layer (type)                       Output Shape                    Param #     
#> ================================================================================
#>  conv2d (Conv2D)                    (None, 64, 64, 16)              448         
#>  max_pooling2d (MaxPooling2D)       (None, 32, 32, 16)              0           
#>  flatten (Flatten)                  (None, 16384)                   0           
#>  dense (Dense)                      (None, 16)                      262160      
#>  Output (Dense)                     (None, 3)                       51          
#> ================================================================================
#> Total params: 262,659
#> Trainable params: 262,659
#> Non-trainable params: 0
#> ________________________________________________________________________________

As you can see, we start by entering image data with 64 x 64 pixels into the convolutional layer, which has 16 filters to extract featuers from the image. The padding = same argument is used to keep the dimension of the feature to be 64 x 64 pixels after being extracted. We then downsample or only take the maximum value for each 2x2 pooling area so the data now only has 32 x 32 pixels with from 16 filters. After that, from 32 x 32 pixels we flatten the 2D array into a 1D array with 32 x 32 x 16 = 16384 nodes. We can further extract information using the simple dense layer and finished by flowing the information into the output layer, which will be transformed using the softmax activation function to get the probability of each class as the output.

Model Fitting

You can start fitting the data into the model. Don’t forget to compile the model by specifying the loss function and the optimizer. For starter, we will use 30 epochs to train the data. For multilabel classification, we will use categorical cross-entropy as the loss function. For this example, we use adam optimizer with learning rate of 0.01. We will also evaluate the model with the validation data from the generator.

model %>% 
  compile(
    loss = "categorical_crossentropy",
    optimizer = optimizer_adam(learning_rate = 0.01),
    metrics = "accuracy"
  )

# Fit data into model
history <- model %>% 
  fit(
  # training data
  train_image_array_gen,

  # training epochs
  steps_per_epoch = as.integer(train_samples / batch_size), 
  epochs = 30, 
  
  # validation data
  validation_data = val_image_array_gen,
  validation_steps = as.integer(valid_samples / batch_size)
)

plot(history)

Model Evaluation

Now we will further evaluate and acquire the confusion matrix using the validation data from the generator. First, we need to acquire the file name of the image that is used as the data validation. From the file name, we will extract the categorical label as the actual value of the target variable.

val_data <- data.frame(file_name = paste0("image_class/train/", val_image_array_gen$filenames)) %>% 
  mutate(class = str_extract(file_name, "cat|dog|panda"))

head(val_data, 10)

#>                               file_name class
#> 1  image_class/train/cats\\cats_001.jpg   cat
#> 2  image_class/train/cats\\cats_002.jpg   cat
#> 3  image_class/train/cats\\cats_003.jpg   cat
#> 4  image_class/train/cats\\cats_004.jpg   cat
#> 5  image_class/train/cats\\cats_005.jpg   cat
#> 6  image_class/train/cats\\cats_006.jpg   cat
#> 7  image_class/train/cats\\cats_007.jpg   cat
#> 8  image_class/train/cats\\cats_008.jpg   cat
#> 9  image_class/train/cats\\cats_009.jpg   cat
#> 10 image_class/train/cats\\cats_010.jpg   cat

What to do next? We need to get the image into R by converting the image into an array. Since our input dimension for CNN model is image with 64 x 64 pixels with 3 color channels (RGB), we will do the same with the image of the testing data. The reason of using array is that we want to predict the original image fresh from the folder so we will not use the image generator since it will transform the image and does not reflect the actual image.

# Function to convert image to array
image_prep <- function(x) {
  arrays <- lapply(x, function(path) {
    img <- image_load(path, target_size = target_size, 
                      grayscale = F # Set FALSE if image is RGB
                      )
    
    x <- image_to_array(img)
    x <- array_reshape(x, c(1, dim(x)))
    x <- x/255 # rescale image pixel
  })
  do.call(abind::abind, c(arrays, list(along = 1)))
}

test_x <- image_prep(val_data$file_name)

# Check dimension of testing data set
dim(test_x)

`1`	`#> [1] 530 64 64 3`

The validation data consists of 530 images with dimensions of 64 x 64 pixels and 3 color channels (RGB). After we have prepared the data test, we now can proceed to predict the label of each image using our CNN model.

1
2
3

pred_test <- model %>% predict(test_x) %>% k_argmax()

head(pred_test, 10)

`1`	`#> tf.Tensor([1 1 0 2 0 2 1 0 0 0], shape=(10), dtype=int64)`

To get easier interpretation of the prediction, we will convert the encoding into proper class label.

# Convert encoding to label
decode <- function(x){
  case_when(x == 0 ~ "cat",
            x == 1 ~ "dog",
            x == 2 ~ "panda"
            )
}

pred_test <- sapply(pred_test, decode) 

head(pred_test, 10)

1
2

#>  [1] "dog"   "dog"   "cat"   "panda" "cat"   "panda" "dog"   "cat"   "cat"  
#> [10] "cat"

Finally, we evaluate the model using the confusion matrix. The model perform very poorly with low accuracy. We will tune the model by improving the model architecture.

1
2
3

confusionMatrix(as.factor(pred_test), 
                as.factor(val_data$class)
                )

#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction cat dog panda
#>      cat    87  62     4
#>      dog    47  65     7
#>      panda  43  51   164
#> 
#> Overall Statistics
#>                                                
#>                Accuracy : 0.5962               
#>                  95% CI : (0.5531, 0.6383)     
#>     No Information Rate : 0.3358               
#>     P-Value [Acc > NIR] : < 0.00000000000000022
#>                                                
#>                   Kappa : 0.3951               
#>                                                
#>  Mcnemar's Test P-Value : 0.00000000000001259  
#> 
#> Statistics by Class:
#> 
#>                      Class: cat Class: dog Class: panda
#> Sensitivity              0.4915     0.3652       0.9371
#> Specificity              0.8130     0.8466       0.7352
#> Pos Pred Value           0.5686     0.5462       0.6357
#> Neg Pred Value           0.7613     0.7251       0.9596
#> Prevalence               0.3340     0.3358       0.3302
#> Detection Rate           0.1642     0.1226       0.3094
#> Detection Prevalence     0.2887     0.2245       0.4868
#> Balanced Accuracy        0.6523     0.6059       0.8362

Tuning The Model

Model Architecture

Let’s look back at our model architecture. If you have noticed, we can actually extract more information while the data is still in an 2D image array. The first CNN only extract the general features of our image and then being downsampled using the max pooling layer. Even after pooling, we still have 32 x 32 array that has a lot of information to extract before flattening the data. Therefore, we can stacks more CNN layers into the model so there will be more information to be captured. We can also put 2 CNN layers consecutively before doing max pooling.

model

#> Model: "simple_model"
#> ________________________________________________________________________________
#>  Layer (type)                       Output Shape                    Param #     
#> ================================================================================
#>  conv2d (Conv2D)                    (None, 64, 64, 16)              448         
#>  max_pooling2d (MaxPooling2D)       (None, 32, 32, 16)              0           
#>  flatten (Flatten)                  (None, 16384)                   0           
#>  dense (Dense)                      (None, 16)                      262160      
#>  Output (Dense)                     (None, 3)                       51          
#> ================================================================================
#> Total params: 262,659
#> Trainable params: 262,659
#> Non-trainable params: 0
#> ________________________________________________________________________________

The following is our improved model architecture:

1st Convolutional layer to extract features from 2D image with relu activation function
2nd Convolutional layer to extract features from 2D image with relu activation function Max pooling layer
3rd Convolutional layer to extract features from 2D image with relu activation function Max pooling layer
4th Convolutional layer to extract features from 2D image with relu activation function Max pooling layer
5th Convolutional layer to extract features from 2D image with relu activation function Max pooling layer
Flattening layer from 2D array to 1D array
Dense layer to capture more information
Dense layer for output layer

You can play and get creative by designing your own model architecture.

tensorflow::tf$random$set_seed(123)

model_big <- keras_model_sequential() %>% 
  
  # First convolutional layer
  layer_conv_2d(filters = 32,
                kernel_size = c(5,5), # 5 x 5 filters
                padding = "same",
                activation = "relu",
                input_shape = c(target_size, 3)
                ) %>% 
  
  # Second convolutional layer
  layer_conv_2d(filters = 32,
                kernel_size = c(3,3), # 3 x 3 filters
                padding = "same",
                activation = "relu"
                ) %>% 
  
  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Third convolutional layer
  layer_conv_2d(filters = 64,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 

  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Fourth convolutional layer
  layer_conv_2d(filters = 128,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 
  
  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 

  # Fifth convolutional layer
  layer_conv_2d(filters = 256,
                kernel_size = c(3,3),
                padding = "same",
                activation = "relu"
                ) %>% 
  
  # Max pooling layer
  layer_max_pooling_2d(pool_size = c(2,2)) %>% 
  
  # Flattening layer
  layer_flatten() %>% 
  
  # Dense layer
  layer_dense(units = 64,
              activation = "relu") %>% 
  
  # Output layer
  layer_dense(name = "Output",
              units = 3, 
              activation = "softmax")

model_big

#> Model: "sequential"
#> ________________________________________________________________________________
#>  Layer (type)                       Output Shape                    Param #     
#> ================================================================================
#>  conv2d_5 (Conv2D)                  (None, 64, 64, 32)              2432        
#>  conv2d_4 (Conv2D)                  (None, 64, 64, 32)              9248        
#>  max_pooling2d_4 (MaxPooling2D)     (None, 32, 32, 32)              0           
#>  conv2d_3 (Conv2D)                  (None, 32, 32, 64)              18496       
#>  max_pooling2d_3 (MaxPooling2D)     (None, 16, 16, 64)              0           
#>  conv2d_2 (Conv2D)                  (None, 16, 16, 128)             73856       
#>  max_pooling2d_2 (MaxPooling2D)     (None, 8, 8, 128)               0           
#>  conv2d_1 (Conv2D)                  (None, 8, 8, 256)               295168      
#>  max_pooling2d_1 (MaxPooling2D)     (None, 4, 4, 256)               0           
#>  flatten_1 (Flatten)                (None, 4096)                    0           
#>  dense_1 (Dense)                    (None, 64)                      262208      
#>  Output (Dense)                     (None, 3)                       195         
#> ================================================================================
#> Total params: 661,603
#> Trainable params: 661,603
#> Non-trainable params: 0
#> ________________________________________________________________________________

Model Fitting

We can once again fit the model into the data. We will let the data train with more epochs since we have small numbers of data. For example, we will train the data with 50 epochs. We will also lower the learning rate from 0.01 to 0.001.

model_big %>% 
  compile(
    loss = "categorical_crossentropy",
    optimizer = optimizer_adam(learning_rate = 0.001),
    metrics = "accuracy"
  )

history <- model %>% 
  fit(
  # training data
  train_image_array_gen,
  
  # epochs
  steps_per_epoch = as.integer(train_samples / batch_size), 
  epochs = 50, 
  
  # validation data
  validation_data = val_image_array_gen,
  validation_steps = as.integer(valid_samples / batch_size),
  
  # print progress but don't create graphic
  verbose = 1,
  view_metrics = 0
)

plot(history)

Model Evaluation

Now we will further evaluate the data and acquire the confusion matrix for the validation data.

1
2
3

pred_test <- model_big %>% predict(test_x) %>% k_argmax()

head(pred_test, 10)

`1`	`#> tf.Tensor([0 0 0 0 0 0 0 0 0 0], shape=(10), dtype=int64)`

To get easier interpretation of the prediction, we will convert the encoding into proper class label.

# Convert encoding to label
decode <- function(x){
  case_when(x == 0 ~ "cat",
            x == 1 ~ "dog",
            x == 2 ~ "panda"
            )
}

pred_test <- sapply(pred_test, decode) 

head(pred_test, 10)

`1`	`#> [1] "cat" "cat" "cat" "cat" "cat" "cat" "cat" "cat" "cat" "cat"`

Finally, we evaluate the model using the confusion matrix. This model perform better than the previous model because we put more CNN layer to extract more features from the image.

1
2
3

confusionMatrix(as.factor(pred_test), 
                as.factor(val_data$class)
                )

#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction cat dog panda
#>      cat   168 168   169
#>      dog     0   0     0
#>      panda   9  10     6
#> 
#> Overall Statistics
#>                                              
#>                Accuracy : 0.3283             
#>                  95% CI : (0.2884, 0.3701)   
#>     No Information Rate : 0.3358             
#>     P-Value [Acc > NIR] : 0.659              
#>                                              
#>                   Kappa : -0.0082            
#>                                              
#>  Mcnemar's Test P-Value : <0.0000000000000002
#> 
#> Statistics by Class:
#> 
#>                      Class: cat Class: dog Class: panda
#> Sensitivity             0.94915     0.0000      0.03429
#> Specificity             0.04533     1.0000      0.94648
#> Pos Pred Value          0.33267        NaN      0.24000
#> Neg Pred Value          0.64000     0.6642      0.66535
#> Prevalence              0.33396     0.3358      0.33019
#> Detection Rate          0.31698     0.0000      0.01132
#> Detection Prevalence    0.95283     0.0000      0.04717
#> Balanced Accuracy       0.49724     0.5000      0.49038

PREDICT DATA IN TESTING DATASET

After we have trained the model and if you have satisfied with the model performance on the validation dataset, we will do another model evaluation using the testing dataset. The testing data is located on the test folder.

1
2
3

df_test  <- read.csv("image_class/metadata_test.csv")

head(df_test, 10)

#>    class                   file_name
#> 1    cat image_class/test/img_01.jpg
#> 2    cat image_class/test/img_02.jpg
#> 3    cat image_class/test/img_03.jpg
#> 4    cat image_class/test/img_04.jpg
#> 5    cat image_class/test/img_05.jpg
#> 6    cat image_class/test/img_06.jpg
#> 7    cat image_class/test/img_07.jpg
#> 8    cat image_class/test/img_08.jpg
#> 9    cat image_class/test/img_09.jpg
#> 10   cat image_class/test/img_10.jpg

Then, we convert the image into 2D array.

test_x <- image_prep(df_test$file_name)

# Check dimension of testing data set
dim(test_x)

`1`	`#> [1] 340 64 64 3`

The testing data consists of 341 images with dimension of 64 x 64 pixels and 3 color channels (RGB). After we have prepared the data test, we now can proceed to predict the label of each image using our CNN model.

1
2
3

pred_test <- model_big %>% predict(test_x) %>% k_argmax()

head(pred_test, 10)

`1`	`#> tf.Tensor([0 0 0 0 0 0 0 0 2 0], shape=(10), dtype=int64)`

To get easier interpretation of the prediction, we will convert the encoding into proper class label.

# Convert encoding to label
decode <- function(x){
  case_when(x == 0 ~ "cat",
            x == 1 ~ "dog",
            x == 2 ~ "panda"
            )
}

pred_test <- sapply(pred_test, decode) 

head(pred_test, 10)

1
2

#>  [1] "cat"   "cat"   "cat"   "cat"   "cat"   "cat"   "cat"   "cat"   "panda"
#> [10] "cat"

Finally, we evaluate the model using the confusion matrix.

1
2
3

confusionMatrix(as.factor(pred_test), 
                as.factor(df_test$class)
                )

#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction cat dog panda
#>      cat   103 104   118
#>      dog     0   0     0
#>      panda   8   2     5
#> 
#> Overall Statistics
#>                                              
#>                Accuracy : 0.3176             
#>                  95% CI : (0.2685, 0.37)     
#>     No Information Rate : 0.3618             
#>     P-Value [Acc > NIR] : 0.9609             
#>                                              
#>                   Kappa : -0.0154            
#>                                              
#>  Mcnemar's Test P-Value : <0.0000000000000002
#> 
#> Statistics by Class:
#> 
#>                      Class: cat Class: dog Class: panda
#> Sensitivity             0.92793     0.0000      0.04065
#> Specificity             0.03057     1.0000      0.95392
#> Pos Pred Value          0.31692        NaN      0.33333
#> Neg Pred Value          0.46667     0.6882      0.63692
#> Prevalence              0.32647     0.3118      0.36176
#> Detection Rate          0.30294     0.0000      0.01471
#> Detection Prevalence    0.95588     0.0000      0.04412
#> Balanced Accuracy       0.47925     0.5000      0.49728

Conclusion

Many data comes in different forms and not only in structured format. Some data may come in form of text, image, or even video. That’s where the conventional machine learning model hit its limit and that’s where Deep Learning shines. Deep Learning can handle different unstructured data by simply adjusting the network architecture.