Using Deep Learning To Take On COVID-19

This article was first published by IBM Developer at developer.ibm.com, but authored by Casper Hansen. Here is the Direct link.

Disclaimer: I am not a medical, radiology or epidemiology professional. This article was an experiment, from an engineering and data scientist perspective, and should be regarded as such.

To apply Deep Learning for COVID-19, we need a good dataset – one with lots of samples, edge cases, metadata, and different looking images. We want our model to generalize to the data, such that it can make accurate predictions on new, unseen data. All the work for this article is available on GitHub.

Unfortunately, not much data is available, but there are already posts on LinkedIn/Medium claiming >90% or in some cases 100% accuracy on detecting COVID-19 cases. Though, these posts usually contains mistakes, which are not acknowledged.

We have actively tried to mitigate some of the usual mistakes, by thinking about and coming up with solutions to the following problems:

You might not believe the following is an issue, but it is a typical fallacy for new people in machine learning and data science. That is, the first rule of machine learning: never, ever test your model’s performance with the same data you used to train it with. Not using a testing dataset, and instead testing and measuring the accuracy of their model on the training dataset, does not give an accurate representation of how well the model generalizes to new, unseen data.
Not using computer vision techniques to achieve better generalization – augmentation being an absolute necessity, especially in our case where there are very few samples for our model to learn from.
Not thinking about what the model learns, i.e. will our model actually learn the pattern of what COVID-19 looks like on an X-Ray image, or is it likely that there is some other noisy pattern in our dataset, that it will learn instead?
One amusing story of early machine learning is called Detecting Tanks: “photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days.”
Not using the correct metrics. If you use the accuracy metric on a heavily imbalanced dataset, then your model might look like it’s performing well in the general case, even though it’s performing poorly on COVID-19 cases. 95% accuracy is not that impressive, if your accuracy on COVID-19 cases are 30%.

Acquiring Data

Here is a list of different data sources that I have accumulated over the period of the coronavirus spreading. Currently, this is the best we are going to get from the internet, because the data being collected in many countries and hospitals are classified information. Even if it was not classified information, you still need consent from each patient.

GitHub: COVID-19 Chest X-Ray and CT images. This dataset is continually updated, as more cases appear.
Kaggle 1: Chest X-Ray Images, positive and negative cases of Pneumonia.
Kaggle 2: Chest X-Ray Images, positive and negative cases of Pneumonia.

Problem With Mixed Data Sources

All the data available on the internet have not been subject to the same preprocessing, and all the images are, as explained below, clearly different in the amount of black bars. Most of our positive COVID-19 data has the whole x-ray take up most of the screen, with little to no black bars on the sides (except for a few). Though, the dataset with negative COVID-19 cases have mostly black bars on the side of each image.

This becomes a problem, because our model later on might learn that it just needs to look at the black bars on the side, to know whether or not a new sample is a positive or negative case of COVID-19.

After manual inspection of the dataset, it has become apparent that almost all the negative cases have these black bars, while approximately only 10-20% of the positive cases have black bars. Wouldn’t it be quite a coincidence, if our accuracy of predicting positive/negative later on turns out to be around the 80-90% mark?

Preprocessing — Improving Our Data

The data problem from earlier is clearly one we need to solve. Can we somehow detect when there are black top, black bottom, or black side bars? And can we somehow automatically correct this?

Here is the key to solving this:

A simple metric: are there a number x or more black pixels in a given row/column?
Yes? Remove that row/column. Repeat over all rows and columns.
Done checking for black pixels? Then we crop the image.
Next image and repeat.

Ok, not that fast. New problem: we need to keep the same aspect ratio. And we can’t upscale or downscale, if the aspect ratio is not the same. This is due to the existing architectures needing an input size of 224×224, which indeed is an aspect ratio of 1×1.

In other words – can we check if we are removing as many columns as we are removing rows? Can we overcompensate, by removing specific rows, if there are too many columns with black pixels, but little to no rows of black pixels?

Removing The Black Bars And Fixing Aspect Ratio

The image below demonstrates the input and output of removing the black bars, with one example where it was particularly successful. Quite powerful, but we are still not adhering to the aspect ratio of 1×1.

The most effective and easy solution I found to fixing the aspect ratio, was calculating the difference between the number of rows removed versus the number of columns removed. Divide the difference by two and remove rows from top and bottom, or remove columns from left and right.

The last image is saved as a 224×224 image, because we can scale it down now that the aspect ratio is 1×1. This saves space and some time when loading the image again in another script.

Running The Script

Here is the thing – it takes about 14 seconds per image, to find the rows and columns for removal. We use a naive approach, which could probably be done more efficiently, but nevertheless, it works for this experiment. We had 8851 images in total in the Kaggle 1 dataset, which turned out to be 39 hours of preprocessing. All data is released and available in the GitHub repository for this article.

We ran the code on both the Kaggle 1 and GitHub datasets, and the result is that we have all these images saved in a 224×224 image format. The below code is only for running it on the Kaggle 1 dataset, but you can substitute the first part of the code to load any dataset of images.

import pandas as pd
import matplotlib.pyplot as plt
import cv2, time, os
import numpy as np

df = pd.read_csv('kaggle-pneumonia-jpg/stage_2_detailed_class_info.csv')
image_folder = 'kaggle-pneumonia-jpg/stage_2_train_images_jpg/'

df = df.loc[df['class'] == 'Normal']

Here is how I loaded the COVID-19 dataset. Note that the rest of the code is the same, this is just loading a different dataset.

columns = ['finding', 'modality', 'filename', 'view']

# only use COVID-19 samples done with X-ray and PA view
df = pd.read_csv('metadata.csv', usecols=columns)
df = df.loc[df['finding'] == 'COVID-19']
df = df.loc[df['modality'] == 'X-ray']
df = df.loc[df['view'] == 'PA']

After loading the dataframe, we load the images, labels and filenames into arrays.

images = []
labels = []
filenames = []

for index, row in df.iterrows():
    image = cv2.imread(image_folder + row.patientId + '.jpg')
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    images.append(image)

    label = row['class']
    labels.append(label)
    
    filename = row.patientId
    filenames.append(filename)

images = np.array(images)

Next, we have the a function get_black_pixel_indices() that loops through rows and columns of pixels – you could imagine an image as a table, that has rows and columns of pixels, that has pixel values.

We count the number of black pixels, defined by the number of pixel values that are less than 40. A pixel value consists of one value for each color channel, e.g. Red, Green, Blue (RGB). We could have an array [39, 67, 45], and the average pixel value would be the average of those numbers, which is 50.33.

When we are done counting the number of black pixels for a given row and column, we check if there are more black pixels than non black pixels, and if that is the case, we choose to remove that row or column.

def get_black_pixel_indices(img):
    rows_remove = []
    cols_remove = []

    for j, counter in enumerate(range(img.shape[0])):
        row = img[counter]
        # shift column dim to first dim, row dim to second dim
        # acts like a 90 degree counterclock-wise rotation of image
        col = img.transpose(1, 0, 2)[counter]

        row_black_pixel_count = 0
        col_black_pixel_count = 0
        for selector in range(img.shape[1]):
            row_pixel = row[selector]
            col_pixel = col[selector]
            
            if np.average(row_pixel) < 40:
                row_black_pixel_count += 1
            if np.average(col_pixel) < 40:
                col_black_pixel_count += 1
        
        if row_black_pixel_count > len(row)/2:
            rows_remove.append(j)
        if col_black_pixel_count > len(col)/2:
            cols_remove.append(j)

    return rows_remove, cols_remove

Before we remove the rows and columns, with a higher black pixel count than non black pixel count, we have to check for rows and columns that, if removed, would obstruct the image. The simplest way we found was that no rows and columns should be removed between column and row 200 and 800. After filtering these obstructing indices, we remove the rows and columns.

def remove_obstructing_indices(rows_remove, cols_remove):
    for i, value in enumerate(rows_remove):
        if 200 <= value <= 800:
            del rows_remove[i]
    for i, value in enumerate(cols_remove):
        if 200 <= value <= 800:
            del cols_remove[i]

    return rows_remove, cols_remove

def remove_black_pixels(img, rows_remove, cols_remove):
    img = np.delete(img, rows_remove, axis=0)
    img = np.delete(img, cols_remove, axis=1)
    
    return img

The last thing for preprocessing, is adjusting the aspect ratio. Quite simply, we know that, in most cases, a human torso is more tall than wide, so we choose to arbitrarily cut the image from the top and bottom. There are also cases where the torso is wider, but those were usually rare when inspecting the dataset.

We start by calculating the column to row difference and row to column difference, and based on that, we find how many rows or columns we need to remove, in order to fix the aspect ratio. We found that we can divide the difference by two, then remove half of the difference from either the top and bottom, or left and right. After that, we might have one leftover row or column if the difference is an odd number, so we compensate by either removing a row from the bottom, or a column from the right.

def adjust_aspect_ratio(img, rows_remove, cols_remove):
    col_row_diff = len(cols_remove) - len(rows_remove)
    row_col_diff = len(rows_remove) - len(cols_remove)
    
    if col_row_diff > 0:
        slice_size = int(col_row_diff/2)
        
        img = img[:-slice_size]
        img = img[slice_size:]
        
        if img.shape[0] != img.shape[1]:
            img = img[:-1]
            
    elif row_col_diff > 0:
        slice_size = int(row_col_diff/2)
        
        img = img[:,:-slice_size,:]
        img = img[:,slice_size:,:]
        
        if img.shape[0] != img.shape[1]:
            img = img[:,:-1,:]
    
    if img.shape[0] == img.shape[1]:
        return img, True
    else:
        return img, False

Finally, we run the loop that calls the four functions previously described. This results in images that have removed black bars from the left, right, top and bottom. We downscale that image to a 224×224 image, then we save the image, along with the label and filename in a .csv file that cumulates all the paths and filenames of all our preprocessed images.

save_folder = 'normal_dataset/'

start = time.time()
for i, img in enumerate(images):
    rows_remove, cols_remove = get_black_pixel_indices(img)
    rows_remove, cols_remove = remove_obstructing_indices(rows_remove, cols_remove)
    
    new_img = remove_black_pixels(img, rows_remove, cols_remove)
    adj_img, shape_match = adjust_aspect_ratio(new_img, rows_remove, cols_remove)

    if shape_match:
        resized_image = cv2.resize(adj_img, (224, 224))
    
        label = labels[i]
        name = filenames[i] + '.jpg'

        cv2.imwrite(save_folder + name, resized_image)
        new_df = pd.DataFrame({'filename': [name], 'finding': [label]})

        if i == 0:
            new_df.to_csv('normal_xray_dataset.csv')
        else:
            new_df.to_csv('normal_xray_dataset.csv', mode='a', header=False)

    print('image number {0}, time spent {1:2f}s'.format(i+1, time.time() - start))

For a final overview of another 40 images of negative and positive cases, we can clearly see that our preprocessing approach helped.

Augmentation Of COVID-19 Samples

To make the most of what we have (99 images), we are going to have to use augmentation of our training data. Augmentation is a manipulation of the image, that will look new to CNN – common examples of augmentation are rotations, shears and flips of images.

Keep in mind that we use augmentation in order to generalize our model better and prevent overfitting. Essentially, each time we use one form of augmentation, we add another similar, but slightly different, image to the training dataset. Note that this is also a demonstration of how you could augment the images, and not the augmentation we will use later.