Abstract
In hospitals, radiologists regularly examine medical images such as X-Rays or CXRs and write radiology reports to summarize their illustrative and conclusive impressions. Nevertheless, this process consumes radiologists? time and is prone to errors. Therefore, the automatic generation of such radiology reports may help reduce the workload for radiologists in making diagnoses and writing relevant reports. However, this remains a challenging task that requires the simultaneous achievement of two important subtasks: understanding the medical images and generating an accurate textual radiology report. Nowadays, advanced Deep Learning (DL) techniques in Computer Vision (CV) and Natural Language Processing (NLP) are widely used for tackling this problem. In our work, we investigate a proposed framework in order to build an encoder-decoder model using an attention mechanism for generating radiology reports from medical images. The proposed framework is based on using Convolutional Neural Networks (CNNs) as an encoder for extracting image features and Long Short-Term Memory (LSTM) as a decoder for generating words. The main contribution of this thesis lies in integrating the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model in the proposed framework to improve the model?s performance. Specifically, in the experimental setup, three different types of pre-trained BERT models have been applied, namely BERT Base, Clinical BERT, and Character BERT. The proposed framework has been applied on a publicly available dataset, Indiana University Chest X-Rays of medical images and their associated radiology reports. Experimental results show that integrating the pre-trained Character BERT model into the encoder-decoder model with the attention mechanism has improved the model?s performance.
