My first GSoC post

Well, it is my first post, hope it will be interesting. I plan just to explain the idea of my project, share some links and useful materials.


The topic of the project

"Several medical procedures in surgery or interventional radiology are recorded as videos used for review, training, and quality monitoring. These videos have at least three interesting artifacts - (1.) anatomical structures such as organs, tumors, tissues, etc. (2.) medical equipment, and (3.) medical information overlaid that describes the patient. It will be immensely helpful for review and search purposes if these can be identified and automatically labeled in the videos." - source.

From the description, it is clear that a resulting set of ML tools should be able to:


  1. Classify the medical procedure being carried out;

  2. Organs, tumors, tissues segmentation, and detection;

  3. Medical equipment classification and segmentation;


An essential feature of the designed product is that the model should be able to work in a VR environment. 


Issues related to implementation

In my opinion, the main issue here is the lack of data. VR datasets are not that rare. However, the domain of VR medical images is significantly less widely spread than the ordinary medical imaging datasets. I propose a simple solution for the beginning of the project: to ‘simulate’ the VR environment.

In other words, we can use an ordinary dataset (for example, Cholec80) and then add Oculus Rift-like distortion so that it looks almost like an image from the VR headset.


Simulated VR example

The figure above shows how I plan to simulate the VR environment. It is seen that the resulting image is similar to the one we can see using the Oculus Rift headset. I think that using this trick; we can use plain image datasets for classification, segmentation, and object deteсtion and then transform it if no VR datasets are available.


Related work

  • Open Visualisation Space and Tumor recognition. I have found an article on OVS+Tumor: a tool for enhanced lung tumor annotation in VR for machine learning training and analysis. If I have understood correctly, this tool can be used to create datasets for 3D images classification in a VR environment. I have requested access to this article from my university. However, it could take some time.

  • Object detection in a VR environment. An article suggests using a special YOLO model for object detection in a VR environment. This article is out of date. However, their dataset could be used to pretrain YOLO-5 on VR images and then to finetune it for tumor/surgery tools detection in VR.

  • More data. It is possible to use more data from CAMMA, simply by simulating the VR environment. At least CAMMA’s Cholec80 surgery scenes dataset will be used in my project to develop a model for scene classification in VR. We can also use any open dataset, for example, the widely known brain tumor dataset, and try simulating a VR environment with this data. 

                   Extraction from Cholec80 dataset 


  • Surgical Tools. There is a nice object detection dataset and a paper accompanying it. I plan to use it for medical equipment detection. Also,another dataset can be used for surgical tools segmentation.


                 Extraction from Surgical Tools detection dataset 


  • Laryngeal Cancer detection. I have found The Laryngeal Dataset, which can be used to detect cancer during laryngeal surgery. It can be adopted to be used in VR via the proposed method.


To conclude the related work section, I would like to identify what I plan to focus on during the GSoC program. First of all, I plan to build a scene classification model from the Cholec80 dataset, adopted for the VR environment. The next step is to build a surgical tools detection dataset.


Conclusion

I believe that this small post can explain the essence of my project, its main aspects and provide you with a vision of my roadmap. Probably, several aspects will change after a meeting with my mentors, but the main "skeleton" of my GSoC project should look like that.
Location: 3MR2+3W Nur-Sultan, Kazakhstan

Comments

Post a Comment

Popular posts from this blog

Summing up my GSoC experience

Image
 Introduction Once again, I am Daniil Orel from Kazakhstan (Nur-Sultan / Astana city). This summer I was working with LibreHealth organisation for an amazing GSoC project. Few words about my host organisation LibreHealth is a  collaborative community, which supports free open source software projects in HealthTech. This organisation has multiple projects, but I was working towards the radiology project. My project Several medical procedures in surgery or interventional radiology are recorded as videos used for review, training, and quality monitoring. These videos have at least three interesting artifacts - anatomical structures such as organs, tumors, tissues, etc.; medical equipment, and medical information overlaid that describes the patient. It will be immensely helpful for review and search purposes if these can be identified and automatically labeled in the videos. My task was to develop ML models which can process videos and that are suitable for VR environment. As a result, I h
Read more

How can we "simulate" VR?

     Having the proper data is important when it comes to developing an ML/DL application. In some cases, the amount of data is limited due to some reasons. For example, if we are trying to detect a rare disease, collecting 1KK samples could be simply impossible. When collecting a proper dataset is too expensive, we cannot afford it as well. What shall we do in such a case?     The answer is simple: we have to generate data. Generative Adversarial Networks  are said to be one of the ways to synthetically generate the data. However, it requires at least some data from the domain of interest to generate new data. In this case augmentations could help.  Data augmentation is the artificial creation of training data for machine learning by transformations (c) It means, that we can change the original data to increase the training sample and / or apply some transformations to another dataset to get something in the domain of our interest. The second technique can be used, for example, to add
Read more

Surgical tools detection

Image
 This week I was training the Surgical Tools Detection model. The dataset was taken from here . To train the model and prepare it for inference I was using Roboflow. Roboflow  is a robust web-platform for collecting datasets, training models and running them into production. Training I have split the data on 69/21/10 for train, val and test sets. Graphs with training metrics is shown below Train / val metrics To be more precise, on test set the model has 93.8% precision with 90% recall and 96.4% mAP.  It is considered as a high performance and means that the model can be used in production. Inference Roboflow itself provides multiple ways for inference. I could use web-API method (requests are sent to the server and the model sends back predictions), but to make it more flexible, I prefer python-package method. It can be easily imported to the main project of LibreHealth and run.
Read more