Daniil Orel GSoC 2022 blog

 Introduction Once again, I am Daniil Orel from Kazakhstan (Nur-Sultan / Astana city). This summer I was working with LibreHealth organisation for an amazing GSoC project. Few words about my host organisation LibreHealth is a  collaborative community, which supports free open source software projects in HealthTech. This organisation has multiple projects, but I was working towards the radiology project. My project Several medical procedures in surgery or interventional radiology are recorded as videos used for review, training, and quality monitoring. These videos have at least three interesting artifacts - anatomical structures such as organs, tumors, tissues, etc.; medical equipment, and medical information overlaid that describes the patient. It will be immensely helpful for review and search purposes if these can be identified and automatically labeled in the videos. My task was to develop ML models which can process videos and that are suitable for VR environment. As a result, I h

 The last part from my work to show is a video-processing tool, where a detection model is used. The detection process Since I am using the roboflow library for predictions, my models is available by url. The issues is that the image can be sent as a file (roboflow processes it as a byte stream), that is why I have to save the image as "tmp.jpg" every time the predict function is called.  Fortunately, the quality of model is high. Here are some examples of images unseen by the model On the images above the bounding boxes are drawn correctly. Moreover, it can be seen that the classes are indicated correctly as well. High quality of the model allows to suggest using it for controlling the surgical tools during trainings.

 If you are reading this post, most probably, you have seen my last post about the video processing pipeline. Actually, this approach has few issues. Segmentation classes are hard do distinguish It is slow For the last two weeks I was working towards fixing these issues. Improving the processing speed To fix the processing speed I only had to adjust few parameters of OpenCV. Initially the FPS of resulting video did not match original (it was less than the original), but after I have fixed it, all processing went faster. Making segmentation easier to read I have adjusted the project so that now two modes are available. Now there are two possible modes: coloured segmentation and segmentation of a single class. Coloured segmentation Example of coloured segmentation On the image above it is easy to differentiate segmentation classes, but here is a single issue. You cannot identify classes if their indexes are not knows (so you have to read the documentation :D) Single class segmentation Th