This is the new website of the UP MS.  |  Back to the old website

“M3D-Vision” – live medical interventions

20 May 2020

“M3D-Vision” – live medical interventions

Because of the coronavirus, there is an urgent need for well-trained health personnel, but the epidemic also makes practice-oriented education difficult. The M3D-Vision developed at the University of Pécs offers the possibility of three-dimensional recording of surgical procedures and interventions, their inclusion in the curriculum and their practice in virtual space, and can be an integral part of health education even in “peacetime”. We spoke with Dr. Péter Maróti, one of those who envisaged the system, at the UP 3D Center.


Written by Gábor Szabó


Now, special attention is paid to any solution, whether for distance education or publishing any content, and if we add targeted healthcare use to it, the M3D-Vision system is possibly even more interesting. What is M3D-Vision briefly?

- Very briefly? Live streaming of medical contents. The more precise answer is that it is a specialized framework that supports education and curriculum editing. An interactive, health 3D content recording and playing toolkit for relieving and renewing health education and care. The M3D-Vision system is a complete hardware and software package that enables the recording of audio and video in a clinical environment with both 2D and 3D technology, as well as user-friendly editing, uploading, and sharing interactive content. It was invented primarily for educational purposes, but in the meantime, it has been found that there is a need for several other uses as well.

In the name, 3D is very targeted then. How does it work?

- From the beginning, our goal was the ability to produce real, three-dimensional content. The product enables users to stream and record live 3D contents under surgical conditions. As devices able to display 3D content are becoming more accessible, such as the Oculus Rift, virtual reality and augmented reality (VR / AR) may become increasingly important in education. Like I said, it is a hardware and a software system at the same time. An intervention can be recorded from two or even more camera angles. The camera can be on the doctor (POV) or placed on a special camera tripod, which can also be controlled by the doctor or an assistant, but it can even be placed in the surgical light.

In addition, images from any recording device can be linked to the data stream: conventional PTZ cameras, (stereo)endoscopic devices or (stereo)microscopes, robotic surgical solutions, or video laryngoscopes to list a few. The recording can be done from the perspective of the one performing the intervention (POV) or from the perspective of the surgical light handle. The control unit is a small box, which the operator can hang on his or her belt, we have also developed a 3D pointing device (we call it a “magic wand”) that can be used to edit or even annotate, comment on the streamed content in real time during the live stream, either with freehand drawings or pre-set, recorded illustrations.

The saved contents can be edited, grouped, and shared with multiple user groups at the same time in a user-friendly way. The streamed content can be displayed on traditional screens (even on a classroom projector), on mobile phones, but it is of course the best in a virtual, 3D environment. It is important that we even took one step further because we designed it so that in the future, the system would be able to record and transmit the parametric data of the environment.

In addition to 3D visual data, it is also possible to extract depth information from the system, thus providing the possibility to transmit, store and analyse spatial information. All the important data of the procedures can be displayed and recorded, played and evaluated over and over again: how the procedure was performed, how big the incision was, what series of movements took place, where the doctor reached. The recordings can then be used to inform patients and to document or even reconstruct the interventions.

The questions may arise as to how difficult is it to use and does it distract?

- We have tried to make the devices as user-friendly as possible, whether we think of hardware or software. It can be easily mastered even without practically any kind of technological background knowledge because simple handling has been considered one of the basics of the development from the very beginning. It is not harder to use or learn than a simple mobile app.

Let's talk about the possibilities of using it in education! Originally, this is where the idea came from if I understood it correctly.

- The core of the idea started to be outlined from the experience of my own practical trainings. The authentic and accurate illustration of various medical interventions is an everyday problem in health higher education. I always had a feeling that there was still room for development in this area. For students, access to surgical areas is limited, and accordingly, it is difficult to acquire knowledge. In this way, the most diverse procedures could also be presented in education very well. This system could also be used in digital curriculum development, as real 3D, annotated learning materials could be created from the recording of hospital interventions.

By the way, the software part is not just about making recordings. We have developed a multiplatform available on the web that allows instructors or doctors working in care, in addition to handling live 3D / 2D surgery streaming, to edit, group and share the recorded materials with students, and to organize the complete documentation of surgeries and interventions,. With the students in mind, we have created an iOS and Android app, which is for viewing live or edited learning materials in VR (e.g., Google Cardboard) or 2D mode. We have even developed an electronic booklet to track performance.

The Medical School of Pécs has been developing its own digital, educational, and virtual curriculum program, PotePedia, for a year now. Can M3D-Vision play a role in this too?

- Absolutely. This solution could effectively supplement the traditional, lexical knowledge bases and contents, and it can also be a very effective streaming platform in simulation education - in which the Medical School Department of Operational Medicine, the Medical  Simulation Education Center, the Mediskills Lab is at the forefront in Hungary as well. We are open to this, some of the testing is still ongoing in the Mediskills Lab.

The quality assurance of the interventions is also an obvious area.

- We do not like talking about this, but malpractice lawsuits are present in Hungary as well. It is a huge expense for the healthcare system and a serious source of tension for the doctors themselves. In most cases, the accurate follow-up of the events that took place during the surgery and the documentation of the interventions are currently done on paper. There is no IT solution in the hands of healthcare professionals and hospital directors that allows for the easy recording of the interventions, so this could be an obvious solution to produce audio-visual, quality assurance contents. I think this would protect many people, including patients too, as the discipline of adhering to protocols can also be strengthened by the presence of the device. The range of users could also be extended to insurance companies, even if they themselves could initiate and fund its introduction to hospitals.

In addition to medical universities and hospitals, insurance companies, who else might be interested in the development?

- With a small modification of the software framework - as it is designed to be quite flexible -, the product can be used in other fields with minimal development as well. What we are now thinking of as target markets are the military industry, the entertainment industry, the full spectrum of higher education, especially in the fields of veterinary, natural sciences and technical trainings.

Robotic surgery systems, such as da Vinci, have been available on the market for some time, and in many cases, they are "taught" and programmed based on real surgeries and the movements of excellent surgeons. Can a series of parametrically recorded surgeries be suitable for this purpose as well?

- Yes, the data obtained from the system can also be used in this field of application.

How unique is this development and what about property rights?

- There are similar systems, initial attempts abroad, but ours is at an advanced stage compared to these. The intellectual property right is currently 100% owned by the University of Pécs. The development is coordinated by the Medical School, with the involvement of the Faculty of Engineering and Information Technology. We also have an industrial partner, a multinational company involved in software development, CAE Engineering Ltd., the national subsidiary of CAE Ltd. (a Canadian group of companies dealing with military, aerospace and medical simulation).

Based on what you said, the work on the program did not start today. When can this be a “boxed” market product, and what else is missing?

- Work has been ongoing since 2017, so the system is now in a very advanced state. We are already testing the prototype live, practically every element is working. The current state has been achieved based on feedback from real practical applications and is suitable for serving research, demonstration and educational purposes in accordance with the primary goals. We plan to further develop our existing prototype in the frame of a university spin-off company. First, we would like to expand the development team, and we need to involve an external service provider in the finalization of the product who has more serious experience in entering it to the market than us, and can also do it quickly. We estimate that if we managed to attract an investment of € 1,150,000, we could complete the development in a year and produce that so-called “boxed” product. However, we could start pre-sales at an early stage, based on our existing results.


References, awards, press/media releases: