Education must also keep pace with today's increasingly widespread and accessible virtual reality-based technologies and new trends, which is why the UPMS has launched an educational and institutional development program.
Almost at the same time, the UP 3D Center - which is connected to the Medical School in Pécs in many ways – has been contacted in connection with several software developments in 2020 so the number of software developers in the center was increased to be able to meet these requests. Unfortunately, this year has brought with it a pandemic, but pressure is often a catalyst for innovation, so developments that are already running have been complemented with new ideas and needs that have fortunately met the opportunities now. One of the results of these was the POTE VR project, at the beginning of which we were still living in the period of quarantine: no wonder it was not only an exciting but also a much-desired opportunity to enter the main building and the Aula of the Medical School in a virtual way. That is, one of the important venues of university life should be accessible from home and have a virtual presence there. This has already been achieved because currently - as a first experiment - visitors can see the exhibition of the photographer Lajos Kalmár, but due to the knowledge of the software and its expected developments, it can be used for more things.
We talked to the developers about this and the background of the work, with dr. Péter Maróti, the professional manager of the UP 3D Center who provided all the support to the team during the development of the software. Dr. Péter Bogár, as a lead developer was responsible for the technical coordination and software development of the development team. Modelling and building scanning were the tasks of Péter Hillebrand and András Szőke, which posed special challenges, and the software development part was reinforced by Attila Péntek and Mátyás Bene.
Written by Gábor Szabó
- How did you come up with the idea?
- Péter Maróti: Remembering when I was about to choose a university, I was curious about where I would go for several years. In addition to the various prospectuses and photos, I would have liked to have a virtual look around the university. We have had the idea for a long time to create some kind of virtual system for students and those who are interested. And in 2020, we were very pleased that our team was invited by the staff of the UPMS so that the orientation days and other important events cannot only be viewed live in the main building, but can also be visited virtually with regard to the current regulations.
- The Thematic Excellence Program has recently ended, in connection with which we have already discussed the UP 3D Center becoming a scientific and technical basic workshop. An important element of this has been the establishment of a dedicated software development team from the excellent developers gathered here. Without this, this project would not have been completed either. What was the initial experience?
- P.M.: Within the framework of UP 3D, a team has already been formed, in which colleagues with different backgrounds work together. Some of them have been working at the center from the beginning of UP 3D while also completing their undergraduate studies. Some of them have done scientific and research work before and earned a PhD degree, others have come to us with company experience. At the center you can meet a physicist, a doctor or even a mechanical engineer, but students from different faculties are also involved in the work of different disciplines.
- These are real, existing spaces, such as the Aula. Based on this, practically anything can be displayed this way, even a classroom or an operating room. How is this virtualized? Do you recreate these spaces or scan them like a three-dimensional photograph?
- Péter Hillebrand, András Szőke: Our tasks were modelling and building scanning. In addition to real spaces, building models designed based on imagination can also be explored with this software. After the 3D scanning, for the visualization we need to optimize the model of the physical space for browsers, then create a texture using high-resolution photos taken on site, and then import it into our system. We then refine and test the model until we are satisfied with its performance. This process can be carried out on any building.
- What was the greatest challenge in creating the system?
- Péter Bogár: We tried to create a display as realistic as possible, but at the same time we had to develop a multiplatform software, i.e. a virtual gallery that could be viewed from a laptop, desktop computer, mobile or even a VR device. For low-performance mobile devices, it was difficult to find a balance between a realistic look and fast, trouble-free operation.
- How does this development work and what does it currently know?
- Attila Péntek: In the completed Aula model, it is possible to view the exhibited photos and videos, for which textual content and audio material can also be uploaded. Virtual galleries can be created in the software administration platform. Entering the gallery on a computer, the user can move with the help of the already familiar “W, A, S, D” buttons or arrows used in games, while on the phone and VR devices, they can feel as if they are really in the Aula.
- To what extent have you drawn from existing virtualization technologies (such as Google's similar solutions)? Does POTE VR use any of these or is it all brand new?
- Péter Bogár, Péter Hillebrand: We used several open source code frameworks during the development, which is almost inevitable during the development of a software today. Throughout the coding, we kept in mind that our program should be run on as many devices as possible. The system uses existing browser-based WebXR-compatible VR rendering that is compatible with most hardware manufacturer solutions, such as Google Cardboard / Daydream, Oculus, SteamVR, and Windows Mixed Reality. These are well-established and well-known solutions used by many people.
- Those who would like to try, how can they get the best experience – say, if they do not have a special device (like VR goggles)?
- Attila Péntek: We can achieve a similar VR experience by placing a mobile phone in a simple CardBoard, but at the same time the cost is minimal because by ordering two lenses online, we can even make the VR goggles for our phone with the help of a downloadable drawing. If we do not feel like DIY at home, we can get CardBoard glasses at a good price in technical or IT stores. The CardBoard glasses are practically a frame made of card stock paper (cut and folded) into which we place a mobile phone and see the shared screen of the phone. The phone's built-in gyroscope allows you to track the movement of the head in VR, providing a much more realistic virtual experience for the user. Human vision is based on an image perceived by two eyes, offset by a few centimetres, so that by showing a different image corresponding to the eyes, our visual senses can be deceived by modern technology. Just like in a 3D cinema.
- The photo exhibition mentioned in the introduction can still be viewed. During the development, it was a basic aspect to be able to expand it later in both its content and functionality. In what direction are you going to continue?
- Péter Maróti: The content can be expanded at any time by the designated staff of the UPMS, a new gallery can be created in the software even in 10 minutes. Due to the positive feedback we received, we would like to create a similar virtual gallery for other faculties as well. We are also thinking of incorporating additional functions. Examples include streaming live video or displaying unique 3D models in cyberspace.
- The current situation has only reinforced the trend that such technologies, which are considered still a bit exotic, are becoming more and more present in education, for example POTE VR has already played a role in the student orientation day. Will it be specifically used in medical education as well?
- Péter Bogár: The main goal of the project was entertainment, to introduce the university mainly to new students, and to overcome the current obstacles. The technology itself can be used to demonstrate a medical site (such as an operating room). We already have plans in this regard. In the future, it may even be possible to create virtual simulations in which the student can see online how the devices installed in each room work.
- What will be the next "big hit"?
- Attila Péntek: We are not idle; we have even started a new project. This is the creation of a virtual copy of the building of the UP Faculty of Engineering and Information Technology. Here we are also thinking about how the building models created by students in architectural exhibitions could be displayed to the public. Our future plans also include virtualizing the events held every year so far. At these virtual events, students will also have the opportunity to contact different clinics or companies. We have also started surveying several health sites, such as operating rooms and hospital rooms; we would like to give everyone the opportunity to learn about these via the Internet.
