How can upper limb rehabilitation be made more accessible for both the patients and healthcare professionals? In this blog post, our talented GameRAT team shares how they tackled on one of the biggest projects at Helsinki XR Center and took traditional physiotherapy to the next level with XR technology!
Text by Leevi Rantala, Annika Hirvelä, Miro Mariapori, Joona Virmajoki, Anna Hyttinen, Pyry Parkkola & Niko Ranta
Gamified Joint Rehabilitation, Analysis and Training, or GameRAT in short, is part of the European Media and Immersion Lab (EMIL) project, with the goal to research the gamification in physiotherapy using VR headset and tracking. Our goal was to develop a web application for healthcare professionals and an XR application for the patients, as well as analysis tools for assisting both parties during gameplay. We collaborated with Virtual Dawn, a company that provides an SDK for streamlining XR development.
And so it begins…
The project kicked off in the beginning of 2024. Our team included Annika, Joona, Leevi and Miro as our new trainees, Pyry and Niko (from Virtual Dawn) together with Juho (Code Lead) and Julia (Art Lead). The first step was to plan out how we would accomplish the tasks. After many meetings, the project started slowly moving forward. A lot of things were new to everyone but the whole team had good spirit even with the monumental project ahead. Early on, we started visiting the Motion Capture studio to tackle tracking the huge library of animations.
Leevi will now tell you more about animation and motion capture!
Hi! I’m Leevi Rantala, a 3D Artist intern here at Helsinki XR Center. I’ve been mostly working on animation, but also on modeling and some design.
Mocap Sudio visits
As GameRAT deals with motion analysis and comparison, we needed high-quality animation data to serve as a base to compare to. We captured performances in Metropolia’s Optitrack-based motion capture studio for all the exercises, with our Senior Lecturer in Rehabilitation, Lasse Liikanen.
Motion Capture being set up. Julia in Mocap suit, Leevi on the computer setting up.
Animations from Mocap data
This data naturally needed visualization, so I designed and built a Puppet character with rigged props to display the exercises on the web and in Unity. Finding the pipeline and retargeting and cleaning the animations were quite an undertaking! I’m rather happy how they turned out, and by the way, the animations and the Puppet are downloadable for free on our GitLab!
Puppet character model.
My experience
I also enjoyed working on the rig and animations for the Frog that Annika designed, a bit of fun in between serious grinding! No, actually this project has been fun from the get-go, and very interesting to work on – I’ve learned a ton. Thanks to Helsinki XR Center for having me, and I’ll be back!
A big part of the project’s success was thanks to good design. All elements in both the Web and XR applications were professionally designed to fit their purpose.
Next, Annika shares her experience in designing the UI/UX for GameRAT!
Hey! I am Annika Hirvelä, a 3D/2D Generalist intern at Helsinki XR Center. I recently graduated from the Metropolia’s XR Design degree program. During my internship, I got the opportunity to continue improving my skills in versatile projects. One of the best parts of this GameRAT project was that I got to do a little bit of everything including game design, UI/UX design, 3D modeling, texturing, creating 2D assets, animating, creating environments and editing videos.
Designing for Web application
Before this project I didn’t have much experience designing UI/UX, so at first it was a bit challenging, but after a short while I got the hang of it. The fact that the web app is created for medical professionals added to the challenge, because the UI design could not look too game-like which would have been more of my strong suit. Nevertheless, I am glad to expand my skills and I feel like this project has been a good growing opportunity for me.
Figma design for the Web application.
Designing for Mixed Reality
Before starting to create assets for the rehabilitation games, we got a task to compose a game design document for each mini game. It was very helpful to first put the game idea into clear sentences and to assemble a list of assets necessary for the game before progressing further. By planning the games thoroughly we avoided doing unnecessary work and the whole team had a better vision of what we were supposed to do. First, when I started to create the game design documents, I thought that they were really clear and included all of the essential information. Soon enough I realized that despite my efforts, people will have different interpretations if every bit of information is not displayed clearly enough. Game designing is an artform which I haven’t fully mastered yet, but I certainly think that I have gotten a lot better at it!
Figma design for the XR application.
Good feedback & reflections on the project
I have done quite a lot of 3D modeling even before this project but it was helpful to receive feedback from our Art Lead Julia. I definitely feel like my UV unwrapping skills have improved thanks to the art reviews we had. I’m overall pretty happy with how the assets turned out.
This project has had its challenges but I think that the whole team has done a great job and we have managed to get a lot of work done. I enjoyed working on this project and I’m grateful for all the new things and skills I learned during this experience which will definitely help me in the future.
Good design is very important but so is implementing that design.
Miro will now tell you about the creation of the Web application
Hey! I’m Miro Mariapori, a Software Developer intern here at Helsinki XR Center and a Software Engineering student at Metropolia University of Applied Sciences. My role has mainly focused on developing the web application for healthcare professionals. I’ve also worked on the backend and API that both the Web and XR applications rely on. In addition, I contributed to the XR app by building the Lobby and API functionalities.
3D on Web
One of the biggest challenges in developing the web app was creating an editor allowing healthcare professionals to design custom physiotherapy exercises in a 3D environment. To make this happen, I learned to use THREE.js, a JavaScript library for 3D graphics. At first, working with 3D in a web environment was unfamiliar to me, but after some trial and error, I got the hang of it. Seeing the editor take shape was definitely one of the most rewarding moments of the project.
Web Application editor page.
Learning XR Development & final insights
While my main focus was on web development, getting to work on the XR app as well gave me valuable experience in a different area. It was interesting to see how the web and XR sides of the project connected and how the API I worked on played a key role in tying everything together.
This project has been a great learning experience for me. There were challenges along the way, but I’m really happy with how much I’ve grown as a developer. I’m grateful for the opportunity to work on something this meaningful, and I know the skills I’ve gained here will definitely come in handy in the future!
Next up is Joona, who will elaborate more on the XR Development side of GameRAT.
Hello! I am Joona Virmajoki, a Game Developer/Programmer intern at Helsinki XR Center. I recently graduated from the XR Design program at Metropolia UAS. During my studies, I fell in love with programming and game development. With over three years of experience in the Unity engine and C# I became responsible for building the Gamerat XR application, aimed at patients undergoing upper-body rehabilitation. opportunity to work on something this meaningful, and I know the skills I’ve gained here will definitely come in handy in the future!
Developing minigames for GameRAT
In the XR application, I was responsible for developing the lively Froggo minigame and integrating features of the Meta Quest 3 headset. I also assisted in the development of two other minigames – Potionmaker and Hypeup. Beyond the games, I implemented several core program components, including the patient movement tracking system. This system records movement data as bone orientations, stores it, and transmits it in JSON format to a custom backend server for professional analysis.
A sneak peek of the Froggo minigame.
Why Unity?
Because of its strong support for multiplatform XR applications and SDKs, we chose to build the application with the Unity engine. The target device for the application is the Meta Quest 3 mixed reality headset. The application makes heavy use of the Meta XR SDK, utilizing features such as hand tracking, body tracking, spatial anchors, etc.
AI data and gathering
Since the application was also planned to feature an AI-powered system for analyzing patient movement performance and predicting potentially pain-inducing movements, we needed a way to record a large dataset of training data. For that purpose, I developed a simple AR utility for the Quest 3 headset, which was used for recording large amounts of movement data.
My conclusion
Although the project came with its own challenges, I’ve learned a great deal of new skills and improved upon existing ones. For example, processing large amounts of movement data at runtime on an Android-based Meta Quest 3 headset required the use of asynchronous operations. Particularly, serializing the .NET-based movement data to JSON and then compressing the JSON into Base64 for faster web transfer required a significant amount of CPU time, which would have completely frozen the application without the use of multithreaded operations.
Building a complex MR application with intersecting systems and components also taught me a great deal about debugging and good programming principles, such as utilizing interfaces, following the single-responsibility principle, and, above all, avoiding global variables. Working with a large SDK like the Meta XR SDK also taught me a lot about navigating extensive codebases and multi-layered abstractions.
Aside from all the technical jargon, I also learned the importance of good communication and project coordination. But above all, I had fun working with my teammates!
Now, let’s hand over to Anna, who is going to tell us more about her dive into tracking methods.
Hello! My name is Anna Hyttinen, and I study Game Development at Metropolia UAS. During my internship here at the Helsinki XR Center, I made a body movement saving and replay application for the GameRAT project with Unity. I learned that the Azure Kinect camera has a body tracking feature and that it is possible to extract body tracking data from it.
Trouble with Kinect
Working with a device that is no longer supported came with some challenges, but the application is usable through the Unity editor, so that it does demonstrate the capabilities of this kind of camera technology. This was relatively independent work that I enjoyed doing.
Kinect application demonstrating a skeleton of a tracked body.
More research
The other thing that I made for GameRAT is that I researched a way to get heart rate out of a smart watch. I managed to make an Android plugin for Unity and a small application for Galaxy Watch 6 that shows the user’s heart rate. It was a great experience to be a part of this project.
Finally, our collaborators Pyry and Niko from Virtual Dawn will talk about their experience with the project.
Hi! My name is Pyry Parkkola, I’m an entrepreneur and a Game Development generalist. I studied 3D Graphics at Hyria Education when I got hired as a VFX artist for the movie Iron Sky. Afterwards, I started my own company making 3D art and animation while self-learning game development; that was around 2012 or so and we’ve been purely a game/simulation development studio since around 2016.
About the project
My role in GameRAT was as a Unity developer and specialist, since the project at parts leverages my company’s, Virtual Dawn’s, SDK tools for Unity development. I’ve worked on two of the three minigames in the project, as well as the LLM-assistant feature. My work ranges from game and level design (which was a group effort) to game development, mixed reality research, UX development and scripting and developing the system for the use of LLM assistant.
Image of the Potion Maker minigame.
Afterthoughts
I’d say I had the fun part of the development process – building the actual game – while others tackled the hard stuff, like the backend, frontend systems, and data analytics. So I can’t complain, it’s been fun, and the team is great!
Handing over to Niko, who will share his experience on this project!
My name is Niko Ranta, I’m an entrepreneur and a Game Development generalist, just like Pyry. I studied General Computer Sciences in Kiipula Vocational School, but got interested in 3D modeling and game development, and decided to self-learn with help of Youtube tutorials. Pyry helped me a lot when I joined Virtual Dawn back in 2016. Since then, I’ve been honing my skills as a part of Virtual Dawn and focused more and more on Unity development as the need for 3D modeling has been less.
Image of the HypeUp minigame.
In the GameRAT project my role has mostly been Unity development and specialist for Virtual Dawn’s SDK tools. I worked on two of the three minigames in the project and helped with the LLM-assistant feature. At the start of the project I did research on mixed reality features on Meta headsets and have since been focused on game and level design on the two minigames.
Project has been fun and the team as a whole is very knowledgeable and easy to work with.
Key Stakeholders and Collaboration
Gamified Joint Rehabilitation, Analysis and Training (GameRAT) is a project funded by EMIL Horizon Europe as a support for third parties. The project lasted 15 months, from January 2024 to March 2025. The project is coordinated by Metropolia UAS, with assistance by the team at University of Bath, related to their exergaming focus in EMIL, The goal of the project is to create a remote rehabilitation system for upper limbs.
GameRAT is a collaboration of:
To see previous news about our trainees’ projects, head over to the Trainee news section.
Follow us on social media for more posts: Facebook | LinkedIn | Twitter | Instagram