Juho Puurunen is one of the trainees working at HXRC in spring 2022. As a part of the Future Touchless Shop project he developed AR apps for measuring rooms and trying on 3D rings. Read more about his experiences below!
Text and pictures by Juho Puurunen
Hey! I’m Juho Puurunen and this spring I was still a 4th year student at Metropolia, studying Information Communication Technology. During the first half of 2022 I did an internship at Helsinki XR Center. I worked on the Future Touchless Shop project, my main focus being developing AR and VR applications as the software developer of the team. We had a lot of interesting concepts to work on!
Measuring real space in augmented reality
First, our trainee team got to work on an idea regarding measuring rooms in mobile AR. This data could be sent to virtual reality, so a user could easily furnish their rooms in a virtual space. My job was to start creating an Android software for the measurement app using Unity. I had done a similar kind of project before, where you could place furniture in a space using AR. Getting real-life measurements in AR was a fun challenge!
We got pretty far! In the picture below, the AR measures the distance as being 96 cm – and with measuring tape it’s also 96 cm.
Screenshots from the AR app, and real-life comparison.
After the AR part, we were supposed to create a VR application you could send the AR measurements to and generate rooms with said data. However, this part of the Future Touchless Shop was canned, as there was not enough interest in it. Though I had already made a working VR space where the scene would generate a room with the given measurements. I enjoyed writing the scripts for these in C#, as in my degree studies we usually write everything in JavaScript.
3D ring fingers
Then we started working on Mediapipe, Google’s machine learning software that can track different things from a live camera feed. I used its JavaScript library to create a browser-based application for trying on different rings – I wanted it to work on a mobile browser, so the user wouldn’t have to download any new applications.
Mediapipe isn’t actually meant to be used with 3D objects, so the documentation didn’t help much, though it did help me out with starting the tracking. Melina made me some 3D rings and after I got the gist of how tracking works in a browser and what data I can get from mediapipe, I could use a Three.js JavaScript library to point the rings on where a finger should be.
Hand and ring assets inside an AR development software.
View of a 3D ring on a real hand using AR.
Since the camera feed itself isn’t 3D, and Mediapipe’s “3D coordinates” weren’t really 3D either, it was not easy to get the ring to look real. Also, based on the hardware, it would oftentimes be laggy. So, we were asked to take a different kind of approach: a 3D software in a browser, where you could view the rings, rotate them, and take a closer look.
I had the idea to create a 3D viewer where you could choose a ring, and then have the option to also view it using AR – since there’s a lot of ring-fitting apps without AR functionality on the web already. This project is still in development, but we have all kinds of prototypes coming up, so stay tuned!
Final words
I have really enjoyed my internship at HXRC. Me and the other interns had a great team spirit, and all the projects we worked on were thought-provoking. Even though I worked remotely the whole time, I feel close to the team, as we discussed not only work matters, but had a good meme game going every day. Having meetings three times a week also makes working on a project agile, and we got results and reviews fast!
To see previous news about our trainees’ projects, head over to the Trainee news section.
Follow us on social media for more posts: Facebook | LinkedIn | Twitter | Instagram