How to build a DIY Magic Mirror out of commodity hardware and augmented reality APIs
As augmented reality technology continues to advance, it opens up possibilities for a dazzling array of practical and innovative applications. One great example is a “Magic Mirror”. An augmented reality Magic Mirror is a reflective TV screen linked to a camera and controlled by augmented reality software. The TV screen can behave like a regular mirror, but it can also enhance the reflection by showing video feeds from different angles, adding accessories like earrings purses or hats, applying makeup or changing the color of outfits.
See a demo of this technology below.
With their features, limited only by the imagination of marketing teams, Magic Mirrors are expected to replace traditional mirrors in dressing rooms in retail stores. They will help draw customers in and enrich the shopping experience, leading to increased engagement, loyalty and sales.
Retailers interested in a Magic Mirror system, can choose to purchase one out-of-the-box from a number of suppliers, or build their own using commodity components and developing a custom augmented reality software. In this blog post we’ll share our experience building a “Magic Mirror kit” out of commodity hardware components and open AR frameworks that is cheap, fast and can be enhanced to fit every retailer’s specific needs, as an alternative to purchasing a closed vendor solution.
Build your own Magic Mirror or buy an out-of-the-box product?
With growing interest in Magic Mirrors, there are many companies that offer out-of-the-box Magic Mirror augmented reality solutions, like Memory Mirror. Using these products can get expensive quickly. With price of $10K - $15K per mirror, a retailer with 1,000 stores and 2-4 dressing rooms per store, may be looking at a $2M - $4M price tag.
At the same time, the cost of the hardware is only around $2K - $3K depending on the configuration, or 20% of the total price. The remaining cost of the solution is in the software. So, the key question the customers should be asking is this: what will it take to build the custom software in-house relative to the overall cost of the out-of-the-box product?
It turns out that you can build a magic mirror relatively inexpensively from scratch. Using a simple mobile application and commercially available hardware, we were able to produce a simple magic mirror for a fraction of the cost in very little time using a small mobile team.
Before we go into more detail about how we developed this solution for the customer, let’s first look further into the features and some examples of magic mirrors available on the market.
Breakdown of Magic Mirror features and capabilities
The following gives a breakdown of the more common features and uses for magic mirrors. First we will review the basic features and then list some of the more advanced features possible using augmented reality (AR) technology.
Basic features
- “Reflect”, i.e. deliver a real-time view of the environment in front of the mirror
- Capture video and playback the recently recorded footage
- Display recorded video and the live view from the camera side by side simultaneously
Advanced AR features
- Augment the reflection (real-time view) with virtual objects e.g. to apply virtual make-up, change the color of clothing items, or put on virtual glasses or watches
- Recognize human faces and bodies
- Motion detection, turning on when someone passes by
- Understand gestures and movements
- Recognize speech and respond to people’s questions
- Display promotions, banners, images, or videos
- Access the internet and communicate with other devices
Types of Magic Mirrors
Tablet sized
A magic mirror can essentially be as small as a tablet. In this case, it is a camera, a mirror, and a computer in a single device. Tablet-based magic mirrors are already being used for makeup and hairstyling in hairdressing salons where they use augmented reality technology to for example apply virtual makeup to the customer.
Of course, any tablet can also be used at home. It’s also possible to cast the screen of a tablet or a smartphone to home TV with Chromecast or Apple TV appliance. Additionally, there are some smart TVs, which could receive a video stream from mobile devices without any hardware in between.
MemoMi [Photos: courtesy of MemoMi] PHOTO: COURTESY OF MEMOMI
Two-way magic mirrors
There are also larger magic mirrors that utilize a semi-transparent two-way film. These appear identical to a traditional glass mirror until they are powered on. Once active, these mirrors have a “smart area”, which takes up a portion of the mirror’s surface and can be used to display a range of different visual elements such as text, video, or other types of multimedia.
Marina Nazario/Business Insider
Public magic mirrors
In public areas such as a retailer outlet or shopping mall, magic mirrors can be much larger (in the range of 65” - 75”). These are intended for public shared access rather than personal use so generally do not allow the public any direct control over them. Because of this, they usually do not include any form of user interface. In some cases, they may recognize gestures or allow limited user control via their personal smartphone or mobile device.
The sensor revolution, Raphael, Rina, fast company
How to build a public Magic Mirror in-house with commodity components
With this understanding of the Magic Mirror market, let’s now take a good look at how to build the Magic Mirror in-house with commodity components. The analysis below is performed for the “public mirror”, although the same approach works for all other types.
The solution is based on three major components: (1) a TV with a camera, on a stand or inside a frame, that acts as a “mirror”; (2) a smartphone with a built-in camera that acts as a computing platform, as well as the “eye”, and sometimes can take user input to configure or control the mirror; and (3) connectivity between a TV and a smartphone, including the network and connectivity software like Chromecast or Apple TV.
The complete solution can be based on either Android or Apple platform technologies. We’ll start with Android stack and later explain how the same architecture can be adopted to iOS stack.
Now we need to integrate these pieces together, starting from a basic ability to get the video feed from the camera, manipulate the image and project it back to the TV screen. That’s done quite simply by building an app based on pure Android SDK without any third-party libraries. We combine CameraFragment API for building modular applications with TextureView (API to display a content stream, along with CameraDevice from the android.hardware.camera2 package to be able to manipulate the video feed from the camera with a fine-grain control of image capture and post-processing at high frame rates.
To seamlessly link the Android device and the monitor, we use a Google Chromecast adapter, which is the cheapest and simplest way to stream smartphone screen content to any TV/monitor with an HDMI input. The Chromecast dongle is plugged into the HDMI port and powered by a separate USB cable. If the monitor has a USB port, the dongle's power cable can be plugged directly into it.
To bridge the smartphone and the monitor using Chromecast:
- Plug the Chromecast dongle into a monitor
- Register the Chromecast
- Enable screen sharing on your Android phone
- Fasten your Android phone on a tripod (or in some other way) in portrait orientation
- Enable Android's camera
For more detailed instructions, refer to this Google guide: How to cast: A quick start guide
As we want to rotate the monitor to portrait mode, there are some changes that need to be made as the default settings for monitors are set to the landscape orientation. To make sure the smartphone and monitor views are synchronized, we need to cheat the native Android orientation and make some changes in our application. To do this, we declare Activity orientation as android:screenOrientation="landscape" but leave the phone in portrait orientation.
Now the monitor works as if it were a mirror, i.e. the monitor renders the real view seen by the smartphone camera. Now to add a little more “magic” to the mirror. By recording several seconds of mirroring and setting the camera to use a delay, the person can turn around and then be able to view themselves. This allows them to turn a full 360 degrees and then wait a set number of seconds to be able to see themselves from any angle displayed on the screen.
The MediaRecorder component is what we need to record the camera view as it allows us to save live video to external storage. It is a very simple process to replay recorded video as MediaPlayer is a standard Android component. Then the only other component that we need is ViewVideoFragment.
Volla! We have a basic Magic Mirror that allows the consumer to walk up to it and spin around for 8 seconds, and then see the recording of these 8 seconds along with a real-time view with full 360-degree view. This “360-Mirror” is a complete system, extremely cost-effective and fully ready for installation at the retail store. It will take a small team of mobile developers a few days of work to put together.
The process using the iOS platform is very similar to the Android process outlined above.
All that needs to be done is to replace the Android hardware and software components with iOS ones. So this involves using an iPhone rather than an Android smartphone and switching the casting device from Chromecast to Apple TV.
From this point on, you have a platform that can be enhanced in various ways to add “augmented reality jazz” and features such as change in color of a piece of clothing worn, add an accessory, apply virtual makeup and change scenery to the beach or a nighttime party. We will not get into the details on such implementations in this blog. Suffice to say that the sky’s the limit and many dazzling features can be added over time using mobile development and specialized augmented reality APIs.
Summary of the argument for building a similar solution in-house
The main argument for building Magic Mirror in-house boils down to two factors:
- Cost: our experience shows that you can build a basic Magic Mirror in-house very quickly with a small team and save millions on large installations that include hundreds of stores and thousands of mirrors
- Flexibility: you can start small and incrementally roll out new features that are unique designed to capture the hearts and wallets of your customers, based on your specific products
Some retailers with small store footprint and lack on qualified development resources would undoubtedly be better off buying a commercial offering. At the same time, a large number of tier-1 retailers may be well-justified in building their own Magic Mirror along the lines described in this post.
Victor Chistyakov, Dmitry Sumtsov