Merged VR: Augmented Reality Cubed

There is considerable development activity at the high end of hardware and content creation for virtual reality and augmented reality, as well as such AR aliases as mixed reality, extended reality and others.

Most industry investment is aimed at leading-edge gaming and industrial application development. However, the low-end of the market is also worth looking at, to see how people and institutions without big budgets — consumers and education — might adopt these technologies sooner, rather than waiting years for all the advancements to trickle down to lower price points.

An example of AR done on a more human-friendly scale is Merge VR. Its founders seem to have taken the hint from The Graduate (“One word: plastics”), added software and a bit of creativity, and then applied it to AR.

First, Merge VR built a soft plastic headset (which they call “goggles”) that fits any smartphone, even the largest. The optics worked well for me. The physical controls to adjust the optics are easy to use, the headset is comfortable to wear, the plastic is easy to clean, and it is inexpensive (US$60 retail in mid August 2017). Two indirect touch buttons provide limited in-app touch control while the smartphone is mounted in the headset. Cord management is easy — both for keeping the smartphone powered and for plugging in headphones.

It comes in a variety of colors; I have seen at least 10 colors in Merge VR’s offices. The device looks friendly, not high-tech. What’s more, the headset has met the safety requirements of a broad swath of regulators for use by kids as young as 10 years old.

TiriasAnalyst Paul Teich Wearing Merge VR Goggles With Earbuds

A Deceptively Simple Cube

A front-panel insert, when installed, keeps the inside of the headset dark for VR applications, especially when it’s used with smaller smartphones. For AR applications, wearers can remove the insert and expose the smartphone’s camera through proper orientation of the smartphone in the headset.

There is huge industry-wide investment in developing transparent displays for AR. Transparent displays promise to deliver full field of vision (FoV) graphic overlays — or at least as much field of vision as wraparound sunglasses afford. The challenge is that creating totally clear displays is very hard; so is creating full FoV displays. Current transparent displays are somewhat dark; wearing them is very much like wearing sunglasses indoors.

Another way to achieve transparency is to use the smartphone camera to transmit the image to the phone’s display, but cameras are still much lower resolution than looking at reality through a physical lens — especially smartphone cameras.

While camera resolution is increasing, a single smartphone camera cannot provide high-quality stereoscopic vision. In this early stage of AR, I think “through the lens” viewing wins over transparent displays, simply because smartphone-based AR viewers can be built and sold at mass market consumer prices.

What can a smartphone based, single-camera AR solution do?

Smartphones now have enough processing power to run many classes of video image recognition in real time. The graphics processing power used to deliver mobile games also can create real-time 3D overlays that track objects in a video stream.

The process of aligning graphics onto video artifacts is called “registration.” Modern smartphones have the compute power to do three-dimensional registration on three axes of rotation.

Running video image recognition and registration in real time without consuming all a smartphone’s batter power requires simplifying the image recognition task.

Merge VR designed a plastic artifact specifically for easy real-time video image recognition to support simple, low-cost education and entertainment AR apps. The artifact is a cube, called, simply enough, the “Merge Cube.”

The Merge Cube is 72mm (2.83 inches) on a side and easy for a child or adult to hold. It is manufactured with unique, easily differentiated patterns embossed into each side. The flat tops of the patterns are reflective silver; the space between patterns is matte grey.

Merge VRGoggles With Merge Cube

Designing the artifact was not enough. Merge VR’s software development team packaged video image recognition and image registration software into the Merge Cube Software Developer Kit . Merge integrated its Merge Cube SDK with Unity’s game development platform and Vuforia’s mobile vision processing SDK. Applications that integrate the Merge Cube SDK run on top of Unity’s game engine on a smartphone. There are many advanced vision processing libraries emerging for smartphones, and Merge VR is likely to leverage more of them in the future.

Most AR registration marks are two-dimensional; rotating a 2D registration mark a few degrees can throw tracking off. The Merge Cube can be tracked in any orientation, at any angle. The tracking software knows that the Merge Cube is a cube, and that each side is unique. It knows what order the sides are in — which sides are connected to which other sides — so that as the Merge Cube spins, tracking software can anticipate the next side to be exposed.

Merge VR designed the registration marks to be identified easily without requiring general purpose pattern recognition algorithms. It is only scanning the incoming video feed for six unique and not naturally occurring patterns. Recognizing hand gestures, for example, would require much more compute capability and would consume more battery power.

Developers create a 3D geometry in Unity and wrap it around the Merge Cube. Unity then takes care of the work to rotate the 3D geometry as the Merge Cube rotates. Tracking is fast because it is performed at standard full video resolution and frame rates (typically 1080p), not at the full photographic resolution of the smartphone image sensor.

Merge VR downsizes the video stream to save power and time for image recognition, and then removes the color components to process the video in greyscale. Using the Merge Cube merely needs a moderate quality smartphone camera sensor — cameras above 12 megapixel are fine.

Smartphones Catching Up

The Merge Cube SDK has been optimized for most phones launched within the past two years. As a baseline, Apple iPhone 6 and Samsung Galaxy S6 are among the oldest phones that can run the Cube SDK.

The challenge for Merge VR is that many smartphones still in use do not have the full complement of features required to support VR: Android 7.0+, high-resolution display, acceptable graphics performance, and a full set of positional sensors (accelerometer, compass and gyroscope).

Developers and customers can check here to see if their smartphones support the Merge Cube.

There is only one version of Merge Cube, because app developers can overlay it with anything they want.

Any recent smartphone, mounted in a Merge VR headset (with the camera exposed) and running an app developed with the Merge Cube SDK, can turn the Merge Cube into… something else, anything else. Wearers spin the cube in any direction and interact with the graphics superimposed on the Merge Cube, all at a very fluid fame rate.

The Merge Cube can be used without the Merge VR headset — for instance, by holding the smartphone without a headset at all. It can be used with any other smartphone-based VR headset, but that headset would have to expose the smartphone camera. I have not tried surgery on an actual cardboard version of Google Cardboard, but that should work.

In the U.S., Walmart is rolling out Merge Cube this week starting in West Coast stores and moving east and into online ordering. It sells for about $15. Walmart sells the Merge VR headset for about $60. Merge VR also offers bundled and direct discount pricing for educators and school districts.

Any developer or hobbyist smartphone app programmer wanting to experiment with AR can do so with under $100 of Merge VR gear, a reasonably current smartphone, and the above-mentioned software tools.

More than 500 studios are developing apps for Merge Cube, with dozens of apps scheduled for release in the next 30 days, Merge VR said.

I have played with prototype units; now I am looking forward to buying one and trying out the new apps.