tag:theconversation.com,2011:/global/topics/mixed-reality-49120/articlesMixed reality – The Conversation2023-06-09T03:20:29Ztag:theconversation.com,2011:article/2073072023-06-09T03:20:29Z2023-06-09T03:20:29ZThe Apple Vision Pro hasn’t really impressed consumers, but that isn’t the goal – for now<figure><img src="https://images.theconversation.com/files/531060/original/file-20230609-28-hnfwyk.jpeg?ixlib=rb-1.1.0&rect=43%2C54%2C7186%2C4758&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><span class="source">Jeff Chiu/AP</span></span></figcaption></figure><p>Apple’s new Vision Pro mixed reality headset has generated a significant amount of buzz. Announcing it at this year’s <a href="https://developer.apple.com/wwdc23/">Worldwide Developers Conference</a>, chief executive <a href="https://www.bbc.com/news/technology-65809408">Tim Cook said</a> the virtual and augmented reality headset will <a href="https://www.youtube.com/watch?v=TX9qSaGXFyg">allow users to</a> “see, hear and interact with digital content just like it’s in your physical space […] seamlessly blending the real and virtual worlds”.</p>
<p>The Vision Pro is the first new product category Apple has introduced since the Apple Watch in 2014. It marks the company’s foray into <a href="https://www.apple.com/au/newsroom/2023/06/introducing-apple-vision-pro/">spatial computing</a>. <a href="https://www.wsj.com/articles/apple-is-breaking-its-own-rules-with-a-new-headset-80c9b36c">Analysts</a>, <a href="https://www.google.com/finance/quote/AAPL:NASDAQ?sa=X&ved=2ahUKEwj8ztzBybL_AhWFd94KHUKbCv0Q3ecFegQIKhAh">markets</a> and <a href="https://www.gizmodo.com.au/2023/06/early-testers-vision-pro-apple/">consumers</a> have been quick to react – and not all positively. </p>
<p>On one hand, the headset has been <a href="https://techcrunch.com/2023/06/05/first-impressions-yes-apple-vision-pro-works-and-yes-its-good/">lauded for</a> its <a href="https://www.theverge.com/2023/6/5/23750003/apple-vision-pro-hands-on-the-best-headset-demo-ever">technical features</a>. It’s less clunky than competitors’ offerings and has a range of advanced capabilities, including hand and eye tracking, and the seamless combination of <a href="https://www.wsj.com/video/series/joanna-stern-personal-technology/apple-vision-pro-headset-first-look-impressive-immersive-and-heavy/776FE781-C5F2-4048-9161-08563DA7364E">virtual and augmented reality</a>.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=337&fit=crop&dpr=1 600w, https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=337&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=337&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/531062/original/file-20230609-22-b9kivr.jpeg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">The mixed reality headset has had a mixed reception, although it has generally impressed on the technical front.</span>
<span class="attribution"><span class="source">Jeff Chiu/AP</span></span>
</figcaption>
</figure>
<p>However, others can’t help but point out the hefty price tag of US$3,500 – and the fact that the general public has <a href="https://www.wired.com/story/do-people-actually-want-to-wear-vr-headsets/">simply not</a> <a href="https://www.wired.com/story/apple-vision-pro-doomed/">embraced</a> mixed reality headsets.</p>
<p>Globally, the demand for these headsets has been slowing. Fewer than <a href="https://www.idc.com/promo/arvr">nine million units</a> were shipped in 2022 (mostly by <a href="https://forwork.meta.com/quest/quest-pro/?ref=AVsRL14V5xl7CziavTtRzDrQ8O5yjImi3boLHA-qhBivxUIybXgQgnTdAcluip9cjZjrA7c67zPnPs5o6hzHTzHhbEBZ9MYq98Y7CrFJcfq2-vlUkmOQWpH81Ve59PH6QRUSjSXFlwQ1ejW5iuueSbY7DHyjlTvaROakHgFhkimSBnVniKx8eBZ_zYlTzI3Cktg9jAlUj3RfA2exKTP5Ps3WH1IndddONEOFiKsrIUPfQFsYk_qPBBw8VOTATf6p_jfNrHzpnFSqu8I4nHc02nw0BgZr3aaQzBvbaxkMnK5SutEvjTUiGtqbUB3VJjeUA-EF4BSkodpCwrtrBJa0zDGpk3yzgc6KwY16h9SP5Fr4en9VGIAt6jPTQSfpYa3Kb9UqVG">Meta</a>, Apple’s biggest competitor in this category).</p>
<p>Meta sees spatial computing as a big part of the tech future, despite <a href="https://www.theguardian.com/technology/2022/oct/27/metas-shares-dip-is-proof-metaverse-plan-never-really-had-legs-facebook">market analysts and critics</a> calling for the <a href="https://www.wired.com/story/what-is-the-metaverse/">metaverse</a> to be abandoned. Last week it released the Quest 3 at a relatively low cost of US$499. With continued heavy spending on the metaverse, developers of Quest 3 Reality Labs recorded an operating loss of <a href="https://www.cnbc.com/2023/06/01/meta-quest-3-unveiled-ahead-of-apples-planned-vr-headset-debut.html">US$3.99 billion</a> in the first quarter of 2023. </p>
<h2>So if there is no demand, who is Apple targeting?</h2>
<p>While Meta’s recent history might seem like a cautionary tale, timing and strategy are critical when it comes to technological innovation. And compared to Meta, Apple’s strategy seems prudent.</p>
<p>Apple is likely betting the app developer community will provide it with the <a href="https://appleinsider.com/articles/23/06/06/even-with-so-many-demonstrated-use-cases-apple-vision-pro-might-not-yet-have-a-purpose">use cases</a> it needs to turn the Vision Pro (and subsequent iterations) into its next big income generator – and perhaps change how we interact with this technology forever.</p>
<p>Getting developers to build exciting complementary offerings, such as apps and device add-ons, would give Apple a springboard to convince users of the Vision Pro’s <a href="https://www.smartinsights.com/manage-digital-transformation/digital-transformation-strategy/digital-marketing-models-technology-acceptance-model/">value</a>. But this won’t work without developers’ buy-in, which leads us to believe the Vision Pro is (at least for now) aimed at Apple’s <a href="https://appleinsider.com/articles/22/06/06/apple-now-has-over-34-million-registered-developers">34 million</a> registered <a href="https://developer.apple.com/wwdc23/">app developers</a>, rather than the broader user market. </p>
<p>It’s expected many of the apps on the App Store will work on <a href="https://kanebridgenews.com/apple-releases-vision-pro-headset-first-major-new-product-in-a-decade/">Vision OS</a>, the Vision Pro’s operating system, by the time the product is launched. Apple is already <a href="https://arstechnica.com/gadgets/2023/06/vision-pro-developer-kits-will-help-devs-get-their-apps-ready-before-launch/">supporting developers</a> with programs and tools to redesign apps for compatibility with the Vision Pro, and create new ones. </p>
<p>Users are attracted to a product that provides more app variety, and their migration to it further piques developers’ interest. Typically, this becomes a self-reinforcing cycle. Such a multiplication of value for consumers, coupled with Apple’s manufacturing capabilities, could allow the Vision Pro to rise to dominance. </p>
<p>And this isn’t just speculation; Apple has used this approach before.</p>
<h2>Leveraging an app-driven ecosystem</h2>
<p>Apple has a history of leveraging its app-driven <a href="https://www.investopedia.com/articles/personal-finance/042815/story-behind-apples-success.asp">ecosystem business model</a> to give its products the upper hand. One early example of this was the iPod and iTunes, wherein the Apple Music store, cloud connectivity and massive storage capacity (at the time) created an environment that locked users in. </p>
<p>More importantly, with the sophistication of the hardware and software, the ease of use and the novelty of the experience, users were happy to be locked in. </p>
<p>This approach has been repeated time and again with other Apple products, such as the <a href="https://theconversation.com/latest-updates-apple-is-trying-to-reclaim-its-major-innovator-status-by-making-you-wash-your-hands-141293">Apple Watch</a>. Once more, Apple drove innovation by linking the hardware to other devices and systems, introducing unique features and providing high-quality apps to generate interest. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/the-iphone-turns-15-a-look-at-the-past-and-future-of-one-of-the-21st-centurys-most-influential-devices-183137">The iPhone turns 15: a look at the past (and future) of one of the 21st century's most influential devices</a>
</strong>
</em>
</p>
<hr>
<h2>Competition heats up</h2>
<p>Ultimately, users will judge the value of the Vision Pro through a combination of objective and subjective information. According to initial reviews, the Vision Pro operates well, and Apple is using branding and marketing tactics to further create a perception of value.</p>
<p>All things considered, Apple’s entry into the mixed reality market represents a big threat to competitors. It has a track record of building hardware at scale and with progressively affordable prices. And let’s not forget its base of some two billion active devices to which the Vision Pro can link. </p>
<p>Apple’s massive ecosystem – built on devices, apps, developers and manufacturing partners – won’t be running dry anytime soon. And by the very fact of its existence, the Vision Pro has a shot at success.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/the-new-iphone-se-is-the-cheapest-yet-smart-move-or-a-premium-tech-brand-losing-its-way-136507">The new iPhone SE is the cheapest yet: smart move, or a premium tech brand losing its way?</a>
</strong>
</em>
</p>
<hr>
<img src="https://counter.theconversation.com/content/207307/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Martie-Louise Verreynne receives funding from the ARC and NHMRC. </span></em></p><p class="fine-print"><em><span>Margarietha de Villiers Scheepers has received funding from State and Local Governments for specific research projects.</span></em></p>Many point to Meta’s failings to make a case for mixed reality headsets having no future. But Apple’s approach is arguably much more strategic.Martie-Louise Verreynne, Professor in Innovation and Associate Dean (Research), The University of QueenslandMargarietha de Villiers Scheepers, Associate professor, University of the Sunshine CoastLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/2071972023-06-08T17:00:56Z2023-06-08T17:00:56ZApple’s new Vision Pro mixed-reality headset could bring the metaverse back to life<figure><img src="https://images.theconversation.com/files/530701/original/file-20230607-16537-he81j2.jpg?ixlib=rb-1.1.0&rect=0%2C8%2C5548%2C3685&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The Apple Vision Pro headset is displayed in a showroom on the Apple campus on June 5, 2023, in Cupertino, Calif.</span> <span class="attribution"><span class="source">(AP Photo/Jeff Chiu)</span></span></figcaption></figure><iframe style="width: 100%; height: 100px; border: none; position: relative; z-index: 1;" allowtransparency="" allow="clipboard-read; clipboard-write" src="https://narrations.ad-auris.com/widget/the-conversation-canada/apple-s-new-vision-pro-mixed-reality-headset-could-bring-the-metaverse-back-to-life" width="100%" height="400"></iframe>
<p>The metaverse — <a href="https://www.newscientist.com/article/2286778-what-is-a-metaverse-and-why-is-everyone-talking-about-it/">a shared online space incorporating 3D graphics where users can interact virtually</a> — has been the subject of increased interest and the ambitious goal of big tech companies for the past few years.</p>
<p><a href="https://theconversation.com/facebook-relaunches-itself-as-meta-in-a-clear-bid-to-dominate-the-metaverse-170543">Facebook’s rebranding to Meta</a> is the clearest example of this interest. However, <a href="https://www.cnbc.com/2023/02/01/meta-lost-13point7-billion-on-reality-labs-in-2022-after-metaverse-pivot.html">despite the billions of dollars that have been invested</a> in the industry, the metaverse has yet to go mainstream. </p>
<p>After the struggles <a href="https://www.nytimes.com/2022/10/09/technology/meta-zuckerberg-metaverse.html">Meta has faced in driving user engagement</a>, many have written off the metaverse as a viable technology for the near future. But the technological landscape is a rapidly evolving one and new advancements can change perceptions and realities quickly. </p>
<p><a href="https://www.nytimes.com/wirecutter/blog/apple-wwdc-2023/">Apple’s recent announcement of the Vision Pro mixed-reality headset</a> at its annual Worldwide Developers Conference — the company’s largest launch since the Apple Watch was released in 2015 — could be the lifeline the metaverse needs.</p>
<h2>About the Vision Pro headset</h2>
<p>The Vision Pro headset is spatial computing device that allows users to interact with apps and other digital content using their hands, eyes and voice, all while maintaining a sense of physical presence. It supports 3D object viewing and spatial video recording and photography. </p>
<p>The Vision Pro is a mixed-reality headset, meaning it combines elements of augmented reality (AR) and virtual reality (VR). While VR creates a completely immersive environment, <a href="https://doi.org/10.1007/0-387-30038-4_10">AR overlays virtual elements onto the real world</a>. Users are able to control how immersed they are while using the Vision Pro.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/TX9qSaGXFyg?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">A video from Apple introducing the Vision Pro headest.</span></figcaption>
</figure>
<p>From a technological standpoint, <a href="https://www.apple.com/apple-vision-pro/?mtid=2092506t42625&aosid=p238&mnid=sKiK0nq00-dc_mtid_2092506t42625_pcrid_661145521361_pgrid_149209007039_pexid__&cid=wwa-ca-kwgo-avalanche-slid----Announce">the Vision Pro uses two kinds of microchips:</a> the M2 chip, which is currently used in Macs, and the new R1 chip. </p>
<p>The new R1 chip processes input from 12 cameras, five sensors and six microphones, which reduces the likelihood of any motion sickness given the absence of input delays.</p>
<p>The Vision Pro display system also features a whopping 23 million pixels, meaning it will be able to deliver an almost real-time view of the world with a lag-free environment.</p>
<h2>Why do people use new tech?</h2>
<p>To gain a better understanding of why Apple’s Vision Pro may throw the metaverse a lifeline, we first need to understand what drives people to accept and use technology. From there, we can make an informed prediction about the future of this new technology.</p>
<p>The first factor that drives the adoption of technology is how easy a piece of technology will be to use, <a href="https://www.jstor.org/stable/2632151">along with the perceived usefulness of the technology</a>. Consumers need to believe technology will add value to their life in order to find it useful.</p>
<p>The second factor that drives the acceptance and use of technology is <a href="https://doi.org/10.3390/jtaer17020036">social circles</a>. People usually look to their family, friends and peers for cues on what is trendy or useful.</p>
<p>The third factor is <a href="https://doi.org/10.1080/13683500.2023.2165483">the level of expected enjoyment of a piece of technology</a>. This is especially important for immersive technologies. Many factors contribute to enjoyment such as system quality, immersion experiences and interactive environment.</p>
<p>The last factor that drives mainstream adoption is <a href="https://doi.org/10.1016/j.jbusres.2019.01.017">affordability</a>. More important, however, is the value derived from new technology — the benefits a user expects to gain, minus costs.</p>
<h2>Can Apple save the metaverse?</h2>
<p>The launch of the Vision Pro seems to indicate Apple has an understanding of the factors that drive the adoption of new technology.</p>
<figure class="align-center ">
<img alt="A white man in a polo shirt poses in front of a displayed VR headset" src="https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/530723/original/file-20230607-21-mjk31r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Apple CEO Tim Cook poses for photos in front of a pair of the company’s new Apple Vision Pro headsets in a showroom on the Apple campus on June 5, 2023, in Cupertino, Calif.</span>
<span class="attribution"><span class="source">(AP Photo/Jeff Chiu)</span></span>
</figcaption>
</figure>
<p>When it comes to ease of use, the Vision Pro offers an intuitive hand-tracking capability that allows users to interact with simple hand gestures and an impressive eye-tracking technology. Users will have the ability to select virtual items just by looking at them.</p>
<p>The Vision Pro also addresses another crucial metaverse challenge: the digital persona. One of the most compelling features of the metaverse is the ability for users to connect virtually with one another, but <a href="https://www.cnn.com/2022/08/25/tech/vr-avatars/index.html">many find it challenging to connect with cartoon-like avatars</a>. </p>
<p>The Vision Pro is attempting to circumvent this issue by allowing users to create hyper-realistic digital personas. Users will be able to scan their faces to create digital versions of themselves for the metaverse.</p>
<p>The seamless integration of the Vision Pro into the rest of <a href="https://medium.com/swlh/the-irresistible-lure-of-the-apple-ecosystem-81bf8d66294a">the Apple ecosystem</a> will also likely to be a selling point for customers.</p>
<p>Lastly, the power of so-called “Apple effect” is another key factor that could contribute to the Vision Pro’s success. Apple has built <a href="https://www.statista.com/statistics/267966/brand-values-of-the-most-valuable-technology-brands-in-the-world/">an extremely loyal customer base over the years</a> by establishing trust and credibility. There’s a good chance customers will be open to trying this new technology because of this.</p>
<h2>Privacy and pricing</h2>
<p>While Apple seems poised to take on the metaverse, there are still some key factors the company needs to consider. </p>
<p>By its very nature, the metaverse <a href="https://www.thedrum.com/opinion/2022/07/21/the-metaverse-and-consumer-data-here-s-what-you-need-know">requires a wealth of personal data collection</a> to function effectively. This is because the metaverse is designed to offer personalized experiences for users. The way those experiences are created is by collecting data.</p>
<p>Users will need assurances from Apple that their personal data and interactions with Vision Pro are secure and protected. <a href="https://www.cnbc.com/2021/06/07/apple-is-turning-privacy-into-a-business-advantage.html">Apple’s past record of prioritizing data security may be an advantage</a>, but there needs to be continuous effort in this area to avoid loss of trust and consumer confidence.</p>
<p>Price-wise, the Vision Pro costs a whopping US$3,499. This will undoubtedly pose a barrier for users and may prevent widespread adoption of the technology. Apple needs to consider strategies to increase the accessibility of this technology to a broader audience.</p>
<p>As we look to the future of this industry, it’s clear the metaverse is anticipated to be fiercely competitive. While Apple brings cutting-edge technology and a loyal customer base, Meta is still one of the original players in this space and its <a href="https://www.meta.com/ca/quest/quest-3/">products are significantly more affordable</a>. In other words, the metaverse is very much alive.</p><img src="https://counter.theconversation.com/content/207197/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Omar H. Fares does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>After struggling to go mainstream, Apple’s recent announcement of the Vision Pro mixed-reality headset could be the lifeline the metaverse needs.Omar H. Fares, Lecturer in the Ted Rogers School of Retail Management, Toronto Metropolitan UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/2071252023-06-06T18:03:00Z2023-06-06T18:03:00ZApple Vision Pro headset: what does it do and will it deliver?<figure><img src="https://images.theconversation.com/files/530291/original/file-20230606-15-6s4g00.jpg?ixlib=rb-1.1.0&rect=17%2C8%2C5964%2C3961&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The headset can blur the lines between virtual reality and augmented reality.</span> <span class="attribution"><a class="source" href="https://epaimages.com/search.pp?flush=1&multikeyword=apple&startdate=&enddate=&autocomplete_City=&metadatafield5=&autocomplete_Country=&metadatafield44=">JOHN G. MABANGLO / EPA IMAGES</a></span></figcaption></figure><p>Apple recently unveiled its <a href="https://www.apple.com/apple-vision-pro/">Vision Pro headset</a> at the Worldwide Developers Conference in California. With it, Apple is venturing into a market of head-mounted devices (HMDs) – which are usually just displays, but in this case is more of a complete computer attached to your head – as well as the worlds of virtual reality (VR), augmented reality (AR) and mixed reality (MR). </p>
<p>The new Apple product will fuel the hopes of many working on these technologies that they will some day be routinely used by the public, just as the iPhone, iPad and Apple Watch helped bring smartphones, tablets and wearable tech into mainstream use.</p>
<p>But what does the Vision Pro actually do, and how much mass appeal will it have?</p>
<p>VR immerses users in an entirely computer-generated world, isolating them to a large degree from their physical surroundings. AR superimposes computer-generated elements onto the real world while the latter remains visible, with the purpose of enhancing the context of our physical surroundings.</p>
<p>A term often used interchangeably with AR is mixed reality, referring to a set of immersive technologies including AR, that provide <a href="https://arxiv.org/abs/1804.08386">different “blends” of physical and virtual worlds</a>. These three technologies are often <a href="https://www.w3.org/immersive-web/">collectively referred to as XR</a>.</p>
<p>The blending of VR and AR seems to be a key part of Apple’s thinking, with the <a href="https://www.apple.com/apple-vision-pro/">Vision Pro</a> allowing users to adjust their level of immersion by deciding how much of the real world they can see. This transitioning between the two experiences will probably be a trend for future HMDs.</p>
<p>The physical world is “seen” through an array of 12 cameras located behind a ski-goggle-like glass fascia, acting as a lens. When the Vision Pro is in VR mode, people approaching you in the real world are automatically detected and displayed as they get close. </p>
<p>A feature called EyeSight also displays the wearer’s eyes through the glass lens when needed, to enable more natural interaction with people around them – a challenge for many HMDs.</p>
<p>In terms of technical specifications, the Vision Pro is impressive. It uses a combination of the M2 microchip and a new chip called the R1. M2 is running <a href="https://developer.apple.com/visionos/">visionOS</a>, which Apple calls its first spatial operating system, along with computer vision algorithms and computer graphics generation. </p>
<p>R1 processes information from the cameras, an array of microphones and a LiDAR scanner – which uses a laser to measure distances to different objects – in order to make the headset aware of its surroundings.</p>
<p>More importantly, the Vision Pro boasts an impressive display system with “more pixels than a 4K TV to each eye”. Its ability to track where the wearer’s eyes are looking allows users to interact with graphical elements just by looking at them. The headset can receive gesture and voice commands and features a form of 360-degree sound called spatial audio. The quoted unplugged operating time is two hours.</p>
<h2>Wearable ‘ecosystem’</h2>
<p>Packed, in typical Apple fashion, in curved aluminum and glass, the headset has an eye-watering price of US$3,499 (£2,819) and represents a collection of many premium features. But Apple has a history of developing products with increasingly versatile capabilities to sense what’s going on in their real-world surroundings.</p>
<figure class="align-center ">
<img alt="Tim Cook (L) and Apple Senior VP of Software Engineering Craig Federighi speak during the conference keynote address." src="https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=377&fit=crop&dpr=1 600w, https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=377&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=377&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=473&fit=crop&dpr=1 754w, https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=473&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/530338/original/file-20230606-15-sfqgzo.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=473&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Tim Cook (L) and Apple Senior VP of Software Engineering Craig Federighi speak during the conference keynote address.</span>
<span class="attribution"><a class="source" href="https://epaimages.com/search.pp?flush=1&multikeyword=apple&startdate=&enddate=&autocomplete_City=&metadatafield5=&autocomplete_Country=&metadatafield44=">JOE MABANGLO / EPA IMAGES</a></span>
</figcaption>
</figure>
<p>Apple also focuses on making its devices interoperable – meaning they work easily with other Apple devices – forming a wearable “ecosystem”. This is what really promises to be disruptive about the Vision Pro. It is also akin to what had been promised and hoped for by pioneers in the idea of <a href="https://www.media.mit.edu/wearables/">wearable computing back in the 1990s</a>.</p>
<p>Combining the headset with the iPhone, which still forms the backbone of Apple’s ecosystem, and the Apple Watch could help create new uses for augmented reality. Likewise, linking the headset to many programming tools demonstrates the company’s desire to tap into an existing community of developers of augmented reality applications.</p>
<p>Many questions remain, however. For example, will it be able to access mixed reality applications via a web browser? What will it be like to use from an ergonomic point of view?</p>
<p>It’s also unclear when the Vision Pro be available outside the US or whether there will be a non-Pro version – as the “Pro” part of the title implies a more “expert”, or developer market.</p>
<p>The Vision Pro is a gamble, as XR is often seen as something that promises but rarely delivers. Yet, companies such as Apple and those that are probably its primary competitors in the XR domain, Meta and Microsoft, have the clout to make XR popular for the general public.</p>
<p>More importantly, devices such as the Vision Pro and its ecosystem, as well as its competitors could provide the foundation for developing <a href="https://theconversation.com/metaverse-five-things-to-know-and-what-it-could-mean-for-you-171061">the metaverse</a>. This is an immersive world, facilitated by headsets, that aims for social interaction that’s more natural than with previous products.</p>
<p>Sceptics will say that Vision Pro and EyeSight make you appear like a scuba diver in your living room. But this could finally be the time to dive into the deep waters of XR.</p><img src="https://counter.theconversation.com/content/207125/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Panagiotis Ritsos receives funding from the DSP Centre, Bangor University, which has been partly funded by the European Regional Development Fund through Welsh Government and also by the North Wales Growth Deal through Ambition North Wales, Welsh Government and UK Government.
</span></em></p><p class="fine-print"><em><span>Dr. Peter W. S. Butcher receives funding from the DSP Centre, Bangor University, which has been partly funded by the European Regional Development Fund through Welsh Government and also by the North Wales Growth Deal through Ambition North Wales, Welsh Government and UK Government.</span></em></p>Will Apple’s Vision Pro set the new standard for the future of virtual reality?Panagiotis Ritsos, Senior Lecturer in Visualisation, Bangor UniversityPeter Butcher, Lecturer in Human Computer Interaction, Bangor UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1700022021-10-20T12:42:10Z2021-10-20T12:42:10ZCan Facebook’s smart glasses be smart about security and privacy?<figure><img src="https://images.theconversation.com/files/427288/original/file-20211019-13-1810ezy.jpg?ixlib=rb-1.1.0&rect=0%2C5%2C3796%2C4650&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Is he looking at you or looking at personal information about you?</span> <span class="attribution"><a class="source" href="https://www.gettyimages.com/detail/illustration/man-in-sunglasses-royalty-free-illustration/97231965">CSA Images via Getty Images</a></span></figcaption></figure><p>Facebook’s smart glasses ambitions are in the news again. The company has launched a worldwide project dubbed <a href="https://ai.facebook.com/research/publications/ego4d-unscripted-first-person-video-from-around-the-world-and-a-benchmark-suite-for-egocentric-perception">Ego4D</a> to research new uses for smart glasses. </p>
<p>In September, Facebook unveiled its <a href="https://about.fb.com/news/2021/09/introducing-ray-ban-stories-smart-glasses/">Ray-Ban Stories glasses</a>, which have two cameras and three microphones built in. The glasses capture audio and video so wearers can record their experiences and interactions. </p>
<p>The research project aims to add augmented reality features to smart glasses using artificial intelligence technologies that could provide wearers with a wealth of information, including the ability to get answers to questions like “Where did I leave my keys?” <a href="https://ai.facebook.com/blog/teaching-ai-to-perceive-the-world-through-your-eyes">Facebook’s vision</a> also includes a future where the glasses can “know who’s saying what when and who’s paying attention to whom.”</p>
<p>Several other technology companies like <a href="https://www.google.com/glass/start/">Google</a>, <a href="https://www.microsoft.com/en-us/hololens">Microsoft</a>, <a href="https://www.spectacles.com">Snap</a>, <a href="https://www.vuzix.com/products/blade-smart-glasses-upgraded">Vuzix</a> and <a href="https://www.lenovo.com/us/en/thinkrealitya3/">Lenovo</a> have also been experimenting with versions of augmented or mixed reality glasses. Augmented reality glasses can display useful information within the lenses, providing an electronically enhanced view of the world. For example, smart glasses could draw a line over the road to show you the next turn or let you see a restaurant’s Yelp rating as you look at its sign.</p>
<p>However, some of the information that augmented reality glasses give their users <a href="https://www.theverge.com/2021/10/14/22725894/facebook-augmented-reality-ar-glasses-ai-systems-ego4d-research">could include identifying people in the glasses’ field of view</a> and displaying personal information about them. It was not too long ago that Google introduced Google Glass, only to face a <a href="https://www.newyorker.com/business/currency/whats-the-problem-with-google-glass">public backlash</a> for simply recording people. Compared to being recorded by smartphones in public, being recorded by smart glasses <a href="https://doi.org/10.1145/2556288.2557352">feels to people like a greater invasion of privacy</a>.</p>
<p>As a researcher who <a href="https://scholar.google.com/citations?user=r42-hFIAAAAJ&hl=en">studies computer security and privacy</a>, I believe it’s important for technology companies to proceed with caution and consider the security and privacy risks of augmented reality.</p>
<h2>Smartphones vs. smart glasses</h2>
<p>Even though people are now used to being photographed in public, they also expect the photographer typically to raise their smartphone to compose a photo. Augmented reality glasses fundamentally disrupt or violate this sense of normalcy. The public setting may be the same, but the sheer scale and approach of recording has changed. </p>
<p>Such deviations from the norm have long been <a href="https://digitalcommons.law.uw.edu/wlr/vol79/iss1/10">recognized by researchers</a> as a violation of privacy. My group’s research has found that people in the neighborhood of nontraditional cameras want a more <a href="https://doi.org/10.1145/3415187">tangible sense of when their privacy is being compromised</a> because they find it difficult to know whether they are being recorded. </p>
<p>Absent the typical physical gestures of taking a photo, people need better ways to convey whether a camera or microphone is recording people. Facebook has already been <a href="https://techcrunch.com/2021/09/20/facebook-warned-over-very-small-indicator-led-on-smart-glasses-as-eu-dpas-flag-privacy-concerns/">warned by the European Union</a> that the LED indicating a pair of Ray-Ban Stories is recording is too small.</p>
<p>In the longer term, however, people might become accustomed to smart glasses as the new normal. Our research found that although young adults worry about others recording their embarrassing moments on smartphones, <a href="https://www.usenix.org/biblio-1979">they have adjusted</a> to the pervasive presence of cameras. </p>
<h2>Smart glasses as a memory aid</h2>
<p>An important application of smart glasses is as a memory aid. If you could record or “lifelog” your entire day from a first-person point of view, you could simply rewind or scroll through the video at will. You could examine the video to see where you left your keys, or you could replay a conversion to recall a friend’s movie recommendation.</p>
<p>Our research studied volunteers who wore lifelogging cameras for several days. We uncovered several privacy concerns – this time, <a href="http://dx.doi.org/10.1145/2702123.2702183">for the camera wearer</a>. Considering who, or what algorithms, might have access to the camera footage, people may worry about the detailed portrait it paints of them. </p>
<p>Who you meet, what you eat, what you watch and what your living room really looks like without guests are all recorded. We found that people were <a href="http://dx.doi.org/10.1145/2632048.2632079">especially concerned about the places being recorded</a>, as well as their computer and phone screens, which formed a large fraction of their lifelogging history.</p>
<p>Popular media already has its take on what can go horribly wrong with such memory aids. “<a href="https://www.youtube.com/watch?v=RRNYmFrfbCg">The Entire History of You</a>” episode of the TV series “Black Mirror” shows how even the most casual arguments can lead to people digging through lifelogs for evidence of who said exactly what and when. In such a world, it is difficult to just move on. It’s a lesson in the importance of forgetting.</p>
<p>Psychologists have pointed to the <a href="https://www.psychologytoday.com/us/blog/the-memory-factory/202108/memory-depiction-in-the-black-mirror-episode-the-entire-history-you">importance of forgetting</a> as a natural human coping mechanism to move past traumatic experiences. Maybe AI algorithms can be put to good use identifying digital memories to delete. For example, our research has devised AI-based algorithms to detect <a href="https://www.ndss-symposium.org/ndss2014/programme/placeavoider-steering-first-person-cameras-away-sensitive-spaces/">sensitive places</a> like bathrooms and <a href="http://dx.doi.org/10.1145/2858036.2858417">computer and phone screens</a>, which were high on the worry list in our <a href="https://www.ndss-symposium.org/ndss2014/programme/placeavoider-steering-first-person-cameras-away-sensitive-spaces/">lifelogging study</a>. Once detected, footage can be selectively deleted from a person’s digital memories.</p>
<h2>X-ray specs of the digital self?</h2>
<p>However, smart glasses have the potential to do more than simply record video. It’s important to prepare for the possibility of a world in which smart glasses use facial recognition, analyze people’s expressions, look up and display personal information, and even record and analyze conversations. These applications raise important questions about privacy and security.</p>
<p>We studied the use of smart glasses by people with visual impairments. We found that these potential users were worried about the <a href="https://doi.org/10.1145/3373625.3417003">inaccuracy of artificial intelligence algorithms</a> and their potential to misrepresent other people. </p>
<p>Even if accurate, they felt it was improper to infer someone’s weight or age. They also questioned whether it was ethical for such algorithms to guess someone’s gender or race. Researchers have also debated <a href="https://dl.acm.org/doi/10.1145/3442188.3445939">whether AI should be used to detect emotions</a>, which can be expressed differently by people from different cultures. </p>
<h2>Augmenting Facebook’s view of the future</h2>
<p>I have <a href="https://vr4sec.hcigroup.de/proceedings/VR4Sec_paper_7.pdf">only scratched</a> <a href="https://vr4sec.hcigroup.de/proceedings/VR4Sec_paper_2.pdf">the surface</a> of the privacy and security considerations for augmented reality glasses. As Facebook charges ahead with augmented reality, I believe it’s critical that the company address these concerns. </p>
<p>[<em>Over 115,000 readers rely on The Conversation’s newsletter to understand the world.</em> <a href="https://theconversation.com/us/newsletters/the-daily-3?utm_source=TCUS&utm_medium=inline-link&utm_campaign=newsletter-text&utm_content=100Ksignup">Sign up today</a>.]</p>
<p>I am heartened by the <a href="https://research.fb.com/blog/2020/09/announcing-the-winners-of-the-explorations-of-trust-in-ar-vr-and-smart-devices-request-for-proposals/">stellar list of privacy and security researchers</a> Facebook is collaborating with to make sure its technology is worthy of the public’s trust, especially given the company’s <a href="https://theconversation.com/facebooks-scandals-and-outage-test-users-frenemy-relationship-169244">recent track record</a>.</p>
<p>But I can only hope that Facebook will tread carefully and ensure that their view of the future includes the concerns of these and other privacy and security researchers.</p>
<p><em>This article has been updated to clarify that future Facebook augmented reality glasses will not necessarily be in the Ray-Ban Stories product line and that, while the company’s goals include identifying people, the Ego4D research data was not collected using facial recognition technology.</em></p><img src="https://counter.theconversation.com/content/170002/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Apu Kapadia receives funding from NSF and DoD. He has also received two Google Faculty Research Awards and one Microsoft Research grant. Kapadia is a shareholder in Snap, Inc.</span></em></p>Smart glasses like Facebook’s Ray-Ban Stories could be used to record you surreptitiously. If the company adds facial recognition, you could be even more exposed.Apu Kapadia, Professor of Computer Science, Indiana UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1583082021-04-07T03:47:26Z2021-04-07T03:47:26Z‘Potential for harm’: Microsoft to make US$22 billion worth of augmented reality headsets for US Army<figure><img src="https://images.theconversation.com/files/393693/original/file-20210407-19-d0anu6.jpg?ixlib=rb-1.1.0&rect=40%2C11%2C1876%2C905&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><span class="source">Microsoft</span></span></figcaption></figure><p>Microsoft has <a href="https://www.cnbc.com/2021/03/31/microsoft-wins-contract-to-make-modified-hololens-for-us-army.html">reportedly</a> been awarded a ten year contract worth close to US$22 billion, to provide 120,000 military-grade augmented reality (AR) headsets to the US Army.</p>
<p>Popularised through mobile apps such as <a href="https://theconversation.com/pokemon-go-wants-to-make-3d-scans-of-the-whole-world-for-planet-scale-augmented-reality-experiences-is-that-good-140636">Pokémon Go</a> and <a href="https://www.snapchat.com/l/en-gb/create/">face filters on social media</a>, AR technology is fundamentally about superimposing digital images over real-world environments.</p>
<p>The AR interface commissioned by the US military is referred to as an “Integrated Visual Augmentation System” (IVAS). It will use Microsoft’s HoloLens headset technology as its base hardware.</p>
<p>As the Army’s <a href="https://www.peosoldier.army.mil/News/Article-Display/Article/2556870/ivas-production-contract-award/">press release</a> notes, the device will be used to coordinate soldiers and implement sensing technologies on the battlefield. </p>
<p>Features will supposedly include thermal sensors, machine learning (to create realistic training simulations) and a digital heads up display to enhance soldiers’ “situational awareness”.</p>
<p>The news follows Microsoft’s previous announcement of a <a href="https://www.theverge.com/2018/11/28/18116939/microsoft-army-hololens-480-million-contract-magic-leap">US$480 million military contract</a> to develop and supply IVAS prototypes to the army in 2018. </p>
<p>Between this older contract, the new one, and Microsoft’s <a href="https://arstechnica.com/tech-policy/2020/04/dod-internal-investigation-finds-dod-handled-jedi-contract-properly/">$10b JEDI cloud computing</a> contract for Azure, Microsoft is set to fortify its position as one of the highest value US defence contractors (alongside <a href="https://www.technologyreview.com/2019/10/08/75349/meet-americas-newest-military-giant-amazon/">Amazon</a>).</p>
<h2>AR and warfare’s relationship isn’t new</h2>
<p>AR interfaces first emerged in the 1960s with Ivan Sutherland’s demonstration of the Sword of Damocles <a href="https://www.researchgate.net/figure/The-worlds-first-head-mounted-display-with-the-Sword-of-Damocles-Sutherland-1968_fig2_292150312">tracking system</a>. This was developed in the Massachusetts Institute of Technology’s <a href="https://www.ll.mit.edu/">Lincoln Laboratory</a>, funded by the US Department of Defence. </p>
<p>The head-mounted display – the first of its kind – was pitched for use in simulating flight instruments and conditions.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=439&fit=crop&dpr=1 600w, https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=439&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=439&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=551&fit=crop&dpr=1 754w, https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=551&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/393218/original/file-20210401-23-ecvlui.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=551&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">American computer scientists Ivan Sutherland’s Sword of Damocles was the first example of a head-mounted display (HMD).</span>
</figcaption>
</figure>
<p>Fast forward to the mid-80s and the Lincoln Lab produced American AR manufacturer Kopin. In 1990 this commercial company received <a href="https://www.kopin.com/innovation">a US$50 million contract</a> from the Department of Defence to develop micro LCDs to be used in wearable computers for the infantry. </p>
<p>AR continues to see uptake today, in a trend which geographer <a href="https://www.versobooks.com/books/365-cities-under-siege">Stephen Graham</a> refers to as the “militarisation” of everyday life. And this is especially noticeable with technologies governing urban societies. </p>
<p>Technology firm Vuzix is one major player in the security and enterprise sector. As its <a href="https://ir.vuzix.com/sec-filings/annual-reports/xbrl_doc_only/13739">annual report</a> states, the company develops products for “governmental entity customers that primarily provide security and defence services, including police, fire fighters, EMTs, other first responders, and homeland and border security”.</p>
<p>In one particularly troubling development, last year it was <a href="https://www.buzzfeednews.com/article/carolinehaskins1/clearview-facial-recognition-insight-camera-glasses">reported</a> that <a href="https://clearview.ai/">ClearView AI’s</a> facial recognition software was being tested to run on Vuzix hardware. </p>
<p>This controversial company trains its artificial intelligence software on a dataset of more than <a href="https://www.buzzfeednews.com/article/ryanmac/clearview-ai-fbi-ice-global-law-enforcement">three billion images</a> from websites including YouTube, Facebook and Instagram. Its marriage with Vuzix points to a future where law enforcement officers use wearable AR with built-in facial recognition capbilites. </p>
<p>The use of AR to govern everyday life has also emerged during the COVID-19 pandemic (which we’ve detailed in this <a href="https://ses.library.usyd.edu.au/bitstream/handle/2123/22485/ETHICAL%20IMPLICATIONS.pdf?sequence=2&isAllowed=y">report</a>). For example, companies such as <a href="https://techcrunch.com/2020/04/16/chinese-startup-rokid-pitches-covid-19-detection-glasses-in-u-s/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAADGreB-OrCZgjgGiQXrLxceifR0qKQyXJ1l7hJv5tWYg6S_Xxqpw0iwv5LTzSCHqxcuhwDwXtZkk-t-mry85jc747U9V-CB8GLSGt2-ki6j0R5SegyozZ_rrkKjfbV447KmZXhCb3MNZrZL41C93WCZ4YJdcykaJ9ED_kMP-9Bxf">Rokid</a> and <a href="https://kcwearable.com/">KC Wearables</a> have developed wearable devices that purport to track the body temperatures of people in the view of the device’s wearer.</p>
<h2>What’s so bad about Microsoft’s AR?</h2>
<p>We know little about the current prototyping and field testing of Microsoft’s IVAS. But the interface raises the same concerns that surround other technologies designed to preemptively target and classify potential “threats” – such as drones.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=349&fit=crop&dpr=1 600w, https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=349&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=349&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=439&fit=crop&dpr=1 754w, https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=439&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/393443/original/file-20210405-15-1k0t2fh.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=439&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">This is what the current IVAS prototype looks like.</span>
<span class="attribution"><span class="source">Microsoft</span></span>
</figcaption>
</figure>
<p>As the US Army’s <a href="https://www.scribd.com/document/405194307/US-Army-Augmented-Reality-IVAS-Statement-of-Objectives">IVAS objectives</a> read, key outcomes are to increase mobility, situational awareness and lethality – that is, deadliness on the battlefield. The document states:</p>
<blockquote>
<p>Soldier lethality will be vastly improved through cognitive training and advanced sensors, enabling squads to be the first to detect, decide and engage. Accelerated development of these capabilities is necessary to recover and maintain overmatch.</p>
</blockquote>
<p>The objectives are framed around soldiers’ efficiency, coordination and safety. This is similar to the usual framing of other predictive sensing technologies, including facial recognition. </p>
<p>The reality, however, is there would be potentially disastrous outcomes if such a system were to misidentify a target. In a 2019 <a href="https://edition.cnn.com/2019/02/25/tech/augmented-reality-microsoft-us-military/index.html">CNN interview</a>, Microsoft CEO Satya Nadella tried to play down concerns. </p>
<p>“We made a principled decision that we’re not going to withhold technology from institutions that we have elected in democracies to protect the freedoms we enjoy,” he said. His statement failed to acknowledge the potential risk which may result from the military’s use of AR. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/what-you-see-is-not-always-what-you-get-how-virtual-reality-can-manipulate-our-minds-63652">What you see is not always what you get: how virtual reality can manipulate our minds</a>
</strong>
</em>
</p>
<hr>
<h2>Not child’s play</h2>
<p>Following Microsoft’s initial IVAS contract in 2018, Microsoft employees wrote a <a href="https://www.theverge.com/2019/2/22/18236116/microsoft-hololens-army-contract-workers-letter">letter</a> to company executives opposing the use of AR for warfare.</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1099066343523930112"}"></div></p>
<p>While the letter itself was ineffectual, the recent rise of collective <a href="https://logicmag.io/the-making-of-the-tech-worker-movement/full-text/">worker resistance</a> in Silicon Valley shows promise. More strikes and walkouts in response to unethical developments could help push back against big tech’s self-serving visions of the future.</p>
<p>Similar to virtual reality, AR has so far enjoyed cover from critique by being taken as a benign gaming or entertainment technology. </p>
<p>The latest IVAS contract is an urgent reminder that developments in this technology should be taken seriously. And its potential for harm must not be downplayed.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/facebooks-virtual-reality-push-is-about-data-not-gaming-145730">Facebook's virtual reality push is about data, not gaming</a>
</strong>
</em>
</p>
<hr>
<img src="https://counter.theconversation.com/content/158308/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Augmented reality has largely enjoyed cover from critique by being taken as a benign gaming technology. But recent developments suggest we need to get serious about it.Ben Egliston, Postdoctoral research fellow, Digital Media Research Centre, Queensland University of TechnologyMarcus Carter, Senior Lecturer in Digital Cultures, SOAR Fellow., University of SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1532562021-01-20T16:44:39Z2021-01-20T16:44:39ZHolographic history is making ‘Night at the Museum’ a reality<figure><img src="https://images.theconversation.com/files/379756/original/file-20210120-15-lillbv.jpg?ixlib=rb-1.1.0&rect=0%2C0%2C5138%2C3428&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/london-united-kingdom-circa-august-2017-696072751">elRoce/Shutterstock</a></span></figcaption></figure><p>For millions of children, being dragged to a museum is a singularly painful experience, marked by time standing still rather than history coming to life – as it does in the film “Night at the Museum”, starring Ben Stiller. </p>
<p>But that all could change with the development of new “mixed reality” (MR) technology, which could inject new forms of interactive storytelling into our museums – introducing holographic tour guides and immersive digital displays in place of finger-smudged glass cabinets and faded information boards.</p>
<p>Unlike the total immersion of virtual reality (VR), or the computer screen required for augmented reality (AR), MR uses a head-mounted glass display, similar to the Google Glass spectacles, which enables the user to see their real-world surroundings while virtual features are overlayed on top, creating a sense of mixed perception.</p>
<p>To manage this feat, MR devices are equipped with sensors which are crucial for tracking the user’s movement. MR devices are also equipped with cameras, which detect cues from the environment to assist in the seamless superimposition of virtual features onto the physical world. All of this is achieved through the projection of 3D graphics onto the glass display – a process that creates a hologram.</p>
<p>As a result, visitors at future MR-enabled museums might find themselves hunting for lost pirate gold, evading booby-traps, or wading through crocodile-infested waters in search of informative clues. We tried to incorporate some of these thrilling features into <a href="https://link.springer.com/article/10.1007/s10055-020-00497-9">our own experimental MR study</a> in Cairo’s Egyptian Museum – which saw guests dodging warring charioteers, and dusting for ancient relics.</p>
<p>Our MR museum experiment in Cairo used <a href="https://www.microsoft.com/en-us/hololens">Microsoft HoloLens</a> hardware, and new software called “MuseumEye”. Ancient Egypt was a wonderful period of human history to bring back to life with the help of technology – though our visuals were more akin to “The Mummy” than “Night at the Museum”.</p>
<h2>The mummy returns</h2>
<p>In <a href="https://link.springer.com/article/10.1007/s11042-019-08026-w">our MR museum experience</a>, holograms of King Tutankhamun and his consort Queen Ankhesenamun lead visitors around the museum, discussing how they used to live and demonstrating how they would use some of the 120,000 tools and artefacts on display. </p>
<p>Making these ancient artefacts more accessible to visitors required us to 3D scan many of the exhibits, so that they could be rendered digitally as manipulable 3D objects. Scanned antiques were then fed into a programme called “Unity” which helped us map hand movements on to tasks – a pinch, for instance, was coded as a command to make an artefact smaller. The MR exhibition’s moving characters were modelled in the programmes “zBrush” and “Autodesk Maya”. As a result of this exhaustive coding work, visitors could “pull items off the shelf” and examine them in detail, using hand gestures sensed by the HoloLens.</p>
<p>Elsewhere, we animated reenactments of how the ancient Egyptian empire once went to war, with soldiers and chariots racing around exhibition rooms. We also offered guests the opportunity to become an archaeologist for the day, unlocking treasures hidden around the museum. By scoring points on discoveries made, we turned the museum learning process into a game – something that’s been shown to <a href="https://link.springer.com/chapter/10.1007/978-3-319-94619-1_29">improve educational outcomes</a> from museum visits.</p>
<p><a href="https://dl.acm.org/doi/10.1145/3359590">Some 171 museum patrons evaluated the MuseumEye application</a>, with over 80% finding the MR experience a significant improvement in interactivity, entertainment, and educational value. And there are other benefits: <a href="https://www.inderscience.com/offer.php?id=104600">in a separate study</a>, we found MR guides more cost effective than human tour guides, with the extra ability to provide real-time information on guest behaviour and section overcrowding.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/g2Yz8kr1v4I?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Guests at the Egyptian Museum in Cairo sampling an MR tour.</span></figcaption>
</figure>
<h2>Holographic history</h2>
<p>A number of other museums have begun experimenting with MR. The National Holocaust Centre and Museum used the HoloLens to powerful effect in their “<a href="https://www.springerprofessional.de/en/immersive-storytelling-in-augmented-reality-witnessing-the-kinde/18527040">Witnessing the Kindertransport</a>” exhibition. Over in New York, the Intrepid, Air & Sea Museum hosted an MR installation called “<a href="https://www.microsoft.com/inculture/arts/smithsonian-museum-day-women-making-history-dr-mae-jemison/">Defying Gravity: Women in Space</a>”. And in Washington DC, the National Geographic Museum hosted “<a href="https://falconscreativegroup.com/selected-work/becoming-jane-the-evolution-of-dr-jane-goodall">Becoming Jane</a>” as a way to immerse visitors in the life and work of chimpanzee researcher Dr. Jane Goodall.</p>
<p>Beyond museums, MR is being put to use with the <a href="https://holo.dentist/">HoloDentist </a> project, which helps trainee dental surgeons safely operate on virtual patients. The “<a href="https://www.island-labs.com/3d-planning-rooms/">.rooms</a>” project, meanwhile, uses MR to assist interior designers, allowing people to walk around and change the furniture and features in their home. </p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/kkIVhkHsV6I?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">The MR exhibition ‘Defying Gravity: Women in Space’ at New York’s Intrepid, Air & Sea Museum.</span></figcaption>
</figure>
<h2>Back to reality</h2>
<p>As MR technology remains in its infancy, the devices currently available to developers are both limited and expensive – which is why the Egyptian Museum is not currently deploying the MR our team developed. One Microsoft HoloLens costs around US$3,500 (£2,570), and large museums may require as many as 300 headsets to run effective MR tours.</p>
<p>However, this technology does appear set for rapid growth. After Facebook’s Oculus Rift VR headsets failed to capture the public’s imagination, the company recently announced the release of their own brand of <a href="https://www.theverge.com/2020/9/16/21439929/facebook-ar-smart-glasses-ray-ban-announcement">MR smart glasses</a>, scheduled for later this year. In 2020, Nreal launched the <a href="https://www.theverge.com/2020/8/10/21362407/nreal-light-ar-glasses-lg-uplus-samsung-galaxy-note-retail-launch-availability-price">U+ Real Glass</a> MR headset in Korea as a cost-effective device for everyday use, with plans to expand sales into Japan and Europe over the coming years. </p>
<p>Our MuseumEye software has shown how MR technology can bring history to life, delivering benefits to both museums and their guests. With the release of new, reasonably-priced MR hardware, and the further development of hologram software, we expect museums to increase their engagement with MR in the coming years – making ‘Night at the Museum’ into an everyday reality.</p><img src="https://counter.theconversation.com/content/153256/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Ramy Hammady receives funding from Newton Mosharafa Fund.
1st Affiliation: Solent University & 2nd Affliation: Helwan University</span></em></p><p class="fine-print"><em><span>Carl Strathearn is affiliated with Edinburgh Napier University</span></em></p>Emerging ‘mixed reality’ technology promises to bring history back to life.Ramy Hammady, Lecturer in Computer Games, Solent UniversityCarl Strathearn, Research Fellow, Computing, Edinburgh Napier UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1457302020-09-18T06:13:45Z2020-09-18T06:13:45ZFacebook’s virtual reality push is about data, not gaming<figure><img src="https://images.theconversation.com/files/358785/original/file-20200918-24-14hty4m.jpg?ixlib=rb-1.1.0&rect=14%2C0%2C1583%2C1061&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><span class="source">Facebook</span></span></figcaption></figure><p>Facebook has <a href="https://www.facebookconnect.com/en">announced</a> the latest version of its successful standalone virtual reality (VR) headset, the <a href="https://www.theverge.com/21437674/oculus-quest-2-review-features-photos">Oculus Quest 2</a>. The new device packs more computing power and a sharper screen than its predecessor, and is also US$100 cheaper.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/ATVGl9wOJsM?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Facebook’s Oculus Quest 2 (from AUD$479) is a powerful wireless VR headset for gaming and, Facebook hopes, much more.</span></figcaption>
</figure>
<p>The Oculus Quest 2 is the latest step in Facebook’s long-term strategy of making VR more accessible and popular. Facebook recently brought all its VR work under the umbrella of <a href="https://about.fb.com/realitylabs/">Facebook Reality Labs</a>, it has announced new applications like the <a href="https://techcrunch.com/2020/09/16/facebook-debuts-infinite-office-a-virtual-reality-office-space/">Infinite Office VR workplace</a>, and will also <a href="https://www.theverge.com/2020/8/19/21375118/oculus-facebook-account-login-data-privacy-controversy-developers-competition">require a Facebook login</a> for future Oculus devices.</p>
<p>The compulsory link to Facebook has many consumers concerned, considering the social media giant’s <a href="https://theconversation.com/why-the-business-model-of-social-media-giants-like-facebook-is-incompatible-with-human-rights-94016">chequered history with privacy and data</a>. VR and its cousin, augmented reality (AR), are perhaps the most data-extractive digital sensors we’re likely to invite into our homes in the next decade. </p>
<h2>Why does Facebook make virtual reality headsets?</h2>
<p>Facebook acquired VR company Oculus in 2014 for <a href="https://www.theguardian.com/technology/2014/jul/22/facebook-oculus-rift-acquisition-virtual-reality">an estimated US$2.3 billion</a>. But where Oculus originally aimed at gamers, Facebook boss Mark Zuckerberg wants VR for social media. </p>
<p>At the same event last year, Zuckerberg said Facebook sees VR as a pathway to a new kind of “social computing platform” using the enhanced feeling of “presence” that VR affords. For Facebook, the introduction of VR-based computing will be like the leap from text-based command line interfaces to the graphical user interfaces we use today. </p>
<p>This may well be right. VR affords a strong feeling of embodied presence that offers new possibilities for entertainment, training, learning and connecting with others at a distance.</p>
<p>But if the VR future is the one Facebook is “<a href="https://www.facebook.com/TechAtFacebook/videos/807207240014945/">working in the lab</a>” on, it will function via the company’s existing social computing platform and business model of extracting data to deliver targeted advertisements. </p>
<h2>Virtual reality collects real data</h2>
<p>A VR headset collects data about the user, but also about the outside world. This is one of the key <a href="https://ses.library.usyd.edu.au/bitstream/handle/2123/22485/ETHICAL%20IMPLICATIONS.pdf?sequence=2&isAllowed=y">ethical issues</a> of emerging “mixed reality” technologies.</p>
<p>As American VR researcher <a href="https://vhil.stanford.edu/pubs/2018/protecting-nonverbal-data-tracked-in-virtual-reality/">Jeremy Bailenson has written</a>: </p>
<blockquote>
<p>…commercial VR systems typically track body movements 90 times per second to display the scene appropriately, and high-end systems record 18 types of movements across the head and hands. Consequently, spending 20 minutes in a VR simulation leaves just under 2 million unique recordings of body language. </p>
</blockquote>
<p>The way you move your body can be <a href="https://dl.acm.org/doi/10.1145/3290605.3300340">used to identify you</a>, like a fingerprint, so everything you do in VR could be traced back to your individual identity. </p>
<p>Facebook’s Oculus Quest headsets also use outward-facing cameras to track and map their surroundings.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/nrj3JE-NHMw?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Facebook VR headsets use Simultaneous Localisation and Mapping (SLAM) to track the movements of the headset in 3D space. This is also another opportunity for collecting data about the world.</span></figcaption>
</figure>
<p>In late 2019 <a href="https://www.roadtovr.com/oculus-quest-camera-privacy-rift-s-facebook/">Facebook said</a> they “don’t collect and store images or 3D maps of your environment on our servers today”. Note the word <em>today</em>, which tech journalist <a href="https://www.roadtovr.com/oculus-quest-camera-privacy-rift-s-facebook/">Ben Lang notes</a> makes clear the company is not ruling out anything in the future. </p>
<h2>Virtual reality leads to augmented reality</h2>
<p>Facebook wants to collect this data to facilitate its plans for augmented reality (AR).</p>
<p>Where VR takes a user to a fully virtual environment, AR combines virtual elements with our real surroundings. </p>
<p>Last year Facebook unveiled the Live Maps application, a vision of an expansive surveillance apparatus presumably powered by AR glasses and data collected from Oculus Insight. Live Maps will provide many minor conveniences for Facebook users, like letting you know you’ve left your keys on the coffee table. </p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/JTa8zn0RNVM?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Facebook’s Live Maps application is a vision of a Facebook owned AR platform.</span></figcaption>
</figure>
<p>Now Facebook have announced their first steps towards making this a reality: <a href="https://about.fb.com/realitylabs/projectaria/">Project Aria</a>. This will involve people wearing glasses-like sensors around Seattle and the San Francisco Bay area, to collect the data to build what Wired co-founder Kevin Kelly calls <a href="https://www.wired.com/story/mirrorworld-ar-next-big-tech-platform/">“the mirrorworld</a>”, the next big tech platform.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/pokemon-go-wants-to-make-3d-scans-of-the-whole-world-for-planet-scale-augmented-reality-experiences-is-that-good-140636">Pokémon Go wants to make 3D scans of the whole world for 'planet-scale augmented reality experiences'. Is that good?</a>
</strong>
</em>
</p>
<hr>
<p>People are rightly concerned about the ethical implications of this kind of data extraction. Alongside Project Aria, Facebook launched its <a href="https://about.fb.com/realitylabs/responsible-innovation-principles/">Responsible Innovation Principles</a> page, and they’re already quick to emphasise that faces and license plates will be blurred in this data collection. </p>
<p>As we have <a href="http://marcuscarter.com/wp-content/uploads/2020/09/Oculus-Imaginaries-New-Media-and-Society-Egliston-and-Carter-Final-Accepted-Version.pdf">argued elsewhere</a>, framing questions about VR and AR surveillance in terms of individual privacy suits companies like Facebook very well. That’s because their previous failings are actually in the (un)ethical use of data (as in the case of <a href="https://www.wired.com/amp-stories/cambridge-analytica-explainer/">Cambridge Analytica</a>) and their <a href="https://www.tandfonline.com/doi/full/10.1080/24701475.2019.1593667">asymmetric platform power</a>.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/why-the-business-model-of-social-media-giants-like-facebook-is-incompatible-with-human-rights-94016">Why the business model of social media giants like Facebook is incompatible with human rights</a>
</strong>
</em>
</p>
<hr>
<h2>We need more than just ‘tech ethics’</h2>
<p>Groups like the <a href="https://xrsi.org/who-we-are">XR Safety Initiative</a> recognise these emerging issues, and are beginning work on <a href="https://xrsi.org/publication/the-xrsi-privacy-framework">standards, guidelines and privacy frameworks</a> to shape VR and AR development.</p>
<p>Many emerging technologies encounter what is known as <a href="https://en.wikipedia.org/wiki/Collingridge_dilemma">the Collingridge problem</a>: it is hard to predict the various impacts of a technology until it is extensively developed and widely used, but by then it is almost impossible to control or change.</p>
<p>We see this playing out right now, in efforts to <a href="https://theconversation.com/facebook-and-google-used-to-be-the-future-of-news-but-now-media-companies-need-more-strings-to-their-bow-145024">regulate Google and Facebook’s power over news media</a>.</p>
<p>As David Watts argues, big tech designs its own rules of ethics to <a href="https://theconversation.com/how-big-tech-designs-its-own-rules-of-ethics-to-avoid-scrutiny-and-accountability-113457">avoid scrutiny and accountability</a>: </p>
<blockquote>
<p>Feelgood, high-level data ethics principles are not fit for the purpose of regulating big tech … The harms linked to big tech can only be addressed by proper regulation.</p>
</blockquote>
<p>What might regulation of Facebook’s VR look like? Germany offers one such response – their antitrust regulations have resulted in Facebook <a href="https://www.competitionpolicyinternational.com/oculus-to-pause-german-sales-over-antitrust-issue/">withdrawing the headset from sale</a>. We can only hope the technology doesn’t become too entrenched to be changed, or challenged.</p>
<p>But regulation has not always stopped Facebook in the past, who paid out <a href="https://www.nytimes.com/2020/01/29/technology/facebook-privacy-lawsuit-earnings.html#:%7E:text=Facebook%20said%20on%20Wednesday%20that,social%20network's%20data%2Dmining%20practices.">US$550 million</a> to settle a lawsuit for breaching biometric privacy laws. In the multi-billion dollar world of big-tech, it’s all a cost of doing business. </p>
<p>Another question we might ask ourselves is whether Facebook’s virtual-reality future and others like it really need to exist. Maybe there are other ways to avoid forgetting your keys.</p><img src="https://counter.theconversation.com/content/145730/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Marcus Carter is a board member of the Digital Games Research Association of Australia.</span></em></p><p class="fine-print"><em><span>Ben Egliston does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The Oculus Quest 2 headset is the latest step in the construction of a ‘mirrorworld’ built on high-tech surveillance and targeted advertising.Marcus Carter, Senior Lecturer in Digital Cultures, SOAR Fellow., University of SydneyBen Egliston, Postdoctoral research fellow, Digital Media Research Centre, Queensland University of TechnologyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1015282019-03-04T18:58:14Z2019-03-04T18:58:14ZVirtual reality adds to tourism through touch, smell and real people’s experiences<figure><img src="https://images.theconversation.com/files/261811/original/file-20190304-110119-1uwzd7h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Virtual reality can bring historical sites to life.</span> </figcaption></figure><p>Back in 2001, an acquaintance who worked for Lonely Planet told me about a surprise discovery. The travel guide business had an audience of people who would buy their travel books, but never travel. Lonely Planet dubbed them “virtual tourists”. </p>
<p>Now Lonely Planet, and others, have become excited by tourism powered by <a href="https://www.lonelyplanet.com/news/2016/04/11/travel-tourism-virtual-reality/">virtual reality</a> (VR) – both on this planet and, thanks to NASA, on <a href="https://www.lonelyplanet.com/news/2018/06/25/virtual-space-tourist-nasa-exoplanet/">others</a>.</p>
<p>VR films are also being developed by travel companies, such as <a href="https://www.cnbc.com/2016/01/08/virtual-reality-devices-could-transform-the-tourism-experience.html">Thomas Cook</a>. And Tourism Australia has partnered with Google to understand the <a href="http://www.tourism.australia.com/content/dam/assets/document/1/6/y/7/t/2003897.pdf">marketing potential of VR</a> (well, 360 degree panoramic videos). </p>
<p>But VR tourism isn’t only about recreating a virtual version of reality that renders travel to the destination unnecessary. It can enhance tourism in other ways – by allowing tourists to handle precious historical artefacts in virtual form, or by retelling contested histories from previously unexplored perspectives.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/tourist-attractions-are-being-transformed-by-immersive-experiences-some-lessons-from-scotland-110860">Tourist attractions are being transformed by immersive experiences – some lessons from Scotland</a>
</strong>
</em>
</p>
<hr>
<h2>What is virtual tourism?</h2>
<p>In contrast to Lonely Planet’s definition, let’s consider virtual tourism to be the application of virtual reality – including augmented reality (AR) and mixed reality (MR) – to tourism.</p>
<p>The term virtual reality is most commonly used to describe what happens when you are completely immersed in a virtual environment you can see through a headset. Enhanced forms of virtual reality allow you to interact with that environment using extra equipment, such as gloves fitted with sensors.</p>
<p>Virtual reality is also used as a catch-all term to describe the overall spectrum of digitally mediated reality, which includes virtual reality, as well as mixed reality and augmented reality. </p>
<p>Augmented reality and <a href="https://medium.com/@marknb00/what-is-mixed-reality-60e5cc284330">mixed reality</a> are computer-generated visualisations that augment our sense of the real world around us or merge the real and virtual together. You still wear a headset, but rather than blocking out the world, an AR or MR headset enables you to see visualisations within your real world surroundings.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=258&fit=crop&dpr=1 600w, https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=258&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=258&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=325&fit=crop&dpr=1 754w, https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=325&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/239856/original/file-20181009-72117-nc4vy0.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=325&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">PhD student Mafkereseb Bekele demonstrates a digital underwater landscape augmented over the real world as it would appear through a Microsoft Hololens headset.</span>
<span class="attribution"><span class="license">Author provided</span></span>
</figcaption>
</figure>
<p>Augmented reality and mixed reality is usually visual, but you can now get <a href="https://www.theverge.com/2018/3/12/17106688/bose-ar-audio-augmented-reality-glasses-demo-sxsw-2018">audio augmented reality</a>, that will play audio recordings through special glasses about sites you’re looking at. There is even <a href="https://www.theatlantic.com/technology/archive/2015/03/finding-out-what-the-past-smelled-like/387352/">olfactory-augmented reality</a> that can enhance your experience with smell.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/vr-technology-gives-new-meaning-to-holidaying-at-home-but-is-it-really-a-substitute-for-travel-101258">VR technology gives new meaning to ‘holidaying at home’. But is it really a substitute for travel?</a>
</strong>
</em>
</p>
<hr>
<h2>Moving beyond realism</h2>
<p>Virtual reality can be more than a mirror that gives you a realistic interactive simulation of the current world: it can bring the past into the present. </p>
<p>As Sir David Attenborough has <a href="https://mashable.com/2018/05/24/david-attenborough-hold-the-world-vr/">noted</a>: </p>
<blockquote>
<p>The one thing that really frustrates you in a museum is when you see something really fascinating, you don’t want to be separated from it by glass. You want to be able to look at it and see the back of it and turn it around and so on.</p>
</blockquote>
<p>The London Natural History Museum’s app <a href="http://www.nhm.ac.uk/discover/news/2018/march/explore-the-museum-with-sir-david-attenborough.html">Hold the World</a> gives users a chance to move and manipulate virtual objects that are fragile, expensive or remote. </p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/nr3l98Omm-0?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
</figure>
<p>Virtual tourism is also breathing new life into mythology and folklore. In Denmark, there are plans to turn a virtual reality exhibition <a href="http://www.attractionsmanagement.com/index.cfm?pagetype=news&codeID=332808">exploring Viking history and Norse mythology</a> into a permanent theme park. Visitors will be able to fight giants and dragons, and explore a complete “Nordic” landscape. </p>
<p>Virtual tourism can allow people to hear fresh interpretations of history. For example, the augmented reality app <a href="http://www.abc.net.au/news/2018-04-25/app-shares-forgotten-stories-indigenous-anzacs/9690766">Dilly Bag</a> connects users with the stories of Indigenous Australian servicemen via a smartphone. </p>
<p>Stories can be told from the <a href="https://www.bbc.com/news/uk-wales-41635746">perspective of flying animals</a>, or provide thrills and spills that appear more dangerous, immediate and visceral than the real thing (see this <a href="https://www.itnews.com.au/news/inside-chinas-first-virtual-reality-theme-park-478484">VR rollercoaster theme park in China</a>). </p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/JhtYQczgNNw?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
</figure>
<p>Whether virtual tourism proves to be only a pale imitation of the real thing depends on how imaginative we are.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/how-virtual-reality-is-giving-the-worlds-roller-coasters-a-new-twist-101395">How Virtual Reality is giving the world's roller coasters a new twist</a>
</strong>
</em>
</p>
<hr>
<h2>How common is virtual tourism?</h2>
<p>Given the expense and complexity of virtual reality, augmented reality and mixed reality arguably have more potential for virtual tourism. </p>
<p>Wi-Fi, which is required for many virtual tourism experiences, is now commonplace, and many people do have their own devices. But content must be tailored to specific devices – smartphones can overheat from processing so much data, and the size of tablets can make them unwieldy. </p>
<p>The number of exciting technological showcases is matched by the number of failed or broken equipment and deserted VR centres. Hyped promises proliferate – apparently every year is the year that <a href="https://www.forbes.com/sites/charliefink/2017/10/20/war-of-arvrmrxr-words/%20-%20ba5adc68d074">VR, AR and MR</a> will <a href="http://www.abc.net.au/news/2018-01-12/augmented-reality-why-2018-might-be-year-ar-goes-mainstream/9321472">break though</a>. </p>
<p>Yet any VR software and hardware currently full of promise seems to get old very, very, quickly. If we are to move past one-hit AR wonders such as <a href="https://theconversation.com/what-went-wrong-with-pokemon-go-three-lessons-from-its-plummeting-player-numbers-67135">Pokémon Go</a>, we need scalable yet engaging content, stable tools, appropriate evaluation research and robust infrastructure.</p>
<p>Formats such as <a href="https://webvr.info/">WebVR</a> and <a href="https://artechglobal.com.au/2018/07/20/browser-augmented-reality-2/">Web XR</a> promise to supply content across both desktops and head mounted displays, without having to download plugins. </p>
<p>But before we see virtual tourism become widespread, we need to change our preconceptions about what virtual reality is. Let’s not limit VR experiences to recreations of the real world, instead let’s open our minds to history, mythology and fresh perspectives from real people.</p><img src="https://counter.theconversation.com/content/101528/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Erik Malcolm Champion received funding for a Ph.D. Scholarship at the University of Melbourne, from an Australian Research Council SPIRT grant, with industry partner Lonely Planet Publications, 2001-2004,</span></em></p>Virtual reality can be more than a mirror that gives you a realistic simulation of the current world: it can bring the past into the present.Erik Malcolm Champion, UNESCO Chair of Cultural Heritage and Visualisation, Curtin UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/998272018-11-16T11:44:38Z2018-11-16T11:44:38ZWhat is augmented reality, anyway?<figure><img src="https://images.theconversation.com/files/245627/original/file-20181114-194494-1p82jkx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Is that Pikachu on the street right next to you?</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/montreal-ca-july-28-2016-closeup-466869869">Marc Bruxelle/Shutterstock.com</a></span></figcaption></figure><p>Augmented reality systems show virtual objects in the real world – like cat ears and whiskers on a Snapchat selfie, or how well a particular chair might fit in a room. The first big break for AR was the “Pokémon GO” game, released in 2016 with a feature that let players see virtual Pokémon standing in front of them, ready to be captured and played with. Now, technology companies like <a href="https://www.microsoft.com/en-us/store/collections/vrandmixedrealityheadsets">Microsoft</a> and <a href="https://research.mozilla.org/mixed-reality/">Mozilla</a> – the company behind the Firefox browser – and even retail businesses like <a href="https://www.theverge.com/2017/9/20/16339006/apple-ios-11-arkit-ikea-place-ar-app">IKEA</a> and <a href="https://www.forbes.com/sites/andyrobertson/2017/12/01/new-lego-augmented-reality-app-is-the-best-open-world-lego-video-game/">Lego</a> are exploring the potential of AR.</p>
<p>Where I <a href="https://scholar.google.de/citations?user=uHCRUlQAAAAJ&hl=en">do research</a>, an <a href="https://www.mi2lab.com/">AR lab</a> at the University of Michigan School of Information, it seems everyone knows about AR and is excited about the technology becoming popular among the general public. My colleagues and I watch <a href="https://www.youtube.com/watch?v=eDxzlwEriAw">videos of impressive AR demonstrations</a>, <a href="https://vimeo.com/228921453">try out new applications</a> and <a href="https://vimeo.com/293709725">play with new devices</a>. The research community’s enthusiasm may be why several experts – including some I talk with – say they expect AR to be <a href="https://arstechnica.com/gaming/2018/03/epic-games-ceo-ar-glasses-will-eventually-replace-smartphones/">commonplace in five years</a>, or envision AR <a href="https://www.valuewalk.com/2017/04/ar-glasses-replace-smartphones/">glasses replacing smartphones</a> within a decade.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/eDxzlwEriAw?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Exploring the possibilities of augmented reality.</span></figcaption>
</figure>
<p>But as an AR researcher with expertise in both industry and academia, I disagree with those optimistic views. Most people in the U.S. haven’t heard of AR – and most of those who have <a href="https://www.valuewalk.com/2017/02/1900675/">don’t really know what it is</a>. And that’s just one barrier between augmented reality today and a future where it is everywhere. Overall, there are three major challenges to be overcome.</p>
<h2>Hardware difficulties</h2>
<p>When I first tried AR glasses three years ago, they quickly overheated and shut down – even when trying to do something fairly basic, like placing two virtual objects in a room. While there has been a lot of improvement in this respect, other problems have emerged. The HoloLens system – one of the most advanced AR headsets – essentially requires a user to carry a Microsoft Kinect system and a computer on their head, which is <a href="https://thenextweb.com/microsoft/2016/04/01/tried-hololens-just-not-ready-mixed-reality/">quite heavy</a> and <a href="https://mashable.com/2018/05/08/microsoft-hololens-field-of-view-big-weakness/">limits the user’s field of view</a>. A different issue are <a href="https://dl.acm.org/citation.cfm?doid=3229089">AR experiences that work across systems</a>.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=384&fit=crop&dpr=1 600w, https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=384&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=384&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=482&fit=crop&dpr=1 754w, https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=482&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/245625/original/file-20181114-194519-sgke0o.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=482&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Microsoft’s HoloLens system requires wearing a computer on your head.</span>
<span class="attribution"><a class="source" href="http://www.apimages.com/metadata/Index/Microsoft-Developer-Conference/ee7e70f861244b4c80554517208e245c/2/0">AP Photo/Elaine Thompson</a></span>
</figcaption>
</figure>
<p>Even “Pokémon GO,” the most popular app that actually uses AR, <a href="http://mobileenerlytics.com/a-first-inside-look-at-pokemon-go-battery-drain-you-wont-catch-many-if-your-battery-dies-so-quickly/">drains smartphone batteries extremely rapidly</a>. And the AR function doesn’t make the game much better – or really different at all – though it is neat at first to see a Pikachu standing on the lawn in front of you. With so little benefit and such a severe hit to device performance, every player I know, including me, has turned off the AR mode.</p>
<h2>Lack of real uses so far</h2>
<p>Just as people turn off AR in “Pokémon GO,” I’ve never seen or heard of anyone actually using <a href="https://itunes.apple.com/us/app/ikea-place/id1279244498?mt=8">IKEA’s furniture app</a> as it’s allegedly intended; the app has just 3,100 reviews in Apple’s app store, far fewer than the 104,000 for “Pokémon GO.” It’s supposed to be useful to people seeking to redesign their living spaces, letting them use their smartphones to add virtual furniture to actual rooms.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/245597/original/file-20181114-194516-1bwd4jk.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Does that chair look good right there?</span>
<span class="attribution"><a class="source" href="https://www.ikea.com/gb/en/images/gb-img-fy15/ikea-ikea-place-app-augmented-reality__1364442642812-s4.jpg">IKEA</a></span>
</figcaption>
</figure>
<p>Apple and Google have released AR toy and demo apps built with their new platforms <a href="https://developer.apple.com/arkit/">ARKit</a> and <a href="https://en.wikipedia.org/wiki/ARCore">ARCore</a> – such as <a href="https://www.youtube.com/watch?v=0aBxeJDdYWw">playing with virtual dominos</a>. They are engaging, and the 3D models look great. They do what they’re designed to do, but their functions aren’t especially useful.</p>
<p>This is partly due to the fact that AR, like the internet, is just a basic technology that needs people to create uses for it. The internet <a href="https://en.wikipedia.org/wiki/ARPANET">started as Arpanet in 1969</a>, but began to grow widely only when Tim Berners-Lee invented the “<a href="https://webfoundation.org/about/vision/history-of-the-web/">World Wide Web</a>” – a now-dated term – <a href="https://home.cern/topics/birth-web">in 1989</a>. And it wasn’t until the 2000s that regular people who used the internet could also <a href="https://www.oreilly.com/pub/a/web2/archive/what-is-web-20.html">create online content</a> for others to consume. That level of development and innovation has not yet happened for AR, though <a href="https://blog.mozilla.org/blog/2017/10/20/bringing-mixed-reality-web/">Mozilla is taking initial steps</a> in this direction by trying to <a href="https://mixedreality.mozilla.org/">bring AR to everyday web browsers</a> like Firefox.</p>
<h2>Marketing challenges</h2>
<p>Even people who use Snapchat don’t think of it as an augmented reality app – though that’s exactly what it is. It’s AR technology that figures out where to put the dog ears, heart eyes or whiskers on their friends’ faces – and sends rainbow vomit out of their mouths. People who don’t know what augmented reality is, or who have never consciously experienced it – even if they use it daily – aren’t going to make a purchase just because a product has some AR capability.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=391&fit=crop&dpr=1 600w, https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=391&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=391&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=491&fit=crop&dpr=1 754w, https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=491&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/245612/original/file-20181114-194503-rvdc8r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=491&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Putting graphics on a Snapchat selfie involves using augmented reality.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/montreal-canada-july-1-2016-snapchat-450910099">dennizn/Shutterstock.com</a></span>
</figcaption>
</figure>
<p>There’s also some confusion in labeling and marketing of AR technologies. Many people have started to hear about virtual reality, which is generally an immersive fully virtual world that doesn’t include aspects of the user’s real environment. The distinctions get fuzzier with mixed reality – sometimes labeled “MR” but other times “XR.” Originally the term meant <a href="https://medium.com/@marknb00/what-is-mixed-reality-60e5cc284330">anything in between a fully real and a fully virtual</a> experience – which could include AR. But now Microsoft is saying products and apps are MR if they provide <a href="https://www.microsoft.com/en-us/mixed-reality">both augmented and fully virtual experiences</a>. That leaves customers unclear what’s being advertised – though they’ll know it might not be very useful and may run their phone batteries down quickly.</p>
<p>I’m with my AR-optimist friends and colleagues in seeing a lot of potential for the future, but there’s a long way to go. They – and I – are already working hard on making the hardware better, finding useful applications and clarifying product labeling. But it will take lots of this hard work and probably many more years before mainstream America lives in a truly augmented reality.</p><img src="https://counter.theconversation.com/content/99827/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Maximilian Speicher is affiliated with C&A Europe.</span></em></p>Before augmented reality products and apps take over the world, they’ll have to get out of their own way.Maximilian Speicher, Sponsored Affiliate Researcher in Interactive and Social Computing, University of MichiganLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/995032018-07-26T20:53:50Z2018-07-26T20:53:50ZHow playful design is transforming university education<figure><img src="https://images.theconversation.com/files/229502/original/file-20180726-106499-zf3vx2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Video games have inspired a revolution in university teaching. Pictured here is a scene from the popular game Fortnite Battle Royale.</span> <span class="attribution"><span class="source">(Sergey Galyonkin, Epic Games Berlin via Wikimedia Commons)</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>A group of 25 interns sit at <a href="https://www.baycrest.org">Baycrest Health Sciences</a>, a research centre for aging in Canada, their eyes glued to their smart phones. They are playing <em>SOS</em> — <a href="http://clri-ltc.ca/?resource=sos-gaming-app-introductory-video-educators">an award-winning game that simulates real-world gerontology practice</a> — where they compete with other students to earn virtual currency. </p>
<p>Across town, a group of professors sit around a table at <a href="https://www.georgebrown.ca">George Brown College</a>, designing a role-playing game with a virtual hospital called <em>The Grid</em>, based on a Matrix-like theme of saving the world from ignorance, for an accredited program in health sciences. Yet another team of game programmers are hard at work at <a href="http://humber.ca">Humber College</a>, building a virtual reality experience of a subway car after a bomb incident. Players wear goggles, moving from person to person, saving some and tagging others for care later on. </p>
<p>Welcome to the new world of serious games and mixed reality. Serious educational games (SEGs) are games designed for learning. Mixed reality is a blend of virtual reality, augmented reality and what are called immersive technologies. </p>
<p>When you combine these, you get a sense of where university education is going. It is called playful design and it’s a <a href="https://www.business.com/articles/5-statistics-that-prove-gamification-is-the-future-of-the-workplace/">multi-billion dollar industry, that was used by over 40 per cent of the top 1,000 companies</a> in 2015. It is also likely coming to a college near you. </p>
<h2>Better than lectures</h2>
<p>There are thousands of peer-reviewed studies on the <a href="https://dl.acm.org/citation.cfm?id=3178707">effectiveness of SEGs</a>, showing that they do three things better than conventional teaching in higher education. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/229498/original/file-20180726-106508-18nohyx.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Serious educational games harness the addictive quality of consumer video games such as such as The Legend of Zelda series.</span>
<span class="attribution"><span class="source">(Flickr/Tofoli.douglas)</span></span>
</figcaption>
</figure>
<p>First, they enable skill acquisition, that is, <a href="https://doi.org/10.1016/j.compedu.2012.03.004">they encourage students to use what they have learned and repeat that many times to master it</a>.</p>
<p>Second, they engage and motivate students more strongly than most lectures. </p>
<p>Third, they reward the learner with achievements every time they play. As far as other comparisons with conventional learning go, they are at least as effective as lectures in most cases for teaching the basics. </p>
<p>In short, there are no compelling reasons not to use games for learning, but there are understandable questions about when and how to do so.</p>
<h2>Achievement, exploration, social connection</h2>
<p>SEGs are divided into categories. They range from card games based on a pharaoh’s empire in which you build an organization, to <a href="https://peartherapeutics.com/reset/">mobile addiction treatment systems</a> approved by the United States Food and Drug Administration, to <a href="http://www.simleader.ca/en/ambulance-paramedic/">training simulations for ambulance drivers and paramedics</a> created by Montreal-based SimLeader. </p>
<p>“Gamification” researchers, such as <a href="https://www.sfu.ca/education/faculty-profiles/dkaufman.html">David Kaufman</a> from Simon Fraser University, have held multi-million dollar grants to investigate their use in education over the past decade. </p>
<p>Some of the ideas used in gamification are simple. In “Point, Badge, Leaderboard (PBL)” systems, points lead to badges, which lead to a position in a group. You might earn points for learning about the biology of a moth, then more points when you also learn about the biology of arachnids. This provides the kind of moment-to-moment feedback you get in video games. </p>
<p>In fact, most of the ideas that underlie this field came from video games. There is an impressive body of research showing that <a href="https://theconversation.com/play-games-with-your-kids-this-summer-to-boost-their-brains-94568">video games do improve our brains</a>, despite what well-meaning parents might say when they see children glued to the screen. </p>
<p>And when we take these elements present in good video games — of achievement, exploration, competition and social connection — and use them for learning, an entirely new way of thinking about education emerges.</p>
<h2>An addictive quality</h2>
<p>It is called <a href="https://journals.ala.org/index.php/ltr/article/view/5629/6946">playful design</a> in some fields, based on the idea that we do like play, that it is human to play, not just childish. </p>
<p>But what is meant by play in this context? It is making learning fun, something you want to do more of, not less of. It is seen as a remedy for “training malaise,” the disappointing finding that the majority of what we teach people is not remembered. We forget things that are banal and remember things that have significance. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=600&fit=crop&dpr=1 600w, https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=600&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=600&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=754&fit=crop&dpr=1 754w, https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=754&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/229499/original/file-20180726-106521-12dzvs4.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=754&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">A teaching application showing the interface screen for game-based learning in medicine in partnership with ARC Reach, an Edmonton-based technology company.</span>
<span class="attribution"><span class="source">(Baycrest Health Sciences)</span>, <span class="license">Author provided</span></span>
</figcaption>
</figure>
<p>Serious games work by practising skills and tracking achievement, but also by giving learning an addictive quality. The “one more move” thinking that keeps video gamers up all night is harnessed for learning. <a href="https://www.gamasutra.com/blogs/BenLewisEvans/20130827/198975/">It is based on the release of the neurotransmitter dopamine</a>. What dopamine does is make us want the “next good thing.” Once we get it, we lose interest and want “another good thing.”</p>
<p>This is called <a href="https://www.theseus.fi/bitstream/handle/10024/147016/tanskanen_selja.pdf?sequence=2in">a compulsion loop</a> in game design, and can be the basis of making university courses not only relevant, but engaging. It is also a form of active learning, which has been shown to <a href="https://doi.org/10.1073/pnas.1319030111">increase grades and decrease failure rates</a> in a major review of research in the field.</p>
<h2>Playgrounds of learning</h2>
<p>The best SEGs now are called “open world games” and are a blend of mixed reality and game design. Imagine living in an alternate reality, where you are not you, but an image of you called an avatar, and this avatar is having adventures building something, defeating a plague or combating addiction. </p>
<p>This is called a “<a href="https://gameanalytics.com/blog/how-to-perfect-your-games-core-loop.html">conglomerate of player satisfaction loops</a>” and, in short, gives you so many things to do in this alternate world that you don’t know where to start. These are truly <a href="https://www.igi-global.com/chapter/team-based-learning-and-team-based-playing/196415">open exploratory playgrounds of learning</a>, you can study things in any order you want, there is no scripting of learning in the best SEGs. You simply enter an imaginary digital world and begin to engage with it.</p>
<p>Preliminary studies by professors <a href="https://www.ryerson.ca/information-technology-management/faculty-research/deborah-fels/">Deb Fels</a>, <a href="https://procom.ryerson.ca/people/robert-bajko">Rob Bajko</a> and other faculty at Ryerson University for internal purposes have shown that the majority of students prefer to learn using games. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=448&fit=crop&dpr=1 600w, https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=448&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=448&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=563&fit=crop&dpr=1 754w, https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=563&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/229504/original/file-20180726-106517-1ix16sj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=563&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Students learning to manage care of the frail elderly using a card-based game.</span>
<span class="attribution"><span class="source">(Baycrest Health Sciences)</span>, <span class="license">Author provided</span></span>
</figcaption>
</figure>
<p>One long term study <a href="https://doi.org/10.1016/j.chb.2016.08.049">tracked a class of students using game-based learning over a three-year period</a> and found there were many types of players. </p>
<p>Some students were thrilled by the game, some less so, but in all cases, the better they did in the game, the better they did in the courses. This has been replicated many times by the author and other teams. </p>
<p>We learn a lot when we love what we are learning. Its a basic benchmark of achievement in higher education. Human beings love doing certain things, and learning to become a master of their own world, however fantastical it might be, is one of them. </p>
<p>Game worlds and mixed reality are rapidly developing fields which any educator with even a passing interest in leveraging student success would be advised to track as it unfolds.</p><img src="https://counter.theconversation.com/content/99503/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>David Chandross does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>University course designers are harnessing the addictive quality of video games to develop ‘Serious educational games’ that engage and motivate students.David Chandross, Professor of Education, Toronto Metropolitan UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/903152018-02-02T01:38:45Z2018-02-02T01:38:45ZDesigners of mixed reality experiences shouldn’t overlook the communal nature of video games<figure><img src="https://images.theconversation.com/files/203906/original/file-20180129-89582-3g8m8z.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Season two of the Netflix series Stranger Things featured a video game called Dragron’s Lair.</span> <span class="attribution"><a class="source" href="https://media.netflix.com">Netflix</a></span></figcaption></figure><p>Augmented reality, virtual reality and mixed reality headsets are gradually becoming more widespread. But the communal pleasure of video games is at risk when our friends disappear into solitary playing experiences that we can’t see. </p>
<p>Our research shows that there are solutions to this problem that could enable spectators to be involved in games, even if only one person has a headset.</p>
<h2>Single-player games can be group activities</h2>
<p>If you’re a fan of the Netflix series <a href="https://www.netflix.com/title/80057281">Stranger Things</a>, you’ll know that some of the key scenes in season two involved the characters of the show playing video games in an arcade. </p>
<p>The star game of season two, called <a href="http://store.steampowered.com/app/227380/Dragons_Lair/">Dragon’s Lair</a>, was a beautifully animated single-player cartoon game drawn by <a href="http://www.imdb.com/name/nm0089940/">Don Bluth</a> and played by the protagonists one at a time, as the other kids watched over the shoulder.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/vgS2L7WPIO4?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Stranger Things Season 2 Trailer.</span></figcaption>
</figure>
<p>Another game you might have found in that arcade just a few years later in 1991 is one of the first commercial virtual reality games: <a href="https://www.arcade-history.com/?n=dactyl-nightmare&page=detail&id=12493">Dactyl Nightmare</a>. </p>
<p>In Dactyl Nightmare, the player donned a headset and wandered around a playscape, looking for and shooting pterodactyls. If you were lucky, the arcade would have two machines and you’d be able to play with a friend, each starting in a different part of the arena in <a href="https://www.youtube.com/watch?v=v6t69mp0ZhE">VR Multiplayer</a>.</p>
<p>But would the Stranger Things kids have played it? Perhaps not. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/oculus-and-our-troubles-with-virtual-reality-87305">Oculus and our troubles with (virtual) reality</a>
</strong>
</em>
</p>
<hr>
<p>Dactyl Nightmare was a distinctly solo experience, where you disappeared into the headset immersed in a world beyond the arcade. Even if you were playing with a friend, you were in different parts of the game. Anyone not playing was unable to watch or participate, they could only stand there watching you wearing a headset and waving your arms around. </p>
<p>One could argue that in 27 years, not much has changed.</p>
<h2>Watch as I enter a digital world you can’t see</h2>
<p>Even though virtual reality has improved since the original Dactyl Nightmare – and augmented reality and mixed reality has arrived to supplement it – the underlying experience is still <a href="https://zerolatencyvr.com/">very solitary</a>. </p>
<p>New devices such as the <a href="https://developer.microsoft.com/en-us/windows/mixed-reality/shared_experiences_in_mixed_reality">Microsoft HoloLens and other solutions by Microsoft</a> promise to seamlessly integrate the digital and the physical world, but they also typically limit the vision of those digital objects to the person wearing the headset. </p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/0AWhsBNU1jU?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Mixed Reality by Microsoft.</span></figcaption>
</figure>
<p>This has been <a href="https://www.forbes.com/sites/forbesagencycouncil/2017/04/26/addressing-the-lack-of-communal-experience-in-virtual-reality/#7737492128f0">acknowledged as a problem</a>. </p>
<p>Some solutions, such as the proposed Facebook social VR system <a href="https://www.facebook.com/spaces">Spaces</a> solve it by inserting multiple users into the experience using multiple headsets, and/or by having all those users inserted into a <a href="http://www.sciencedirect.com/science/article/pii/S0360131515000755">blended</a> <a href="https://altvr.com/">virtual world</a>. </p>
<p>But few seem to consider the Stranger Things viewer aspect. That is, sometimes the viewer doesn’t want to be part of the experience, they just want to view what’s going on. A good example is the arena of eSports, where it’s <a href="https://gadgets.ndtv.com/games/features/e3-2017-e-sports-twitch-youtube-facebook-creators-1712010">reported by E3</a> that the industry for watching others play video games will accelerate from roughly $200 million in revenue in 2015 to $1 billion by 2018.</p>
<h2>I don’t play football, but I still want to watch it</h2>
<p>Given the emergence of AR/VR/MR as <a href="https://blogs.unity3d.com/2017/09/05/looking-to-the-future-of-mixed-reality-part-i/">prominent new technologies to watch in 2018</a>, we have been considering this problem in our work at <a href="http://www.mixedrealityresearch.com/">The Mixed Reality Research Lab</a>. As part of a <a href="http://epublications.bond.edu.au/fsd_papers/532/">paper we presented</a> at the <a href="http://2017conference.ascilite.org/">ASCILITE</a> conference late last year, we proposed some solutions.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=450&fit=crop&dpr=1 600w, https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=450&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=450&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=566&fit=crop&dpr=1 754w, https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=566&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/203341/original/file-20180124-107963-1cid4q5.JPG?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=566&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">The Authors Presenting Multi-User Mixed Reality at ASCILITE 2017.</span>
</figcaption>
</figure>
<p>One option is to feed the view of the user to a third party screen, either in 2D or in another headset. Whilst this is possible (<a href="https://twitter.com/financialtimes/status/926982112271577088">and often what is done today</a>), it doesn’t give the spectator control of their own experience, and can actually <a href="https://www.sciencenews.org/article/virtual-reality-has-motion-sickness-problem">make some people sick</a>. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/virtual-reality-has-added-a-new-dimension-to-theme-park-rides-so-whats-next-for-thrill-seekers-89222">Virtual reality has added a new dimension to theme park rides — so what's next for thrill-seekers?</a>
</strong>
</em>
</p>
<hr>
<p>Another option is to recognise the role of a spectator in a simulation, and build the simulation to allow the spectator to be involved.</p>
<p>At it’s simplest level, this could simply be a passive interface into the world that the headset wearer is seeing. Imagine holding up your phone or tablet up to somebody wearing a headset, launching an app, and seeing from a spectator view all the digital objects they can see and are interacting with. </p>
<p>Microsoft has been experimenting with this using a HoloLens attached to a GoPro camera and a computer to generate a <a href="https://developer.microsoft.com/en-us/windows/mixed-reality/spectator_view">spectator view</a>, but it’s still very much a hack. They’ve also looked at adding mixed reality to <a href="https://uploadvr.com/microsoft-launches-sci-fi-reality-rooms-employee-vr-ar-development/">special SciFi “reality rooms”</a>.</p>
<p>At a higher level, this could allow the spectator to participate in the simulation, possibly by providing them with some information that the headset wearer doesn’t have. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/its-time-to-get-ready-for-augmented-reality-89760">It's time to get ready for augmented reality</a>
</strong>
</em>
</p>
<hr>
<p>Imagine a variation on the popular game <a href="http://www.keeptalkinggame.com">Keep Talking and Nobody Explodes</a> where the spectators can use a secondary device to see the bomb but not touch it, requiring them to circle around and give directions to the headset user in real-time, which they need to execute straight away. This model separates the spectator and the user whilst still allowing them both to be <a href="https://www.oculus.com/experiences/gear-vr/814885695293688/">involved in the simulation</a> - no more standing there whilst somebody else waves their arms around.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/hqelfBKuiic?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Keep Talking and Nobody Explodes.</span></figcaption>
</figure>
<p>Most importantly, this means that the situation can be adjusted for whatever makes the most sense in the moment. That might mean everyone wearing a headset, but it might also mean one user playing the game whilst the others passively or actively interact. </p>
<p>And just like Dustin and his friends from Stranger Things, one good player and a bunch of spectators is fine, as long as you’re all having fun!</p><img src="https://counter.theconversation.com/content/90315/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>New gaming headsets promise to seamlessly integrate the digital and the physical world, but they also typically limit the vision of those digital objects to the person wearing the headset.Michael Cowling, Senior Lecturer in Educational Technology, CQUniversity AustraliaJames Birt, Assistant Professor Interactive Media and Design, Bond UniversityLicensed as Creative Commons – attribution, no derivatives.