Science & Tech Spotlight: Extended Reality Technologies

Why This Matters

Extended reality (XR) technologies combine elements of the real world and the digital world to create new kinds of interactivity and perception. XR provides new ways to access and use information to train, educate, entertain, and collaborate. But XR could also make more data vulnerable to cyberattacks and privacy threats, as well as create new avenues for online harassment.

The Technology

What is it? Extended reality (XR) is the overarching term for a spectrum of technologies that link or integrate the digital world and the real world. These include augmented reality (AR), mixed reality (MR), and virtual reality (VR) technologies, all of which provide different degrees of sensory immersion and interaction between the real world and digital content. AR overlays digital content onto representations of the real environment, using smartphones, tablets, or AR glasses. In MR, a dedicated headset recognizes its environment and enables the interaction between digital content and the real world in multiple dimensions. VR completely obscures the real world, immersing users in digital environments using head-mounted displays (see fig. 1).

XR technologies can be used for workplace collaboration, training, education, therapeutic treatments, and data exploration and analysis. XR also enables the creation of online universes, or "metaverses," where users can interact with each other.

Figure 1. The continuum of extended reality.

How does it work? XR devices send digital information, which can be accessed and contained locally, or shared and transferred over a network, to the human senses. They can also use sensors to gather information and accept commands. These devices create customized experiences by enabling real-time responses to virtual stimuli. Each requires supporting technologies, which vary by type of experience and real-world application. For example, appliance technicians might wear a holographic display headset configured with video cameras and distance and positioning sensors. The XR-enabled headset could then help guide them through an inspection or repair.

In medicine, XR is emerging as a tool to help train surgeons by enabling interaction with three-dimensional images of anatomical structures (see fig. 2). In addition, specialists may be able to virtually examine organs or body systems in three dimensions using XR, rather than the more limited evaluations allowed by flat screens. Realistic XR renderings could help providers better understand the complexity of each patient, making medical procedures safer and more effective.

Figure 2. Surgeons wear MR headsets to examine a personalized 3D model of a patient’s anatomy. Sensors track hand gestures and eye motion, and listen for verbal commands to control the model and the digital environment.

Another promising health care application is VR-based psychotherapy for addiction disorders, in which therapists work with patients in a digital world. For example, therapists could monitor patients' heart rates and other responses to simulated situations, without exposing patients to an actual event in an uncontrolled environment. This could provide information to therapists and patients to tailor treatment.

In workplace collaboration and training, XR may help workers become more efficient and skilled at lower cost and with less risk. For example, XR systems are currently used to train workers virtually before they actually operate complex systems, such as nuclear power plants. In the construction industry, architects, engineers, and project managers can collaborate in a virtual environment to plan and oversee projects. XR systems could also be used to help K-12 administrators improve security procedures and better protect the physical and operational security of their schools and students.

How mature is it? Widespread use of XR has been limited, although it has been around for decades. As of January 2022, multiple XR hardware systems and software development platforms are available in the market, and the market is expanding rapidly. For instance, one global market research advisor estimates that yearly AR/VR headset shipments will increase from 9.7 million in 2021 to 32.8 million by 2025. Reasons for the anticipated growth include ongoing development of more sophisticated devices (e.g., smartphones), growing needs for a more skilled and productive workforce, and recent disruptions to workplace dynamics caused by social distancing associated with the COVID-19 pandemic.

However, implementation of XR in business and government remains limited. Many of the applications described above are pilot programs, rather than mature uses of XR. Full deployment of these applications would require tailoring for broader use in different environments, and the widespread availability of key enabling technologies such as artificial intelligence and 5G. Artificial intelligence could provide users with customized and highly interactive XR experiences by interpreting large amounts of data. 5G network expansion may provide more reliable high-bandwidth communications to facilitate more diverse and complex XR technologies.

Opportunities

• Access. XR could provide better access to jobs, medical care, and other opportunities for remote communities or people with few or no transportation options.
• Collaboration. XR could provide data sharing and digital workspaces that support collaborative design, planning, and decision-making.
• Data analysis. Analyzing data in XR environments might allow new kinds of knowledge generation or decision-making.
• Therapeutic treatments. Immersive environments can be used in therapy to treat addiction, anxiety, autism, and other conditions.
• Training and education. Expensive or dangerous procedures might be taught more cheaply and safely in XR environments.

Challenges

• Cybersecurity and privacy. XR will require more diverse and complex data, offering new targets for cyberattack and exploitation.
• Enabling technologies. Certain technologies necessary for XR to achieve its full potential are not yet mature, including artificial intelligence and 5G. Further, these technologies may not be affordable or accessible to all users.
• Effects on users. XR environments can create new space for negative social interactions that are already common online. For example, some users have reported sexual harassment and bullying within those environments. In addition, some users experience motion sickness and disorientation while navigating immersive XR, and the effects of long-term XR immersion on cognitive functions are unknown.

Policy Context and Questions

• What incentives or barriers exist to the XR industry coalescing around common standards for software and content development that address ethics and vulnerabilities, among other issues?
• What barriers and challenges, such as 5G maturity and improving user experience, exist to the appropriate expanded adoption of XR in the private and public sectors, including health care, education, and the military?
• What new cybersecurity, privacy, harassment, and other threats are XR technologies vulnerable to, or could be inappropriately used for, and what new or enhanced safeguards might be needed as a result?
• What steps could be taken to improve the affordability and accessibility of XR and its enabling technologies, particularly 5G networks?

For more information, contact Brian Bothwell at (202) 512-6888 or BothwellB@gao.gov.