Federal extended reality: Merging the real and virtual world

These tools are already enabling federal agencies to reimagine how they deliver on their missions. For example, a large federal agency used mixed reality headsets that project 3D images into a physical space to overcome travel constraints created by COVID-19. In this case, U.S.-based members of the agency conducted real-time visual inspections of facilities on the other side of the world, supported by visual data delivered by an on-site inspector — a vivid demonstration of the power of XR to enhance government operations regardless of geographic limitations.

In the post-pandemic world, federal agencies cannot afford to be followers when it comes to XR, and they no longer have the luxury to wait and see. Government will need extended reality to meet mission needs. To better understand XR’s potential for federal government, Accenture surveyed 115 federal technology leaders to understand how they are using or planning for extended reality; this report shares the findings, as well as XR use cases, challenges, and recommended next steps for federal agencies.

To make the most of this opportunity, federal leaders need to consider how XR will impact and can enable their mission, and what technologies and skillsets are needed to master this new domain. They can begin by looking at their current business challenges, and exploring new ways of enabling services for employees and customers using the latest technologies. Leaders must begin a process of cultural change, looking for opportunities to leverage XR in support of increasingly diverse mission needs.

Extended reality is an umbrella term covering a range of immersive experiences. All represent digital extensions of the real world, but their uses may vary. Among federal agencies already working in the XR space, Accenture found 60 percent have used augmented reality, followed by mixed reality at 54 percent, and virtual reality at 41 percent.

• Virtual reality (VR) represents the far end of the spectrum: It offers a fully immersive experience, with the user deeply engaged in a simulated environment. In this mode, users operate within an artificial three-dimensional milieu via a headset or special goggles. Fully immersive experiences can be supplemented by sensor-enabled haptic gloves and other technologies that help deliver a realistic, albeit digitally simulated, experience.
• Augmented reality (AR) offers a blending of the physical and digital worlds. Users can still see their actual surroundings, but the scene is overlaid with three-dimensional, interactive digital content. This includes concepts of mixed reality using advanced sensor and imaging technologies that enable spatial awareness, allowing the user to interact with that digital content naturally while still operating in real-world space.

VR can be especially useful in cases where the user needs to engage fully in a scenario that might be difficult to create in the real world. For example, when training for dangerous or highly unusual missions, it may be easier, safer, and more cost effective to ramp up a simulated digital experience than to pursue conventional training. In addition, VR can enhance remote collaboration by bringing geographically disparate groups together with less cost and travel.

AR can be valuable when you need to execute complicated procedures with specialized equipment, knowledge, or experience. In this scenario, the workers’ real-world efforts can be informed and guided by a digital overlay, with contextual data and other key content made available in real time to increase operational efficiency and adherence to procedures. This can include on-demand remote collaboration and digital step-by-step guides that bring the expertise required to complete complex tasks.

Most in government already recognize the need for bold action when it comes to integrating XR and see the application of XR expanding over time.

Empowered with tools to extend the human senses and reshape our relationship to our physical environments, government has the opportunity to dramatically enhance its performance through XR. The technology can be a powerful enabler of training, bring digital twins to life in complex ecosystems, and empower workers to increase their expertise, speed, and precision. From field inspectors to warfighters, the potential benefits extend across a range of federal missions.

The best way to understand the moment we’re in is to view it as similar to the advent of the smartphone. No one at first predicted how disruptive a portable computing device would be; yet today, smartphones support the federal government in any number of areas, driving business processes that support ever-higher levels of productivity. In the future, will we always carry a smartphone to perform many day-to-day functions, or will that interface be replaced by simple, smaller headsets that expand our interactions using XR?

The impact of XR promises to be just as far-ranging and equally disruptive, revolutionizing the employee experience. Digital simulations will give workers new tools in support of their efforts, while simultaneously empowering those same workers to deliver vastly improved services. In the “new normal” — at a time when remote connections are essential — agencies can leverage XR to maintain critical business operations and services where they are needed. Digital experiences will loosen the shackles of geography, freeing federal workers to access information from anywhere; to collaborate through tangible, hands-on interactions; to interact with each other and with citizens in profoundly powerful new ways.

Government leaders have begun to envision or adopt new use cases for XR, with successful implementations already driving enhanced employee and customer experiences. Accenture found that 59 percent of federal technology leaders use XR for remote work and collaboration, while 53 percent use it for operations/inspections, and another 53 percent for security and surveillance.

Federal use cases underscore XR’s potential, making clear that:

Faced with a potential looming wave of retirements, and under pressure to increase their pipelines for talent, federal agencies can leverage XR to transform the way they deliver training. Virtual experiences offer the promise of higher-value learning experiences, especially in scenarios that are inherently dangerous or difficult to recreate. Personnel can learn without having to worry about making mistakes that would be costly in the real world and can even come together virtually to train regardless of geographic distance.

XR training also represents a potential cost savings: It cuts down on travel time and reduces the expense associated with setting up and executing training experiences. Moreover, XR training is immersive, and it can be easily repeated. These traits tend to make it highly effective. Research from Stanford University and Technical University Denmark found learners recall more when using virtual teaching methods than with traditional methods, resulting in a 76 percent increase in learning effectiveness.

Digital twins, or virtual models of objects, processes, and ecosystems fed with real-time data, are improving government’s ability to understand and predict outcomes. XR can bring these models to life as immersive, 3D digital twins in both live operational environments and simulated exercises with synthetic data used to model any number of scenarios.

Consider how a digital twin of a warehouse’s manufacturing processes or a city’s transportation networks could be made more interactive through XR – allowing users to step into the model, view from different perspectives, and potentially even interact.

While emerging use cases help to demonstrate the promise of XR, there are still technical and cultural challenges that need to be overcome in order to achieve its widespread implementation.

In Accenture’s research, federal leaders raise a number of key concerns with increasing their XR adoption.

In terms of policy, IT teams should be thinking now about the ramifications for security and device management. XR technologies including goggles and headsets represent yet another endpoint in the ever-expanding galaxy of devices. They’ll need to verify that those devices are secure, and they will need to implement policies and procedures for device management — ensuring devices are all accounted for and up to date. A robust device management plan ensures devices can be appropriately distributed, patched, maintained, and wiped if necessary. Agencies have a head start here: The same practices that currently support vast inventories of agency-issued mobile devices likely will extend into the XR realm.

Bandwidth considerations also come into play. Agencies deploying XR will need to ensure users have ample connectivity in support of an immersive experience. Depending on the volume of XR experiences deployed in a given location, network management may require some special attention. Agencies should be looking now at 5G network connectivity to deliver the low latency and high bandwidth needed to enable XR in edge and remote scenarios.

Finally, the race is on for the next wave of hardware and software. Companies like Facebook, Microsoft, HTC, and Varjo are heavily investing in the space which will drive out smaller, more flexible headsets. Apple and Google are other examples of companies that continue to invest in XR, including expanding software capabilities like ARKit and ARCore.

It’s significant that we are seeing the presence in the market of such major players. They and others are delivering software development kits for building AR capabilities, with ready-to-deploy applications already on the market. And the hardware constraint is rapidly disappearing: With XR no longer a niche space, there are commercial products available that can be made government-friendly for federal implementations.

For federal agencies looking to leverage the emerging power of XR, strategic investments today can help to set the stage for tomorrow’s successes. They can begin with some key steps:

Agencies can begin their XR journey by defining the mission cases where it will be most impactful. The time is ripe to initiate a culture shift from leadership out to the field, with all stakeholders encouraged to identify likely use cases for this powerful new capability.

On the training side, the low-hanging fruit is any experience that is prohibitively expensive or dangerous to recreate. In terms of field operations, agencies can be looking for areas where more immersive data visualization can improve job performance and decision-making. Situations like these are ripe for XR enhancements. For advancing situational awareness, agencies can explore which information should be brought together in new, visual ways to make the data more actionable or digestible.

It’s helpful to think of this process in a three-year timeline. This scope allows agencies to map out how their missions can be transformed both in the near- and long-term, while still allowing flexibility to integrate not-yet-realized technologies beyond the three-year mark.

Iteration is key here. Rather than revamp an entire work process, it may make sense to carve off some small piece, to look for an area that is most readily fixable. It helps, too, to find areas where progress is most easily measurable, where metrics show that key procedures aren’t being met, or where there are high-value assets at risk. Rich data builds the business case for future XR investments.

Agencies should begin investing in people: They can start to build the teams they will need to support widespread XR adoption. One first step is establishing a Center of Excellence (CoE) that can lead XR strategy, development, and program management for the agency.

This group can bring together the business and IT side to explore how this emerging technology can be applied for the agency. Like with any new technology, people need to understand the realm of the possible to reimagine how they execute their agency’s mission.

The CoE will need traditional IT skills such as infrastructure, network, and application developers, as well as systems engineers and project managers, who understand how these skills can be adapted for XR development and deployment specifically.

However, XR also requires new skills in addition to IT, including:

• Creative storytellers to plan out the immersive experience in an engaging and logical way
• Graphic 3D artists to design the virtual environments, objects, and features
• 3D game engine developers to cohesively bring the storytelling and artistry to life
• User experience designers to customize the XR experiences to the individual

To build these teams, like with every other new technology, there’s an element of upskilling your people. The CoE can begin now to socialize with employees what the XR landscape looks like and how it can benefit the mission.

Agencies may need to recruit outside of their traditional pipelines. They can look to the video game industry as both a source of inspiration and as a potential talent pool; bringing XR to life will require the storytelling skills and experience with 3D engines such as Unity and Unreal Engine that have long been prevalent in this field.

Agencies need the appropriate technology architecture and infrastructure to support XR adoption.

Cloud, edge computing, and 5G networking will all play a key role in enabling XR. Widespread, successful XR deployments will require future-ready compute capabilities and network infrastructure, with the ability to support high-performance, high-definition operations both within the traditional perimeter and at the edge.

For example, leaders need to assess the bandwidth impacts of these devices on networks and may need to consider 5G implementations where WiFi is not viable but high bandwidth and low latency is still needed for a high-quality XR experience. When data processing needs surpass network capacity, agencies can still provide a high-fidelity experience by offloading compute intensive programs to the edge. Ultimately, understanding how an agency’s technology architecture can intersect to provide an impactful XR experience, regardless of bandwidth, location, or use case, will be key to long-term success.

In terms of hardware, agencies can assess if their XR implementations can be viewed in 2D on existing screens, or if they want to invest in 3D experiences that require more specific devices, such as headsets. For those that are exploring headsets and goggles, it’s important to understand that the available options are rapidly evolving. These should not be viewed as long-term or permanent investments, but rather as an opportunity to take advantage of the best of the market at this point in time.

Lastly, XR leaders must understand how the technology will integrate within an agency’s existing cybersecurity framework. Now is the time to put in place the foundational elements, best practices, and security policies so that as new and better hardware and software emerges, IT will be able to upgrade seamlessly and securely.

Critical to the success of XR is the use of human-centered design as a means of ensuring XR experiences are engaging and effective. This requires an understanding of users’ needs, and a program in place to train workers how to interact with and interpret XR technology and data. As deployments scale, agencies must ensure they are equipping users with the appropriate knowledge and skills to make the most of their investment.

For example, the user experience for XR can differ dramatically compared to what people are used to from web or mobile interactions. Agencies should plan to onboard capabilities incrementally when it comes to XR, particularly for more immersive experiences, to avoid sensory overload.

It is important that agencies bring together the right skills to create the content and experiences – including experts in spatial cognition understanding, 3D experiences, designers, and the subject matter at hand.

Even with the right expertise creating and deploying XR, though, it is equally important to have the right analytics and measurements in place to validate the outcomes.

XR is an emerging technology with rapidly evolving software and hardware. Given that, agencies should be constantly monitoring and iterating on their XR deployments to ensure they are effectively meeting mission needs and optimizing use of available tools.

Continuous feedback can be a valuable tool for measuring success at the user level. With sensor-enabled devices, it may be possible to collect real-time feedback, which in turn can be leveraged to continually fine-tune the user experience.

In practice this means program leaders should engage with the employees, soldiers, and other end users either via discussion or questionnaire after each XR experience. They need to explore functionality: Could users see the images? Did the flow of the experience make sense? They can also ask about impact: Did this experience help them more effectively complete a task, for example? This human feedback loop can inform future iterations and provide a greater understanding of which technologies or visualizations are most useful in a particular scenario.

Agencies also need to consider results: Did the XR experience fulfill its intended function in terms of either training or mission support? By monitoring metrics including retention of training materials, productivity, speed of decision-making, or quality of customer service interactions, agencies can better understand how the addition of XR is improving mission outcomes.

XR is rapidly advancing and will transform how enterprises operate. Federal agencies should move forward now on this powerful technology or risk being left behind. They have the chance today to move the needle on training, customer service, field operations, knowledge sharing, and situational awareness, with virtual experiences elevating performance across diverse missions.

Given the rapidly evolving XR landscape, agencies should consider teaming with a partner that has a depth of experience in the field, and a track record of delivering needed technologies, particularly at scale. Prototyping and testing technology is one facet of the process, but for a robust federal implementation, scalability and sustainability over the long term will be the hallmarks of success. Agencies will want to team with industry partners who have successfully navigated that territory.

XR will add new dimensions to how the federal government can operate, with enhanced insights and immersive experiences improving how employees can train, execute in their jobs, and interact with customers and each other. And the truth is – XR is becoming increasingly ubiquitous and the convergence of edge and cloud architecture with high-bandwidth, low-latency 5G networks will further accelerate the widespread adoption of XR experiences. Agencies can’t ignore it and should look now at how it can best be integrated in their missions. By adopting this emerging technology earlier on, agencies will be able to experiment with and evolve their usage as new capabilities emerge and XR’s momentum builds.

¹ Source: IDC, Worldwide Augmented and Virtual Reality Spending Guide
² The Extended Reality Opportunity Today: Your Employees, Forrester Research, Inc., February 21, 2020