It is challenging for navigators to constantly translate data from equipment to the world outside while engaged in maritime operations. Augmented Reality (AR) is a promising new technology that might improve this situation by directly overlaying the users’ field of vision with digital content related to the world outside
Research suggests AR may improve operator performance and situation awareness by supporting navigation focus, reducing information overload and linking real[ds_preview] and digital information. There exist early examples of AR in maritime operations, but it has not yet been widely applied.
At a first glance it would seem like AR would be beneficial in maritime operations. Yet, there are multiple barriers that make safe and efficient implementation a challenge for maritime operations. Issues like ergonomics, contrast and the ability to compensate for ship motions are well known issues that need to be solved. As the hardware gradually improves we expect that AR will be feasible in an increasing number of operational scenarios. However, even in the case of improved AR hardware there is still a need to adapt AR systems to maritime user needs and their environmental and technological context. Our work addresses three challenges that the industry should overcome in order to make AR more useful and safer in maritime operations.
Three challenges
First, a ship bridge is commonly made up of many different systems delivered by multiple actors. Since this is unlikely to change in the imminent future, there is a need to address AR as a user interface platform that multiple independent systems can share. Second, since AR applications in effect use the entire world as a canvas to display information, there is a need to define rules for how AR applications may be superimposed over the real world without interfering with each other, or with important objects in the real world. Third, since the operations of a ship and the tasks the crew is engaging with are dynamic and related to what is happening inside and outside the ship, there is a need for rules that enable AR to adapt to the context where it is used.
Set of rules
Overall, there is a need for developing a system architecture that enables an efficient use of multiple AR applications in ship bridges. This article shows a proposal for how to meet this need, based upon research carried out in the SEDNA project funded by EU. The core of our proposal is a set of rules for designing user interfaces (UIs) for AR applications based upon the OpenBridge open source design system currently being collaboratively developed and implemented by a large user community in the maritime industry.
The AR UI architecture proposal is built upon three main sets of rules that guide:
1. What type of information may be displayed in AR,
2. How to specifically display and cluster the information based upon its content, and
3. How to make the information reactive to the position of the user on the ship bridge.
For the first set of rules governing the type of information to be displayed, we developed a number of predefined AR components. For example, an »App display« renders an existing application (ECDIS, radar, etc) in its entirety in AR. »AR Map« can be used to display location-based information on a 2D or 3D map. »Annotation« shows a small piece of information connected to a physical object in the world – for example information about a vessel passing by. Together, the predefined AR components enable to display information in ways that are predictable for the users and well adapted to the specificities of the user’s work environment.
Because there is a finite space around users, we developed a second set of rules to govern where to display information, and how to cluster it. For example, the water surface should have a limited overlay because it might occlude important objects like small boats. Other areas might be less critical for operation, such as the sky, or the bridge´s ceiling and walls.
Information and masked areas
To deal with this problem, we proposed that specific AR components should be displayed only in specific »information areas«. For example, we propose that the area located near the horizon should only contain annotation components, keeping the annotations close to the objects on the water while not overlapping them. The area located under the horizon band should contain only ocean overlay components.
This is a critical area where there is a high probability that the graphics may occlude important objects in the water. We also proposed to define »masked areas« where no information may be displayed, and we defined rules for the free positioning of certain AR components.
Finally, the proposal acknowledges the fact that the users of the bridge will need to have information follow them while they move around in the bridge to engage with different operational tasks.
To deal with this challenge we proposed »AR zones« where the structure of the information areas as well as their informational content change as users enter and leave these zones. We created three zones implemented in a typical bridge shaped with a fore and aft part, and two wings. The »workplace zone« is centred on traditional workplaces in the bridge, such as the bridge wings or navigation stations.
»AR« zones
In this zone, we propose that specific AR components may be pinned to locations predefined by the user, thus creating a versatile and tailormade work station. The »by window zone« is any zone close to the windows. The »in-between zoneq is any other area inside the bridge. Since the physical, collaborative and operational conditions in these areas are hard to predict, we propose that these zones allow for a more flexible use of the digital space, still following a number of rules within each zone.
Field tests
Together, the design principles presented in our proposal make up an early example of UI system architecture for maritime AR. We are currently testing the AR principles with users in the field onboard ships and in realistic simulations of operations in our lab using virtual reality. Our objective is to improve the AR system architecture to make it part of the OpenBridge design guideline. In doing so, we hope to make AR an integrated part of maritime multivendor workplaces and fit seamlessly with screen-based systems. This is important requirement for making AR a safe and efficient part of maritime operations.
Kjetil Nordby Institute of Design, Oslo School of Architecture and Design
