\documentclass[conference]{IEEEtran} \IEEEoverridecommandlockouts % *** CITATION PACKAGES *** % \usepackage{cite} % *** GRAPHICS RELATED PACKAGES *** % \ifCLASSINFOpdf \usepackage[pdftex]{graphicx} % declare the path(s) where your graphic files are \graphicspath{{./figures/}{./images/}{./}} % where to search for the images % and their extensions so you won't have to specify these with % every instance of \includegraphics \DeclareGraphicsExtensions{.pdf,.jpeg,.png,.jpg} \else % or other class option (dvipsone, dvipdf, if not using dvips). graphicx % will default to the driver specified in the system graphics.cfg if no % driver is specified. % \usepackage[dvips]{graphicx} % declare the path(s) where your graphic files are % \graphicspath{{../eps/}} % and their extensions so you won't have to specify these with % every instance of \includegraphics % \DeclareGraphicsExtensions{.eps} \fi % *** PDF, URL AND HYPERLINK PACKAGES *** % \usepackage{url} % correct bad hyphenation here \hyphenation{op-tical net-works semi-conduc-tor} \begin{document} \title{A Virtual Reality Video System for Deep Ocean Remotely Operated Vehicles} % author names and affiliations % use a multiple column layout for up to three different % affiliations \author{ \IEEEauthorblockN{Eric J. Martin\IEEEauthorrefmark{1}, Benjamin Erwin\IEEEauthorrefmark{1}, Kakani Katija\IEEEauthorrefmark{1}, Amy Phung\IEEEauthorrefmark{2}, Everardo Gonzalez\IEEEauthorrefmark{2},\\ Susan Von Thun\IEEEauthorrefmark{1}, Heidi Cullen\IEEEauthorrefmark{1}, and Steven H.D. Haddock\IEEEauthorrefmark{1}}\\ \IEEEauthorblockA{\IEEEauthorrefmark{1} Monterey Bay Aquarium Research Institute\\ Moss Landing, California\\ Email: emartin@mbari.org\\}\\ \IEEEauthorblockA{\IEEEauthorrefmark{2} Olin College of Engineering\\ Needham, Massachusetts\\ } } % conference papers do not typically use \thanks and this command % is locked out in conference mode. If really needed, such as for % the acknowledgment of grants, issue a \IEEEoverridecommandlockouts % after \documentclass % for over three affiliations, or if they all won't fit within the width % of the page, use this alternative format: % %\author{\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1}, %Homer Simpson\IEEEauthorrefmark{2}, %James Kirk\IEEEauthorrefmark{3}, %Montgomery Scott\IEEEauthorrefmark{3} and %Eldon Tyrell\IEEEauthorrefmark{4}} %\IEEEauthorblockA{\IEEEauthorrefmark{1}School of Electrical and Computer Engineering\\ %Georgia Institute of Technology, %Atlanta, Georgia 30332--0250\\ Email: see http://www.michaelshell.org/contact.html} %\IEEEauthorblockA{\IEEEauthorrefmark{2}Twentieth Century Fox, Springfield, USA\\ %Email: homer@thesimpsons.com} %\IEEEauthorblockA{\IEEEauthorrefmark{3}Starfleet Academy, San Francisco, California 96678-2391\\ %Telephone: (800) 555--1212, Fax: (888) 555--1212} %\IEEEauthorblockA{\IEEEauthorrefmark{4}Tyrell Inc., 123 Replicant Street, Los Angeles, California 90210--4321}} % use for special paper notices %\IEEEspecialpapernotice{(Invited Paper)} % make the title area \maketitle % As a general rule, do not put math, special symbols or citations % in the abstract \begin{abstract} Telepresence has always been a core aspect in the design and use of Remotely Operated Vehicles (ROVs) for deep-sea ocean exploration since they have replaced manned submersibles as the preeminent technology for deep sea research. Operators and users of these platforms require significant training to be able to navigate the environment through a multitude of sensors and two-dimensional video presentations. In this arrangement, conventional video cameras do not provide spatial context of study objects since they are flattened in presentation, limiting scientific observations. Display of these many video feeds also necessitates construction of large and costly control rooms, which in turn requires space on constrained vessels. Using multiple two-dimensional perspectives force operators to develop spatial relationships to perform manipulations. A new camera system has been developed at the Monterey Bay Aquarium Research Institute through a collaborative development with Blue Ring Engineering. It is deployed from an ROV, and designed specifically for generating video for virtual reality head-mounted displays. The resultant system facilitates collection of three-dimensional, 180° field-of-view stereo video delivered in real-time to the control room for science class ROVs. The imaging system, ROVVR, is a single 4000 m depth-rated housing with two 4K resolution cameras with fisheye lenses spaced at near-human interpupillary distance. As part of this effort, we have developed a Virtual Control Room (VCR), which presents the three-dimensional VR feed, with two-dimensional standard ROV video feeds as foreground panels, along with contextual data using augmented reality. Together, our software and hardware solutions can be used to improve vehicle piloting, reduce the time it takes for subsea manipulations, in-situ coarse size estimations, and improve portability of deployed systems. \end{abstract} \section{Introduction} Since 1989, the Monterey Bay Aquarium Research Institute has conducted science operations in the deep sea, employing remotely operated vehicles as a primary exploration and discovery platform. These teleoperated robots help scientists explore the deep ocean more safely and efficiently than their predecessor, the manned submersible \cite{Sharp1977}. Scientific ROVs are required to provide the highest fidelity measurements across all sensors. No sensor engages the attention of operators (or pilots) and researchers more than the video systems on these platforms. These cameras are typically broadcast-quality sensor modules fit to lenses that provide a wide view ($>80^\circ$) with zoom capabilities \cite{Robison2017}. ROV video has been a consistent source of materials used in outreach and ocean engagement as well. These videos have inspired stewardship, conservation, and a general appreciation for the deep ocean \cite{DelGado}. A common problem for all ROVs is that these wide angle cameras still do not provide enough context for the pilot to be truly aware of the entire subsea environment. The current solution to this problem is to judiciously use more cameras, placed where space allows on the vehicle, and aimed at providing video for regions of interest. ``Fisheye'' ultra-wide cameras can be used to improve subsea vehicle context, however they have proven difficult to interpret when presented on two-dimensional video monitors due to distortion effects. Through regular calibration, these effects can be somewhat removed, but complications in manufacturing remain \cite{Hughes2010}. Multiple cameras as a solution forces a user to divert their attention across many monitors in the teleoperation space, known as the ROV control room (Figure \ref{fig:controlroom}). The user must also have the ability to piece together these disparate video sources in such a way as to provide spatial awareness for the subsea vehicle. %Since light sources are limited, they are often aimed to illuminate the space optimally for a single camera perspective. Hence, the peripheral cameras suffer from poor lighting. %An added complication is that marine environments, especially those in the Monterey Bay National Marine Sanctuary \cite{NOAAOfficeofNationalMarineSanctuaries2019}, are filled with particulates that create optical backscatter, which limits the effective amount of light that can be sourced from the constrained frame of an ROV. \begin{figure}[btp] \centering % avoid the use of \begin{center}...\end{center} and use \centering instead (more compact) \includegraphics[width=\columnwidth]{images/control_room.jpg} \caption{The ROV control room aboard the R/V \textit{Western Flyer}.} \label{fig:controlroom} \end{figure} Here we present a system (ROVVR) that combines emerging Virtual Reality (VR) tools with science-class remotely operated vehicles (ROVs). %Both technologies offer the opportunity for telepresence, given recent advances in high quality camera miniaturization, high-speed computing, and head-mounted displays. In Section \ref{sec:currentefforts}, we present limitations of conventional ROV video displays, which motivate the development of ROVVR. A discussion of the design and integration of the system is presented in Section \ref{sec:camera}. Operational benefits from using VR are discussed in Section \ref{sec:controlroom}. Section \ref{sec:motivations} will present use cases and improvements to both human-in-the-loop control and real-time science tools, as well as the impact VR will have on ocean engagement and outreach. \subsection{Previous Efforts} Since ROVs are fundamentally a teleoperated platform, efforts to bring underwater video systems into head mounted displays have been attempted before. While these efforts have showed great promise \cite{Iii1993}, they have not gained community-wide adoption due to earlier technology limitations in ROV video quality and head mounted display sophistication. More recently, many efforts have taken place on small ROVs, leveraging the monoscopic video commonly available \cite{7404466}. A common difficulty in the use of head mounted displays is the development of motion-sickness in operators. This fatigue can be associated with multiple factors: (1) degree of motion capture, (2) the interpupillary distance (IPD) of the display being well matched to the user, and (3) cameras whose inter camera distance (ICD) best approximates human IPD \cite{Best1996, Wann1995}. These findings were thoroughly explored during the first boom of head mounted displays (HMDs) in the 1990’s. Since the reinvigoration of virtual reality in the domestic entertainment market, and the wide availability of consumer grade head mounted displays, it is clear that earlier technological limitations (e.g., advent of compact, high quality video sensors and video displays) have been reduced significantly. Improvements to graphics processing units are also critical to reducing latency, which can also contribute to VR fatigue. %In addition, a number of other efforts have begun to leverage the newest generation of head-mounted displays to provide new haptic control modes for robots, and ROVs in particular \cite{Martins2015, Solstad2016}. These projects have focused on the possibility of building better human-interface devices, and will influence future work for this effort. \subsection{Conventional ROV Video Challenges}\label{sec:currentefforts} In order to access deep marine waters of 4000 m, MBARI uses ROVs that have changing tool-sleds that are dependent on the dominant science needs of principal investigators. The VR efforts at MBARI have initially involved ROV \textit{Doc Ricketts} (Figure \ref{img:docricketts}) as well as ROV \textit{MiniROV}, and can be operated independent of variable tool-sleds for system flexibility \cite{MontereyBayAquariumResearchInstitute2019}. More generally, ROVs require propulsion in order to navigate their environments, and thrusters provide control through 5 degrees of freedom. Most ROVs are designed to maintain neutral buoyancy to allow for maximum authority in propulsion. Strategic placement in flotation and ballast help create stable vehicles with strong righting moments. Another common aspect of science-class ROVs are manipulator arms, to aid in sample collection, site investigations, and other physical tasks \cite{Sivcev2018}. Particularly relevant to deep sea robotics, no light penetrates below ~200-meters (aphotic zone)\cite{TrujilloAlanP2008Eoo}, so all illumination for navigation and observations must be provided by the ROV itself \cite{Ballard1991}. Power and telemetry are provided for vehicles at these depths through a single tether with conductors and fiber-optic cables, respectively. \begin{figure}[tb] \centering \includegraphics[width=\columnwidth]{ricketts_nosled} \caption{The ROV \textit{Doc Ricketts}.} \label{img:docricketts} \end{figure} %%Figure xx: The ROV Doc Ricketts. The fiber-optics cabling inside these ``tethers'' provide bidirectional communication and allow for COTS telemetry solutions to transceive a multitude of electrical information both down to the ROV (control) and up to the ship (sensors, video). Common sensors on modern ROVs that are necessary for successful operations include inertial navigation sensors (INS), compasses, pressure sensors (depth), and tracking beacons (for subsea geolocation). Acoustic sensors provide scalar measurements such as altitude, and also three-dimensional target imaging, operating similarly to shipboard radar technologies. Finally, many lower-quality cameras are placed around higher-definition ones to provide a complete visual picture for an operator to monitor. ROVs often are designed for all interactions to take place on the front of the platform. As such, most light sources are oriented in the same direction. To help extend the utility of the limited high-definition cameras, they are often placed on pan-and-tilt units to help extend the field of view. These actuators commonly do not respond quickly and smoothly, and are often controlled separately from the yaw of the vehicle, making target tracking difficult. Also, the light distribution in the front of the ROV is often set statically, which helps provide the most even light pattern for targets in the center axis of the ROV. Movement of the focal point of these primary cameras often pushes subjects outside of optimal illumination. Many of these shortcomings can be addressed by utilizing modern applications of virtual reality. \section{A Deep Sea 180° 3D VR Camera}\label{sec:camera} In 2019, we collaborated with Blue Ring Engineering, a virtual reality company specializing in underwater 3D and VR imaging technologies, to design a system that will generate a live VR feed to a HMD for use in ROV operations. The system also provides functionality for onboard recording video, providing content for future science and outreach products. Our ROV-compatible VR system uses a two-camera setup, each equipped with a fisheye lens capable of providing 180 hemispherical degrees of coverage (Figure \ref{img:camera}). The VR system is mounted to the upper frame of ROV \textit{Doc Ricketts} (Figure \ref{fig:rovdiag}). This position optimally places the system to provide a complete view of the front of the vehicle and the illuminated scene. The video feeds from each camera are synced and then converted to a fiber optic signal that is transmitted on specific wavelengths of light through ROV \textit{Doc Ricketts's} fiber optic system. Power and ethernet communications are provided by the science interface module on the ROV. In the ROV control room, the signal from each camera is converted back into an HD-SDI signal and sent to a capture and playback card (Blackmagic). This card saves the full-resolution 4K video feed from each camera for later processing and archival purposes, and streams a real-time 2K video feed to a high-performance computer. The virtual control room system described in Section \ref{sec:controlroom} combines these feeds in real-time and displays them to create a 3D environment for a VR headset. This workflow is depicted in Figure \ref{fig:workflow}. \begin{figure}[htb!] \centering \includegraphics[width=\columnwidth]{images/cam_merged.jpg} \caption{The ROVVR system (a) and a sample image (b) taken in the MBARI Test Tank facility.} \label{img:camera} \end{figure} %%Figure XX: The ROV-VR system (a) and a sample image (b) taken in the MBARI Test Tank facility. \subsection{The Virtual Control Room}\label{sec:controlroom} % Mock up % \begin{figure}[tb] % \centering % \includegraphics[width=\columnwidth]{vr_controlroom} % \caption{A mock up of the proposed virtual reality control room head mounted display interface.} % \label{fig:controlroom} % \end{figure} % Screenshot of app \begin{figure}[tb] \centering \includegraphics[width=\columnwidth]{images/rov-vr-snapshot.png} \caption{View of the virtual control room from within the VR headset. Heading is displayed across the top, attitude is displayed in the lower right-hand corner, the tether turns count is displayed on the bottom, and the ROV depth and distance from its tether weight is displayed on the left. Hand tracking enables these foreground displays to be easily rearranged.} \label{fig:virtualcontrolroom} \end{figure} % Original Text: %Currently, modern ROV control rooms require the use of dedicated ship space, multiple computers, monitors, and routers, along with all the infrastructure and technical know-how required to operate them. Modern VR environments can potentially provide a way of combining the multiple camera feeds, telemetry, and data management systems within a virtual space. With the recent additions to the VR landscape of hand and eye tracking \cite{Beattie2015}, it is now not only possible for an ROV pilot, scientist, or other party, to interact with a real world remote environment, but also do so with their own customised virtual layout, and potentially collaborate within a shared virtual space. Currently, modern ROV control rooms require significant amounts of dedicated ship space for multiple computers and monitors, where each computer-monitor pair displays a single camera feed or graphic visualization of telemetry data. Using a VR headset condenses the hardware necessary for ROV piloting into a single desktop and VR headset. A VR-based control room enables pilots to leverage the wide field view footage from the fisheye cameras more effectively since they can view the footage without the distortion that results from projecting it onto a 2D screen. Therefore, we developed a VR-based control room (VCR) in Unity that overlays real-time ROV telemetry data on an immersive 1:1-scale stereoscopic view from the camera system described in Section \ref{sec:camera}. In the existing control room, ROV telemetry data (e.g. heading, depth, altitude, etc.) is transferred from the ROV to the ship through a fiber optic cable. Once onboard the ship, the data is converted into Lightweight Communications and Marshalling (LCM) \cite{5649358} packets that are distributed across the ship's local network. The control room desktop computer receives these packets and overlays them on the stereoscopic view created by the stereo cameras, which makes the data viewable from the VR headset. This data pipeline is illustrated in Figure \ref{fig:workflow}. In order for the video feeds to be displayed properly in VR, two important pre-processing steps are needed for each camera feed: undistortion and reprojection. The undistortion step uses a third-degree polynomial to account for imperfections in the fisheye lens and create an image that closely resembles the ideal fisheye projection. The reprojection step takes the adjusted image and reprojects it to a hemispherical projection using Bourke's process \cite{bourke2009}, which is needed to properly display it in the headset. The reprojected image from the right camera is used to create the virtual environment for the right eye, and similarly the reprojected image from the left camera is used for the left eye. The cameras are mounted as close to the average person's IPD as physical constraints from the pressure housing would allow; however, since each person's IPD is slightly different, an x and y offset is added to each image to mitigate the ``double-vision'' effect that results from this difference. The result is a 3D, 1:1-scale stereoscopic view in VR that provides pilots with depth perception, and gives them an immersed vehicle perspective. There are also numerous affordances associated with integrating the telemetry data overlays into the VR environment. In the VCR implementation, we used the Leap Motion VR Developer kit to enable pilots to rearrange the virtual telemetry data overlays with their hands. This adds flexibility to how pilots display data, improving upon the fixed nature of the monitors in modern control rooms. The virtualized environment also lets pilots save individual display preferences, allowing them to quickly load their preferred configuration when starting their piloting shifts. While existing control rooms are limited to the 2D displays for telemetry data, the VR control room can utilize 3D features to create more intuitive interfaces. In our implementation, we utilize this to display heading as a 3D ring and display attitude by changing the orientation of a miniature 3D model of the ROV (Figure \ref{fig:virtualcontrolroom}). % TODO: add a paragraph on future work? Though this is also covered in the motivations section % This was the start, but there are a number of other benefits VR can have with some more work on implementation. % Modified The current iteration of the VCR can allow for significant cost savings by reducing the amount of equipment, ship space, and cognitive load posed by traditional ROV control rooms. Additionally, it provides anyone in the virtual control room with increased situational awareness and the ability to tailor their own work environment based on their professional needs. Utilizing the 3D panoramic camera system as a backdrop to telemetry overlays allows for the virtualization of data placed within a real setting (Mixed Reality). Future iterations can make improvements by overlaying more complex information such as distant georeferenced waypoints, historical data, and topographical maps on top of the panoramic backdrop. The current application is limited to a single user in VR at one time. A companion application has been developed that allows other users outside of the HMD to view and collaborate with the VR user. A simple pointer tool exists as well that can be used in a way to ``paint'' targets of interest for the user in virtual reality. %The camera itself also provides the potential to create coarse 3D depth maps of the scene it is filming, which can provide volumetric measurements through photogrammetry \cite{Rossi2018}. % Old figure % \begin{figure}[bt] % \centering % \includegraphics[width=\columnwidth]{rovvr-product-diag} % \caption{Video workflow in the proposed virtual reality system.} % \label{fig:workflow} % \end{figure} % New figure - updated data flows to the best of my knowledge % Link to edit figure: https://drive.google.com/file/d/1ma8E1eYNdt1g_Ozg5TyTmEgHwdCpeslO/view?usp=sharing \begin{figure}[bt] \centering \includegraphics[width=\columnwidth]{figures/stereo-data-Rev2.png} \caption{Data pipeline for the virtual reality system.} \label{fig:workflow} \end{figure} %%Figure 4: Video workflow in the proposed virtual reality system. % Original % \begin{figure*}[btp] % \centering % \includegraphics[width=\textwidth]{rov_diag_frame_cropped.png} % \caption{The ROV \textit{Doc Ricketts} (a) front and (b) side views. The proposed field of view for a new VR camera suited to the available light is shown.} % \label{fig:rovdiag} % \end{figure*} \begin{figure*}[btp] \centering \includegraphics[width=\textwidth]{images/rov-diag-v1.jpg} \caption{(a) The ROV \textit{Doc Ricketts} with the VR camera system. (b) Front and (c) side views of the mount are shown with annotations denoting the relative positions of the lighting and HD camera, and the new cameras' field of view.} \label{fig:rovdiag} \end{figure*} \section{Potential Applications}\label{sec:motivations} This generation of subsea hemispherical stereo video can be leveraged for new scientific insights and outreach, and further work in this area will be a primary focus now that the system is fully integrated onto the ROV. Our multidisciplinary team of engineers and scientists have interests in a number of areas, including exploring improvements to vehicle control, sampling, and in-situ video analysis products. In addition, a number of other efforts have begun to leverage the newest generation of head-mounted displays to provide new haptic control modes for robots, and ROVs in particular \cite{Martins2015, Solstad2016}. These projects have focused on the possibility of building better human-interface devices. \subsection{ Manipulation Accuracy } Robotic manipulator arms (Figure \ref{fig:rovdiag}) are the primary tool for an ROV pilot in almost all use cases. Stereo vision systems have been adapted onto ROVs in the past since stereo vision gives the user the perspectives needed to bring a manipulator arm to its target in 3D space \cite{Lee2000,Ishibashi2009}. ROV pilots are skilled at leveraging the many perspectives from different cameras to develop sufficient spatial awareness for manipulations. However, these disparate cameras often do not capture all useful views, and operating errors are often made. Some errors merely expend valuable in-water time, but other errors can result in visual loss or damage of delicate samples and even damage to the vehicle due to collisions or entanglement. \subsection{ Decision Support} Aside from time lost to visual scanning of multiple monitors, ROV dives often experience delays due to a lack of spatial awareness of the environment surrounding the vehicle. Due to transmission losses in seawater, ROV lights illuminate roughly 10 meters in front of an ROV. This lack of visibility means that operators must rely on geolocation from acoustic tracking to understand where the vehicle is in space. This new video system, combined with mixed reality tools, should enable the creation of an artificial landscape in the virtual space. This virtual space can present the vehicle relative to a priori obtained seafloor topography, real-time path histories, sample locations, and historical data. Future efforts will involve evaluating whether virtual spaces can support more efficient decision-making during subsea ROV operations. \subsection{Portability} Space aboard expeditionary vessels is limited in a number of ways. Current ROV control rooms require multiple monitors to display information to operate the vehicle. Beyond operators, principal investigators require these views to make decisions critical to guiding ROV operations. However, with virtual reality, the ``top-side’’ equipment needed to display video, software, and control tools can be much reduced and aided by augmented reality data overlays of the stitched footage. This solution could easily increase in scale and number of participants if footage is simultaneously brought into multiple head-mounted displays. \subsection{Science Uses} \subsubsection{Advance a detailed quantitative view of marine communities} Present image-based quantification methods can give an estimate of organisms per time, and a gross approximation of number per volume. To understand species interactions such as predator-prey and mate-finding, this requires knowing the 3D spatial distribution of animals, including nearest-neighbor distances. Calibrated VR footage will enable quantification of a number of features in video, including average distances from organisms in the field of view. This information has the potential to revolutionize our understanding of how animals are interacting in the deep sea, where visual signaling and more covert modes of communication are under-explored. \subsubsection{Enhance understanding of the fluctuations and fate of forage species} Within the captured 3D panoramic scene, our camera system will allow scientists to view a recorded deep water scene with contextual information. For example, our extended goals include augmented reality (AR) software tools and a calibrated camera system. This will allow scientists to measure and potentially overlay that information on items in the scene, enabling quantification of biomass as well as raw abundance estimates. \subsubsection{Map the seafloor in support of biological and geological investigations} Increased support of our work in creating Photogrammetric 3D models of deep water features using existing ROV cameras, will allow us to repeatedly generate detailed information of both biology, and geology (Figure \ref{img:bouldermodel})\footnote{Model can be viewed online at: https://skfb.ly/6KTNJ}. %%Figure xxx: Photogrammetric reconstruction of a large boulder in the Sur Ridge segment of Monterey Bay.\footnote{Model can be viewed online at: https://skfb.ly/6KTNJ} \begin{figure}[tb] \centering \includegraphics[width=\columnwidth]{boulder} \caption{Photogrammetric reconstruction of a large boulder in the Sur Ridge segment of Monterey Bay.} \label{img:bouldermodel} \end{figure} \subsubsection{Devise informatics tools for understanding the ocean and its inhabitants} Post processed 3D video from our system can be overlayed with graphics and text. Visual elements can even become interactive and historical data can be layered. This will allow content viewers the ability to look at any item in the scene, interact with it, and learn more information through “pop-up” text windows or cued audio. Future efforts will involve merging MBARI’s AI-driven computer vision elements to the 3D live view for autonomous identification and sizing of elements that enter the scene \cite{DBLP:journals/corr/abs-2007-00114}. \subsection{Outreach} With the ability to walk around and interact freely within captured media and 3D data, VR has the potential to be the ultimate tool to view the ocean and its processes. This new system will present content that allows MBARI and its collaborators to create immersive experiences that emplace users in this unique and mostly unreachable environment. VR is being widely explored for its use in a variety of education and training experiments. Case studies of its use in classroom environments are readily available, and some even with overlap for issues such as climate change, ocean health, \cite{Markowitz2018} and exploration and discovery \cite{TheHydrous2019}. Better education encourages stewardship of the oceans, a societal need expressed thoroughly by the oceanographic community \cite{Mengerink2014}, and this VR camera system can provide unprecedented access to mostly inaccessible places in the deep ocean. \section{Conclusions} Virtual reality applications have re-entered the public space and continue to become a more mainstream technology, as many HMD vendors and developers build a more established ecosystem of products. Here we have presented how VR will address a number of needs for exploring the deep sea using remotely operated vehicles, thereby addressing gaps in ROV users’ ability to interact fully with the environment in the deep-ocean. The ROVVR camera system will be evaluated for many of the applications presented here. The system as-built does possess some limitations. Stretching a 4K resolution camera across such a large field of view leads to distortions around the extents of the view. Also, targets often seem pixelated as a result of the display limits of the head mounted displays. A common point of feedback from early trials of the system is that user-perceived resolution does not provide enough detail on small targets. These issues can be addressed in part through upgrades to both camera modules and HMDs as next generation devices become available. In our early use of this system, we have observed promising improvements across many aspects of ROV use. ROV pilots have reported a sense of better emplacement in the marine environment, and a higher sense of situational awareness. Those that have performed subsea manipulations have felt the stereo enabled VR video is an improvement over the conventional approaches. We have begun efforts with researchers in human-robot-interface science to quantify those improvements. Collaborating scientists have also expressed enthusiasm for the virtual reality system. Scientists have commented on the utility of a much better sense of scale for the underwater spaces that they study. The ability to observe a wider field of view is of great value when studying complex ecosystems. We do not expect this system to be a panacea for all the challenges to ROV-based ocean exploration, but these motivations were chosen because they best leverage the system that is practical to develop at this time. The system could be scaled, thereby enabling more immersive, 360$^\circ$, three-dimensional experiences. This system, as described, enables the MBARI Virtual Reality team the tools needed to evaluate how to best aim future developments, and explore deep waters in new ways. \section*{Acknowledgments} Support for the ROVVR project at MBARI is provided by the David and Lucile Packard Foundation. We would like to thank the pilots of the ROV \textit{Doc Ricketts} for their continued support. We would also like to thank fellow Olin SCOPE team members Erika Lu, Nathan Shuster, and Cameron Wierzbanowski, and their advisor Dr. Lynn Andrea Stein for their contributions to the VR-based control room. Support for the Olin SCOPE team was provided by La Fondation Dassault Systèmes. \bibliographystyle{IEEEtran} \bibliography{refs} \end{document} % no keywords %\section{Introduction} % no \IEEEPARstart %This demo file is intended to serve as a ``starter file'' %for IEEE conference papers produced under \LaTeX\ using %IEEEtran.cls version 1.8b and later. % You must have at least 2 lines in the paragraph with the drop letter % (should never be an issue) %I wish you the best of success. %\hfill mds %\hfill August 26, 2015 %\subsection{Subsection Heading Here} %Subsection text here. %\subsubsection{Subsubsection Heading Here} %Subsubsection text here. % An example of a floating figure using the graphicx package. % Note that \label must occur AFTER (or within) \caption. % For figures, \caption should occur after the \includegraphics. % Note that IEEEtran v1.7 and later has special internal code that % is designed to preserve the operation of \label within \caption % even when the captionsoff option is in effect. However, because % of issues like this, it may be the safest practice to put all your % \label just after \caption rather than within \caption{}. % % Reminder: the "draftcls" or "draftclsnofoot", not "draft", class % option should be used if it is desired that the figures are to be % displayed while in draft mode. % %\begin{figure}[!t] %\centering %\includegraphics[width=2.5in]{myfigure} % where an .eps filename suffix will be assumed under latex, % and a .pdf suffix will be assumed for pdflatex; or what has been declared % via \DeclareGraphicsExtensions. %\caption{Simulation results for the network.} %\label{fig_sim} %\end{figure} % Note that the IEEE typically puts floats only at the top, even when this % results in a large percentage of a column being occupied by floats. % An example of a double column floating figure using two subfigures. % (The subfig.sty package must be loaded for this to work.) % The subfigure \label commands are set within each subfloat command, % and the \label for the overall figure must come after \caption. % \hfil is used as a separator to get equal spacing. % Watch out that the combined width of all the subfigures on a % line do not exceed the text width or a line break will occur. % %\begin{figure*}[!t] %\centering %\subfloat[Case I]{\includegraphics[width=2.5in]{box}% %\label{fig_first_case}} %\hfil %\subfloat[Case II]{\includegraphics[width=2.5in]{box}% %\label{fig_second_case}} %\caption{Simulation results for the network.} %\label{fig_sim} %\end{figure*} % % Note that often IEEE papers with subfigures do not employ subfigure % captions (using the optional argument to \subfloat[]), but instead will % reference/describe all of them (a), (b), etc., within the main caption. % Be aware that for subfig.sty to generate the (a), (b), etc., subfigure % labels, the optional argument to \subfloat must be present. If a % subcaption is not desired, just leave its contents blank, % e.g., \subfloat[]. % An example of a floating table. Note that, for IEEE style tables, the % \caption command should come BEFORE the table and, given that table % captions serve much like titles, are usually capitalized except for words % such as a, an, and, as, at, but, by, for, in, nor, of, on, or, the, to % and up, which are usually not capitalized unless they are the first or % last word of the caption. Table text will default to \footnotesize as % the IEEE normally uses this smaller font for tables. % The \label must come after \caption as always. % %\begin{table}[!t] %% increase table row spacing, adjust to taste %\renewcommand{\arraystretch}{1.3} % if using array.sty, it might be a good idea to tweak the value of % \extrarowheight as needed to properly center the text within the cells %\caption{An Example of a Table} %\label{table_example} %\centering %% Some packages, such as MDW tools, offer better commands for making tables %% than the plain LaTeX2e tabular which is used here. %\begin{tabular}{|c||c|} %\hline %One & Two\\ %\hline %Three & Four\\ %\hline %\end{tabular} %\end{table} % Note that the IEEE does not put floats in the very first column % - or typically anywhere on the first page for that matter. Also, % in-text middle ("here") positioning is typically not used, but it % is allowed and encouraged for Computer Society conferences (but % not Computer Society journals). Most IEEE journals/conferences use % top floats exclusively. % Note that, LaTeX2e, unlike IEEE journals/conferences, places % footnotes above bottom floats. This can be corrected via the % \fnbelowfloat command of the stfloats package. %\section{Conclusion} %The conclusion goes here. % conference papers do not normally have an appendix % use section* for acknowledgment \section*{Acknowledgment} %The authors would like to thank... % trigger a \newpage just before the given reference % number - used to balance the columns on the last page % adjust value as needed - may need to be readjusted if % the document is modified later %\IEEEtriggeratref{8} % The "triggered" command can be changed if desired: %\IEEEtriggercmd{\enlargethispage{-5in}} % references section % can use a bibliography generated by BibTeX as a .bbl file % BibTeX documentation can be easily obtained at: % http://mirror.ctan.org/biblio/bibtex/contrib/doc/ % The IEEEtran BibTeX style support page is at: % http://www.michaelshell.org/tex/ieeetran/bibtex/ %\bibliographystyle{IEEEtran} % argument is your BibTeX string definitions and bibliography database(s) %\bibliography{IEEEabrv,../bib/paper} % % manually copy in the resultant .bbl file % set second argument of \begin to the number of references % (used to reserve space for the reference number labels box) %\begin{thebibliography}{1} %\bibitem{IEEEhowto:kopka} %H.~Kopka and P.~W. Daly, \emph{A Guide to \LaTeX}, 3rd~ed.\hskip 1em plus % 0.5em minus 0.4em\relax Harlow, England: Addison-Wesley, 1999. %\end{thebibliography} % that's all folks \end{document}