Preamble

The following work was presented at the Hydrographic Conference HYDRO 2024, 5–7 November 2024, Rostock-Warnemünde, Germany in the oral session Data fusion and management.

1 Introduction

TrueOcean is an ocean data platform that is being developed by north.io GmbH, which is a German based software company. The TrueOcean platform serves as a comprehensive data warehouse designed to simplify data management, sharing, and collaboration. Furthermore, it provides a foundation for data processing and analysis using scalable computational resources, enabling efficient extraction of valuable information directly from raw sensor data.

Managing marine data poses significant challenges due to the absence of standardized data formats for the diverse range of sensors used to acquire it. Common sensor types include multibeam echosounders, side-scan sonars, magnetometers, sub-bottom profilers, as well as lidar, optical, and SAR satellite sensors. Each sensor type can again store data in a different format, such as *.gsf, *.xtf, *.all, or *.s7k for multibeam echosounders. This variety of data formats makes interoperability and scalability difficult (Fig. 1, Raw Data). In addition, these formats are not optimized for scalable cloud environments, highlighting the need to unify various sensor data into a cloud-optimized format for efficient storage and streamlined data handling (Fig. 1, Preprocessed Data). Furthermore, proper metadata management is equally important, with alignment to existing standards (ISO, 2019; INSPIRE, 2025) being a critical objective. Another key consideration for marine data, as with any geospatial data, is the visualization of locations on a map. Efficient indexing of geospatial data is essential to make files easy to locate, access, and use within the platform.

Once data is effectively organized, sharing and collaboration typically become the next priorities, given that marine projects often involve multiple stakeholders. TrueOcean facilitates sharing through its built-in map functionality, common protocols like SFTP/RSYNC, as well as via standardized APIs such as Web Map Services (WMS) and Web Feature Services (WFS; Fig. 1, Publish).

Unifying data into a cloud-optimized format also enables integration with scalable computing environments, supporting data processing and advanced analysis workflows (Fig. 1, Computation). A unified and open-source data format offers the added advantage of eliminating vendor lock-in, ensuring interoperability across workflows and compatibility with a variety of tools without requiring proprietary software.

By combining functionalities for data management, sharing, collaboration, and scalable processing/analysis, the TrueOcean platform establishes itself as a central hub for accessing marine data and extracting actionable insights for a wide range of use cases.

2 Technology

The TrueOcean platform is designed as a SaaS (Software-as-a-Service) and hence consists of several microservices that interact with each other. These microservices allow for a more flexible structure regarding updates, maintenance, and the implementation of new functionalities. The main technical components regarding the different topics “Data Management”, “Data Sharing/Collaboration”, and “Processing/Analysis/AI” are described in more detail below.

2.1 Data management

To manage the diverse set of marine data formats from different sensor types, unification of this data is required. Therefore, a microservice denoted as fconv that can convert the data from the different sensor types to a common data format, the Apache Parquet format (Apache Parquet, 2025), is being developed. The Parquet format is chosen as it is optimized for cloud applications, an open-source data format, constantly maintained due to its founding through the Apache Foundation and it is already widely used in the technology industry. The optimization for cloud applications implies a high compressibility, fast query operations for I/O operations, support of complex nested data structures, interoperability with various languages (e.g., Java, Python, Scala) and big data environments (Apache Spark, Hadoop, Hive), leading to efficient usage for parallel computing. In addition to the data format conversion each file that contains geographical coordinates is indexed using H3 indexing. H3 indexing divides the Earth’s surface into a hierarchical grid of hexagons, which provides an efficient and flexible way to index and organize spatial information (H3, 2025). The division into hexagons reduces edge effects compared to traditional square grids, and it is more natural for representing spherical surfaces. A main advantage of H3 is its hierarchical structure, which allows for varying levels of precision. The zoom levels for the size of a hexagon vary between 0.58 m to 1281 km, which refers to the edge length of the hexagon. A list of currently supported data formats by the fconv service is given in Table 1.

Despite the data itself, metadata is also important to align with the FAIR principles (Wilkinson et al., 2016) and make the data findable, accessible, interoperable, and reusable. There exist different metadata standards for geospatial which are in use by different stakeholders. Common standards are INSPIRE (INSPIRE, 2025) and ISO19115 (ISO, 2019). A vocabulary that is developed in the context of the European Commission’s “Interoperable Europe” initiative to map the metadata attributes is the GeoDCAT-AP 3.0.0 (GeoDCAT-AP, 2025). While this is not fully implemented into the TrueOcean data platform, the goal is to thrive towards interoperability with these existing metadata standards. An additional standard to discover, browse, and query metadata is the Catalog Services for the Web (CSW) defined by the Open Geospatial Consortium (Catalog Services, 2025). The TrueOcean platform allows to interoperate with this CSW standard.

The TrueOcean platform is designed to be deployed on cloud infrastructure, with the option to also be implemented on on-premises systems. Given that marine survey data can range from gigabytes to several terabytes, a scalable storage solution is essential. For this purpose, the TrueOcean platform utilizes Ceph as its distributed storage system (Ceph, 2025). Ceph is an open-source project known for its scalability, allowing data to be stored across distributed systems. It supports multiple storage types, including object, block, and file storage. Additionally, Ceph offers robust fault tolerance and redundancy, ensuring data integrity and preventing loss. When deployed on appropriate hardware, Ceph can deliver high data throughput, making it well-suited for large-scale data processing and ensuring seamless interoperability within the system. Data can be uploaded to the storage location using several common protocols, including HTTP, SFTP, and RSYNC.

2.2 Sharing and collaboration

Having data organized within the TrueOcean platform, sharing and collaboration are commonly the next steps that are required. The user roles available on the platform, which are admin/member/guest allow for different variations on how to share the data. On the platform the data is organized in projects and user defined folder structures within each project. To share and collaborate on data, users can be invited to the respective project with the intended user rights (e.g., view, edit, download). The intended user rights can be defined on a folder and subfolder level. In addition to an invitation of users, the data can also be shared using the same protocols as for the data upload, which are HTTP, SFTP and RSYNC. When a share is created, the credentials to access the data can be shared by the data owner with whoever the data owner wants (Fig. 2, left). For specific geospatial data types, like raster data, standard protocols like WMS and WFS are available to share the data. Also, external data can be integrated into the TrueOcean platform using the WMS/WFS protocols. The data that is being stored in the projects can also be displayed on maps within the platform. Different maps can be created, like projects (Fig. 2, right), while a map can contain data from different projects and vice versa.

2.3 Data processing and analysis

The processing of large-scale data will be handled by the Geodata Processing Engine, currently under development at north.io. Designed as an independent technological foundation, the Geodata Processing Engine is being developed in respect of interoperability with the TrueOcean platform. As its core, the Geodata Processing Engine combines the technologies of Apache Spark (Apache Spark, 2025) and Kubernetes (Kubernetes, 2025) to distribute computing power and horizontally scale data processing tasks across multiple servers. This combination ensures stable performance while mitigating the limitations of single-server setups, such as constrained hardware capacity, and provides a degree of fault tolerance. Kubernetes streamlines resource management and cloud deployment, offering dynamic resource allocation, automatic scaling, and built-in fault tolerance. Meanwhile, Apache Spark automatically translates user-defined algorithms into optimized, scalable computations, efficiently distributing tasks across available resources. Additionally, the Apache Parquet format, used as the unified data format, is optimized for seamless integration with the Apache Spark framework.

The efficiency of algorithm translation and resource allocation also depends on the data structure itself. To maximize the performance of the Geodata Processing Engine, the system automatically minimizes data shuffling and optimizes data storage arrangements for greater efficiency. Examples of algorithms that are being developed for multibeam echosounders range from preprocessing (e.g., offset corrections, navigation interpolation, georeferencing), to data quality control (e.g., data point density, beam footprints), and analytics (e.g., gradient, sphericity).

3 Use cases

3.1 Offshore: Enhancing data management, sharing, and analysis during acquisition

In offshore geophysical surveys, vast quantities of data are collected from a range of sensors, e.g., multibeam echosounders (MBES), side-scan sonar (SSS), sub-bottom profilers (SBP), magnetometers, and seismic streamers. Managing these datasets in real time is critical for ensuring survey integrity and immediate decision-making.

3.1.1 Data management

Traditional data acquisition methods have often depended on local storage onboard survey vessels. Such setups come with inherent risks including data duplication, limited storage capacity, and the potential for loss during adverse conditions. TrueOcean addresses these challenges by providing a cloud-based data management framework that automates the ingestion of raw data. As data streams in from various sensors, TrueOcean assigns structured metadata tags, ensuring that every dataset is immediately backed up and catalogued. This approach minimizes manual intervention and expedites data retrieval, thereby reducing downtime and enhancing operational efficiency offshore.

3.1.2 Sharing and Collaboration

One of the steady challenges in offshore operations is bridging the gap between the vessel and onshore support teams. With TrueOcean, real-time data synchronization becomes a cornerstone of offshore collaboration. Through secure cloud connections, survey teams can share data instantaneously with remote experts and project managers. This real-time sharing means that preliminary assessments and feedback can be provided even before the survey vessel leaves the area, allowing early detection of data gaps or anomalies. The use of standardized data formats within TrueOcean further facilitates interoperability among various stakeholders, ranging from regulatory bodies to subcontractors, ensuring that the same high-quality data is available to everyone being involved.

The rapid evolution of cloud computing has transformed the way geophysical data is acquired, managed, and interpreted. As surveys grow in scale and complexity, robust platforms like TrueOcean serve as a blueprint for integrating scalable data management, real-time collaboration, and advanced processing techniques into every stage of geophysical operations. This article outlines how offshore operations, onshore validation efforts, and cloud-based processing services can benefit from a unified approach that TrueOcean exemplifies.

3.2 Onshore: Validating, managing, and collaborating on survey data for QA/QC

Once raw data reaches onshore processing centers, the focus shifts from acquisition to a thorough validation process. This stage is crucial for ensuring the integrity and quality of the datasets before they are used for final interpretations and reporting. Here, a cloud-based system modeled after TrueOcean’s robust data management protocols plays an indispensable role.

3.2.1 Data management

Onshore operations benefit immensely from a structured, centralized repository for geophysical data. TrueOcean’s architecture supports a wide array of industry-standard file formats, ranging from near-seabed sensor formats to sub-seabed formats like SEG-Y seismic files to GeoTIFF raster images and vector files, allowing for the efficient organization and retrieval of complex datasets. The platform allows linking the numerous documents that needs to be prepared by contractors and client to be geotagged and linked to the same area of interest along with sensor files acquired during operations making these available in near real time for all stakeholders of the project to have the same visibility on the progress of the survey but also leverage the platform to expedite decision making for reruns, infills and compliance with contractual agreements by minimizing conflict during any decision making process.

3.2.2 Sharing and collaboration

Historically, data sharing in onshore environments often meant physically transferring hard drives or using disparate data management systems that did not communicate well with one another. TrueOcean’s cloud-centric approach allows multiple users to access, annotate, and collaborate on the same datasets simultaneously, regardless of their location. Survey contractors, academic researchers, government agencies, and technical experts can all work on a shared platform, ensuring that every piece of data is scrutinized and validated. Furthermore, integrated version control systems maintain a detailed audit trail of every change made to the data, fostering transparency and facilitating error tracking. This seamless collaboration not only accelerates QA/QC workflows but also builds trust among stakeholders through consistent, documented validation processes.

3.3 Data processing: A scalable cloud solution for geophysical data interpretation

As the demands of geophysical surveys continue to expand, the need for a scalable, cloud-based data processing service becomes paramount. Building on the principles demonstrated by TrueOcean, our upcoming processing service is designed to streamline complex workflows while supporting a broad range of file formats and processing techniques.

3.3.1 Data management

The cloud-based processing service leverages a centralized data repository to handle diverse geophysical file formats detailed in Table 1 and extendable to other file formats depending on the scope of work. By integrating TrueOcean’s approach to data management, the service ensures that data, from raw inputs to processed outputs, is stored in a coherent and accessible manner. This organization simplifies not only data retrieval but also the archiving of historical datasets, enabling users to track trends over time or revisit previous survey results with ease. The support for widely recognized file formats coming from near seabed sensors, like S7k, ALL, XTF, Jsf, GSF and sub-seabed formats like SEG-Y, along with formats like XYZ, Geotiff, and SHP underline the system’s commitment to interoperability with existing geospatial and survey software.

3.3.2 Sharing and collaboration

A key innovation of the processing service is its web-based dashboard, which exemplifies TrueOcean’s commitment to transparency and user empowerment. The dashboard provides stakeholders with real-time insights into processing progress and allows for the secure submission of raw data. Through role-based access controls, the system ensures that every user—from project managers to field engineers—has access only to the data relevant to their responsibilities. This targeted sharing minimizes the risk of data misinterpretation and helps maintain a secure processing environment. Additionally, the service features automated report generation for survey files troubleshooting in PDF, GIS-compatible files, and interactive 3D models. These capabilities facilitate a comprehensive review of survey results without overwhelming users with technical details.

3.3.3 Data processing and analysis

The Geodata Processing Engine will expedite critical tasks such as bathymetric contour generation, seismic visualization, and the creation of data deliverables highlighting quality metrics like TVU, THU and TPU of bathymetric survey and statistical indicators of seabed conditions and morphometric surfaces.

To ensure the efficiency of these tools, we are conducting a comprehensive benchmark testing phase to evaluate the time efficiency and quality enhancements offered by the Geodata Processing Engine compared to traditional processing methods. Preliminary results are promising, aligning with our expectations and providing valuable insights as we rigorously stress-test our architecture to identify and resolve potential issues. This meticulous approach aims to deliver a reliable and resilient solution that meets the evolving demands of the renewable energy sector.

Traditional desktop solutions, while historically reliable, face significant challenges in handling the increasing volume and complexity of data generated by modern sensors. Scaling computational power to accommodate this growth is essential, especially as geophysical scopes of work in the offshore industry become more intricate and keener to simultaneous operations approaches. The Geodata Processing Engine, being integrated into the TrueOcean platform, addresses this challenge by leveraging distributed computing capabilities, allowing for horizontal scaling of tasks. This approach enables the efficient processing and analysis of raw data to actionable deliverables with less turnaround time than its counterpart. In the context of renewable site characterization, timely data processing is crucial for informed decision-making related to permitting, construction design, and logistical planning during the installation and throughout the whole lifecycle of an offshore windfarm, the challenge has become data centric.

4 Conclusion and outlook

The integration of cloud-based platforms into geophysical data workflows signifies a transformative shift in offshore and onshore operations. Platforms exemplifying effective data management, seamless sharing, and advanced processing capabilities within a scalable framework are at the forefront of this evolution. Offshore, real-time data management and analysis not only safeguard data integrity but also enhance operational decision-making. Onshore, structured data management and collaborative validation ensure that each dataset is rigorously vetted before informing critical interpretations. Moreover, cloud-based processing services open new avenues for automated, AI-enhanced data analysis, capable of handling diverse file formats and substantial datasets.

By aligning real-world use cases with these functionalities, it becomes evident that the future of geophysical data management lies in scalable, interoperable systems supporting every stage of the survey lifecycle. As the offshore geophysical industry continues to embrace digital transformation, such platforms will be instrumental in delivering high-quality, actionable insights while maintaining the operational agility required in today’s dynamic survey environments.

In addition to these capabilities, addressing the challenge of vast amounts of valuable data stored in isolated archives by Hydrographic Offices and private entities is crucial. Recognizing the potential of these untapped resources, establishing a marketplace for data that has been acquired and resides in silos can facilitate the optimization of survey planning across various regions worldwide by providing access to historical datasets. Consequently, the hard work and resources invested in acquiring these insights are preserved and utilized, preventing them from fading into obscurity.

By transforming dormant data into accessible assets, such initiatives not only enhance the efficiency of future surveys but also honor and leverage past efforts, ensuring that valuable information continues to contribute to advancements in geophysical research and operations.

Acknowledgements

We like to acknowledge the entire team at north.io GmbH for development of technology used for the TrueOcean platform and the Geodata Processing Engine.

References

Apache Parquet (2025). Apache Parquet Documentation. The Apache Software Foundation. https://parquet.apache.org/docs/ (last accessed 10 March 2025).

Apache Spark (2025). Unified engine for large-scale data analytics. The Apache Software Foundation. https://spark.apache.org/ (last accessed 11 March 2025).

Catalog Services (2025). OGC Catalogue Services 3.0. Open Geospatial Consortium. https://www.ogc.org/publications/standard/cat/ (last accessed 10 March 2025).

GeoDCAT-AP (2025). GeoDCAT-AP 3.0.0. https://semiceu.github.io/GeoDCAT-AP/releases/3.0.0/ (last accessed 11 July 2022).

Ceph (2025). Ceph – The Future of Storage™. Ceph Foundation. https://ceph.io/en/ (last accessed 12 March 2025).

H3 (2025). H3 indexes points and shapes into a hexagonal grid. https://h3geo.org/ (last accessed 10 March 2025).

INSPIRE (2025). Metadata Technical Guidelines. European Commission, European Union. https://knowledge-base.inspire.ec.europa.eu/metadata-technical-guidelines_en (last accessed 10 March 2025).

ISO (2019). ISO 19115-1:2014 Geographic information – Metadata. Part 1: Fundamentals. International Organization for Standardization, ISO/TC 211, 35.240.70. https://www.iso.org/standard/53798.html (last accessed 31 March 2025).

Kubernetes (2025). Kubernetes Documentation. The Linux Foundation ®. https://kubernetes.io/docs/home/ (last accessed 11 March 2025).

Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18