Integration and Visualization of NDE Data in Digital Building Models – A conceptual view

,


Introduction
In recent years, several non-destructive evaluation (NDE) techniques for application in civil engineering (also: non-destructive testing, NDT) have been developed to operational maturity and measuring devices for their application are available. At the same time there is a growing awareness that the state of health and the service life of structures, especially infrastructure constructions, can be improved through regular examinations. For bridges, for example, databases have been developed that record structural data and damage, from which condition grades are calculated for the structures [1].
It is now possible to unite both approaches -NDE techniques and structural databasesin the sense of digital transformation into an all-digital workflow. The task is to combine and integrate the various data for a structure, organize the data, store it permanently and make it accessible, visualize and process the data, and use the data for applications such as inspection planning and life-time prediction.
Digital building models can be the central element here. In the context of Building Information Modeling (BIM) [2], they can represent both the planning data and the condition data of the infrastructure over the entire life cycle. In the utilization phase, the integration of information from structural analysis and monitoring enables a detailed, up-to-date description of the condition of the structure, which can be used as a digital twin. The BIM process is well suited for this task because it is established, appropriate, and structured. It is already used for construction planning and execution and is mandatory in Germany, for example, for major public bridge and tunnel construction projects. BIM is also being applied to incorporate monitoring data from bridges [3]. The application for historic buildings is proposed [4].
NDT-CE 2022 -The International Symposium on Nondestructive Testing in Civil Engineering Zurich, Switzerland,  2 This contribution attempts to analyse NDE procedures for testing structures and to develop a workflow for integrating NDE results into BIM models, paying special attention to evaluation stages, data types, and data interfaces. The BIM process is briefly presented. Requirements for data storage of BIM models are formulated from a NDE perspective, and the possible use of NDE data is addressed. Some approaches for partial implementations are presented that illustrate the possibilities and benefits of digital building models. Finally, topics for further activities, research and development are described. The contribution is presented from a specific German perspective.

NDE procedures for testing structures
There are several options for the examination of buildings. The used methods can be grouped into three categories:  Classic structural assessment including visual inspection, coring, mechanical and chemical testing  Non-destructive evaluation and testing (NDE/NDT), about 10 techniques  Continuous monitoring with stationary sensors for strain, acceleration, crack opening Only NDE is discussed here; however, the full range of data sources must always be considered when designing the preparation and use of digital building models. In the following, all results of NDT and NDE techniques in the field of civil engineering are referred to as NDE results.

The current approach
Today, NDE investigations are mainly initiated on specific occasions, especially when problems arise. The tested structure is described in more or less detail in the tender or in discussions prior to the measurements; often, but not always, plans are available in the form of drawings. During the measurements, a protocol is kept manually in which important parameters of the measurements are noted. Often photos are taken as a supplement. The positioning of the measurement points is done with a folding rule, tape measure, or distance meter and is timeconsuming. After measurement and evaluation, the contractor hands over his NDE results in paper form, sometimes also electronically as a PDF file ( Figure 1). Evaluated data may be added as a spreadsheet (CSV file format). The client draws his own conclusions from the results and initiates subsequent actions. Archiving is carried out independently at different locations.
There are some issues with this approach:  Measurements are often not carried out with foresight, but only in the event of damage.  Most steps in the workflow are manual and analog.  Graphics and images of NDE results are two-dimensional, the results are usually not embedded in the context of the building.  Circumstances and parameters of the measurements are often not reported.  Results can only be interpreted by humans, information about the interpretation of the NDE results can be lost due to chained subcontracts.  Storage and archiving is done by each actor, long-term access is not guaranteed.
NDT-CE 2022 -The International Symposium on Nondestructive Testing in Civil Engineering Zurich, Switzerland,  3 One improvement is the regular inspection of structures such as bridges and tunnels in accordance with DIN 1076 [5], the results of which are stored in the SIB Bauwerke database [1]. Usual result of an evaluation is a condition grade of the structure.

Vendor-specific cloud solutions
Proprietary cloud solutions are available from measurement equipment providers such as Olympus, Proceq and Waygate Technologies. They only support equipment from the respective manufacturer. The software focuses on the analysis and annotation of signals, defect detection and sizing, report generation, and inspection history. Transfer and storage of measurement data in the vendor-operated cloud are supported. Some allow the addition of positional data, the combination of results from different techniques, and the linking of NDE results with CAD (computer aided design) data of tested parts or structural elements and with photos of the test site.
These solutions provide a useful IT infrastructure, but they are closed systems: The systems of different vendors are not interoperable and there are no interfaces for integrating the results into digital building models. Questions arise as to the long-term support for the data, especially when it comes to subscription-based software or equipment changes and possible access to the data by providers.

Building information modeling (BIM)
Building Information Modeling (BIM) is a process that describes methods and tools for efficient digital collaboration in the construction sector. It is being continuously developed by the international standardization organization buildingSMART.
BIM has three main components: Data Structure, Data Exchange Definitions and Terminologies. Within the Data Structure, the digital building information model, referred to NDT-CE 2022 -The International Symposium on Nondestructive Testing in Civil Engineering Zurich, Switzerland,  4 here as the BIM model, is used for the data exchange between all processes and stakeholders. As a schema for data exchange, buildingSMART has defined the Industry Foundation Classes (IFC) as an ISO standard [6]. According to the IFC standard, building data may be stored as ASCII files, e. g., in the form of STEP files [7], XML files [8], or as an ontology, for example via RDF [9] or ifcOWL [10]. The contents includes geometry, for instance boundary representations or constructive solid geometries, and semantic information such as materials, connections, or relationships. The Data Exchange Definitions specify which data is exchanged when between which actors and in which detail. This depends on the respective processes and national requirements. Therefore, buildingSMART does not define a completed standard that specifies the requirements, but provides guidance on how to create an Information Delivery Manual (IDM) [11] and a Model View Definition (MVD) [12]. The IDM defines which actor needs which data at which time. The MVD defines how these requirements are implemented for a specific model use based on IFC; MVDs may be used for later software development.
Especially for an automated exchange of data it is necessary to clearly define the terms to be used. For this purpose, Terminologies are defined using the buildingSMART Data Dictionary (bsDD) which buildingSMART provides for this purpose. The bsDD standardizes the terms similar to a thesaurus. The definitions can be created hierarchically and contain additional conditions or parameters. Different domains are possible in the bsDD for different application areas and country-specific definitions. A bsDD is currently created by prospective users on their own responsibility after training by buildingSMART. Testing software for terms and rules can be developed and implemented on the basis of a bsDD.
In order to implement BIM for a process (e .g., the inspection of a reinforced concrete bridge) or a process class (inspections in general), it is necessary to 1. identify all processes and actors involved, 2. define all necessary data for the exchange between the processes, 3. determine how this data is to be stored in an IFC file and 4. specify which terms are to be used for the data.
Based on these specifications for the BIM process, construction models can then be created, exchanged, and used.
For collaboration on a BIM project, a Common Data Environment is best suited. "The Common Data Environment (CDE) is defined as a common digital project space that provides distinct access areas for the different project stakeholders combined with clear status definition and a robust workflow description for sharing and approval processes" [2]. A CDE contains all necessary data for building related data. Currently, CDEs are primarily used for design, planning, and construction processes of structures. It is desirable that it also be used during operation. All building related data may be included in a CDE system including testing information and results from NDE processes. This approach eases the communication and collaboration between the owner and testing engineers.

BIM workflow for NDE results
The BIM process can be applied to the processing of NDE results. Figure 2 shows an overview of the entire process. It starts with the measurements and their evaluations for the generation of the NDE results which are integrated into the BIM model. The data in the BIM model can then be visualized and used for further applications. In turn, the BIM model can also support the planning and design of NDE measurements, and generated data from the applications can be fed back into the BIM model. This workflow is elaborated in detail in the next chapter. The BIM process requires the definition of actors to tailor processes and data to their respective needs. Actors in this sense are e. g. the test engineer, bridge inspector, structural engineer, stakeholder, site manager, maintenance staff, and database administrator.

A structured view on NDE results
In order to develop a workflow compatible with a BIM model and its use, it is necessary to analyze and structure the processing steps of NDE results. Starting point of the workflow are NDE results as they are available after measurement and evaluation.
Beyond the current procedure, two main aspects have influence on the workflow:  NDE results will be stored in digital databases in a structured way, and the data transfer should preferably be done automatically.  Usage can be diverse and unpredictable at present, later users will need to rely on the validity of the data in order to benefit.
As a consequence, the results must have been created in a traceable manner, be complete and self-contained. This means that the data must contain all the signals, values, results, parameters, and documentation that others need to understand and work with it (data completeness). The quality of the testing and evaluation procedures should be ensured by the use of recognized, validated, and documented procedures. As part of data quality, the measurement accuracy of the NDE results should be provided. It is also desirable to ensure that the original data are unchanged from their original state (data integrity), and that all additions, changes, and editions to the original data are marked and can be traced back to the respective authors (traceability).
NDT-CE 2022 -The International Symposium on Nondestructive Testing in Civil Engineering Zurich, Switzerland, August [16][17][18]2022 6 Both can be ensured by technical measures. These steps make sure that NDE results yield unambiguous output that can be interpreted at any later time in terms of the test objective.
One additional point is the development of an uninterrupted digital process chain throughout the workflow. While most measurement and evaluation parameters are digitally stored by the software together with the signals and values, protocol notes must be converted to a suitable digital data format. The position of the measurement sensors is generally measured manually and documented in analog form. Here, unambiguous and accurate digital logging relative to reference points in the BIM model will become necessary.
In order for NDE results to be used independently of NDE testers in different abstraction levels, further process steps are necessary beyond the measurement. We propose to go through the following essential stages in the workflow ( Figure 3) to make sure that NDE results can be transferred to a BIM model in a meaningful way:  Measurement: Carrying out the measurements according to recognized procedures, providing complete documentation.  Evaluation: Performing additional processing (like transformations, imaging) to achieve valid NDE results. Includes documentation.  Interpretation and assessment: Interpreting the NDE results to draw conclusions in terms of the test objective, naming and assessing the results, possibly employing additional information (e. g. interpreting an ultrasonic indication as a tendon duct utilizing the expected depth, locating a rebar and deciding that it is too close to the surface).  Abstraction: Creating independent objects from the interpreted results, possibly using additional information. These objects will be integrated in the BIM model and provide NDE results as geometric and semantic parts of the structure (e. g. generating CAD models of localized tendon ducts using the projected diameter).
Each of these workflow stages must conform to quality requirements such as documentation and completeness as mentioned before. Now that the NDE results can be integrated into BIM models, an overview of the data types and data formats must be created, as integration mechanisms must be developed for each data type and format. At any point in the workflow, NDE techniques result in data and metadata which can be classified as  Values: Numerical measurement output, e. g. time signals, reinforcement cover, potential values, reconstructed image amplitudes.  Graphics: Graphics illustrating the results, e. g. measurement points, markers, thickness maps, contour lines, imaging results, crack paths, photographs.  Parameters: Parameters describing the measurement, the result, and the circumstances under which both were achieved, e. g. measurement settings, names, dates, labels.  Notes: Explanatory notes, reference to explained objects must be given.
Values and graphics can have different data formats and dimensions: Both can be points and scalars (0-D), vectors, signals, and lines (1-D), tables, images, and planes (2-D), or volume data (3-D) ; see Table 1. Parameters and notes come as numeric values, strings, and continuous text. The data can be stored in a number of file formats, a selection of which is given in the table.
The original measurement data (raw data) as stored by the devices has so far mostly been in a vendor-specific data format and file format that is not disclosed.  All these data must be available in digital form and be given standardized names so that they can be unambiguously allocated.
One way to achieve the use of recognized result forms, data formats and file formats is to use standardized data formats:  DICONDE (Digital Imaging and Communication for Non-Destructive Evaluation) [13] is a NDE data format based on the medical data format DICOM. It includes measures for data safety and integrity, and method-specific modules can be included. DICONDE is still in development and adoption.  OPC UA (Open Platform Communications Unified Architecture) [14] is a framework of interfaces and communication models that warrants semantic interoperability by companion specifications. Measures for data safety are included. OPC UA is used in production environments for automation purposes.
NDT-CE 2022 -The International Symposium on Nondestructive Testing in Civil Engineering Zurich, Switzerland, August [16][17][18]2022 8 The suitability of these data formats for the integration of NDE results into BIM models cannot be concluded today.

Integration in BIM-models
The task is to process the data in the data formats and file formats of the previous chapter. When NDE results are to be integrated in BIM models, the first question is which data formats (some of them are listed in Table 1) are to be supported. Choices are:  Standardization of all data input from all NDE results. This would involve international agreement on standards for many NDE techniques and many devicesa major task.  Development of specific translation procedures for each provider of NDE results.
Each NDE tester would be responsible for providing compliant data, with some parts of the data possibly already being present in a standardized data format.
One could perhaps say that the first option is the long-term goal and the second option is the way to get there. The data interface on the BIM model side is clearly IFC (Figure 4). Currently, the IFC standard does not provide mechanisms for the integration of NDE results. Since an extension of the IFC standard is possible but time-consuming, the obvious way is to use already defined IFC entities. Without going into too much detail here, our suggested approach would be as follows:  Using the IFC entity IfcPropertySet to embed particular alpha-numeric NDE results  Store important and compact NDE parameters directly in IFC also by using IfcPropertySet  Integrate larger NDE data sets and further results in IFC via file links This allows the content of the NDE results to be technically integrated into BIM models.
NDT-CE 2022 -The International Symposium on Nondestructive Testing in Civil Engineering Zurich, Switzerland,  9 The second level is to integrate the meaning of NDE data into the BIM model. If one were to regard the two sides, NDE and BIM, as different languages, one would speak of a translation. However, since both the data and the parameters represent formal systems, we call this process a transformation ( Figure 5).

Figure 5: Transformation of NDE results and integration in a BIM model including data interfaces
The transformation is carried out in two phases:  Data transformation: Transforming data where necessary, e. g. conversion of measurement coordinate systems to coordinate systems of the BIM models, conversion of units, data structures, and file formats. The conversion rules are assumed to be known or defined in the attached parameters.  Metadata Transformation: Transforming the meaning of NDE parameters and results into the semantic structure of the BIM model, e. g. translating parameter names, assigning corresponding parameters, and transforming parameter sets in a hierarchical BIM structure. The NDE parameters are mapped to BIM parameters as defined in the bsDD.
To this end, an IDM and MVD for NDE aspects have to be developed to clarify processes, data exchange, and required data. Furthermore, a domain in the bsDD must be created that comprises naming conventions and the used vocabulary. This bsDD domain will be a sort of dictionary, containing knowledge of all relevant NDE terms. It can be extended to an ontology [15]

Data storage of BIM-models
The data of a BIM model will be stored in a database. The data model and the choice of the database are not critical as long as standardized interfaces are used and a number of requirements are met. Unlike monitoring tasks, real-time capability is not essential. Ownership and storage location of such databases are still to be discussed, they could be located with the owner of the construction or a service provider.
In addition to the mandatory IFC interface, further interfaces are to be provided for the transfer of other (possibly large) files into and out of the database. These interfaces and data formats should be standardized and well documented. The availability of SDKs (Software Development Kits) simplifies the practical use.
Data storage should meet some requirements:  Access: Access models for operators, data providers, and data users  Availability: Availability of data over long periods of time. How can this be achieved with changes in the standardization of BIM models and data formats?  Data sovereignty: The rights of ownership and use of the data must be guaranteed  Data security: Protection against database failure and unauthorised changes  Scalability: scalability of the database with growing data volume A Common Data Environment (CDE) could be used to combine building information and NDE data. Based on the BIM model, further information about NDE processes, parameters, and results can be provided. For example, when analyzing a column of a bridge, the user can click on the column and the available NDE results and information will be displayed. Such a CDE facilitates data exchange to the test engineer by providing three-dimensional information about the structure with a highlighted area or component for examination.

Use of NDE data from BIM-models
During the use phase of a structure, the essential task of the BIM model is to provide a condition assessment of the structure. For this purpose, the stored NDE data can be extracted from the BIM model and be used by various user groups in different ways. Some possible uses are:  Visualization of the BIM model and the model data  On-site data retrieval, edit, update, and addition  Condition rating resulting in a condition grade of the structure or its parts  Early warning for high load or failure  Planning and assistance of maintenance and NDE tests  Flaw detection, monitoring of changes over time  Load simulation, static recalculation  Prediction of remaining lifetime Enriched results can be fed back into the BIM model to enhance the data. Similar to the integration of NDE results in Figure 5, the output formats and file formats must also be considered when using them. The use of NDE results is easier here because the BIM data model is consistent and well documented. This also allows semantic interoperability, enabling automated processes to interpret the data.

Current approaches
To date no complete integration of NDE results in BIM processes and models is known. In the following, some approaches for partial implementations are presented that illustrate the possibilities and benefits of digital building models.

Conversion of classical approaches into BIM models
Visual inspections of bridges, tunnels, and other engineering structures are carried out in Germany on a regular basis according to DIN 1076 [5]. The results, in particular defects, are entered manually for each component into the SIB-Bauwerke database, where condition grades are calculated for the entire structure. Current efforts are directed at converting the contents of the database into BIM models.
A number of structures, especially bridges, are equipped with sensors that allow continuous real-time monitoring of the bridge's condition (Structural Health Monitoring, SHM). The evaluation software can generate an alarm if necessary. An example for the extension of such a monitoring system into a BIM model is given in ch. 5.3.

Crack detection
Reinforced concrete floors of multi-storey car parks are exposed to an increased risk of corrosion due to exposure to road salt. Active corrosion causes cracks which in turn allow more salt-laden water to penetrate and eventually cause the destruction of the reinforcement and concrete. For effective repair it is necessary to record the crack paths and crack widths and to measure the corrosion activity. in a digital building model (reproduced from [16] with permission) The company ifsb GmbH, Barleben, Germany, has developed an optical method for crack detection with automated evaluation, which yields digital, georeferenced crack maps and crack statistics [16]. In addition, a potential field measurement is carried out to determine the corrosion activity. The recording of the geometry with a laser scanner allows the construction of an asbuilt digital building model which includes the height profile of the floor. The measurement results can be merged and overlaid in the digital building model in order to to plan the repair effectively ( Figure 6).

Multi-sensor condition monitoring and visualization
The smartBRIDGE project is a comprehensive demonstration project that illustrates the implementation of condition monitoring and visualisation using a digital building model ( [17], [18]). Being implemented at the Köhlbrand bridge of Port of Hamburg, Germany, the project demonstrates a methodical approach to digitalization and linking of construction and condition information.
In the project, periodic inspections according to DIN 1076, continuous structural health monitoring, and results from diagnostic examinations are combined in a digital building model to form a digital twin of the bridge. A central storage manages data of heterogeneous data types and formats. The partially automated data processing outputs aggregated condition indicators based on the diagnostic data. Users can access detailed sensor, construction, and condition information by navigating in a photo-realistic 3-D environment created from laser scans and photographs (Figure 7). According to individual user needs, the structural information is visualized in different aggregation levels. Dashboard showing current state of the structure and links to selected sensors (smartBRIDGE-project, reproduced from [17] with permission) In the future, the project will be extended to include existing bridge inspection data and the standardization of interfaces in order to fully comply with the BIM process.

Augmented reality visualization
One possibility to use the BIM model is the visualization of the model data by an IFC viewer in the office or on site. The visualization is particularly intuitive with augmented reality (AR) glasses, through which NDE results can be superimposed directly onto the real building. Figure 8 shows the augmented reality visualization of ultrasound and radar SAFT (Synthetic Aperture Focusing Technique) reconstructions on a concrete test specimen containing internal objects [19]. In the AR glasses, the real object is superimposed with the three-dimensional ultrasonic image and the two-dimensional radar depth section of the interior of the component (actual data) as well as the 3-D geometry of the test object and its internal components (target data). The different layers can be switched by tapping a virtual menu. The geometric relationship between the exterior and interior views is maintained during position changes, so that the AR glasses act as a viewing window into the interior of the component. The threedimensional positioning of the virtual geometries is once done via an optical marker and then via an environment model generated by the AR glasses.
Prospectively, AR visualization will be implemented for arbitrary BIM models using an IFC interface. Different levels of detail will be provided for different user groups, and annotations and edits will be made possible.

Figure 8:
Augmented Reality visualization using an AR headset, and visualizations of the object model and the virtual menu (photomontage, see text) [19] 6 Summary and outlook The established BIM process has the potential to represent both the planning and the condition data of infrastructure over the entire life cycle. A structured view on NDE procedures for testing civil structures was attempted, focusing on evaluation levels, data types, and data interfaces. Based on the results, a workflow for the integration of NDE results into BIM models was developed. Examples of partial implementations demonstrate the possibilities and benefits of using digital building models.
The primary goal is to compile NDE results in such a way that they can be unambiguously interpreted in terms of the test objective at any time. For implementation, the following activities, research, and development steps are necessary:  Consistent use of validated NDE techniques and evaluation procedures, development of test procedures  Digitization of the sensor position logging  Establishment of a recognized workflow for the integration of NDE results into digital building models, which includes quality assurance such as specification of measurement accuracy and provisions for data integrity and traceability  Harmonization and standardization of NDE data and parameters as well as data interfaces, utilisation of standardized data formats where possible  Development of a bsDD that supports and validates the transformation of the NDE metadata into BIM models  Education and training on the NDE techniques Standardization of all data inputvalues, graphics, parameters, notesfrom all NDE techniques in a single step seems unrealistic. As a pragmatic solution, communities of each NDE technique can contribute to the eventual harmonisation of the above-mentioned points. The objective is an automation digital transformation of the entire NDE workflow. Using all diagnostic methodsclassic structural assessment, non-destructive evaluation and testing, and continuous monitoringthe goal can be achieved to create digital twins for important structures that provide all the necessary information to assess the structure's condition at any time.