Possibilities and Limitations of Automatic Feature Extraction shown by the Example of Crack Detection in 3 DCT Images of Concrete Specimen

To assess the influence of the alkali-silica reaction (ASR) on pavement concrete 3D-CT imaging has been applied to concrete samples. Prior to imaging these samples have been drilled out of a concrete beam pre-damaged by fatigue loading. The resulting high resolution 3D-CT images consist of several gigabytes of voxels. Current desktop computers can visualize such big datasets without problems but a visual inspection or manual segmentation of features such as cracks by experts can only be carried out on a few slices. A quantitative analysis of cracks requires a segmentation of the whole specimen which could only be done by an automatic feature detection. This arises the question of the reliability of an automatic crack detection algorithm, its certainty and limitations. Does the algorithm find all cracks? Does it find too many cracks? Can parameters of that algorithm, once identified as good, be applied to other samples as well? Can ensemble computing with many crack parameters overcome the difficulties with parameter finding? By means of a crack detection algorithm based on shape recognition (template matching) these questions will be discussed. Since the author has no access to reliable ground truth data of cracks the assessment of the certainty of the automatic crack is restricted to visual inspection by experts. Therefore, an artificial dataset based on a combination of manually segmented cracks processed together with simple image processing algorithms is used to quantify the accuracy of the crack detection algorithm. Part of the evaluation of cracks in concrete samples is the knowledge of the surrounding material. The surrounding material can be used to assess the detected cracks, e.g. micro-cracks within the aggregate-matrix interface may be starting points for cracks on a macro scale. Furthermore, the knowledge of the surrounding material can help to find better parameter sets for the crack detection itself because crack characteristics may vary depending on their surrounding material. Therefore, in addition to a crack detection a complete segmentation of the sample into the components of concrete, such as aggregates, cement matrix and pores is needed. Since such a segmentation task cannot be done manually due to the amount of data, an approach utilizing convolutional neuronal networks stemming from a medical application has been applied. The learning phase requires a ground truth i.e. a segmentation of the components. This has to be created manually in a time-consuming task. However, this segmentation can be used for a quantitative evaluation of the automatic segmentation afterwards. Even though that work has been performed as a short term subtask of a bigger project funded by the German Research Foundation (DFG) this paper discusses problems which may arise in similar projects, too.


Introduction
For the experimental investigation of damaging influences on concrete, corresponding test specimens are mechanically and chemically loaded and subsequently examined using NDT methods.In many cases, 3D-CT or laminographic techniques are used as imaging methods.The resulting data volumes of several gigabytes in typically more than 1000 layer images can easily be visualized with modern computers and thus are subject to a qualitative inspection.The assessment of damages in concretes, however, requires a quantitative analysis of cracks for comparison purposes with e.g.other concrete mixtures or otherwise induced damages.A necessary segmentation for the most complete possible recognition of certain characteristics in the data for their qualitative and quantitative evaluation can no longer be carried out manually by experts.Therefore an automatic segmentation of the interesting characteristics is a solution for which the question of the reliability of the algorithms used has to be examined.The problem is discussed on the basis of a crack detection algorithm based on template matching and image processing.3D-CT images of damaged concrete specimens and artificial test data sets, in which cracks can be identified reliably and unambiguously by simple means due to their homogeneous structure, serve as the basis for the evaluation of the approach.In the project mentioned below, the knowledge of the material surrounding the detected cracks is also part of the evalution.This helps to assess the character of the detected cracks, e.g.wether they run directly through an aggregate or along the edge of an aggregate stone within the so-called Interfacial Transition Zone (ITZ).In addition to automatic crack detection, this also requires segmentation of the entire test specimen into the concrete components cement stone matrix, 1 More info about this article: http://www.ndt.net/?id=23664 9th Conference on Industrial Computed Tomography, Padova, Italy (iCT 2019) aggregate and pore space.Since such segmentation is hard to perform manually due to the amount of data, whereas the training data has to be segmented manually, some results of a segmentation using a trained convolutional neural network are shown.The results obtained are not yet perfect and should be further improved in a possible follow-up project.
This work was carried out as part of a now completed subcontract within a subproject of the DFG project FOR1498.However, the aspects described in the following paper are of a more fundamental nature and also relevant for similar projects.

Related Work
Essentially, this article is based on the knowledge and experience gained in previous projects on crack detection in threedimensional CT data sets [PBE * 12], [MEL * 14], [ETR * 17], [PBB * 13], [EGD * 11], [PBE * 14].A large part of the literature known to the author covers 2D crack detection, which is often carried out during so-called on-site inspections of structures by evaluating images of the surface [HLWW17], [Zhe14], [GSMM12], [CC17], [MK11].A good overview of crack detection algorithms can be found in [MP17] 3D crack detection in homogeneous materials such as in [FE17] is often performed with combinations of simple image processing algorithms, which also provides good results with such data.One approach to crack detection is percolation, applied in [PD * 16] to noisy 3D computer tomographic images of carbon anodes.In [CC17] a promising machine learning approach based on Convolutional Neural Networks (CNN) is used for crack detection in 2D images.

Crack Detection Used Method
Template matching as a method for shape-based recognition of structures is not a novel method and is very often used in many applications, but to the best of the author's knowledge not in the field of crack detection in 3D CT images of concrete.The template matching approach described in depth in [PBE * 12] was used for crack detection.For the binarization of the calculated correlation coefficients, a hysteresis threshold method is used.Subsequently, smaller connected components, i.e. small cracks, are removed, because cracks are very often detected falsely in noise and are therefore very small.Especially when using small templates, and are therefore considered to be incorrect.The parameters having the largest impact on the result are: the size of the template (thickness and length), the threshold value and the size of the cracks to be removed.

Experimental Results
Figure 1: Segmented cracks on a slice.Left: short template (14 voxels) Right: longer template (24 voxels) with same thickness In general it can be said that smaller (shorter) templates work very well on cracks winding on a small scale.but lead quite often to false-positive cracks, particularly in the cement matrix.Larger templates detect long, rather smooth sections of cracks.and deliver less false-positive cracks.See Fig. 1.
Figure 2: Segmented cracks on a slice assembled from 40 computations with templates of different sizes.The colour depicts the certainty derived from the correlation values.The higher the correlation value, the higher the certainty.
Therefore, using only one template of a fixed size does not allow a complete crack detection.A remedy is the combination of many templates of different sizes.See Fig. 2. Due to the lack of a ground truth of cracks stemming from a real 3D CT dataset, an artificial dataset has been created.For this a photo taken from a surface of a concrete wall where the cracks can clearly be distinguished from the surrounding was used.The photo has than been replicated into Z direction to resemble a 3D dataset.Overlaid with noise as the background the result was used as a test dataset.To create a ground truth the cracks were detected using simple image processing techniques (thresholding, morphological closing).After that the results of a crack detection using template matching with many templates were compared with the ground truth.The accuracy T P+T N Total , where T P means "true positive" and T N "true negative", is approximately 99 percent.Since the test dataset has not the typical components of concrete no "false positive" detection can be made due to imaging artifacts or features of the e.g.aggregates.Therefore, the relevance of that test dataset is somewhat limited here, but can nevertheless be used to assess the detection accuracy.For the crack detection and the creation of the test data AmiraZIBEdition [SWH05] with its special modules for template matching1 was used.

Limitations of the Template Matching Approach
The templates used in the project for shape-based crack detection resembling on a small scale a leaf-like crack structure.This applies to cracks that run inside test specimens.If, on the other hand, they run towards the edge of the specimen, their shape may change to an often delta-like structure (see Fig. 4), such that TM crack detection does not detect cracks in these areas.In the example shown, the crack approaching the edge can only be segmented with an additional manual post-processing step.Though it is not mentioned here before, a further aspect is the limited detection of bifurcations of cracks with leaf-shaped templates.

Automatic Concrete Segmentation Crack Characterisation According to Surrounding Material
In the project on which this work is based, it was important not only to investigate the actual cracks themselves, but also to categorize the cracks in relation to their surrounding materials.This analysis is important because the mechanisms that cause  the crack formation are dependent of the surrounding material.It is even possible that for some concrete mixes the cracks in the aggregate are mostly caused by chemical processes, although the cracks in the cement matrix must have developed due to mechanical stress.For a precise understanding of the material behaviour, it is important to know whether the cracks proceed inside the Interfacial Transition Zone (ITZ) between aggregate and cement matrix.The ITZ is considered to be the weak link in the properties of concrete [ZH08].Simulations indicate that microcracks are first detected in the ITZ around aggregates and then gradually pass through the cement matrix to unite with other ITZ microcracks to create cracks on the macro scale [Maz86] [ALB11].However, 9th Conference on Industrial Computed Tomography, Padova, Italy (iCT 2019) this behaviour has not yet been sufficiently experimentally proven.In order to specify the crack embedding, for each crack voxel, the material in the vicinity on both sides in the direction of the crack normal and its opposite, has to be determined.That leads to an additional labelling of each voxel previously labelled as a crack voxel.A prerequisite is the segmentation of the concrete sample into its components aggregate, pores and cement matrix (see Fig. 5).Although such a segmentation can be performed manually, for large datasets it is very time-consuming; automatic segmentation is in this case the better, not to say only option.

Automatic Segmentation of Concrete Components
An automatic segmentation of the concrete components by means of the grey values in the CT data can only be carried out with the necessary security for pores, since their grey values are always close to zero, while the other components have very different grey values due to their internal structure.In order to be able to automatically segment, a machine learning (ML) approach was used, which has been used successfully in the field of automatic segmentation of medical and biological data sets [TMZ18].This approach is based on the U-Net-concept, which is not described in detail here (see [RFB15] for details) for CNN's.The CNN was implemented using Keras [Cho15] and Tensorflow [MA16].The training data were derived from a manual segmentation of a section (256 3 voxels) of a data set used for the crack detection examples presented here, too.Since the CNN works on 2D image data, a part the data section (128 3 voxels) was taken and split up into all three directions into 128 images of 128 x 128 pixels each, such that 384 images were available.Through rotations and split operations (augmentation), the number of images could be increased to 3072 images and finally used as training data.The manual segmentation as well as all visualizations of this paper have been carried out with AmiraZIBEdition [SWH05].The automatic segmentation, which also takes place on 2D image was applied to four sections of the same size (128 3 voxels) of the same data set.This data section was split up into 128 images in each of the three spatial directions, analogous to the acquisition of training data.Thus each voxel of the test data set was labelled three times from all spatial directions.Utilizing a majority vote (voxels are labelled at least two times with the same label) the uncertainty of the segmentation could be reduced.See Fig. 6.Because only one eighth of the manually segmented dataset was used to train the CNN, the remaining eighth could be used as ground truth for comparison purposes.The accuracy range was 93 to 97 percent for the segmentation of the aggregates compared with the manual segmentation, regarded here as the ground truth.The partly better automatic segmentation compared to the manual segmentation is interesting to notice.See the dark blue areas in the upper half of Fig. 8. Appplying the training data to data sets stemming from different concrete specimen lead to absolutly poor and unusable results 9th Conference on Industrial Computed Tomography, Padova, Italy (iCT 2019)  although the structure of the concrete seems to be similar to the training data by visual inspection.

Limitations of Automatic Segmentation utilizing an ML approach
The biggest challenge in automatic segmentation is the availability of sufficient training data.Since the creation of training data has to be done by humans, these data are only as good as the humans who generate them.Due to the amount of data, fatigue and attention deficits during the very time-consuming manual segmentation step play an important role in assessing the quality of the data as ground truth.

Discussion and Outlook On Crack Detection
Although crack detection via a simple threshold method is always the better choice for data with a similar structure to the test data set generated here, comparative data could be generated by using template matching based crack detection and comparing it with the test data.Which template size is the most suitable depends on the crack characteristic.Since this is often unknown or cracks with different characteristics may be present in the data set, the combination of a larger number of calculations with templates of different sizes in ensemble calculations is a solution.See Fig. 2. All statements refer to the test data set and are by no means generally valid.A meaningful preprocessing of the raw data as well as an adapted selection of the template sizes must be determined for each data set.A parameter-free out-of-the-box crack detection with the method presented here cannot be recommended.Especially not if the crack structure in the data set changes in such a way that it no longer corresponds to a leaf-shaped crack structure as required by form-based template matching.A knowledge about the surrounding material of the detected cracks helps to assess the certainty of the detection.A visual examination of the results shows that crack detection provides more reliable results in the area of the aggregate than in the area of the cement matrix or at the edges of the aggregate.

On Automatic Segmentation of Concrete Components
The segmentation of the material surrounding the crack with techniques from machine learning (ML) and artificial intelligence (AI), which was necessary to determine the crack embedding, was not a direct component of the project underlying the article then.However, since such segmentation of the concrete components is essential and manual segmentation is difficult to achieve due to the amount of data involved, an attempt was made (to the authors' best knowledge for the first time) to use these techniques for such a purpose.While in biological and medical projects automatic segmentations with ML and AI have been carried out for a long time and existing public training data can be used, there are no such data for concrete components as far as known.In the available time only a very small amount of training data (in medicine often several thousand data sets are usual) could be generated for exactly one concrete mixture of one test specimen used here.The quality of the automatic segmentation obtained is good enough to further pursue these techniques.Depending on the aggregates and cements used for concrete mixes, the structure of the concrete components is highly variable.Resolution, noise and imaging artifacts are further aspects that influence the quality of the raw data and must also be taken into account in the training data.In order to use ML and AI techniques for segmentation, manually generated training data would have to be generated for as many concrete mixes and imaging parameters as possible.It is important to ensure a high quality of training data, which can only be ensured with the help of experts, taking into account human imperfections, too.

Further Work
In possible follow-up projects, crack detection should also be attempted with the help of ML and KI.Manually corrected cracks from the template matching approach could be used as training data.Furthermore, the development of a database with training data for different concretes should be examined.

Figure 3 :
Figure 3: Segmented cracks in a test dataset on a slice assembled from many computations with templates of different sizes.Top: The grey value image; Bottom: The detected cracks compared to the ground truth, Red: In ground truth and detected by means of TM; Light blue: In ground truth but not detected by TM; Dark blue: detected by TM but not in ground truth

Figure 4 :
Figure 4: Undetected cracks at the edge of the specimen, even if assembled from many computations utilizing different templates.

Figure 5 :
Figure 5: Visualization of the material surrounding the cracks.Transparent-blue: Aggregates; transparent-green: Pores; red: Cracks inside aggregates; lightblue: cracks between aggregates and cement matrix; lightgreen: cracks near to aggregates and pores.

Figure 6 :
Figure 6: Automatic segmentation of concrete components using a CNN.Left: Slice through the data; Right: Slice with segmentation of the aggregates; Light blue: aggregate segmented only once from one of the spatial directions; Dark blue: Segmented from two of the spatial directions; Red: Segmented in all directions.

Figure 7 :
Figure 7: 3D visualization of the automatic segmentation of concrete components as depicted in Fig. 6.

Figure 8 :
Figure 8: Comparison of an automatic segmentation with a manual one.Left: Slice with manually segmented aggregates in light grey; Right: Automatic segmentation vs. the manual segmentation; Light blue: Not segmented by CNN but in ground truth (false negative); Dark blue: Segmented from CNN, not in ground truth (false positive); Red: correctly classified (true positive).