Table of Contents ECNDT '98 Session: Reliability and Validation Copenhagen 26 - 29 May 1998

Modelling of NDT Reliability (POD) and applying corrections for human factors M. Wall, F. A. Wedgwood, S. Burch AEA Technology plc, National NDT Centre, Culham Science Centre, Abingdon, OX14 3DB United Kingdom Tel: +44(0)1235 464097 Fax: +44 (0)1235 463799 EMAIL: martin.wall@aeat.co.uk, alan.wedgwood@aeat.co.uk

Abstract

Inspection by non-destructive testing techniques is not perfect and it has become common practice to quantify the reliability of flaw detection in terms of the probability of detection (POD) and the probability of false indication (PFI). Experimental measurement of these quantities is expensive with poor statistics; therefore attention has recently turned to modelling them. A number of computer models are now available, increasingly used and validated for prediction of POD and PFI. These cover an increasing range of inspection techniques and run in real time on standard PCs. This paper reviews the current status, benefits, approaches to, and applications of POD modelling. Examples are given of the use of POD models developed by AEA Technology and of current development of specific models within Europe for aerospace application. Methods are discussed for correction of model data for human and environmental effects to give POD predictions now comparable with those obtained in experimental trials.

Introduction

Inspection is an important contributor to the integrity and safe operation of engineered components and structures, providing information on current condition and the presence of defects. Despite the faith sometimes placed by engineers and plant operators, NDT inspection like any diagnostic technique is not perfect. Inspection will find defects that may have gone undetected but for a variety of reasons including physics of the technique, geometry and human frailty defects may be missed. Only in the 1970's and 1980's did it become common to ask (and less common to answer) questions about the reliability of the inspection process [1].

By understanding and quantifying this reliability it is possible to assess what importance and value to put on inspection findings and also to improve the reliability and performance. In the last twenty years it has become common practice to quantify reliability in terms of two factors: the probability of detection (POD) and probability of false indication (PFI) as illustrated in Figure 1.

Figure 1: Schematic diagram illustrating (a)POD/PFI and (b) relative operating characteristic (ROC) approaches to quantifying inspection reliability

The reliability of NDT inspection has traditionally been evaluated by experimental trials, notably in the nuclear, aerospace and offshore sectors: for example, PISC I-III, NORDTEST, FAA/Sandia, NIL and ICON. Examples of experimentally derived POD data may be found elsewhere in these proceedings. The benefit of such trials is that they are representative of the actual inspection environment: this is balanced by high cost, often with poor statistics and considerable scatter in results. Frequently there is no accompanying data on PFI.

In the last five years there has been a change of emphasis from experiment towards the development of computer models for predicting POD most notably within the UK National NDT Centre [2,3] and also in the USA [3]. The increased power of computers, PC's and workstations, means that simulations of inspection processes are possible. A number of models are now available for POD using a range of approaches; these models are being increasingly used and validated. Such models mimic the inspection process and allow for statistical uncertainties. Methods are evolving to correct model predictions for human error.

This paper reviews these developments. The values of POD now being obtained by modelling are not now dissimilar in accuracy to those obtained in experimental trials, but it may be hard to model the real inspection process in detail. It is expected that the use of reliability models will become an increasing trend in the validation of inspection over the near future. This is analogous to the way finite-element stress analysis has complimented or replaced structural testing in many plant applications. Models also have their own distinct uses.

Benefits of modelling POD

So, having established that POD can be modelled, what benefit does it bring ? The advantage of using models is that parametric studies of performance can be made with relative ease and little expense. Typically, work would be performed that varied a parameter (such as scan rate, defect orientation, threshold setting) to obtain consistent estimates of their effect on POD and PFI. This then allows a considered optimisation of the inspection to be performed with regard to costs and benefits. A second very important advantage of model calculations is that there are very little experimental data on false-calls PFI and the model is usually the only source of data for them. Modelling also allows assessment of historical data, optimisation at the design stage and allows valuable experimental data to be extended to new applications. Given the poor statistics and large scatter in many POD trials it is arguable whether these provide more accurate values than a modelling or simulation approach.

Approaches to modelling of POD

There are many established models for inspection techniques which can estimate the signal associated with individual defects in a component, for example AEA Technology's ultrasonic ray tracing model RAYTRAIM. It is only recently that models have evolved which include the additional factors (noise, geometry, defect visibility and human factors) needed to make effective predictions of POD. The approaches presently available to make predictions of POD take one of several forms and can be summarised as follows :
• Physical models for POD and PFI
• Signal/Noise models
• Image classification models/ Inspection simulators('Visual' POD, 'Spot the Ball')
• Expert judgement
• Statistical models (Curve-fitting)
• Human reliability models

Current Status

Physical POD Models


Figure 2: Schematic illustrating the the basis of a POD model

Physical models for POD and PFI provide data through a physical model of the inspection process, including background noise and criteria for defect detection. During the last 5 years a suite of models has been developed within the National NDT Centre for predicting inspection reliability in terms of probability of detection POD and false calls (PFI) [2-4]. These models use established and well validated physical models as their basis but also include variable factors such as noise, geometry, defect visibility and detection criteria necessary to make predictions in reliability terms. All models run on a standard PC in real time and cover a range of inspection methods including ultrasonics (pulse-echo and time-of-flight), radiography and magnetic techniques. Customised models have been developed for specific applications including validation of a procedure for detection of complex weld defects, inspection of steel railway line, composite materials, and inspection of concrete structures. Reviews of these developments may be found in References [2,3].

The approach is similar to that used to predict POD by Thompson[3] in the US aerospace industry. Our models allow analysis of image-based data, produce simulated images and allow correction for human and environmental effects. The estimates of POD combine estimates of signals expected from specific defects and transducers, estimates of background noise and a 'threshold criterion'. The simulation of defects is being constantly improved based on the analysis of 'real defect' data from operating plant.

Signal/Noise models
These convert signal and noise values to POD and PFI using statistical methods. The signal and noise values can be derived from models or experiment, for example measurements on samples with reference defects. The approach in calculating POD is similar to the physical models above. This method avoids the statistical difficulties associated with conventional POD trials and allows predictions to be made for new inspection techniques that may be too complex to physically model. This approach has been used for many years in the USA aerospace industry , through the ã v a approach [Berens 3]. We have used this approach to make POD predictions for magnetic methods including MFE and as the basis for a basic spreadsheet model derived for member organisations of the Harwell Offshore Inspection Service (HOIS), managed by AEA Technology. A similar methodology has been used in Germany for aerospace component inspection. A modular approach can be adopted, with input data to the POD model derived from a physical model or experiment..

Image classification models/ Inspection simulators ('Visual' POD, 'Spot the Ball')
These represent methods for analysis of image-based inspection data such as radiographs, to give information in terms of POD and PFI. Inspection simulators are a special class of computer model that simulates the inspection process by presenting simulated inspection results to the operator. Interpretation of image-based data is more subtle and requires a more complex detection criteria than analysis of signal/noise data above. The detection criterion may be simply exceeding a threshold signal level at a number of locations, over a number of pixells or over a specified area ( as in the rastor scanning model developed by Ogilvy) or more closely configured to actual inspection system operation. Colin Windsor [2] has recently pioneered a neural-network based approach for detection of defects in image-based techniques, used in the radiographic model NNXPOSE. This uses receptive fields to search for and enhance specific defect types such as cracks porosity or slag inclusions and more closely reproduces the human interpretative skills of the inspector.

Figure 3: Schematic illustrating the the basis of a POD model and the use of simulated images for POD trials to aid in the validation of model predictions. ('Visual POD', 'Spot-the-ball contest').

The production of simulated images and inspection data is an important feature of our POD models. The images give confidence that the simulation and POD estimates are reasonable. The programmes can present a series of simulated images to the inspector like a 'spot the ball' contest, essentially simulating a POD trial (Figure 3). The POD and PFI is automatically calculated, we call this 'Visual POD'. This provides a second independent method for the model to estimate POD and comparison can provide information on human reliability. A similar simulation method was used in the PISC III programme on human reliability [5].

Expert Judgement
Expert judgement has been used where input on POD is required for fracture mechanics or risk-based assessments and is not available from experiment. Provided the judgement comes from trained inspectors and sensitivity analysis is used this can be an effective method. The National NDT Centre maintains a computer database of POD information (PODDATA) which can be used to aid such judgements.

Statistical models
These use methods for statistical analysis or curve-fitting to experimental data, with the aim of making this data accessible for use in other applications (such as fracture mechanics). These do not model the inspection process as such.

Human reliability models
These take account of the effects of human error in the inspection process, and correct predicted POD values for these effects. An example is the methodology applied by AEA Technology to utilise human error data from the PISC III work [5] discussed later.

POD Models for Aerospace

A major POD modelling contract is underway by AEA Technology, National NDT Centre for the European Space Agency ESA (ESTEC Contract 12228/96). This commenced in February 1997 and has a 20 Month duration. Models are being developed to predict POD and PFI for composite aerospace components and validated by experimental trials. Two specific NDT techniques are included: ultrasonic C-scan and X-radiography. In addition a 'technique-independent' model, based on the signal/noise approach above, will allow POD predictions to be made from image-based or signal-noise data for evolving techniques such as transient thermography. All models will produce simulated images (see example in Figure 3) which can be used in place of real samples for POD trials and provide an independent route ('Visual POD) for verification of the model POD predictions. It is anticipated the models will lead to improvements in the quality and understanding of aerospace NDT and could be adapted to other materials and structures.

Examples of Model Application

Parametric Studies


Figure 4: Effect of varying a single parameter: defect tilt.POD model calculations using PODUT

Figure 5: Model predictions of POD,PFI and ROC for voids in steel plate using radiographic POD model XPOSE. The detection threshold is varied

The specific application of a POD model to examine the effect of a single parameter is illustrated in Figure 4 . This shows the effect on POD for ultrasonic inspection of defect orientation; allowing a crack to be mis-orientated by up to +/-15 (. Such data would be complex to determine by experiment. Figure 5 shows the effects of varying inspection threshold and defect size on the POD for radiographic inspection calculated using model XPOSE, in this case plotted in terms of an ROC curve. Figure 6 shows an example simulated radiograph and corresponding predicted POD and PFI curves. XPOSE sets up the inspection in the same way as a radiographer, then produces simulated radiographs as well as POD and PFI predictions. The resulting POD values may be compared to the visual perception of the defect in the radiograph. Standard defects include voids, porosity, inclusions, lack of fusion defects and cracks. The simulated radiograph also shows the series of Image Quality Indicator (IQI) lines used as reference in the inspection as in routine radiographic work.

a	b
Figure 6: Example output from AEA Technology radiography POD model XPOSE showing: (a) simulated radiograph and (b) corresponding POD, PFI curves as a function of defect size. The iImage quality indicator (IQI) lines numbered 1-7 are also shown in the radiograph.

Comparison of Techniques
Figure 7 considers inspection for a surface crack-like defect in 25mm steel plate misorientated by up to 10 ( and compares the use of time-of-flight diffraction (TOFD), pulse-echo ultrasonics and radiography. As the defect is tilted away from normal radiography and conventional UT become progressively less suited, whereas TOFD which is dependent on diffraction from the defect tips remains effective. This is illustrated by the calculated POD's. These model predictions are almost identical to experimental POD data shown by Olaf Forli for the 3 techniques in the present conference.

Correction for Human Factors

Figure 7: Comparison of techniques: POD model calculations for inspection for mis-orientated cracks in 25mm plate.

Inspection reliability is clearly quite complex to model and must include consideration of procedure, material , human and environmental factors [Figure 8]. Human factors have recently been shown to be very important and dependent on a large number of factors such as fatigue, environment, stress and complexity of the task. We distinguish between tasks involving the 'eye' and ' hand-eye coordination'. It is unlikely that a theoretical model of this will be developed in the near future. However, experimental data is being amassed in a number of fields including NDT, for example the PISC III trials [5]. The physical models described above give a 'theoretical' or 'physically achievable' POD, allowing variation in parameters such as defect orientation or visibility. In order to give model predictions comparable to those that would be found in experimental trials we use a human factor curve as a multiplier H for models based on the physics and engineering of an inspection.
PODactual = POD model * H

The simplest correction uses a constant multiplier (or POD reduction factor), typically 95% or lower for more severe environments. A better correction method uses curves such as the data for human error expressed in terms of POD in Figure 9, derived from PISC III data. This recognises that the effects of human error are greater for small defects close to threshold. In this case the human error POD is used as the multiplier H. An example showing the correction of a model prediction of POD from radiographic model XPOSE for human error is given in Figure 10. Experience to date shows that, where account is taken, through this use of a multiplier, the agreement of model predictions with experimental POD work is improved. This is an important area for future work. At the recent European-American workshop in Berlin [3] a general equation was suggested in which the theoretical POD was reduced to take account of environmental effects:

POD actual = POD intrinsic - g(AP)- h(HF)

Figure 8: Illustration of the environments that may be experienced in actual plant inspections.

Figure 9: POD curves for human reliability derived from human reliability data from experiments in PISC III using AEA Technology ultrasonic simuilator PCSIMONE. I/Io is the signal (I) divided by the threshold (Io)

Where g(AP) and h(HF) are factors relating to the application of the technique (environment, surface, couplant, geometry etc.) and human error respectively.

Validation

Validation of the models is important in gaining increased acceptance for their use. The validation needs should be considered in the context of the application. For example, in probabilistic fracture mechanics assessments or economic assessments sensitivity studies are often included and therefore upper and lower bound data may be used in place of absolute values. Second, much scatter can arise in experimental POD trials: therefore model calculations only need to be similarly accurate to offer a realistic alternative.

Our POD models are based on physical models for ultrasonic and radiographic inspection which have been proven and validated over many years. We have taken the additional step of moving from a simple signal/noise calculation to a prediction of POD. To validate existing POD models we have used the following approaches:

Figure 10: Example showing correction of model POD curve for human error (HR) and comparison with experimental NORDTEST POD data [7]. Correction using mean and lower bound human reliability curves in Figure 9. Radiography of 25mm steel plate.

• Comparison with experimental POD data.
• Parametric studies
• Compare simulated images with actual images (e.g radiographs)
• Use simulated images in POD trials ('Visual POD').

Where comparison with experiment is possible, reasonable agreement has been found between model calculations and experiment and parametric variations observed in experimental data are reproduced by models. One such comparison is given in Figure 10 and for the TOFD model similar comparisons have been given [4]. For precisely defined input data model POD curves are generally sharper than experiment, but more realistic curves are obtained if 'randomising factors' such as defect orientation, background, attenuation or defect visibility are allowed to vary between practical limits found in real components e.g Figure 4.

Applications of POD Models

POD models have application in all areas where experimental NDT reliability data is currently used, for example: (i) inspection validation , (ii) comparing inspection techniques, (iii) input to Safety cases (insurers, regulatory bodies), (iv) economic assessment, (v) input to fracture mechanics and integrity assessments, (vi) prediction of flaw distributions, (vii) optimisation of inspection performance and procedures and (viii) to aid introduction of new technology.

There are several applications where modelling is the only possible approach: (i) parametric and sensitivity studies, (ii) assessment of historical inspection data (e.g. ageing plant), (iii) extrapolation and interpolation from experimental data, (iv) optimisation of inspection at the design stage, (v) POD trials using simulated inspection data ("Spot-the Ball") and (vi) to aid design of and support experimental POD trials. Currently our POD models have been used in all these applications and the approach has been extremely valuable in support of safety cases and inspection validation.

Other NDT Performance Factors

The key performance factors in assessing the value of an NDT technique in operating plant are sensitivity, speed, coverage and reliability [2,3]. The POD curve may be considered to combine sensitivity and reliability. The balance between these factors is important in determining the most appropriate inspection in operating plant. One way to combine the parameters is by performance-based economic assessment methods such as the Inspection Value(IVM) method described elsewhere in this conference.

CONCLUSIONS

1)	A number of computer models are now available for prediction of inspection reliability in terms of probability of detection (POD) and false calls (PFI). These cover an increasing range of inspection techniques and run in real time on a standard PC. The models are being increasingly used and validated. Methods are evolving to correct model predictions for human error.
2)	The model approach provides complimentary data to experimental assessments and allows existing experimental data to be more widely used. The models can provide specific data not available from experimental measurements such as parametric studies, assessment of historical data and optimisation at the design stage.
3)	POD models are already being used in economic assessments, integrity assessments, to support safety cases and validation of inspection procedures and plans. The values of POD now being obtained by modelling are not dissimilar in accuracy to those obtained in experimental trials.
4)	POD models should be seriously considered as an integral part of future POD trials. This could reduce the number of test samples required, help gain acceptance and familiarity for the modelling approach, provide validation and lead to improvements in the model predictions and correction methods used for human and environmental effects.
5)	It is anticipated that he use of POD models will increase significantly in future years. There is a need for development of customised models for specific applications and industries, increased validation as use increases and more work on human and environmental effects. Support from industry to meet these objectives is sought.

Acknowledgements

The authors would like to thank the European Space Agency (ESA/ESTEC) for permission to include information on Project 12228/96 and the members of the Harwell Offshore Inspection R&D Service who have part-sponsored the development of the economic assessment method (IVM) mentioned in this paper. The current HOIS members are Amerada Hess, British Gas, BP, DNV, Norsk Hydro, Phillips Norway, Phillips UK, RTD, Saga, Saudi Aramco, Shell, Statoil, Texaco Britain & HSE.

References

1. Silk M G, Stoneham A M and Temple JAG: 'The Reliability of Non-Destructive Examination', Adam Hilger Press, Bristol, UK, 1987, ISBN 0-85724-533-8
2. Wall M: Modelling of Inspection Reliability, Engineering Science and Education Journal, Vol. 6 No. 2 pp63-72 April 1997
3. Proceedings European-American Workshop on Determination of reliability and validation methods for NDE, Berlin, June 18-20 1997. (Papers by Wall, Thompson, Rummel, Berens and Forli).
4. Silk M S: Estimates of the probability of detection of flaws in TOFD data with varying levels of noise; BINDT Journal INSIGHT vol. 38 no. 1 January 1996.
5. R A Murgatroyd et al: Human Reliability in Inspection: Results of Action 7 of PISC III Programme, 1994.