 ·Table of Contents ·Workshop - Reliability | Worth of Modelling for Assessing the Intrinsic Capability of NDTMartin Wall, Steve BurchAEA Technology plc, National NDT Centre,E1, Culham Science Centre,Abingdon, Oxfordshire,OX14 3ED, United Kingdom Email: martin.wall@aeat.co.uk ,steve.burch@aeat.co.uk Contact |
| (1) |
Where IC is the Intrinsic Capability of the inspection system, generally considered as an upper bound. AP is the effect of application parameters such as access restrictions and surface state. HF is the effect of human and environmental factors. The latter factors APand HF will generally reduce the capability of the NDE system.
The reliability and intrinsic capability of inspection techniques has traditionally been determined by experimental trials and measurements, notably in the nuclear, aerospace and offshore sectors; for example PISC 1-3, NIL, ICON, NORDTEST, USAF and the Sandia Trials. Such trials provide practical measures of the capability of commercial inspection equipment and the actual inspection environment, but are expensive, can have poor statistics and may exhibit considerable scatter in results. Examples of experimentally derived POD data may be found elsewhere in these proceedings.
This paper reviews and considers the benefits and limitations of the alternative approach of using computer modelling to determine intrinsic capability and reliability of inspection. Models have been used for many years as part of the validation and qualification of NDE methods. The use of models to determine reliability in terms of probability of detection (POD) is more recent. In particular areas this is well accepted, for example the ã v a approach [2,3] has been used for many years for evaluating POD in the US aerospace industry. Over the last 10 years increasingly sophisticated models have been developed for POD notably at Iowa State University in the US [4] and at the UK National NDT Centre[4,5]. Such models run on standard PC's, produce simulated data and can be linked into CAD packages and probabilistic models.
Examples will be given from recent model development in Europe and the USA, including models for inspection of composite materials for space applications, the use of simulators, applications in the offshore, aerospace, transport and power industries, and the development of empirical models to correct for human and environmental effects. The benefits and limitations of these approaches are considered and contrasted with the uncertainties that can also arise in experimental measurements.
Despite the faith sometimes placed by engineers and plant operators, NDT inspection like any diagnostic technique is imperfect. Inspection will find defects that may have gone undetected but for a variety of reasons including physics of the technique, geometry and human frailty defects may be missed. Only in the 1970's and 1980's did it become common to ask (and less common to answer) questions about the reliability of the inspection process [3 ]
The key performance factors in assessing the value of an NDT technique in operating plant are sensitivity, speed, coverage and reliability [5,6]. The balance between these factors is important in determining the most appropriate inspection in operating plant. One way to combine the parameters is by performance-based economic assessment methods such as the Inspection Value(IVM) method.
By understanding and quantifying the reliability it is possible to assess what importance and value to put on inspection findings and also to improve the reliability and performance. In the last twenty years it has become common practice to quantify reliability in terms of two factors: the probability of detection (POD) and probability of false indications (PFI) as illustrated in Figure 1. In practice a compromise is needed to maximise POD whilst minimising false calls. The latter can significantly impact on cost and the viability of the inspection, particularly for automated inspection systems. Data is usually plotted as a POD curve, with POD plotted against defect size. This may be considered to combine sensitivity and reliability. An alternative is the reliability operating characteristic (ROC) curve which plots POD against false calls.
Fig 1: Schematic diagram illustrating (a)Probability of Detection POD/false calls PFI and (b) relative operating characteristic (ROC) approaches to quantifying inspection reliability.
|
It is becoming increasingly common to obtain POD and false call information from modelling rather than experiment or trial.
Modelling allows assessment of the reliability of historical data, the value of inspection data on manufactured components. It is possible to estimate the remaining defect population and what may have been missed by past or proposed inspections where the inspection equipment may not exist. Models allow the reliability of inspection systems to be optimised at the design stage and allows valuable experimental data on reliability to be extended to new applications e.g. Rummel [7 ]. Models are complementary to experimental measurements and can be used in combination to produce well focussed performance trials. Models particularly simulators can be valuable training aids for inspectors and can be used as part of the qualification and validation of NDT methods for a given application; for example as input to ENIQ qualification in the nuclear industry. Given the poor statistics and large scatter in many POD trials it is arguable whether experiments provide more accurate values than a modelling or simulation approach.
In experimental measurements, it is difficult to isolate the intrinsic capability from application and human factors. Models in most cases allow determination of the intrinsic capability (IC) so an estimate can be made of these other factors.
For example, the ultrasonic ray tracing model MUSE linked to CAD packages can be used to plot out the beam paths within complex components and to establish the reflected and mode converted beam paths. This would establish the time delays for each of the echoes displayed on the A-scan and probable location of defects. The amplitude of the various echoes from the geometric features can be predicted by calculating losses at interfaces, the use of standard reflection coefficients for basic reflectors and by the use of a amplitude modelling software. The software package can be used to predict the amplitude of possible defects in pulse echo mode, the combination of time delay and amplitude, and construct similar displays to those produced from the inspections under assessment.
Such models are extremely valuable in understanding the inspection process and data interpretation. Huge cost savings are possible by reducing the need for testing when changes are made in component design. Whilst such models may estimate the signal or simulate data, they give no indication of the reliability or intrinsic capability (IC) of the inspection. This needs consideration of other factors such as detection criteria and background noise. Model approaches to predict reliability are considered in the following sections.
The input and basis to such models to give predictions of POD and false calls is summarised schematically in Figure 2. As well as the signal and background noise for the inspection it is necessary to simulate the detection process and the way in which the data, signal or image-based, will be interpreted by the inspector or by the system in the case of automated inspection systems. Reliability prediction could be seen as an add on module to conventional models. All the inputs shown in Figure 2 could come in principle from model or experiment; particularly true for novel techniques where a physical modelling capability is not yet established.
| 
Fig 2: Schematic illustrating the basis of , and typical inputs to an inspection reliability POD and simulation model.
| | |
The detection criteria used should reflect the way in which the inspection will be carried out and data analysed. This has varied from simple thresholding, repeat detection, integration methods to simulate the visual detectability of a defect in image-based data by a human inspector, to neural network methods.
Figure 3 shows the inputs and outputs to the ultrasonic POD model developed by the AEA Technology's National NDT Centre for ESA [11].under ESTEC Contract 12228/96. An example of the resulting model POD curves and corresponding simulated UT C-Scans are shown in Figure 4.
Fig 3: Inputs and outputs from physical POD and simulation models developed on ESA/ESTEC Project 12228/96.
| 
Fig 4: Example output from POD model for ultrasonic C-Scan inspection developed for ESA/ESTEC: Left POD curves for different detection criteria, Right simulated UT C-Scan of delamination defect in CFRP composite panel .
|
The different types of reliability model are discussed below with examples of their benefits and application. Modern POD models may often use a combination of these methods and produce a full simulation of the inspection data. For example in the ESTEC case physical models were developed to predict POD and PFI for composite aerospace components and validated by experimental trials. Two specific NDT techniques were included: ultrasonic C-scan and X-radiography. These models enable the effectiveness of inspections carried out in manufacture of space components to be assessed and optimised, and confirm that unacceptable flaws are unlikely to go undetected into service..
All the models produced simulated images (Figure 4) which could be used in place of real samples for POD trials and provide an independent second route ('Spot the Ball') for verification of the model POD predictions. In addition a 'technique-independent' model, based on the signal/noise and image classification approaches discussed below, allowed POD predictions to be made from experimental image-based or signal-noise data for evolving techniques such as transient thermography. It is anticipated these models will lead to improvements in the quality and understanding of aerospace NDT and could be adapted to other materials and structures.
These models use established and well validated physical models as their basis but also include variable factors such as noise, geometry, defect visibility and detection criteria necessary to make predictions in reliability terms. All models run on a standard PC in real time and cover a range of inspection methods including ultrasonics (pulse-echo and time-of-flight), radiography and magnetic techniques. Customised models have been developed for specific applications including validation of a procedure for detection of complex weld defects, inspection of steel railway line, composite materials, and inspection of concrete structures. Reviews of these developments may be found in References [5,6]. The models allow analysis of image-based data, produce simulated images and allow correction for human and environmental effects. The estimates of POD combine estimates of signals expected from specific defects and transducers, estimates of background noise and a 'threshold criterion'. The simulation of defects is being constantly improved based on the analysis of 'real defect' data from operating plant. Silk [8] developed a model for time of Flight Diffraction TOFD which could produce simulate images and predict POD under different inspection and noise conditions.
The approach is similar to that used to predict POD by Gray and Thompson[3] in the US aerospace industry who have also developed many reliability models including specific models for composites.
Such models allow parametric studies so that the effect of an individual parameter on inspection capability can be evaluated. In combination they can allow comparison of techniques. The specific application of a POD model to examine the effect of a single parameter is illustrated in Figure 5. This shows the effect on POD for ultrasonic inspection of defect orientation; by allowing a crack to be mis-orientated by up to +/-15 °
. Mean and upper and lower bound values are shown. Such data would be complex to determine by experiment. Figure 6 shows the effects of varying inspection threshold and defect size on the POD for radiographic inspection calculated using model XPOSE, in this case plotted in terms of an ROC curve. Figure 7 considers inspection for a surface crack-like defect in 25mm steel plate misorientated by up to 10° and compares the use of time-of-flight diffraction (TOFD), pulse-echo ultrasonics and radiography. As the defect is tilted away from normal radiography and conventional UT become progressively less suited, whereas TOFD which is dependent on diffraction from the defect tips remains effective. This is illustrated by the calculated POD's. These model predictions were very similar to experimental POD data shown by Olaf Forli in the 1997 European-American workshop [2].
XPOSE [5] sets up a radiographic inspection in the same sequence of steps as a radiographer and produces simulated radiographs as well as POD and PFI predictions . This is a valuable training and set-up aid. The resulting POD values may be compared to the visual perception of the defect in the simulated radiograph. Standard defects include voids, porosity, inclusions, lack of fusion defects and cracks. The simulated radiograph also shows the series of Image Quality Indicator (IQI) lines used as reference in the inspection as in routine radiographic work. Simulated data can be produced to optimise conditions before carrying out the actual inspection.
Fig 5: Effect of varying a single parameter: defect tilt. POD model calculations using PODUT.
Fig 6: Model predictions of POD,PFI and ROC for voids in steel plate using radiographic POD model XPOSE. The detection threshold is varied.
Fig 7:
Comparison of techniques: POD model calculations for inspection for mis-orientated cracks in 25mm plate.
The detection criterion may be simply exceeding a threshold signal level at a number of locations, over a number of pixells or over a specified area ( as in the rastor scanning model developed by Ogilvy) or more closely configured to actual inspection system operation. Colin Windsor [5] has recently pioneered a neural-network based approach for detection of defects in image-based techniques, used in the radiographic model NNXPOSE. This uses receptive fields to search for and enhance specific defect types such as cracks porosity or slag inclusions and more closely reproduces the human interpretative skills of the inspector.
The 'Technique Independent 'model developed for ESA is an example of this model type. This runs as separate software package using image display & analysis software NTPLOT developed by AEA Technology for other applications. Reliability predictions are based on analysis of experimental image data from components containing defects of known size & location such as a test or calibration block. The user interactively selects the defect region & neighbouring background regions. The model then makes a statistical derivation of POD & PFI parameters using detection simulation methods as used in the physical models. The principle is similar to the theoretical reliability models but the input is from experiment rather than model.
The programmes can present a series of simulated images to the inspector like a 'spot the ball' contest, essentially simulating a POD trial (Figure 3). The POD and PFI is automatically calculated, to show the POD and level of false calls achieved by the inspector. This provides a second independent method for the model to estimate POD and comparison with the theoretical model predictions can provide valuable information on human reliability and the differences between qualified and novice inspectors. A similar simulation method was used in the PISC III programme on human reliability [12] using a physical ultrasonic simulator PCSimone.
Figure 4 shows an example simulated radiograph and corresponding predicted POD and PFI curves. In the NNXPOSE model developed by Windsor Neural Network defect recognition methods were used on simulated data to give separate POD predictions. NNXPOSE was also used to analyse old radiographs from ageing power plant to assess the likely reliability of manufacturing inspections.
Empirical methods have been developed based on empirical data which seek to take account of the effects of human error in the inspection process, and correct predicted model POD values for these effects. An example is the methodology applied by AEA Technology which utilises human error data from the PISC III work and other sources to correct model predictions [5]. In order to give model predictions comparable to those that would be found in experimental trials this used a human factor curve as a multiplier H for models based on the physics and engineering of an inspection.
The simplest correction uses a constant multiplier (or POD reduction factor), typically 95% or lower for more severe environments. A better correction method uses curves such as the data for human error expressed in terms of POD in Figure 8, derived from PISC III data. This recognises that the effects of human error are greater for small defects close to threshold. In this case the human error POD is used as the multiplier H. Correction values for Human error have also been derived by comparison of data from simulation ('spot the ball') model trials with experimental data or from comparison of field and lab trials. An example showing the correction of a model prediction of POD from radiographic model XPOSE for human error is given in Figure 9. Experience to date shows that, where account is taken, through this use of a multiplier, the agreement of model predictions with experimental POD work is improved. This is an important area for future work.
Fig 8: POD curves for human reliability derived from human reliability data from experiments in PISC III using AEA Technology ultrasonic simulator PCSIMONE. I/Io is the signal (I) divided by the threshold (Io).
Fig 9: Example showing correction of model POD curve for human error (HR) and comparison with experimental NORDTEST POD data [7]. Correction using mean and lower bound human reliability curves from Figure 8. Radiography of 25mm steel plate.
|
Most current POD models are based on physical models for inspection which have been proven and validated over many years. The models take the additional step of moving from a simple signal/noise calculation to a prediction of POD. To validate existing POD models the following approaches have been used:
Fig 10: Model validation. Comparison of model and experimental POD curves for ultrasonic C-Scan inspection for delamination defects in CFRP plate.
|
A number of computer models are now available for prediction of inspection reliability in terms of probability of detection (POD) and false calls (PFI). These cover an increasing range of inspection techniques and run in real time on a standard PC. The models are being increasingly used and validated and often linked in a modular approach to and other NDT and inspection simulation models. Methods are evolving to correct model predictions for human error.
It is concluded that modelling increasingly offers a realistic alternative to experimental trials. This needs to take account of the context in which the data will be used. The model approach provides complimentary data to experimental assessments and allows existing experimental data to be more widely used. The models can provide specific data not available from experimental measurements such as parametric studies, assessment of historical data and optimisation at the design stage. Other uses include validation and qualification of inspections and such models are a valuable aid in the qualification and training of inspectors.
Reliability models are already being used in economic assessments, flaw population estimates, integrity assessments, to support safety cases and validation of inspection procedures and plans. The values of POD now being obtained by modelling are not dissimilar in accuracy to those obtained in experimental trials if empirical corrections are made for human and environmental effects.
Computer models should be seriously considered as an integral part of future POD trials and be developed for a broader range of NDT methods. This could reduce the number of test samples required, help gain acceptance and familiarity for the modelling approach, provide validation and lead to improvements in the model predictions and correction methods used for human and environmental effects. Experimental reliability trials are realistic but expensive and time consuming and can suffer poor statistics and much scatter.
There is a need for development of customised models for specific applications and industries, increased validation as use increases and more work on human and environmental effects. Support from industry to meet these objectives is sought.
| © AIPnD , created by NDT.net | |Home| |Top| |