·Home ·Table of Contents ·Workshop - Reliability

Human Factors: The NDE Reliability of Routine Radiographic Film Evaluation Ferenc Fücsök, Budapest Power Plant Ltd, Hungary Christina Müller(Nockemann), Martina Scharmach, BAM, Berlin, Germany Contact

Introduction

Many radiographic exposures and film interpretations are made daily in a typical industrial test laboratory. Yet, questions remain regarding the precise probability of detecting specific discontinuities, including the reliability of each individual inspector or laboratory. Although each laboratory's most experienced inspectors evaluate each radiograph, the actual reliability of these inspectors remains somewhat unknown.

In order to estimate the performance of inspectors and laboratories involved in NDE, volunteer laboratories from Croatia, Hungary and Poland conducted a series of round-robin inspections using X-ray films of welds. A set of well-characterised X-ray films was sent to each laboratory for their individual assessment. The selected films were initially evaluated by BAM's Reliability Laboratory with the assistance of an image-processing computer program. This established a baseline of "true values" for each weld image. Each image, in turn, was evaluated by individuals at the participating laboratories.

Using the results obtained from the round-robin study ROC diagrams were established to give an overview about the overall reliability of routine inspections.
The paper is organized as follows: The next section will give a short background of the ROC method. Then the practical procedure of the RRT is described. Finally the results in terms of the ROC diagrams are presented and discussed.

Background of he ROC-Method

To describe the efficiency of an NDT system it is necessary to distinguish between the device parameters as signal to noise ratio or spatial resolution which guarantee merely the functioning of the NDE equipment and the actual testing performance in defect detection and classification. The NDE diagnosis system acts from the interaction of rays ore waves with a defect in the material up to an indication in an inspection report with an eventful history of the useful indication signal and the noise signal. Considering the example of radiographic weld testing the physical process can be theoretically modeled or described by empirical parameters to a certain accuracy up to the creation of the image on the film.
For the evaluation of the whole testing chain - especially for such NDE-systems where standards are not yet ready and the experience of the "NDE world" is poor - including the human inspector and its interaction with complex technique it is helpful to perform a statistical evaluation of defect findings.

Fig 1: The four cases of NDT-diagnosis


The statistics of evaluation is based on the Receiver Operating Characteristic (ROC) (1-5) which is deviated from the general theory of signal detection. The general four possible situations in NDT diagnosis are presented in Fig. 1. The idea of the ROC method is to characterize the accuracy of an inspection system by evaluating the true positive detection rate versus the false positive detection rate for a set of possible decision criteria (decision whether the signal is noise or a defect signal) which represents a varying sensitivity or recording level. The creation of an ROC curve in this way is shown in Fig. 2 where - following the curve from the lower left corner to the upper right - the sensitivity of the system raises.

Fig 2: Creation of an ROC curve (theory)

Fig 3: Practical creation of an ROC curve

So - in the lower part of the curve the highest signals (correct indications) are included and only a small amount of noise (false calls). In the higher part more and more all of the defects are taken into account but also a greater amount of false calls has to be paid as price. In practice it is not possible to apply continuously growing signal thresholds and to count correct and false call rates for each. Therefore different discrete categories of signal counting are defined to be applied by the inspectors during the non-destructive testing evaluation as indicated in Fig. 3. These categories might correspond to the visibility of defects on a radiographic film or to an echo height in an ultrasonic A-scan. We call it detectability later on. So we yield five different experimental points in the ROC diagram - in the current RRT investigation we reduced it to four. The maximum point represents the actual possible operating point. From the whole curve shape - which can be obtained by using a special regression method on the basis of the binormal model - the overall capability of the system is indicated. There is e.g. a forecast possible what will happen when the sensitivity of the system will be raised: Is there a gain in defect finding or is only the false alarm rate increasing? Considering the area under the ROC-curve (see Fig. 4) it may vary from 0.5 (pure chance curve 1) up to 1.0 which corresponds to an ideal NDT system belonging to the left corner's step curve. For the fictive systems shown in Fig. 4 the performance of the system increases from curve 1 to curve 7.

Fig 4: Differentiating NDT-systems by ROC curves

The RRT of film evaluation

To try to estimate the reliability of the human factor in radiographic film evaluation an international Round-robin test (RRT) was organised. The RRT is in progress in Croatian, Hungarian and Polish laboratories by voluntary attendees. For the purpose of the RRT a set of films was selected in the Reliability Laboratory of BAM in Germany which provides the scientific and technical support.

The selected 38 films contain more than 200 defect indications of different types and dimensions.

The selected films were scanned by a state of the art film digitizer (LS85 SDR, Lumisys). The digital images were evaluated with the help of a dedicated image processing computer program of BAM. The results of this evaluation are taken as the true values of the discontinuities. The types of the discontinuities were discussed and agreed by a small group of Hungarian experts. At the same time the X-ray films were copied with a laser printer (AGFA Scopix LR5200), for further information see [7]. Four sets of films were printed into AGFA Scopix Laser films which were sponsored by AGFA. In this way all of the participants of the RRT evaluate exactly the same films.

The voluntary evaluators were provided with clear instructions of the procedure and specific forms for the support of the evaluation work and to aid the pre-processing of the results with the computer.

The forms contain the columns of identification, the code of defects/imperfections according to EN 26520:1991, the co-ordinates, the dimensions and the detectability of the discontinuities and for completeness the IQI and the optical density of the films

The inspectors were asked to evaluate and identify each weld image cm by cm which is a very strict prescription. Additionally they were required to fill in a form for the circumstances of the film evaluation including the length of evaluation time. The evaluation work of the 38 films took 5 to 8 hours. The longer the evaluation time the more detailed discontinuities were indicated.

The evaluation results were filled in an Excel table to support the data processing of the indication results for the ROC statistics.

Results and Discussion

So far the results of a total of 19 inspectors of the different laboratories were evaluated. Those inspectors were selected who do commonly the work of radiographic film interpretation. They have from 5 up to 35 years professional experience. For better overview we created three groups:
• maximum 5 years experience,
• from 6 to 22 years experience,
• from 23 to 35 years experience.
The corresponding ROC results are presented in the figures 5 a, b and c. p(TP) means the probability of a true positive indication (correct defect finding) and p(FP) means the probability of a false positive (false alarm) indication as explained above. The right hand side images show the actual experimental operating points of the individual inspectors. Under nuclear power conditions we would select the most upper points as the valid operation points with the highest indication rate. For testing of e.g. normal water pipes we would select one of the lower points with a lower indication rate but also a much lower false call rate for better economy. From the maximum operation points we can estimate for the overall reliability a correct indication rate of about 70% and a false call rate of about 20%. This is quite reasonable within the strict 1 cm score. The error cross indicates the standard deviation for each group in p(FP) and p(TP) respectively. The left hand side diagrams show the full ROC curves calculated by maximum likelihood fit. From the full ROC curves we can learn about the potential of each individual inspector or the whole group. By varying the detectability level it is possible to shift the operation point along the curve. The red curves indicate the mean of each group. From the statistical point of view the results of the three groups are identical. A more detailed look reveals the best indication rates and smallest scatter for group c. This impression is confirmed by the area index. The area under the ROC curve describes the probability of the system to distinguish between a cm weld with defect from a cm weld without a defect. We have about 90% for group c and about 80% for a and b. But more important than the years of experience for the performance of the inspectors seems to be their individual gift and education which we see from the very different curves in group a and b. Specifically inspector 3 of group a and inspectors 5 and 6 of group b have their curves near to the pure chance line.

Fig 5a

Fig 5b

Fig 5c


The same results are arranged in column diagrams of the individual indication rates in the figures 6 a, b and c. From these diagrams we learn that the false call situation is optimum for group c where the inspectors have more then 22 years of experience.

Fig 6a

Fig 6b

Fig 6c

Conclusions

From the international RRT test with 19 human inspectors we learned about the value of years of professional experience for high testing performance and smalll scatter of results. We learned further about the influence of the individual capability of each person. From this level of knowledge we could recommend ROC tests for film evaluators to exclude pure chance results. For a final confirmation of these results we shall wait until the end of our RRT when a higher statistical basis will be available.

Acknowledgement

We thank Dr. Uwe Zscherpel and Gisela Malitte from BAM for supporting the digitization and film copy work. We thank AGFA for providing the film material for the copies. We thank further all the voluntary inspectors for the effort of film evaluation.

References

1. Metz, C.E. 'Basic Principles of ROC analysis' Seminars in Nuclear Medicine 8 4 (1978)
2. Metz, C.E. 'Some practical issues of experimental design and data analysis in radiological ROC studies' Invest Radiol. 24 (1989), pp 234-245
3. Swets, J.A. 'Assessment of NDT systems-Part I' Mater. Eval. 41 11 (1983), pp 1294-1298
4. Swets, J.A. 'Assessment of NDT systems-Part II' Mater. Eval. 41 11 (1983), pp 1299-1303
5. Nockemann, C., Heidt, H., and Thomsen, N. 'Reliability in NDT: ROC study of radiographic weld inspections' NDT&E International 24 5 (1991), pp 235-245
6. Nockemann, C., Heine, S., Wessel, H., Heidt, H. 'Reliability in NDT: Application of the ROC-Method to Radiographic and Ultrasonic Testing', 1992 ASNT Fall Conference, Chicago, IL, USA, 16. 11.-20. 11. 1992, Paper Summaries, pp. 244
7. Zscherpel, U., 'Film Digitisation Systems for DIR: Standards, Requirements, Archiving and Printing', the e-Journal of NDT & Ultrasonics, Vol.5 (2000) No.5 www.ndt.net/v05n05.htm

© AIPnD , created by NDT.net

|Home|    |Top|