Table of Contents ECNDT '98 Session: Reliability and Validation Copenhagen 26 - 29 May 1998

THE CURRENT STATUS OF PERFORMANCE DEMONSTRATION AND EVALUATION DEVELOPMENTS Chris Waites and John Whittle AEA Technology, Risley, Warrington, Cheshire WA3 6AT, UK Email: chris.waites@aeat.co.uk, john.whittle@aeat.co.uk

Abstract

It has long been accepted that the inspectors who carry out NDT must be qualified to ensure they have the necessary competence. However, qualification schemes tend to assess general competence rather than the competence needed for a specific inspection. If the latter poses problems beyond those examined by the qualification scheme, "job specific" additions to the basic qualification are sometimes required. The inspection techniques used in the field usually arise from the involvement of suitably qualified staff in the production of procedures within the constraints of accepted codes and standards.

The traditional approach to the provision of confidence in NDT outlined above is perfectly adequate for the majority of inspections. However, there are two situations where these measures may be inadequate. The first arises whenever the NDT plays a significant role in the demonstration of plant safety and the consequences of failure are severe. The second is where the NDT method is new and there are no relevant personnel qualification schemes or standards yet in existence. In these situations, the need to assess and demonstrate the performance of the NDT techniques applied to specific components is increasingly being required as a necessary precursor to the use of those methods. Such demonstrations usually involve the application of the proposed inspection to test pieces replicating the components in question containing deliberately introduced defects. Exercises such as the PISC series and others in the nuclear area showed that there is often a large gap between the performance which is anticipated and that which is actually achieved in practice. This has led to formal requirements in many countries operating nuclear plant that the performance of the proposed methods should be shown to match the structural integrity requirements. Some involvement of independent third parties in the demonstration process is frequently required. The need for similar arrangements for the inspection of major components in conventional plant is currently being considered. A key input to such decisions will arise from performance evaluation exercises in which industrial NDT methods are applied to realistic test pieces.
This paper describes the status of performance demonstration requirements for the nuclear industry world wide. It also reviews the current position regarding performance assessment for non-nuclear components.

Introduction

NDT is recognised as a vital part of the safe and efficient operation of all kinds of industrial plant and other structures. Whenever there is a possibility that unexpected defects may cause failures to occur under normal or abnormal operating conditions there are both economic and safety incentives to detect such defects at an earlier stage and take appropriate remedial measures. NDT methods provide the means to detect and measure the size of defects and there is therefore a strong incentive to be certain that the NDT is capable of meeting the requirements. Not only does the safety and economic well-being of the plant depend on NDT but also the NDT itself can be very costly to perform and it is important on both counts to be sure that it is effective. The means by which confidence can be provided in the NDT carried out in the field are discussed further in the next section of this paper.

The following section then describes the specific measures that have been introduced to ensure the NDT carried out on nuclear plant in different parts of the world is as effective as possible. The consequences of failure of nuclear plant mean that measures to ensure high inspection capability have been introduced already in many countries. For conventional plant, the benefits of specific performance demonstration requirements are not so readily apparent in many cases. There is a need to assess the adequacy of the traditional approach involving the use of codes and standards before similar measures to those adopted in the nuclear industry would be justifiable. The next section of the paper describes a major exercise being carried out under Health and Safety Executive (HSE) funding in the UK to assess the performance of selected industrial NDT techniques in practice.
Finally, it is apparent that there is considerable momentum in the current trend towards the need to demonstrate the performance of NDT. The last section of this paper discusses likely developments for both nuclear and non-nuclear NDT.

Providing Confidence in NDT

Traditionally, confidence in the quality of NDT has been provided in three ways. First, the inspectors who carry out NDT in the field have been assessed through personnel certification schemes. These involve both theoretical and practical aspects to check that inspectors understand the basis of the NDT method they wish to qualify for and that they are capable of applying it successfully in practice to test pieces containing defects. Different levels of certification exist in a number of schemes in recognition of the fact that there are a number of different activities that inspectors can be called upon to perform, each demanding different knowledge and skills. For example, apart from applying NDT in practice, it is necessary to write procedures setting out the detail of how the particular method is to be applied, often within the constraints of general codes and standards. This requires additional assessment to that involved in practical application. The use of staff qualified in this way is the second means by which confidence is provided in the procedures used. A third way of providing confidence in the quality of NDT is auditing the companies carrying out inspection. This process can involve surveillance of the inspections, independent sample re-inspection, typically 5 to 10% of the original inspection, and verification of inspection reports.

The traditional approach outlined above provides confidence in a general way. It does not necessarily provide specific assurance that a particular NDT technique applied to a particular component will be adequate. This may be acceptable in many applications but there are two situations where more specific assurance is needed. The first of these is when it is particularly important that the NDT be effective because it is a key part of a safety case for plant whose failure is unacceptable. The second is where new techniques are to be used and neither codes or standards or relevant personnel certification schemes exist. In these situations specific performance demonstration may provide a suitable way forward.

The need for specific performance demonstration is often recognised even within the traditional approach. The demands of a particular inspection may be very different to those of the test specimens used in certification. For example, many certification schemes are based on inspection during fabrication of components. The test pieces contain fabrication defects only and there is often no need to measure through-thickness size. However, many inspectors qualified in this way are used for in-service inspection where the defect types are different and there is frequently a requirement for through-thickness size measurement so that defect significance can be assessed. The way that such problems are frequently addressed is through "job-specific" requirements which are additional to the basic qualification. The additional assessment is conducted either by the plant owner, the inspection company or by a central certification arrangement as appropriate. Although the personnel aspects are covered as outlined above, the procedure is still developed as before by a suitably qualified inspector working within the constraints of a code or standard. Another wide class of inspections which are frequently the subject of job-specific qualification are automated ones where the demands on inspectors are related to the particular equipment being used and are quite different from the demands of manual inspection.

Formal performance demonstration takes the above process of job-specific qualification to its logical conclusion and includes inspection procedures, personnel and equipment. It starts with a definition of the objectives of the inspection. The potential defect types are analysed and a defect specification is drawn up which defines all the defect parameters which can influence the inspection method being used. For ultrasonics, for example, these include defect position, orientation, roughness and size. Such specifications should be the basis not only for performance demonstration but also for the design of the inspection. In practice many inspections are designed without such a specification and their basis is questionable as a result.

Once a defect specification is available, the formal process of performance demonstration can begin. It will involve the both use of test pieces containing defects and the use of theoretical assessment to varying degrees which depend on the particular scheme adopted. The different forms of performance demonstration adopted to date for nuclear NDT are discussed in greater detail in the next section.

Performance Demonstration for Nuclear NDT

Preamble

Several systems for performance demonstration have now been developed and applied to nuclear NDT. The first of these was in Britain and was a requirement of the safety case for the pressurised water reactor at Sizewell B. Here it was known as validation and an Inspection Validation Centre (IVC) was established under utility funding to carry out the work for the reactor vessel and some of the other key components. The next implementation of performance demonstration was in the USA and the ASME Code, Section XI which governs in-service inspection and testing of light water reactors was modified by addition of two performance demonstration appendices. In Europe, the utilities operating nuclear plant decided to act collectively to establish a European approach and an organisation known as the European Network for Inspection Qualification (ENIQ) was established to develop a suitable system.

Finally, the nuclear regulator in South Africa required that the Koeberg Nuclear Power Station, the only one in South Africa, should introduce performance demonstration because it was included in the ASME code which has traditionally been applied there. However, Koeberg were allowed to propose an alternative approach to that given in ASME and this has allowed them to develop a novel method, tailored to their own specific needs.
The developments discussed above are described in more detail in the remainder of this section.

Sizewell B

In the early 1970's, the UK was studying the implications of building a PWR as its next reactor. A study group was set up under the chairmanship of Sir Walter Marshall to look at the safety aspects of the reactor pressure vessel which contains the nuclear core and is central to the safety case. Among the subjects they investigated was the effectiveness of the NDT. They calculated that, to achieve a vessel failure probability of less than 10-7 per reactor year, it was necessary that the ultrasonic NDT applied to the vessel had a 95% probability of detecting defects 25mm through the 250 mm thick vessel wall. In the absence of other sources of information, the study group sought the views of NDT experts on the likely performance of NDT under the particular circumstances. A questionnaire was circulated and, from the responses, the study group concluded that the required performance would easily be achieved.

Shortly after the study group reported, a trial was carried out in which large blocks replicating PWR vessel welds and containing large defects were inspected using the techniques prescribed in the ASME Code for PWR vessel inspection. The results painted a very different picture from that gained through the opinions of NDT experts. Several very large defects, some almost through the wall of the test pieces, were missed by a high proportion of the teams who participated. This was the PISC I exercise and its findings prompted further investigations into the causes of the problem and ways of improving performance. Further large scale trials, PISC II and III were carried out through the next two decades.

It turned out that the major cause of the problems was that the inspection was not designed to detect the defects which were in the test pieces. These were large smooth defects perpendicular to the surface of the components. Such defects are difficult to detect using angled beams from the surface because they reflect specularly and little of the reflection returns to the probes. Their detection requires special techniques such as tandem or the use of a high sensitivity technique and when these are included, the defects are detected very easily. This was an early illustration of the benefits of designing an inspection on the basis of a defect specification rather than through blind application of code requirements.

A consequence of the lack of confidence in NDT that resulted from the PISC I exercise was that the utility, then the CEGB, introduced a number of measures to improve the reliability of the key ultrasonic inspections to be applied on the Sizewell B reactor. The principles of diversity, redundancy and independent validation were applied to all inspections which were vital to the safety case. Diversity involves the use of different defect detection mechanisms such as specular response, diffracted edge waves, mode conversions etc. to safeguard against intrinsic weaknesses in one particular mechanism. The use of multiple beam angles is included for the same reason. Redundancy involves repeat, independent application of inspections, often by manual and automated techniques, to avoid problems of human error. Finally, independent validation led to the establishment of the IVC and the development of a system to carry out the work.

The obvious way to validate an inspection is to apply it to a realistic test piece containing defects. Unfortunately, while a negative result can condemn an inspection, as in PISC I, a positive result is not conclusive. This is partly because of the large number of variables that can influence an ultrasonic inspection. For practical reasons it is only possible to introduce a few of these into practical trials and other possible permutations of defect position, orientation, roughness than those included in the trials can be postulated which might foil the inspection. In addition, it requires a very large number of defects to establish high confidence in high reliability - 59 successful detections from 59 defects are needed to demonstrate 95% confidence in 95% reliability. This is impracticable for most inspections.

The impossibility of demonstrating the necessary levels of confidence by test piece trials alone led to another approach which involved the production of a technical justification for the inspection (1). This contains all the evidence for the capability of the inspection which can be assembled. What it contains in any particular case depends on what is available but for the Sizewell B RPV the following were included:

• Physical reasoning explaining in qualitative terms why the inspection parameters were selected starting from the component details and the defect specification.
• Theoretical modelling of the inspection. This involves predicting the response from defects to the chosen ultrasonic technique using validated mathematical models of the inspection. The results can be used to extrapolate test piece results over all the possible permutations of influential variables and so generalise them. They can also be used to predict which of the defects in the specification are hardest to detect. If these are then included in test pieces and successfully detected it allows the argument to be advanced that all other defects in the defect specification would also be detected.
• Parametric studies of the influence of variables such as cladding, surface finish and defect roughness. If the influence of these different parameters can be established through separate studies, their effects can be built into the results without the need for them to be included in the test pieces.
• The results obtained in previous round robin trials such as PISC II in which similar techniques to those proposed for Sizewell B were applied.

Having compiled the technical justification, the test piece trials then give a confirmatory check that the entire system - procedure, equipment and operator - produce good results in practice. The trials results and technical justification together constitute the case for the effectiveness of the inspection.

ASME XI

In the USA in the early 1980's, a number of leaks in the pipework systems of boiling water reactors were discovered in pipework which had previously been examined ultrasonically and found to be defect free. It was found that the problem arose from the growth of stress corrosion cracks and that the ultrasonic methods currently in use found it difficult to detect these because they occurred near counterbores in the pipe which produced confusing signals. This led to a recognition of the need to demonstrate the performance of the inspections used for key components of all nuclear reactors. Mandatory Appendices VII and VIII were added to the 1989 edition of the ASME Section XI Code to describe how to carry out the performance demonstration. As described in Appendix VIII, the methods of performance demonstration require only test piece trials. The defects required in the test pieces are not well controlled and no defect specification is required. There are also a number of other features related to the statistical methods used to assess passes and failures which are not in line with western European thinking and Appendix VIII has not found widespread acceptance outside the USA in spite of the extensive use of ASME XI itself in the operation of light water reactors. It has become apparent that when Appendix VIII is implemented in the USA by the Performance Demonstration Initiative (PDI) body set up by the utilities for this purpose, it uses considerable additional documentation which changes the process described in Appendix VIII significantly. Unfortunately, this documentation is proprietary and so it is not possible to be certain exactly how performance demonstration is carried out in the USA.

ENIQ

The introduction of a requirement for performance demonstration into the ASME Code as discussed above caused considerable interest in European countries operating nuclear reactors. This was because many countries use the code as the basis for their own operations or as a set of minimum requirements which they seek to exceed in some areas. The performance demonstration requirements introduced into the code were not acceptable to many countries on technical grounds as already discussed. Consequently the European nuclear utilities combined under the aegis of the European Commission's Joint Research Centre at Petten to establish a European Network for Inspection Qualification (ENIQ). ENIQ also includes qualification bodies such as the IVC and others with relevant expertise. Because all the different countries are subject to different regulatory and legal requirements it was not possible to produce a detailed set of requirements for qualification. Instead, ENIQ has produced a methodology which sets out a framework for qualification setting out the principles which should be applied and the organisational responsibilities of the different involved parties. The principles include the following:
• Qualification is by a mixture of practical trials and technical justification
• Procedures and equipment can be qualified using "open" trials in which those applying the procedure have knowledge of the defects in the test pieces
• Any specific teat piece trials for personnel should be conducted "blind", i.e. the personnel should have no knowledge of the defects except that which would be available before a site inspection

To assist those developing detailed systems for qualification on the basis of the methodology, ENIQ is developing a series of recommended practices which will provide advice for different component types on material which could be included in technical justifications and the way in which practical trials should be conducted. The aim is to promote a uniform approach throughout Europe without being prescriptive in the detail.

Further assistance is being provided by ENIQ through a pilot study. This involves using the methodology to develop and then implement a qualification system for austenitic pipework welds. This will provide an illustration of how qualification can be carried out in practice within the ENIQ methodology.

In parallel with the ENIQ developments outlined above, the European nuclear regulators have been considering their own position on qualification through the Nuclear Regulators Working Group on Reactor Safety.. They have established a task force to study the issue and this recently produced its report on the subject. The conclusions are consistent with the ENIQ approach. Differences between the regulators and the utilities will inevitably emerge as the detail of individual schemes is developed. However, at this stage, there is the almost unprecedented situation of agreement between the two groups on the principles which should apply.

Sweden

In Sweden, the national regulatory body, SKi, recently introduced a requirement for qualification into the regulations which govern the operation of nuclear plant. A Swedish Qualification Centre (SQC) has been established to implement the requirements. The Swedish utilities are currently developing qualification systems to meet deadlines set by SKi and , in general terms, are operating within the ENIQ guidelines.

South Africa

In South Africa, as discussed above, there is a requirement to carry out qualification on the components identified in the ASME Code, Section XI but also the freedom to propose alternatives to the approach set out in Appendices VII and VIII of the Code. The code requirements are viewed in South Africa as inappropriate to their particular requirements and they have worked with the UK IVC to develop an alternative. In-service inspections on many of the components involved have been carried out for some time and there is a wish to continue to use the same inspection methods in the future if qualification shows that they are adequate. Consequently, qualification is being used to assess the performance of the current methods rather than applying pass/fail criteria determined in advance. The assessment of the adequacy of the methods once their performance has been established will be carried out as a separate exercise on a case by case basis. This avoids many of the pessimisms associated with general pass/fail criteria and hence avoids unnecessarily rejecting perfectly adequate inspections.
The way performance assessment is carried out follows the principles listed above under the ENIQ approach.

Performance Assessment for Non-Nuclear NDT

A number of exercises have been carried out in the past to investigate the performance of the NDT methods used to inspect components in different industries. These include the aircraft, offshore and manufacturing industries. A major concern in operation of many kinds of industrial plant using pressurised components relates to the effectiveness of the NDT carried out in service. Regular inspections are required by law and carried out on a large scale. Unfortunately, the effectiveness of these is not clear, particularly since many of the inspectors are qualified using schemes more relevant to the requirements of fabrication. For example many schemes do not include through-thickness size measurement, a common requirement for assessment of service defects.

In the light of the above discussion, the HSE have funded an investigation into the effectiveness of current in-service inspection methods for pressurised components. This is known as the Programme for the Assessment of NDT in Industry (PANI) and is being carried out by the IVC at Risley. The programme commenced in November 1996. A Management Committee has been formed to control the programme and take the key decisions. This has senior representatives from the major industries concerned i.e. oil, gas, petrochemical, chemical, fossil-fuelled power etc. The components involved in the study will be pressure vessels and pipework in ferritic steel and the NDT method to be investigated will be ultrasonics.

A number of test pieces will be made containing defects and defective components from industry will also be used. The NDT will be investigated through a round robin exercise in which a number of operators inspect the test pieces. Some will use a procedure they devise themselves, others will be provided with a procedure endorsed by the Management Committee to form a control group. Great care will be taken to ensure that the inspection conditions in terms of prior information, access, time available etc. resemble those experienced in an industrial inspection. Care will also be taken to keep details of the defects in the test pieces secret and also to preserve the anonymity of those who participate to prevent them being associated with a particular set of results.
The study will be completed in 1999 and, together with widespread distribution of the final report, a closing seminar will be held to disseminate the conclusions to industry.

The Future for Performance Demonstration

Nuclear NDT

At present the number of countries which require performance demonstration is a fraction of those who operate nuclear plant. However, it is clear that the trend is towards more widespread adoption of some form of demonstration, not least because it is a requirement of the latest editions of the ASME Code, Section XI. There is interest in the far East and in eastern Europe, while most of the western European countries have already taken the decision though some are yet to implement it.

It seems likely that some countries requiring performance demonstration will adopt the methods specified in ASME in the first instance because there are legal requirements to do this. However, it is also apparent that many who do this become disenchanted with ASME as the expense and ineffectiveness become apparent. The ENIQ approach on the other hand requires countries to develop detailed systems of their own and this requires expertise which many do not possess. Some of these will turn to organisations like the IVC who have the necessary experience. Others will delay further development until more international experience of implementing ENIQ-type systems has accumulated. One of the key areas is technical justification which is unfamiliar to many countries but which offers the prospect of using test pieces which are fewer in number but high in effectiveness because they have been designed to focus on the important issues. The consequence is that performance demonstration becomes more effective at lower cost.

Non-Nuclear NDT

Non-nuclear NDT is currently at the stage occupied by nuclear NDT in 1970. Most of those involved in its application are reasonably confident in its performance. Until it is apparent that there is a problem there is little incentive for change. The cost and complexity of performance demonstration has been illustrated by its application to nuclear NDT and this means that it would have to be clear that there are major shortcomings with the current approach before there would be a case for change. The PANI programme is likely to be crucial in determining whether there is a need to make changes and in suggesting what these should be.

References

1. "The Use of Technical Justification for Inspection Qualification"; R. K. Chapman, C. Waites and M. J. Whittle, pp 85-90, Proc. 14th Int. Conf. on NDE in the Nuclear and Pressure Vessel Industries, ASM (1996)