Decision Rules and Statements of Conformity: Pass, Fail?

When a customer asks, “did my instrument pass?”, the answer depends on a choice that is often never made explicitly: the decision rule. Two laboratories can calibrate the same instrument, get the same result and still give conformity status, because they apply different rules.

What ISO/IEC 17025 requires

• Definition (clause 3.7). A decision rule describes how measurement uncertainty is accounted for when stating conformity with a specified requirement.
• Agreement with the customer (clause 7.1.3). When a customer requests a statement of conformity, the specification or standard and the decision rule must be clearly defined. Unless the decision rule is already part of that specification or standard, it must be communicated to and agreed with the customer before work.
• Documented risk (clause 7.8.6.1). The laboratory must document the decision rule it applies, taking into account the level of risk, such as false acceptance and false rejection.
• Reporting (clause 7.8.6.2). The certificate must state which results the statement applies to, which requirements were met or not met, and the decision rule used, unless that rule is inherent in the specification or standard.

ILAC-G8:09/2019, Guidelines on Decision Rules and Statements of Conformity, gives practical guidance on applying these requirements.

Two risks behind every pass or fail

Because every result carries uncertainty, every pass/fail decision carries two risks:

Risk What it means Who is hurt

False accept | The instrument is declared in tolerance when it is | The user: wrong product decisions, safety and | actually outside | compliance exposure

False reject | The instrument is declared out of tolerance when it | The owner: unnecessary adjustment, repair or | is actually inside | replacement

A decision rule is an agreed, documented choice about how to balance these two risks.

The main decision rules

Simple acceptance (shared risk). The result passes if the measured error is inside the tolerance, whatever the uncertainty. It is simple and widely used, but for a result sitting right at the tolerance limit, the chance that the true value is outside approaches 50 %. ILAC-G8 notes that simple acceptance is reasonable only when the uncertainty is small compared with the tolerance, which in practice means a good test uncertainty ratio (TUR).

Guarded acceptance (guard band). The acceptance limit is moved inside the tolerance by a guard band w, most often w = U. A result passes only if the error is within the tolerance minus the guard band. With w = U at k = 2, the probability of a false accept at the acceptance limit is no more than about 2.5 %.

Guarded rejection. The guard band is placed outside the tolerance, so an instrument is rejected only when it is clearly out of tolerance. This protects the owner rather than the user, and it is rarely appropriate for calibration of instruments used in quality decisions.

Binary and non-binary statements

ILAC-G8 describes two ways of reporting:

• Binary: only pass or fail.
• Non-binary: four zones, which show how close the result is to the limit: Zone Condition (w = U) Statement

1 Error + U within tolerance Pass

2 Error within tolerance, but error + U outside it Conditional pass

3 Error outside tolerance, but error − U within it Conditional fail

4 Error − U outside tolerance Fail

Non-binary statements give the user more information, but they also leave a decision to the user for results in zones 2 and 3.

Worked example

A pressure gauge with a tolerance of ±0.50 bar and a calibration uncertainty of U = 0.10 bar (k = 2):

• Simple acceptance: Measured error: +0.45 bar < 0.50 bar, so pass.
• Guarded acceptance (w = U): the acceptance limit is 0.50 − 0.10 = 0.40. Since Measured error 0.45 > 0.40, the result is not a pass. Depending on the rule agreed, it is reported as fail. Measured error Simple acceptance Guarded acceptance (w = U) Non-binary

+0.35 bar Pass Pass Pass

+0.45 bar Pass (0.45 < 0.50) Fail Not a pass (limit 0.40) Conditional pass

+0.55 bar Fail Fail Conditional fail

+0.65 bar Fail Fail Fail

The +0.45 bar result is where the rules disagree. Under simple acceptance it passes. Under a guard band it does not. Under a non-binary rule it is reported as a conditional pass, and the user decides. The same calibration gives different verdicts, which is why the rule must be agreed before work, not after it.

What to tell your calibration laboratory

When you request a calibration with a statement of conformity, state: | 1. | The specification: the tolerance, accuracy class or standard to judge against | 2. | The decision rule: simple acceptance, guarded acceptance (and the guard band), or a non-binary | statement | 3. | The points apply the whole range, or only the range you actually use

If you do not ask for a statement of conformity, the certificate will report errors and uncertainties only. Customer then makes the judgement against his tolerance.