QC Webinar QC Webinar Tony Badrick
Wh t i L b t ? What is a Laboratory? • Equipment • People • Procedures
Wh t i L b t ? What is a Laboratory? • Documented procedures describing all the functions of the laboratory? functions of the laboratory? • ISO 15189
• The laboratory version of ISO 9000 The laboratory version of ISO 9000 • We are not really all that different!
Wh t i ‘Q lit ’ What is ‘Quality’
What are Quality Systems, Quality Wh t Q lit S t Q lit Control & Quality Assurance Control & Quality Assurance There are a number of different activities that There are a number of different activities that laboratories undertake to ensure that results meet acceptable standards. These are a number t t bl t d d Th b y p of closely inter‐related concepts which will now define.
Q lit S t Quality Systems ‐ 1 A Quality System is defined in ISO 15189 as including the following: Quality policy Staff education and training Quality assurance Document control Records, maintenance and archiving Accommodation and environment Instruments, reagents and/or relevant consumables management • Validation of examination procedures • • • • • • •
Q lit S t Quality Systems ‐ 2 A Quality System is defined in ISO 15189 as including the following: • Safety, Safety, transportation, consumables and waste disposal transportation, consumables and waste disposal • Validation of results Quality control (including inter‐laboratory laboratory • Quality control (including inter comparisons) Communications and other interactions with patients • Communications and other interactions with patients, health professionals, referral laboratories and suppliers, and Internal audits.
Quality Control is a procedure that is used to monitor Quality Control is a procedure that is used to monitor analytical measurements and evaluate when medically p p p important errors have occurred. There are three purposes of Quality Control. 1. To monitor the accuracy and precision of the complete analytical process. 2. To detect immediate errors that occur due to test‐system failure adverse environmental conditions and operator failure, adverse environmental conditions, and operator performance. 3. To monitor over time the accuracy and precision of test p performance that may be influenced by changes in test y y g system performance, environmental conditions and variance in operator performance.
Q lit C t l QC Quality Control ‐ The Quality Control procedures that are used in clinical chemistry involve the analysis of control clinical chemistry involve the analysis of control materials and the plotting of their results on QC charts QC rules are then applied to ensure that charts. QC rules are then applied to ensure that analysers are performing according to specifications. ifi ti QC ensures an assay is performing according to specifications by running quality control specimens specifications by running quality control specimens routinely with each ‘run’ of patient samples.
Q lit C t l Quality Control An analytical run has met specifications if the quality control samples are within pre‐defined precision levels around a set mean – they are precise. These precision levels are set using statistical control theory. For repeated measurement of the same material the precision of these results can be obtained by h f h l b b db calculating their Standard Deviation. Statistical control theory then provides rules which are applied to the results of the quality control samples to li d t th lt f th lit t l l t identify if a result lies outside what is statistically likely.
Q lit A Quality Assurance vs Quality Control Q lit C t l • Quality Assurance
• Quality Control
An overall management system to guarantee the system to guarantee the integrity of data
A series of analytical measurements used to measurements used to assess the quality of the analytical data analytical data
The “System”
The “Tools”
E t External Quality Assurance l Q lit A An External Quality Assurance (EQA) or Proficiency Testing (PT) program is a program in which multiple samples are periodically sent to members of a group of laboratories for analysis and/or to members of a group of laboratories for analysis and/or identification, in which each laboratory’s results are compared with those of other laboratories in the group and/or with an assigned value, and reported to the participating laboratory and others. d t d t th ti i ti l b t d th Such a program may also compare an individual Such a program may also compare an individual’ss results with their results with their peer group. Participation in external quality assessment (EQA) is a mandatory requirement for meeting acceptable laboratory performance standards and extends beyond the analytical aspects of performance standards and extends beyond the analytical aspects of laboratory testing into all areas of laboratory function. NPAAC: Requirements for Participation in EQA (Fourth Edition 2009)
Difference between Quality Control y and EQA Quality Control procedures ensure that results for a run or batch are precise and fit for use for a run or batch are precise and fit for use. EQA allows a laboratory to ensure that the results of its assays are consistent with other results of its assays are consistent with other laboratories. As the EQA assigns both target values and values for allowable performance l d l f ll bl f they therefore test both the Accuracy and Precision of the laboratory.
• Quality Control ensures there is consistent i imprecision ii • EQAP can identify bias problems from a method/manufacturer group • If If there is a traceable target then EQAP can there is a traceable target then EQAP can identify true bias
EQA E EQA Examples l The following are examples from the RCPA EQA P Programs which show some of the information hi h h f th i f ti that can be obtained from EQA.
Consistent with method but bias when compared to all laboratories h h db b h d ll l b
Error • QC systems are designed to detect error B By using QC rules! i QC l ! • There are two types of error we can detect: 1 Systematic error or bias 1. Systematic error or bias 2. Random error or imprecision
From Cost‐effective Quality Control. Westgard and Barry. AACC Press. 1986
I Imprecision and Accuracy ii dA Precision is the agreement between replicate measurements and is measured by the standard di db h d d deviation of the replicates. Imprecision is caused by increased random error. Accuracy is agreement between the best estimate of quality (mean) and its true value. Inaccuracy is caused l ( ) d l d by increased systematic error.
T Types of Error fE Some Causes of Random Error: • Matrix interference – there is interference from another component in the quality control sample component in the quality control sample • Mechanical variation – there is fluctuation in ‐ detectors, li h light sources, monochromators, dispensing systems, h di i aspiration systems , mixers or laundering systems . • Electrical interference – there is a random electronic problem with the instrument • Specimen problems – There is carryover from a specimen • Variation in reagent quality • Variation in Calibrator consistency Variation in Calibrator consistency
C Causes of Random Error fR d E Inconsistent Reagent Preparation & Handling • Reagent deterioration / contamination • Lack of instrument maintenance Lack of instrument maintenance • Intermittent dripping ‐ sample / reagent probes • Intermittent Electronic Problems (eg) ‐ i l i bl ( ) Sticky solenoid Si k l id valves, shear valves • Scratched Reagent Mixers h d
T Types of Error fE Some Causes of Systematic Error: • Quality control material not prepared appropriately – too dilute or concentrated or a new batch of quality control material is being used • Reagent lot bias – a new reagent lot has a different background colour different background colour • Calibration – the method has been recalibrated with calibrator material that has deteriorated from when freshly prepared
Method Bias and Systematic Error h d d A certain level of Systematic Error will be inherent to a method causing a consistent deviation from the True value.
This leads to Method Bias where there is a constant difference between results from a particular method and results from between results from a particular method and results from other methods. This Method Bias is caused by: ‐ Differences in the Standardisation Diff i th St d di ti and tracebility dt bilit of the calibrator used in different methods ( what the method is standardised against ) ‐ Types of calibrators used ( Serum, Aqueous , Frozen, Freeze dried ) ‐ Interferences to the method ( Bilirubin, Lipaemia, Haemoglobin, Drugs ) g , g ) But Systematic error in Quality Control is an additional error to this Method Bias.
D i i Designing a QC System QC S t ISO 15189 describes the requirements of an internal Quality Control Program. The laboratory shall design Quality Control Program. The laboratory shall design internal quality control systems that verify the attainment of the intended quality of results It is attainment of the intended quality of results. It is important that the control system provide staff members with clear and easily understood information on which to with clear and easily understood information on which to base technical and medical decisions. Special attention should be paid to the elimination of mistakes in the should be paid to the elimination of mistakes in the process of handling samples, requests, examinations, reports, etc. (ISO 15189: 2007 Clause 5.6.1) t t (ISO 15189 2007 Cl 5 6 1)
D i i Designing a QC System QC S t The laboratory shall document its quality control plan in Th l b t h ll d t it lit t l l i detail, including the levels of quality control materials run each day, frequency of performing QC, types of QC materials each day, frequency of performing QC, types of QC materials and the QC acceptance criteria customised for each examination procedure based on that procedure’s capabilities. Acceptable ranges (confidence limits) must be defined for i t internal quality control material. Where acceptable ranges are l lit t l t i l Wh t bl set to limits other than ± 2SD based on current analytical performance, the rationale for the limits should be performance, the rationale for the limits should be documented. (Supplementary Requirements for accreditation in the field of Medical Testing, NATA)
Q lit C t l S Quality Control Sera Quality Control samples are substances that are inserted into the run alongside the test materials and subjected to exactly the same treatment. A control material must contain an h A l i l i appropriate concentration of the analyte, and a value of that concentration must be assigned to the material Control concentration must be assigned to the material. Control materials act as surrogates for the test materials and must therefore be representative, i.e., they should be subject to the same potential sources of error. To be fully representative, a i l f T b f ll i control material must have the same matrix in terms of bulk composition including minor constituents that may have a composition, including minor constituents that may have a bearing on accuracy. There are other essential characteristics of a control material. It must be adequately stable over the period of interest.
Q lit C t l S Quality Control Sera Control materials need to be different from the calibrator materials to ensure that the QC lib t t i l t th t th QC p procedure provides an independent assessment p p of system performance. (CLSI C24‐A3: 2006) Laboratories should establish their own means Laboratories should establish their own means and ranges rather than use product insert ranges. (CLSI C24‐A3: 2006)
Setting the Mean and SD of a new g batch of QC material (CLSI) Over a 20 days period, assay one set of new QC material on each day along with the existing QC materials. After the runs are accepted based on the existing QC materials, calculate the mean, standard deviation (SD) and i l l l h d d d i i (SD) d coefficient of variance (%CV) for the new QC material. Prepare a tentative QC chart for the analyte. Set mean ± 2SD, the 95% confidence limit as the temporary target ranges the 95% confidence limit, as the temporary target ranges, which should be less than the mean ± ALE. Ideally the new lot of QC material should overlap with the existing lot of QC for at least 20 batches of assays.
General guidelines for planning and g p g design of QC procedures (CLSI) The essential steps for planning a statistical QC procedure are presented as follows: procedure are presented as follows: 1. Define the quality requirement for the test. 2 Determine method precision and bias 2. Determine method precision and bias. 3. Identify candidate QC procedures. 4. Predict QC performance. 5 Set goals for QC performance 5. Set goals for QC performance. 6. Select an appropriate QC procedure.
F Frequency Of QC Of QC Although minimum regulatory standards exist for determining QC testing frequency, g g y gQ g q y, decisions regarding when and how to run QC samples are not standardized. Most QC testing strategies test control samples at fixed time intervals, often placing the samples in the same position on an instrument during subsequent QC events and leaving large gaps of time when control position on an instrument during subsequent QC events and leaving large gaps of time when control samples are never run, yet patient samples are being tested. Strategies for QC Testing Schedules St t i f QC T ti S h d l • Strategy 1: QC events scheduled at fixed time intervals. • Strategy 2: QC events randomly scheduled within fixed time intervals. • Strategy 3: QC events scheduled at random intervals. • Strategy 4: QC events scheduled at a random interval, followed by a series of n QC events scheduled at fixed intervals. at fixed intervals. • Strategy 5: The average interval between QC events was set at eight hours for all of the evaluated scheduling strategies. An Analytical run is generally defined by CLIA as an 8 hour to 24 hour interval during which control materials must be analyzed.
F Frequency of QC samples f QC l The level of QC applied in the laboratory varies according to the number of analytical runs and the specimens to the number of analytical runs and the specimens analyzed per day. The following protocol may be adopted y g by the laboratories according to the total number of specimens analyzed per analyte: Less than 50 per day ‐ apply at least one level of QC apply at least one level of QC • Less than 50 per day once a day. • Between 50‐100 per day ‐ Between 50 100 per day apply two level of QCs at apply two level of QCs at least once a day. • More than 100 per day ‐ M th 100 d apply two level of QCs at least l t l l f QC t l t twice a day.
How to implement an QC program p Q p g 1. Establish written policies and procedures. 2. Assign responsibility for monitoring and reviewing. 2. Assign responsibility for monitoring and reviewing. 3. Train staff. 4 Obtain control materials 4. Obtain control materials. 5. Collect data. 6 Set target values (mean SD) 6. Set target values (mean, SD). 7. Establish Levey‐Jennings charts. 8 R i l l 8. Routinely plot control data. ld 9. Establish and implement troubleshooting and corrective action protocols. ti ti t l 10. Establish and maintain system for documentation.
C t lR l Control Rules The laboratory uses control rules to assess if an analytical run has performed according to l ti l h f d di t p specifications. The laboratory assesses this on the basis of repeated measurements of quality control repeated measurements of quality control material and comparing these against statistical rules.
Q lit C t l Quality Control Each laboratory will have a quality control system which consists of quality control material, quality which consists of quality control material, quality control rules and a process to follow when the rules alert the operator to a failure or warning. p g Quality control material will often consist of Quality control material will often consist of multiple samples containing analytes at different concentration levels concentration levels. Quality Control rules are usually different for l l l ll d ff f different laboratories.
Si l Q lit C t l l Simple Quality Control rule As an example of how these quality control rules work consider the 12s rule which rejects an analytical run (or warns of analytical problem) when 1 control observation exceeds the f l i l bl ) h 1 l b i d h 2s control limits (2 standard deviations from the mean). If a quality control sample is measured many times the mean and standard deviation of these measurements can be and standard deviation of these measurements can be calculated. Then 95% of all measurements will lie within the range mean +/‐ 2SD. This means that 19 of 20 results should fall within this range.
Repeated measurement of QC samples will follow a Gaussian distribution
Si l Q lit C t l l Simple Quality Control rule If we have more than 1 in 20 results lying outside this range then there may be an error as this is this range then there may be an error as this is statistically unlikely (likely only 5% of the time). If we look at those samples that lie within the mean +/‐ 3s then this should include 99% of all values. +/ 3s then this should include 99% of all values. A result that lies outside this range is very unlikely (only likely 1% of the time). This is called the 13s rule.
A Q lit C t l Ch t A Quality Control Chart A quality control chart is a plot (electronically or manually) of the values obtained when the quality control samples are run. The different QC levels may be plotted on individual graphs or combined onto a single graph. This is a plot of concentration against time. The next slide is a Quality Control chart where the mean is 16 14 and the Quality Control chart where the mean is 16.14 and the standard deviation is 1.66. Each time point represents the q quality control samples for consecutive days on the chart. y p y 1, 2 and 3 SDs are marked. ,
Quality Control Chart. Note that on 19/12 the 13s rule was breached.
B ki Breaking a Quality Control rule Q lit C t l l There are two possible situations where a quality control rule is broken. control rule is broken. The mean of the quality control samples has The mean of the quality control samples has shifted. So all the results of the quality control samples are moved up or down relative to the ‘true’ samples are moved up or down relative to the true mean. The standard deviation of the quality control samples has shifted relative to the ‘true’ standard l h hf d l h ‘ ’ d d deviation.
B ki Breaking a Quality Control rule Q lit C t l l If the mean of the quality control samples shifts then there is a systematic error or bias present. then there is a systematic error or bias present The bias may be the same at all concentrations or it may be proportional where the bias differs i b i l h h bi diff at different concentrations. If the standard deviation of the quality control If th t d d d i ti f th lit t l samples broadens then there is random error present.
Oth Q lit C t l l Other Quality Control rules There are other Quality Control rules available. Th They are all based on detecting statistically ll b d d t ti t ti ti ll y q y p unlikely combinations of quality control sample results. These rules are examples of the Westgard These rules are examples of the Westgard rules rules which are available at the Westgard site (http://www.westgard.com/)
Popular control rules for the interpretation of control Popular control rules for the interpretation of control sample measurements 12s
Reject an analytical run (or warn of analytical problem) when 1 control observation exceeds the 2s control limits; usually used as a warning ; y g
1 2.5s Reject an analytical run when 1 control observation exceeds the x + 2.5s control limits (where x = mean) 1 3s
Reject an analytical run when 1 control observation Reject an analytical run when 1 control observation exceeds the x + 3s control limits
1 3.5s Reject an analytical run when 1 control observation exceeds the x + 3.5s control limits exceeds the x + 3 5s control limits
Popular control rules for the interpretation of control Popular control rules for the interpretation of control sample measurements 2 2s
Two control observations are on the same side of the mean and exceed either the x + 2s or x ‐ 2s control mean and exceed either the x + 2s or x ‐ 2s control limits
Reject an analytical run when 4 consecutive control observations are on the same side of the mean and exceed either the x + 1s or x ‐ 1s control limits
4 1s
10
x
Reject an analytical run when 10 consecutive control Reject an analytical run when 10 consecutive control observations are on the same side of the mean
Popular control rules for the interpretation of control Popular control rules for the interpretation of control sample measurements R 4s
Reject an analytical run if the range or difference between the maximum and minimum control between the maximum and minimum control observation exceeds 4s.
X 0.01 Reject an analytical run if the mean of the last ‘n’ control observation exceeds the control limits control observation exceeds the control limits that give a 1% frequency of false rejection (Pfr = 0 01) 0.01) R 0.01 0 01
Reject an analytical run if the range of the last ‘n’ j y g control observation exceeds the control limits that give a 1% frequency of false rejection that give a 1% frequency of false rejection
Diagrammatically these rules look like g y the following: See http://www.westgard.org/index.pl?id=63991
These rules may be used in various h l b d combinations such as the following: combinations such as the following:
• • •
13s / 22s 13s / 22s / R4s 13s / 22s / R4s / 41s / 10x
C bi i th Combining the rules l Using more than one rule increase the power of th the rules to detect error. l t d t t
I In control t l The assay is said to be in control if none of the quality control rules are breached. The 12s rule is often used as a warning rule. That is the assay should be monitored carefully but allow results to be h ld b i d f ll b ll l b released. The 13s rule is a reject rule which means no results should be released. released If any other rule is breached then no results should be If any other rule is breached then no results should be released. This may involve using an algorithm such as the next slide.
In response to the failed QC results, one of three p options can be chosen: 1. CONTINUE ‐ continue without change, if false alarm/rejection is identified. l / j ti i id tifi d 2. PAUSE ‐ Stop performing the assay ‐ troubleshoot and continue when fixed troubleshoot and continue when fixed. 3. STOP ‐ Stop releasing results ‐ troubleshoot and rerun previous samples after corrective and rerun previous samples after corrective action(s).
Wh t t d if QC l i b What to do if a QC rule is breached h d Corrective action must be taken and documented when control results exceed defined tolerance limits. This allows other staff to see what has occurred and aids in future troubleshooting if the problem reoccurs occurred and aids in future troubleshooting if the problem reoccurs. Patient test results will not be reported when controls do not yield acceptable results. lt Other corrective actions in response to Shifts, Drifts and Trends, etc. p , , The laboratory personnel performing the test should determine the appropriate action to be taken for QC data that fall outside the established appropriate action to be taken for QC data that fall outside the established tolerance limits. Corrective action should be documented with the technician's Initials and C i i h ld b d d ih h h i i ' I ii l d Date.
Q lit C t l t i l Quality Control material Quality control material is used to assess whether or not the analytical process is in h th t th l ti l i i q y control. However quality control material is different to patient samples and care must be exercised when interpreting quality control exercised when interpreting quality control results to ensure the material has been used correctly.
Q lit C t l M t i l Quality Control Material Problems occur if the material is not prepared correctly or not stored correctly correctly or not stored correctly. Common problems are: 1 QC material incorrectly prepared 1. QC t i l i tl d 2. Material not stored correctlyy 3. Out of date material used
What to do if a Quality Control rule is y breached? There should be a documented approach to d li dealing with a failed quality control result. The ith f il d lit t l lt Th pp approach needs to be consistent across all staff and times of the day. There will be a number of common procedures There will be a number of common procedures used for any failed quality control rule.
Troubleshooting a failed quality g q y control rule. QC material Check that the correct quality control sera has been used (lot number/level) Check that the quality control sera is fit for purpose (in date, fresh) Procedure: Repeat the quality control analysis with the correct Procedure: Repeat the quality control analysis with the correct material.
Reagent Causes Check the reagent batch number to ensure the batch is current g Check for reagent shortages or incorrect reagents Check that the reagent has not deteriorated Procedure: change the reagent
Troubleshooting a failed quality g q y control rule. Instrument failure Check for blockages in probes or tubing g p g Check for instrument flags Are multiple assays breaching QC rules p y g Procedure: Repair instrument, perform preventative maintenance
Calibration Causes Check when the instrument was last calibrated – was it before this run? Check the instrument is calibrated Check the instrument is calibrated Procedure : recalibrate the assay
An example of an algorithm to follow if there is a quality control failure.
