DIAGNOSTIC VALUE OF DIFFERENT ADHERENCE MEASURES

Download (AACTG) questionnaire,18 and the Self-Reported Adherence. (SERAD)19—have been validated against electronic moni- toring. The SMAQ assessed ...
0 downloads 481 Views 166KB Size
Download PDF
Love PNG Images
AIDS PATIENT CARE and STDs Volume 22, Number 9, 2008 © Mary Ann Liebert, Inc. DOI: 10.1089/apc.2007.0229

Diagnostic Value of Different Adherence Measures Using Electronic Monitoring and Virologic Failure as Reference Standards Ann E. Deschamps, M.S.N.,1 Sabina De Geest, Ph.D., R.N.,2,3 Anne-Mieke Vandamme, Ph.D.,4 Herman Bobbaers, M.D., Ph.D.,1 Willy E. Peetermans, M.D., Ph.D.,1 and Eric Van Wijngaerden, M.D., Ph.D.1

Abstract

Nonadherence to antiretroviral therapy is a substantial problem in HIV and jeopardizes the success of treatment. Accurate measurement of nonadherence is therefore imperative for good clinical management but no gold standard has been agreed on yet. In a single-center prospective study nonadherence was assessed by electronic monitoring: percentage of doses missed and drug holidays and by three self reports: (1) a visual analogue scale (VAS): percentage of overall doses taken; (2) the Swiss HIV Cohort Study Adherence Questionnaire (SHCS-AQ): percentage of overall doses missed and drug holidays and (3) the European HIV Treatment Questionnaire (EHTQ): percentage of doses missed and drug holidays for each antiretroviral drug separately. Virologic failure prospectively assessed during 1 year, and electronic monitoring were used as reference standards. Using virologic failure as reference standard, the best results were for (1) the SHCS-AQ after electronic monitoring (sensitivity, 87.5%; specificity, 78.6%); (2) electronic monitoring (sensitivity, 75%; specificity, 85.6%), and (3) the VAS combined with the SHCS-AQ before electronic monitoring (sensitivity, 87.5%; specificity, 58.6%). The sensitivity of the complex EHTQ was less than 50%. Asking simple questions about doses taken or missed is more sensitive than complex questioning about each drug separately. Combining the VAS with the SHCSAQ seems a feasible nonadherence measure for daily clinical practice. Self-reports perform better after electronic monitoring: their diagnostic value could be lower when given independently.

Introduction

A

DHERENCE TO ANTIRETROVIRAL THERAPY is crucial for good virologic and clinical outcomes,1–3 however, nonadher-

ence is common in HIV.4,5 Successful clinical management therefore demands nonadherence assessment, however, a reference standard to measure it has, as yet not been agreed on.6–8 Of the measurement methods available, electronic monitoring correlates strongest with virologic outcome7 and has been shown in cross-validation studies9–12 to be the most sensitive. Disclosure issues and expense make electronic monitoring problematic for daily practice.13 Therefore, the most frequently used nonadherence measurement method is selfreporting via questionnaires or interviews, which offers high

feasibility, low cost, low patient and staff burden, and a significant correlation with viral load.14,15 However, compared to electronic monitoring, self-reports tend to underestimate nonadherence and validation studies are scarce.7 Only four self-report questionnaires—the simplified medication adherence questionnaire (SMAQ),16 the visual analogue scale (VAS),17 the Adult AIDS Clinical Trials Group (AACTG) questionnaire,18 and the Self-Reported Adherence (SERAD)19—have been validated against electronic monitoring. The SMAQ assessed nonadherence using three qualitative “yes/no” questions about forgetting, being careless or stopping medication, and three quantitative questions asking about missed doses over the preceding week, weekend, and 3 months. Using fewer than 90% doses taken as measured by electronic monitoring as the reference standard,

1University

Hospitals Leuven, Department of Internal Medicine, Katholieke Universiteit Leuven, Leuven, Belgium. of Nursing Science, University of Basel, Basel, Switzerland. 3Center for Health Services and Nursing Research, School for Public Health, Faculty of Medicine, Leuven, Belgium. 4Rega Institute, Katholieke Universiteit Leuven, Leuven, Belgium. 2Institute

735

736

DESCHAMPS ET AL.

the sensitivity was 72% and specificity was 91%. Using electronic monitoring, Walsh et al.17 assessed the diagnostic value of the VAS and the AACTG. The VAS asked participants to mark on a scale from 0% to 100% the proportion of doses they have taken over the last month. The AACTG assessed nonadherence by asking, “How many doses did you miss . . . (1) yesterday? (2) the day before yesterday? and (3) the day before that day.” The reference standard was fewer than 95% doses taken, measured by electronic monitoring. Sensitivity and specificity were 64% and 77% for the VAS and 43% and 96% for the AACTG (respectively in both cases). Oyugi et al.20 compared correspondences between the VAS, the AACTG, and electronic monitoring. All measures were closely related (r  0.77–0.89) but the diagnostic value was not reported. Munoz et al.19 compared the SERAD, assessed as an interview, with electronic monitoring. They found adequate levels of agreement between both measures when adherence was high, but lower levels of agreement when adherence was low. Furthermore, the diagnostic value of the SERAD was not reported. Given its correlation with viral load,15 nonadherence questionnaires are often validated with viral load, based mainly on retrospective data from the medical file.14 However, since the goal of antiretroviral therapy is sustained viral suppression and nonadherence assessment aims at detecting patients at risk for virologic failure, nonadherence questionnaires should be validated against future virologic failure. Godin21 assessed the diagnostic value of seven nonadherence questions against viral load change over the following 6 months. Nonadherence was indicated by an increased viral load and adherence by a decreased, stable, or unstable viral load. The latter indicator for adherence could be particularly problematic, since increases in viral load after a decrease could lead to drug-resistance, resulting in virologic failure. Furthermore, only the question about missed doses over the preceding 7 days reached a sensitivity and specificity of 62% and 67%, respectively.

The Swiss HIV Cohort Study recently integrated two adherence items into its follow-up: the SHCS-AQ assesses overall doses missed and drug holidays over the past 4 weeks: (1) “How often did you miss a dose of your HIV medication? (6-point scale from “never” to “every day”) and (2) “Did you forget 2 doses one after the other? Yes/no.” There was a strong linear relationship between the number of selfreported missed doses and optimal viral suppression (p  0.0001).22 However, data on that relationship’s diagnostic value are not yet available. The aim of this study was to explore the diagnostic value of two nonadherence measures: electronic monitoring and self-reports. Self-reported nonadherence was assessed using three questionnaires: (1) the VAS; (2) the SHCS-AQ; and (3) the European HIV Treatment Questionnaire (EHTQ) (developed for this study). Electronic monitoring and virologic failure at 1 year post-nonadherence assessment, were used as reference standards. Materials and Methods Design This 15-month prospective, descriptive, single-center study assessed nonadherence for the past 3 months using electronic monitoring and self-reported nonadherence for the past month (Fig. 1). Patients were recruited between September 2003 and December 2003. Virologic failure was explored using quarterly viral load assessments until 12 months after the second nonadherence assessment (until March 2004). Sample and setting A convenience sample of 149 patients on antiretroviral therapy (ART) participated. Subjects included were HIVpositive adults, on ART at least 1 month and in follow-up at the University Hospitals Leuven. All had to be Dutch-,

Time 1 (T1): baseline

Time 3 (T3): completion

Self report: VAS

Self report: VAS SHCS EHTQ Demographic variables CD4- cell count Viral load

SHCS EHTQ Demographic variables CD4- cell count Viral load

End of Study

Time 4 (T4): Assessment for virological failure: 12 months

Time 2 (T2): Electronic Monitoring: 3 months

VL

Months 0

VL

3

FIG. 1.

VL

VL

VL

VL

6

9

12

15

Design of the study.

DIAGNOSTIC VALUE OF NONADHERENCE MEASURES

737

French-, or English-speaking and able to manage their medication intake independently. Patients were excluded if visual or cognitive impairment prevented adequate communication or if they were participating in another study involving nonadherence assessment. Written informed consent was obtained from all participants.

The European HIV Treatment Questionnaire (EHTQ) assessed missed doses and DH for each antiretroviral drug separately over the past month. For each drug patients took, they were asked the following questions: (1) “How many times a day should you take these medicines?” and (2) “How many pills should you take each time?” TaA was assessed via the question: “During the last 30 days, how many TIMES did you not take any of these pills?” The DH assessment question was, “During the last 30 days, how many DAYS did you not take any of these pills?” Patients reported using a number. Four definitions of non-adherence were used: (1) mean TaA (TaAMEAN): the sum of TaA of all reported antiretrovirals, divided by the number of antiretrovirals; (2) mean DH (DHMEAN): the sum of DH of all reported antiretrovirals divided by the number of antiretrovirals; (3) monitored TaA (TaAMON): self-reported TaA of the antiretroviral electronically monitored; and (4) monitored DH (DHMON): self-reported DH of the antiretroviral monitored. Similar to the SHCS-AQ, two definitions of nonadherence were used: either less than 100% TaAMEAN or 1 or more DHMEAN per 30 days; or less than 100%TaAMON or 1 or more DHMON per 30 days.

Variables and measurement Collected sociodemographic factors were age, race, gender, transmission mode, marital status, living situation, employment status, and education. Clinical data were months diagnosed with HIV, months on therapy, number of regimen changes, CD4 cell count and viral load. These clinical data were collected from the medical file. Viral load was assessed using the Quantiplex HIV-1 RNA 3.0 test (Chiron, Emeryville, CA; detection limit: 50 copies per milliliter). Virologic failure was defined as at least 1 viral load above 400 copies per milliliter, or as at least 2 consecutive viral loads above 50 copies per milliliter in previously fully suppressed patients. Nonadherence was assessed by electronic monitoring and twice by three self-report types: (1) the VAS, (2) the Mems® (Aardex, Switzerland), and (3) the EHTQ (Fig. 1). MEMS® consists of a pill bottle with a cap containing a microchip that registers the date and time of each bottle opening, showing individual medication-taking dynamics. One protease inhibitor or non-nucleoside analogue inhibitor was monitored per patient during 3 months. Nonadherence was defined as (1) taking adherence (TaA): the percentage of doses taken compared to the total doses prescribed and (2) drug holidays (DH): no medication intake for 24 hours  50% of the dosing interval of the medication. DH data were expressed as DH per 100 days monitored. Patients were instructed to always take their pills from the electronic monitoring-dispenser. Patients were asked to note the date and time of their actual medication intake in a diary, when privacy or disclosure issues prevented them from using the electronic monitoring dispenser, or if the dispenser was accidentally opened. These data were integrated into the electronic monitoring dataset. Reliability of electronic monitoring data was assessed by a structured interview upon completion of the study and was based on an algorithm developed for this purpose.23 If the patient self-reported never or occasionally using the electronic monitoring dispenser and there were no notes, the electronic monitoring data were considered unreliable and excluded from analysis. The VAS assessed overall doses taken over the past month with a visual analogue scale ranging from 0% to 100%.17 Patients had to mark the scale at the point showing their estimated proportion of doses taken. Nonadherence was defined as less than 100% TaA. The SHCS-AQ22 assessed overall doses and DH missed over the past 4 weeks: (1) TaA: “How often did you miss a dose of your HIV medication? (6-point scale from “never” to “every day” and (2) DH: “Did you forget 2 doses one after the other? Yes/no.” Patients admitting that they had forgotten 1 or more dose during the last 4 weeks or that they had missed two consecutive doses—equivalent to less than 100% TaA or 1 or more DH in 4 weeks—were considered nonadherent.

Data Collection Nonadherence was assessed by electronic monitoring for 3 to 4 months, the interval between two consecutive clinic appointments (Fig. 1). Self-reported nonadherence was assessed using the SHCS-AQ, the VAS and the EHTQ immediately before and after the electronic monitoring-period (Fig. 1). Patients first completed the SHCS-AQ, received a 20minute briefing on the use of the electronic monitoring-dispenser, then filled out the VAS and the EHTQ. Upon completion of their participation, patients filled out the SHCS-AQ, then responded to the reliability interview for the electronic monitoring data, and finally completed the VAS and the EHTQ again. We assessed virologic failure for 1 year, using consecutive viral loads collected quarterly. Statistical analysis Descriptive and comparative statistics were applied as appropriate. Comparison for gender, race, transmission pathway, viral load at inclusion (50 or 50 copies per milliliter) and virologic failure (yes/no), was performed using Fisher’s exact test. Kruskal-Wallis testing was used to compare age, months diagnosed, months on therapy, number of changes in ART regimen, and CD4 cell count. We used an iterative partitioning method and visual inspection of the raw electronic monitoring data to develop an electronic monitoring algorithm that categorized patients as adherent or nonadherent. Based on our previous finding of a 40-day intervention effect of electronic monitoring on adherence,24 we analyzed two periods: the total monitoring period and the total monitoring period minus the first 40 days—resulting in two electronic monitoring algorithms: EMTOTAL and EM40. Both algorithms were then post hoc validated with virologic failure at 1 year. The diagnostic value of both electronic monitoring-algorithms and the self-reported nonadherence questionnaires was explored with virologic failure at 1 year as the reference standard, using cross tabs and receiver-operature curves (ROC).

738

DESCHAMPS ET AL.

We also explored the diagnostic value of the questionnaires using electronic monitoring, again with cross tabs and ROC. Statistical significance was established at the p  0.05 level. Analyses were performed using SPSS 11.0 (SPSS, Inc., Chicago, IL).The ethics committee of the University Hospitals Leuven approved this study’s protocol and informed consent procedures. Results Sample Between September 2003 and December 2003, a total of 179 patients with HIV were in follow-up at the University Hospitals Leuven and on ART. Thirty patients were excluded: 7 participated in the pilot study, 8 had languages problems, 13 participated in another study with adherence measurement, and 2 were too ill. Therefore, 149 patients were eligible to participate, 16 of whom declined, resulting in 133 participants (Fig. 2). Ten patients were excluded because they used the electronic monitoring-dispenser inconsistently: 6 patients never used the electronic monitoring device, 2 patients lost the device, 1 patient disclosed having used the device only occasionally for 46 days but did not provide diary notes to correct the electronic monitoring data. One patient systematically manipulated the electronic monitoring device. The electronic monitoring sample thus consisted of 123

patients. Virologic failure could not be assessed in 4; therefore, the final EMTOTAL sample was 119 patients. Seven patients stopped with electronic monitoring within 45 days and were excluded, resulting in reliable EM40 data for 112 participants. Demographic and clinical data Comparisons between participants, excluded patients because of inconsistent use, patients who stopped prematurely, and refusers revealed that more participants were male (p  0.028); men who have sex with ment (MSM; p  0,001), and older (p  0.003). Other sociodemographic variables did not differ. No difference was apparent between the groups for all the clinical variables and none who declined had suffered virologic failure after 1 year. Development and post hoc validation of the electronic monitoring algorithms Using an iterative partitioning method and visual inspection of the compiled electronic monitoring-data (n  119) and of the electronic monitoring data minus the first 40 days (n  112), the electronic monitoring algorithm defining nonadherence was less than 90% TaA or more than 6 DH per 100 days monitored. The algorithms were identical for both periods. Table 1 outlines the electronic monitoring parameters for both algorithms. Overall median TaA was 98.9% and

On ART: 179

Excluded: 30

Eligible patients: 149

Refusers: 16

Participants: 133

Inconsistent use: 10

EM sample: 123

Virological failure not evaluative: 4

EMTOTAL sample: 119

Stopped premature: 7

FIG. 2.

Sample.

EM40 sample: 112

DIAGNOSTIC VALUE OF NONADHERENCE MEASURES

739

differed significantly between the adherent (100%) and nonadherence (55.1%) patients. Post hoc validation against virologic failure at 1 year between adherent and nonadherent patients showed that only 2.2 % of the adherent patients failed virologically, versus 28.6 % of the nonadherent patients.

Looking at each question separately at time 1, the VAS had the highest sensitivity (75%), a moderate specificity (60.6%), and a good AUC (0.678). Second best was the TaA question from the SHCS-AQ (sensitivity, 50%; specificity, 80.8%; and AUC, 0.654). The EHTQ’s TaA question had an excellent specificity (90%) but insufficient sensitivity ( 25%). When the questions were asked a second time after electronic monitoring, their diagnostic value improved. This was especially true for the SHCS-AQ question regarding DH, where sensitivity increased from 12.5% to 62.5%, while the AUC jumped from 0.534 to 0.803. Combining the SHCS-AQ questions on TaA and DH improved their diagnostic value, with an excellent sensitivity (87.5%), specificity (78.6%) and AUC (0.831). Within the EHTQ, combining the TaA and DH questions improved the diagnostic value neither at time 1 nor at time 3, and the sensitivity remained below 50%. Finally, combining the VAS with the SHCS-AQ questions increased both the sensitivity (to 87.5%) and AUC (to 0.736). Combining the EHTQ questions on TaA and DH with the VAS did not improve the diagnostic value of the VAS (data not shown).

Prevalence of nonadherence and virologic failure Overall adherence assessed regarding the various measures was high. Mean TaA was 93% for the EMTOTAL sample (n  119); 91% for the EM40 sample (n  112); 98% for the VAS (n  112) and 99% for the EHTQ (n  112). Median TaA was 99% for the EMTOTAL sample and the EM40 sample and 100% for the VAS and the EHTQ. Only 6.3% reported a DH and 79% reported 100% adherence, as assessed with the SHCS-AQ. The prevalence of virologic failure was 7.6% for the EMTOTAL sample and 7.1% for the EM40 sample. Table 2 outlines the prevalence of nonadherence for the various measures, based on the operational definitions. The prevalence of nonadherence for the EMTOTAL algorithm was 23.5%, compared to 18.8% for the EM40 sample. Examining each nonadherence question alone, we found that the prevalence of nonadherence was lowest regarding DH (5.5% to 9.8%), and highest regarding the VAS (42%). In most cases the prevalence of self-reported nonadherence was higher when the questions were asked a second time after electronic monitoring. Combining the TaA and DH questions in an “or” relationship also lead to a higher prevalence of nonadherence. Furthermore, combining the VAS with the questions of the SHCS-AQ increased the prevalence of nonadherence to 46.4%, compared to 42% for the VAS alone. Diagnostic value of the electronic monitoring algorithms, the VAS, the SHCS-AQ, and the EHTQ with virologic failure at one year Table 2 outlines the diagnostic values of both electronic monitoring algorithms and the three nonadherence questionnaires using virologic failure as the reference standard. The EM40 algorithm performed more reliably than the EMTOTAL algorithm, with excellent sensitivity (75%), specificity (85.6%), and an area under the curve (AUC) of 0.803.

TABLE 1.

Diagnostic value of the VAS, the SHCS-AQ, and the EHTQ using electronic monitoring as the reference standard Table 3 outlines the diagnostic value of the three questionnaires. Since the EM40 algorithm outperformed the EMTOTAL algorithm, we used the EM40 algorithm as reference standard. Looking at each question alone at time 1, the VAS again had the highest sensitivity: 57.1%. The SHCS-AQ TaA question had a lower sensitivity (42.9%) but a better AUC (0.632) due to a higher specificity (83.5%). Although the questions of the EHTQ resulted in a good AUC (0.640), not one question attained a sensitivity above 33%. When the questions were asked a second time after electronic monitoring, the diagnostic value increased consistently. This was again most striking for the SHCS-AQ drug holiday question, with a sensitivity increase from 19% to 33%, specificity of 100% and an AUC increase from 0.579 to 0.667.

POST HOC VALIDATION

OF

BOTH EM ALGORITHMS

EMTOTAL All n = 119 TaA (%) DH (n/100 days)

Adherent n = 91

Nonadherent n = 28

EM40 p-value

All n = 112

98.9 (7.7) 0.0 (3.33)

Median (IQR) 99.4 (1.67) 67.7 (44.43) 0.0 (0.0) 10 (11.6)

0.000a 0.000a

98.9 (3.33) 0.0 (2.36)

7.6

% VF at 1 year 3.3 21.4

0.005b

7.1

Adherent n = 91

Nonadherent n = 21

p-value

Median (IQR) 100 (2) 55.1 (36.39) 0.0 (0.0) 13.8 (8)

0.000a 0.000a

% VF at 1 year 2.2 28.6

0.000b

EMTOTAL, EM algorithm based on total period; EM40, EM algorithm based on total period minus first 40 days; EMTOTAL algorithm and EM40 algorithm: 90% taking adherence or  6 drug holidays/100 days monitored; TaA, taking adherence; DH, drug holidays; VF, virological failure. aMann Whitney test for comparison between adherent and nonadherent patients. bFisher exact test for comparison between adherent and nonadherent patients.

740

DESCHAMPS ET AL. TABLE 2.

DIAGNOSTIC VALUE OF THE EM ALGORITHMS, THE VAS, THE SHCS-AQ, AND THE EHTQ USING VIROLOGICAL FAILURE AT ONE YEAR AS REFERENCE STANDARD (n = 112) Prev NA%

Sens%

Spec%

PPV%

NPV%

AUC (95% CI)

EMTOTAL (n = 119) EM40

23.5 18.8

66.7 75

80 85.6

21.4 28.6

96.7 97.8

0.73 (0.55–0.92) 0.80 (0.62–0.99)

VAS VAS (T1) VAS (T3)

42 38.4

75 62.5

60.6 63.5

12.8 11.6

96.9 95.7

0.68 (0.49–0.84) 0.63 (0.49–0.83)

SHCS-AQ TaA (T1) TaA (T3) DH (T1) DH (T3) TaA or DH (T1) TaA or DH (T3)

21.4 23.2 6.3 6.3 23.2 26.1

50 50 12.5 62.5 50 87.5

80.8 78.8 94.2 98.1 78.8 78.6

16.7 15.4 14.3 71.4 15.4 24.1

95.5 95.3 93.3 97.1 95.3 98.8

0.65 0.64 0.53 0.80 0.64 0.83

(0.44–0.87) (0.43–0.86) (0.31–0.75) (0.59–1.00) (0.43–0.86) (0.69–0.97)

EHTQMEAN TaA (T1) TaA (T3) DH (T1) DH (T3) TaA or DH (T1) TaA or DH (T3)

9.9 15.8 7.3 9.8 10.8 16.8

25 42.9 12.5 28.6 25 42.9

91.3 86.2 93.1 91.6 90.3 85.1

18.2 18.8 12.5 20 16.7 17.6

94 95.3 93.1 94.6 93.9 95.2

0.59 0.65 0.53 0.60 0.58 0.64

(0.36–0.81) (0.40–0.88) (0.31–0.74) (0.36–0.84) (0.36–0.80) (0.40–0.88)

EHTQMON TaA (T1) TaA (T3) DH (T1) DH (T3) TaA or DH (T1) TaA or DH (T3)

9 12.9 5.5 9.8 9 13.9

12.5 42.9 0 28.6 12.5 42.9

91.3 89.4 94.1 91.6 91.3 88.3

10 23.1 0 20 10 21.4

93.1 95.5 92.3 94.6 93.1 95.4

0.52 0.66 0.47 0.60 0.52 0.66

(0.31–0.74) (0.42–0.90) (0.27–0.67) (0.36–0.84) (0.31–0.74) (0.42–0.98)

VAS or SHCS-AQ VAS (T1) or TaA(T1) VAS (T3) or TaA (T3) VAS (T1) or DH (T1) VAS (T3) or DH (T3) VAS (T1) or TaA or DH (T1) VAS (T3) or TaA or DH (T3)

43.8 45.5 43.8 40.2 44.6 46.4

87.5 75 75 75 87.5 87.5

59.6 56.7 58.7 62.5 58.7 56.7

14.3 11.8 12.2 13.3 14 13.5

98.4 96.7 96.8 97 98.4 98.3

0.74 0.66 0.69 0.69 0.73 0.72

(0.58–0.89) (0.47–0.85) (0.48–0.85) (0.50–0.87) (0.58–0.89) (0.56–0.88)

EM, electronic monitoring; VAS, visual analogue scale; SHCS-AQ: Swiss HIV Cohort Study Adherence Questionnaire; EHTQ, European HIV Treatment Questionnaire; VF, virological failure; EMTOTAL, electronic monitoring algorithm based on the total period; EM40, electronic monitoring algorithm based on the total period minus the first 40 days; TaA, taking adherence; DH, drug holidays; both EM algorithms, 90 TaA or  6 DH/100 days monitored; EHTQMEAN, mean nonadherence assessed by the EHTQ; EHTQMON, nonadherence assessed by the EHTQ for the electronic monitored drug; T1, inclusion; T3, completion; Prev NA, prevalence of nonadherence; Sens, sensitively; Spec, specificity; PPV, positive predictive value; AUC, area under, the curve.

Combining the questions on TaA and DH within the SHCS-AQ resulted in increases in sensitivity and AUC, especially after EM. Combining the EHTQ TaA and DH questions also increased the AUC, but the sensitivity remained below 44%. Finally, combining the VAS with the SHCS-AQ questions improved the sensitivity to more than 60%, with an AUC of 0.663. Combining the EHTQ with the VAS did not improve the diagnostic value of the VAS (data not shown). Discussion To our knowledge, this is the first HIV-related study to explore the diagnostic values of electronic monitoring and self report as adherence measures with two reference stan-

dards. Self-reported nonadherence was assessed with three types of questions (overall doses taken, overall doses missed and drug holidays, and doses missed and drug holidays for each antiretroviral separately). As assessed using electronic monitoring, nonadherence was based on a clinically validated electronic monitoring algorithm that allowed more accurate nonadherence patient detection. A first finding in our study is the good adherence in our study population, ranging from 91% to 99%—similar as in some studies12,17,20 but significantly higher than the 70% reported elsewhere.20 The high level of adherence is consistent with the low prevalence (7%) of virologic failure in our study. Adherent and nonadherent patients differed significantly in view of taking adherence, drug holidays, and virologic failure. Furthermore, excluding the first 40 electronic moni-

DIAGNOSTIC VALUE OF NONADHERENCE MEASURES

741

TABLE 3. DIAGNOSTIC VALUE OF THE VAS, THE SHCS-AQ, AND THE EHTQ USING THE EM40 ALGORITHM AS REFERENCE STANDARD (n = 112) Sens%

Spec%

PPV%

NPV%

AUC (95% CI)

VAS VAS (T1) VAS (T3

57.1 57.1

61.5 65.9

25.5 27.9

86.2 87

0.59 (0.46–0.73) 0.62 (0.48–0.75)

SHCS-AQ TaA (T1) TaA (T3) DH (T1) DH (T3) TaA or DH (T1) TaA or DH (T3)

42.9 38.1 19 33.3 42.9 52.4

83.5 80.2 96.7 100 81.3 80

37.5 30.8 57.1 100 34.6 37.9

86.4 84.9 83.8 0 86 87.8

0.63 0.59 0.58 0.67 0.62 0.66

(0.49–0.77) (0.45–0.73) (0.43–0.73) (0.52–0.82) (0.48–0.76) (0.52–0.80)

EHTQMEAN TaA (T1) TaA (T3) DH (T1) DH (T3) TaA or DH (T1) TaA or DH (T3)

33.3 38.9 23.8 33.3 38.1 44.4

95.6 89.2 96.6 95.2 95.6 89.2

63.6 43.8 62.5 60 66.7 47.1

86 87.1 84.3 87 86.9 88.1

0.65 0.64 0.60 0.64 0.67 0.67

(0.50–0.80) (0.49–0.80) (0.46–0.75) (0.48–0.80) (0.53–0.82) (0.51–0.82)

EHTQMON TaA (T1) TaA (T3) DH (T1) DH (T3) TaA or DH (T1) TaA or DH (T3)

28.6 33.3 14.3 33.3 28.6 38.9

95.6 91.6 96.6 95.2 95.6 91.6

60 46.2 50 60 60 50

85.1 86.4 82.7 87 85.1 87.4

0.63 0.62 0.56 0.64 0.63 0.65

(0.48–0.77) (0.47–0.78) (0.41–0.70) (0.48–0.80) (0.48–0.77) (0.50–0.80)

VAS or SHCS-AQ VAS (T1) or TaA(T1) VAS (T3) or TaA (T3) VAS (T1) or DH (T1) VAS (T3) or DH (T3) VAS (T1) or TaA or DH (T1) VAS (T3) or TaA or DH (T3)

61.9 61.9 57.1 66.7 61.9 66.7

60.4 58.2 59.3 65.9 59.3 58.2

26.5 25.5 24.5 31.1 26 26.9

87.3 86.9 85.7 89.6 87.1 88.3

0.61 0.60 0.58 0.66 0.60 0.63

(0.48–0.75) (0.47–0.74) (0.45–0.72) (0.53–0.79) (0.47–0.74) (0.49–0.75)

VAS, Visual Analogue Scale; SHCS-AQ, Swiss HIV Cohort Study Adherence Questionnaire; EHTQ, European HIV Treatment Questionnaire; Sens, sensitivity; Spec, specificity; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve; TaA: taking adherence; DH, drug holidays; EM40, electronic monitoring algorithm based on the total period minus the first 40 days: 90 TaA or  6 DH/100 days monitored; EHTQMEAN, mean nonadherence assessed by the EHTQ; EHTQMON, nonadherence assessed by the EHTQ for the electronic monitored drug; T1, inclusion; T3, completion.

toring days increased the electronic monitoring algorithm’s reliability. This supports our previous finding about the 40day intervention effect of electronic monitoring on medication intake behavior.24 We assume that the adherence behavior after that period is closer to the prestudy behavior. This implies the need either to include a run-in period or to exclude the first 40 days of electronic monitoring data from future studies.7,24 In exploring the diagnostic value of the nonadherence questions, we observed four interesting findings. First, simple questions about overall doses have higher diagnostic values than detailed questions for each separate drug. This was shown by a consistently higher sensitivity regarding the VAS and the SHCS-AQ compared to the EHTQ. In exploring the answers patients gave on the

EHTQ, we found that they consistently reported similar for all antiretrovirals they took simultaneously. This suggests that when patients missed a dose, they missed all the drugs taken at the same time. Two other studies exploring adherence for each antiretroviral separately, one with selfreport25 and one with electronic monitoring26 confirmed this finding. Oyugi et al.20 found no difference between adherence as reported by the VAS or the AACTG, where patients also reported for each antiretroviral separately. However, Oyugi and coworkers’ subjects took only 1 pill twice daily, which may explain the comparable results between the VAS and the AACTG. Second, the diagnostic value of the nonadherence questions increased when combined in an “or” relationship. Combining taking adherence and drug holidays within the SHCS-

742 AQ resulted in excellent sensitivity (87.5%), specificity (78.6%), and AUC (0.831). Third, the diagnostic value of the nonadherence questions improved when they were asked again after an electronic monitoring period, suggesting that their diagnostic value could be lower when given independently. To our knowledge, this is the first validation study with self-report and electronic monitoring to report on the performance of nonadherence questionnaires before and after electronic monitoring. Therefore, future validation studies of self-reports with electronic monitoring should examine its psychometric characteristics before and after the electronic monitoring period. Fourth, after electronic monitoring the SHCS-AQ performed surprisingly well in predicting virologic failure—better even than electronic monitoring (the reference standard) alone. As Bova13 observed, electronic monitoring remains an indirect measurement with some pitfalls. The basic assumption of electronic monitoring is that a recorded event directly reflects medication intake. However, the system can malfunction: 2.4% of the openings of the electronic monitoring dispenser were not recorded.13 In our study we did not measure this failure rate. Moreover, electronic monitoring interferes with medication intake organized by pill boxes. In our study 55 patients (41%) used a pill box while 36% of the participants and 75% of the dropouts and the patients who stopped prematurely admitted to not always having used the electronic monitoring bottle. Therefore, as was stated by DiMatteo,27 triangulation of diverse non-adherence measurements improves their diagnostic value. In a 15-month prospective study, LLabre and colleagues28 showed that combining electronic monitoring with non-adherence questions (self-reported and interview) predicted 45% in HIV viral load variability over time, compared to 20% to 24% for each measurement separately. However, 7 self-reported nonadherence measurements or 4 assessments by electronic monitoring, self-report and interview were needed to achieve an acceptable reliability. Therefore, as also suggested by the authors, their methodology is not feasible for daily practice. The choice of nonadherence measurement methods depends on measurement goals and available resources.8 In resource-limiting settings, large cohort studies and busy clinical practices, rapid screening for nonadherence is of utmost importance. For these settings, triangulation of the VAS and the SHCS-AQ is a valuable and feasible instrument. The instrument takes only a few minutes, does not require complex interpretation and the sensitivity is unsurpassed. When more accurate measurement is needed, i.e., in RCT’s comparing the efficacy of adherence enhancing interventions, triangulation of electronic monitoring with SHCS-AQ results or the EM40 algorithm is most promising given these systems’ high diagnostic values. This study has some limitations. Due to the small sample size, the low prevalence of virologic failure and the high level of adherence in our population, issues remain concerning the generalizability of the results. Self-reported adherence measures usually perform better in a highly adherent population than in a modestly or poorly adherent population, because over reporting is less an issue in highly adherent patients. The high adherence level could be due to only three intravenous drug users in our population, the extensive multidisciplinary counseling and the implementation of a comprehensive ad-

DESCHAMPS ET AL. herence counseling and retention into care program. Therefore, assessing the diagnostic value of the VAS and the SHCSAQ in a less adherent population is recommended. A second limitation of the study is the use of virologic failure as reference standard. Although is it imperative to detect patients at risk of virologic failure in order to support them with adherence enhancing interventions, virologic failure is not influenced solely by adherence: preexisting drug resistance and pharmacokinetic issues may also play important roles. A final limitation of this study is the assessment of nonadherence simply in view of doses taken or missed, without regard for patients’ beliefs, attitudes, and behaviors that are not assessed with these questions. Nonadherence assessment goals include not simply detecting patients at risk of virologic failure, but also identifying modifiable personal, socioenvironmental and health care setting-related determinants, thereby allowing development of individual adherence enhancing interventions. Liu and colleagues29 combined such covariates with self-reported non-adherence questions and found that the combination improved the sensitivity up to 93% compared to a composite adherence score based on electronic monitoring and pill count. Although the specificity dropped to less than 20%, the idea of combining self-reported nonadherence with social demographic, personal and health care-related variables to predict virologic failure merits further exploration, especially because these variables are less susceptible to social and recall bias. In conclusion, asking simple questions about overall doses taken or missed is more sensitive than complex questioning about individual drugs. For simplicity and excellent diagnostic value, combining the VAS with the SHCS-AQ provides a useful and feasible nonadherence measurement for daily clinical practice, large cohort studies and resource-limiting settings. For more accurate measurement, such as for RCT’s, use of the SHCS-AQ after EM or the EM40 algorithm may be preferable. Acknowledgments This study was partly sponsored by the European Commission, DG Sanco, Agreement N° SPC2002334 and an unrestricted grant from Bristol-Myers Squibb Belgium. This work was supported in part by FWO-Vlaanderen grant G.0266.04, and by the Katholieke Universiteit Leuven through Grant OT/04/43. Parts of this study were presented at the First and Second International Conference on HIV Treatment Adherence, Jersey City, March 2006 and March 2007; and at the Eight International Congress on Drug Therapy in HIV Infection, Glasgow, November 2006. References 1. Clavel F, Hance AJ. HIV drug resistance. N Engl J Med 2004;350:1023–1035. 2. Osterberg L, Blaschke T. Adherence to medication. N Engl J Med. 2005;353:487–497. 3. Bangsberg DR. Monitoring adherence to hiv antiretroviral therapy in routine clinical practice: The past, the present, and the future. AIDS Behav 2006;10:249–251. 4. Walsh JC, Dalton M, Gazzard BG. Adherence to combination antiretroviral therapy assessed by anonymous patient self-report. AIDS 1998;12:2361–2363.

DIAGNOSTIC VALUE OF NONADHERENCE MEASURES

743

5. Harzke AJ, Williams ML, Nilsson-Schonnesson L, Ross MW, Timpson S, Keel KB. Psychosocial factors associated with adherence to antiretroviral medications in a sample of HIVpositive African American drug users. AIDS Care 2004;16: 458–470. 6. Kerr T, Walsh J, Lloyd-Smith E, Wood E. Measuring adherence to highly active antiretroviral therapy: Implications for research and practice. Curr HIV/AIDS Rep 2005;2:200–205. 7. Berg KM, Arnsten JH. Practical and conceptual challenges in measuring antiretroviral adherence. J Acquir Immune Defic Syndr 2006;43(Suppl 1):S79–S87. 8. Chesney MA. The elusive gold standard: Future perspectives for HIV adherence assessment and intervention. J Acquir Immune Defic Syndr 2006;43(Suppl 1):S149–S155. 9. Arnsten JH, Demas PA, Farzadegan H, et al. Antiretroviral therapy adherence and viral suppression in HIV-infected drug users: Comparison of self-report and electronic monitoring. Clin Infect Dis 2001;33:1417–1423. 10. Liu H, Golin CE, Miller LG, et al. A comparison study of multiple measures of adherence to HIV protease inhibitors. Ann Intern Med 2001;134:968–977. 11. Hugen PW, Langebeek N, Burger DM, et al. Assessment of adherence to HIV protease inhibitors: comparison and combination of various methods, including MEMS (electronic monitoring), patient and nurse report, and therapeutic drug monitoring. J Acquir Immune Defic Syndr 2002;30:324–334. 12. Deschamps AE, De Graeve V, Van Wijngaerden E, et al. Prevalence and correlates of nonadherence to antiretroviral therapy in a population of HIV patients using Medication Event Monitoring System. AIDS Patient Care STDs 2004; 18:644–657. 13. Bova CA, Fennie KP, Knafl GJ, Dieckhaus KD, Watrous E, Williams AB. Use of electronic monitoring devices to measure antiretroviral adherence: Practical considerations. AIDS Behav 2005;9:103–110. 14. Simoni JM, Kurth AE, Pearson CR, Pantalone DW, Merrill JO, Frick PA. Self-report measures of antiretroviral therapy adherence: A review with recommendations for HIV research and clinical management. AIDS Behav 2006;10: 227–245. 15. Nieuwkerk PT, Oort FJ. Self-reported adherence to antiretroviral therapy for HIV-1 infection and virologic treatment response: A meta-analysis. J Acquir Immune Defic Syndr 2005;38:445–448. 16. Knobel H, Alonso J, Casado JL, et al. Validation of a simplified medication adherence questionnaire in a large cohort of HIV-infected patients: The GEEMA Study. AIDS. 2002;16: 605-613. 17. Walsh JC, Mandalia S, Gazzard BG. Responses to a 1 month self-report on adherence to antiretroviral therapy are consistent with electronic data and virological treatment outcome. AIDS 2002;16:269–277. 18. Chesney MA, Ickovics JR, Chambers DB, et al. Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: The AACTG adherence instruments. Patient Care Committee & Adherence Working Group of the Outcomes Committee of the Adult AIDS

Clinical Trials Group (AACTG). AIDS Care 2000;12: 255–266. Munoz-Moreno JA, Fumaz CR, Ferrer MJ, et al. Assessing self-reported adherence to hiv therapy by questionnaire: The SERAD (Self-Reported Adherence) Study. AIDS Res Hum Retroviruses 2007;23:1166–1175. Oyugi JH, Byakika-Tusiime J, Charlebois ED, et al. Multiple validated measures of adherence indicate high levels of adherence to generic HIV antiretroviral therapy in a resourcelimited setting. J Acquir Immune Defic Syndr 2004;36:1100– 1102. Godin G, Gagne C, Naccache H. Validation of a self-reported questionnaire assessing adherence to antiretroviral medication. AIDS Patient Care STDs 2003;17:325–332. Glass TR, De GS, Weber R, et al. Correlates of self-reported nonadherence to antiretroviral therapy in HIV-infected patients: The Swiss HIV Cohort Study. J Acquir Immune Defic Syndr 2006;41:385–392. Denhaerynck K, Steiger J, Bock A, et al. Prevalence and risk factors of non-adherence with immunosuppressive medication in kidney transplant patients. Am J Transplant 2007; 7:108–116. Deschamps AE, De Geest S, Vandamme A-M, Van Wijngaerden E. Use of Electronic Monitoring induces a 40-days intervention effect in HIV-patients. J Acquir Immune Defic Syndr 2006;43:247–249. Wilson IB, Tchetgen E, Spiegelman D. Patterns of adherence with antiretroviral medications: An examination of betweenmedication differences. J Acquir Immune Defic Syndr 2001; 28:259–263. McNabb JJ, Nicolau DP, Stoner JA, Ross J. Patterns of adherence to antiretroviral medications: The value of electronic monitoring. AIDS 2003;17:1763–1767. DiMatteo MR, Haskard KB. Further challenges in adherence research: Measurements, methodologies, and mental health care. Med Care 2006;44:297–299. Llabre MM, Weaver KE, Duran RE, Antoni MH, PhersonBaker S, Schneiderman N. A measurement model of medication adherence to highly active antiretroviral therapy and its relation to viral load in HIV-positive adults. AIDS Patient Care STDs 2006;20:701–711. Liu H, Miller LG, Hays RD, et al. A comprehensive evaluation of survey questions for adherence to antiretroviral medications and exploratory analyses for identifying optimal sets of survey questions. AIDS Patient Care STDs 2006; 20:760–772.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

Address reprint requests to: Ann E. Deschamps, M.S.N. University Hospitals Leuven Department of Internal Medicine Herestraat 49 3000 Leuven Belgium E-mail: [email protected]

This article has been cited by: 1. Ira B. Wilson, Amanda E. Carter, Karina M. Berg. 2009. Improving the self-report of HIV antiretroviral medication adherence: Is the glass half full or half empty?. Current HIV/AIDS Reports 6:4, 177-186. [CrossRef] 2. Mark J. Atkinson , Jeffrey J. Petrozzino . 2009. An Evidence-Based Review of Treatment-Related Determinants of Patients' Nonadherence to HIV MedicationsAn Evidence-Based Review of Treatment-Related Determinants of Patients' Nonadherence to HIV Medications. AIDS Patient Care and STDs 23:11, 903-914. [Abstract] [Full Text] [PDF] [PDF Plus] 3. Kathleen Nicole Deering , Kate Shannon , Hayley Sinclair , Devi Parsad , Erin Gilbert , Mark W. Tyndall . 2009. Piloting a Peer-Driven Intervention Model to Increase Access and Adherence to Antiretroviral Therapy and HIV Care among Street-Entrenched HIV-Positive Women in VancouverPiloting a Peer-Driven Intervention Model to Increase Access and Adherence to Antiretroviral Therapy and HIV Care among Street-Entrenched HIV-Positive Women in Vancouver. AIDS Patient Care and STDs 23:8, 603-609. [Abstract] [Full Text] [PDF] [PDF Plus] 4. Staci Martin , Deborah K. Elliott-DeSorbo , Sarah Calabrese , Pamela L. Wolters , Gregg Roby , Tara Brennan , Lauren V. Wood . 2009. A Comparison of Adherence Assessment Methods Utilized in the United States: Perspectives of Researchers, HIV-Infected Children, and their CaregiversA Comparison of Adherence Assessment Methods Utilized in the United States: Perspectives of Researchers, HIV-Infected Children, and their Caregivers. AIDS Patient Care and STDs 23:8, 593-601. [Abstract] [Full Text] [PDF] [PDF Plus] 5. Allison J. Applebaum , Laura C. Reilly , Jeffrey S. Gonzalez , Mark A. Richardson , Catherine L. Leveroni , Steven A. Safren . 2009. The Impact of Neuropsychological Functioning on Adherence to HAART in HIV-Infected Substance Abuse PatientsThe Impact of Neuropsychological Functioning on Adherence to HAART in HIV-Infected Substance Abuse Patients. AIDS Patient Care and STDs 23:6, 455-462. [Abstract] [PDF] [PDF Plus]