In order to publish clinical trials at a high-level, it is advisable to conduct some sort of treatment adherence procedure. This will help you and your readers see how adherent your therapists or intervention deliverers were to the manuals provided (even if you used a flexible protocol, it can illuminate how well the important processes were delivered). It is also good to include a measure of therapist/intervention deliverer competence. In many of the ACT clinical trials, researchers have assessed both.
It is virtually impossible to conduct high-quality treatment adherence and competence coding without a coding manual that describes, accurately and with sufficient detail, the constructs of interest that most individuals with sufficient training could understand.
In assessing fidelity to a treatment model, such as ACT, these can be difficult questions to develop. Therefore, we recommend utilizing manuals that have already been tested in other clinical trials. Even if some of the questions pertain to areas you did not cover in your treatment (e.g., a different presenting problem), the format of the response options and the wording of the questions can still be very helpful as you modify some of the content you will assess.
Consistent with recommendations in the literature, we recommend the following:
- Select independent raters whenever possible, meaning the coders are not the same as those who administered the intervention.
- Wherever possible, raters should be blind to treatment condition.
- We recommend creating a training scenario much like any shaping procedure wherein raters watch/listen to recordings (this means that when designing your study, you should plan to audio or video record your sessions) together and talk through decision points for each item to be coded in the segment, then rate a segment selected for training, and assess the similarity of codes, and so on until sufficient reliability is reached.
- Inter-rater reliability should be assessed throughout the training period and discrepancies discussed with a trained master rater, the intervention developer, the adherence manual developer, or some combination of those.
- For calculating inter-rater reliability, search your local library for the latest statistics articles on this topic as conventions can change. In general, Kappas can be calculated for two raters assessing dichotomous ratings (e.g., YES or NO for the presence of a variable of interest such as diagnosis). Inter-class correlation coefficients (ICCs) are calculated for continuous variables -- such as a particular process occurring on a continuum -- and can accommodate more than two raters. Selecting the two-way mixed ICC (an option within SPSS) is preferred when the raters are not randomly selected from the population of all possible raters -- which is most often the case for clinical trials research.
- The training period should last until raters reach at least .80 reliability (a convention in the literature is that reliability of .80 and higher is sufficient).
- Finally, we recommend assessing the current convention in the literature regarding the number of sessions to code for adherence and competence. In several ACT studies to date, the convention is about 20% of each therapist's sessions over the course of the study.
Click on the child pages below for sample treatment adherence and competence materials that have been used in ACT studies.
There is a publication that provides detailed suggestions for conducting adherence ratings, as well as a published manual as a starting point:
Plumb, J. C. & Vilardaga, R. (2010). Assessing treatment integrity in Acceptance and Commitment Therapy: Strategies and suggestions. International Journal of Behavioral Consultation and Therapy, 6, 263-295.