Multisystemic Therapy Clinical Overview, Outcomes, and Implementation Research SCOTT W. HENGGELER* CINDY M. SCHAEFFER † Multisystemic therapy (MST) is an evidence-based treatment originally developed for youth with serious antisocial behavior who are at high risk for out-of-home placement and their families and subsequently adapted to address other challenging clinical problems experience by youths and their families. The social-ecological theoretical framework of MST is presented as well as its home-based model of treatment delivery, defining clinical intervention strategies, and ongoing quality assurance/quality improvement system. With more than 100 peer-reviewed outcome and implementation journal articles published as of January 2016, the majority by independent investigators, MST is one of the most extensively evaluated family based treatments. Outcome research has yielded almost uniformly favorable results for youths and families, and implementation research has demonstrated the importance of treatment and program fidelity in achieving such outcomes. Keywords: Delinquency Multisystemic therapy Outcomes Implementation research Fam Proc x, 2016 INTRODUCTION M ultisystemic therapy (MST Henggeler, Schoenwald, Borduin, Rowland, Cunningham, 2009) is a comprehensive family and community-based treatment originally designed for youth with serious conduct problems who are at imminent risk of out-of-home placement (e.g., incarceration, residential treatment. Adapted versions of the original MST model have since evolved and been successfully applied to youth and families with other serious clinical problems, including child maltreatment, psychiatric disturbance, problem sexual behavior, and pediatric chronic illness. The initial section of this paper outlines the theoretical and empirical foundations of MST, provides an overview of the MST treatment model, and describes its quality assurance procedures. The remainder of the paper summarizes the extensive evidence base for the clinical effectiveness of MST as well as the growing literature on the transport and implementation of MST in community settings. CHARACTERISTICS OF THE MST MODEL Empirical and Theoretical Foundation Multisystemic therapy is designed to comprehensively address the array of risk factors that lead to the clinical problem being addressed. In the case of serious conduct problems, *Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, Charleston, SC. † Department of Psychiatry, University of Maryland School of Medicine, Baltimore, MD. Correspondence concerning this article should be addressed to Scott W. Henggeler, Psychiatry and Behavioral Sciences, Medical University of South Carolina, 176 Croghan Spur Rd. Suite 104, 29407 Charleston, SC. Email henggesw@musc.edu 1 Family Process, Vol. x, No. x, 2016 © 2016 Family Process Institute doi: 10.1111/famp.12232
decades of cross-sectional and longitudinal research has shown that risk factors involve influences at multiple individual (e.g., cognitive biases about aggression, low social skills), family (e.g., low parental supervision, inconsistent family discipline, peer (e.g., association with deviant peers, school (e.g., low academic achievement, and neighborhood (e.g., high drug availability) levels (Deater-Deckard, Dodge, Bates, & Pettit, 1998; Elliott, 1994; Thornberry & Krohn, 2003). Similarly, child maltreatment stems from risk factors across multiple systems including the individual (e.g., parental substance abuse, family (e.g., partner conflict and violence, peer (e.g., social isolation, and neighborhood (e.g., low use of community resources) (Child Welfare Information Gateway, Given that the risk factors for serious clinical problems such as juvenile offending and child maltreatment exist within and across multiple domains, Bronfenbrenner’s (social ecological model provides a useful organizing framework for MST. According to this model, behavior is largely determined by the functioning of the proximal systems (i.e., family, peer, school, and neighborhood) in which individuals are embedded and the reciprocal interplay between these systems. Consistent with this view, MST contends that to optimize outcomes, interventions must have the capacity to target risk factors within (e.g., parenting practices) and between (e.g., caregiver interactions with school) multiple domains. Factors in the broader ecology (e.g., caregiver work hours, lack of prosocial activities in neighborhood) that create barriers to the effective functioning of proximal systems also must be addressed to increase the probability of favorable change. From an ecological perspective, it is also important to understand behavior within its naturally occurring context. This view has direct implications for the design of MST interventions. MST uses a home-based model of service delivery that emphasizes ecological validity in the assessment of behavior and delivery of interventions. Assessments are considered ecologically valid when they integrate information from multiple sources (e.g., siblings, extended family, teachers) and consider parent and youth functioning in a variety of real-world settings (e.g., at home, in school, during neighborhood activities. Similarly, MST interventions are provided where problems occur (i.e., homes, schools, community locations) and, whenever possible, are delivered by key members of the ecology (e.g., a parent administers drug tests to a youth and rewards clean screens a spouse enacts a family safety plan when a maltreating parent uses drugs). MST Theory of Change A central assumption of MST is that caregivers are the key to achieving and sustaining positive long-term outcomes. Thus, interventions focus intensely on empowering caregivers to obtain the resources and skills needed to more effectively parent, care for, and manage their children. As caregiver competencies (e.g., ability to provide consistent supervision abstinence from drug use) increase, the therapist guides caregiver efforts to address other factors that might be contributing to the clinical problem, such as a youth’s associations with deviant peers or parental unemployment. The ultimate goal is to create a context that supports adaptive, rather than deviant, youth and parent behavior (e.g., relationships with prosocial peers, effective parenting. Treatment also aims to surround caregivers with support from family, friends, and members of the community to help sustain the changes achieved during treatment. Characteristics of MST Clinical Implementation Treatment delivery Multisystemic therapy for juvenile offenders is delivered by a team consisting of two to four full-time Master’s level therapists, a part-time Master’s or doctoral level supervisor, www.FamilyProcess.org 2 / FAMILY PROCESS
and administrative support. Therapists in this standard version of MST (i.e., aimed for youth presenting serious antisocial behavior) typically carry caseloads of four to six families each. Treatment duration is relatively brief, ranging from 3 to 6 months. However, the intervention process is intensive and often involves 60 to 100 hours of direct contact with the family and other members of the ecology. In adaptations of MST for other clinical populations, treatment tends to be more intensive and longer in duration. For example, MST for child abuse and neglect (MST-CAN) uses a team of 3 Masters-level therapists, a full time supervisor, a full time crisis case manager, and 20% dedicated time from a psychiatrist who works with both children and adults. Each therapist’s caseload is capped at four families, and treatment lasts an average of 6 –9 months. The relatively greater intensity of MST-CAN and other MST adaptations is due to the complex clinical needs of the populations served. For example, it is not uncommon for the MST-CAN team to provide full courses of treatment to several individual family members (e.g., for substance abuse, trauma symptomatology, and/or mental health problems, in addition to core MST ecological interventions. Members of an MST team usually work for private service provider organizations contracted by public juvenile justice, child welfare, or mental health authorities. Therapists provide 24-hour/day and 7 day/week availability, which allows them to work with families at times the family finds convenient and to respond to clinical crises in a timely fashion. As noted previously, MST services are provided in home- and community-based settings, which enhances the ecological validity of assessments and interventions, helps overcome barriers to service access, and facilitates family engagement in treatment. Treatment principles Multisystemic therapy is highly individualized and does not follow a manualized treatment plan. Instead, nine treatment principles provide the underlying structure and framework upon which therapists build their interventions (see Appendix 1). A key treatment principle is that all aspects of MST must be strength-based. Therapists communicate an optimistic perspective to the family and other members of the ecology throughout the assessment and treatment process. Therapists identify potential strengths within the contexts of the child (e.g., hobbies and interests, academic skills, parent (e.g., employed, motivated), family (e.g., problem-solving ability, affective bonds, peers (e.g., prosocial activities, achievement orientation, school (e.g., management practices, after-school activities, and the neighborhood/community (e.g., concerned and involved neighbors, recreational opportunities. Identified strengths then are leveraged in the design of interventions. For example, an extended family member might be enlisted to assist with monitoring the youth after school until a caregiver returns home from work. Clinical Procedures and Interventions The nine MST treatment principles are applied using an analytical/decision-making process that structures the treatment plan, its implementation, and the evaluation of its effectiveness. Specific goals for treatment are set at individual, family, peer, and social network levels. However, as noted previously, caregivers are viewed as key to achieving desired outcomes and as crucial for the generalizability and sustainability of treatment gains. Early in the treatment process, the problem behaviors to be targeted are specified clearly from the perspectives of key stakeholders (e.g., family members, teachers, juvenile justice, or child welfare authorities, and ecological strengths are identified. Then, based on multiple perspectives, the ecological factors that seem to be driving each problem are organized into a coherent conceptual framework. For example, a youth’s car stealing Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 3
behavior might be associated with alack of caregiver monitoring, association with peers who have been arrested, and poor school engagement. Similarly, excessive use of force in parenting might be related to parental substance abuse, ineffective child behavior management skills, and untreated child attention deficit disorder. Next, the MST therapist, with support from other team members, designs specific intervention strategies to target those drivers of identified problems. Strategies incorporate interventions from empirically supported, problem-focused treatments such as structural/strategic and behavioral family therapies, behavioral parent training, and cognitive behavioral therapy. Psy- chopharmacological interventions might also be incorporated into treatment when evidence suggests that biological factors are contributing to identified problems. Importantly, these empirically supported interventions are highly integrated and are delivered in conjunction with interventions that address other pertinent ecological drivers of the identified problems (e.g., supporting caregivers in advocating for more appropriate school services, helping a parent find employment and drug-free recreation to support sobriety). Intervention effectiveness is monitored continuously from multiple perspectives. When goals are not being met, identified drivers are reconceptualized, and modifications are made until an effective intervention strategy is developed. This reiterative process reinforces two important features of the MST model (1) MST teams strive to never give upon youth and families, doing whatever it takes to help families reach treatment goals and) When interventions are not successful, the failure is the team’s rather than the family s. In other words, when the team develops accurate hypotheses of the drivers, identifies barriers to implementation success, and delivers corresponding interventions appropriately, families tend to achieve their goals, and the issues that led a family into treatment (e.g., juvenile offending, child maltreatment, youth suicide attempt) usually diminish. Training, Supervision, and Ongoing Quality Assurance As discussed more extensively by Schoenwald (in press, several processes and structures are setup within the MST model to support treatment fidelity and help therapists attain desired clinical outcomes. New therapists participate in a day orientation training that provides initial grounding in MST, and all team members participate in quarterly booster trainings. Adaptations for special populations (e.g., MST-CAN, MST-Psychiatric) involve additional initial training. The majority of MST clinical learning, however, occurs as therapists work with families and receive weekly structured supervision and feedback both from the onsite MST team supervisor and an offsite MST expert consultant. Multisystemic therapy training, supervision, and consultation take place within a comprehensive quality assurance/quality improvement (QA/QI) system designed to help ensure that the dissemination of MST occurs with fidelity to the key aspects of the model that are essential in attaining youth and family outcomes. The process underlying this system has been worked out through more than 20 years of experience assisting community-based agencies in developing and maintaining sustainable MST teams. Indeed, 23,000 youth and families are treated annually through MST programs in more than 30 states and 15 nations. In addition to the well-specified initial and ongoing training, supervision, and consultation protocols, key components of the QA/QI system include validated measures of implementation adherence at all levels (therapists, supervisors, and consultants) and a web-based implementation tracking system to provide teams and provider organizations with ongoing team-specific feedback about adherence and youth outcomes. Importantly, many aspects of the QA/QI system have been validated in ongoing research, and numerous studies have validated significant associations between program (e.g., therapist, supervisor, consultant) fidelity and favorable youth outcomes (Schoenwald, in press). www.FamilyProcess.org 4 / FAMILY PROCESS
In addition to supporting practitioner implementation of MST on a case-by-case basis, MST consultants provide extensive organizational support to communities and provider organizations that are interested in establishing MST programs both initially (to setup an MST team and secure stakeholder buy-in) and on an ongoing basis (to evaluate program success and problem-solve threats to sustainability. More information regarding the implementation of MST programs is available from Henggeler, Schoenwald et al. (and the MST Services website www.mstservices.com. OUTCOME RESEARCH Efficacy research aims to determine whether anew or innovative treatment can produce favorable outcomes under relatively ideal conditions. Thus, efficacy studies are usually conducted in university settings with highly trained graduate students or faculty as therapists, intensive supervision to sustain high treatment fidelity, homogenous client samples (e.g., excluding participants with co-occurring disorders, and minimal extraneous disruptions (e.g., billing requirements, organizational mandates) to project functioning. Effectiveness research, on the other hand, aims to determine whether a promising treatment can produce favorable outcomes when delivered in community settings by real world practitioners. By their nature, efficacy studies should produce larger effect sizes than effectiveness studies, and such has been found in MST meta-analyses (Curtis, Ronan, & Borduin, 2004; Van der Stouwe, Asscher, Stams, Dekovic, & van der Laan, 2014). Efficacy/Effectiveness Hybrids The initial MST outcome studies were efficacy/effectiveness hybrids —conducted in university settings under close supervision, but including youth and family samples with a wide range of co-occurring behavioral and emotional problems (Henggeler, 2011). In the first (Henggeler et al., 1986), MST was more effective than diversion services in decreasing the behavior problems of juvenile offenders and improving their family and peer relations. In the second (Brunk, Henggeler, & Whelan, 1987), MST was more effective than behavioral parent training in improving key aspects of parent-child interactions in maltreating families. These studies are noteworthy for setting the stage for the extensive and wide-ranging body of MST-related research produced during the past 30 years. As of January 2016, outcome and implementation studies have been published yielding more than 100 peer- reviewed journal articles —the majority of which were authored by investigators independent of the treatment developers. This article summarizes key aspects of this body of research. For more detailed information, Multisystemic therapy research at a glance, 2016 (http://mstservices.com/files/outcomestudies.pdf) provides a table listing each study citation, design, client sample, comparison condition, length of followup, key findings, and the nature of the therapists and provider organization (e.g., university vs. community based). The initial promising results of MST with juvenile offenders led to several NIH-funded effectiveness trials described subsequently. Soon thereafter, however, Borduin began the Missouri Delinquency Project, which has become the longest continuous MST study —fol- lowing the original 176 violent and chronic juvenile offenders and their families for years post treatment. Results from this efficacy/effectiveness hybrid likely set the ceiling for MST outcomes. At posttreatment, youths and caregivers in the MST condition had decreased behavior problems and psychiatric symptoms, respectively family relations improved and a 63% decrease in youth recidivism was observed (Borduin et al., 1995). At 14-year followup, rearrests decreased by 54% and days incarcerated by 57% (Schaeffer & Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 5
Borduin, 2005); and similar outcomes were sustained though a year followup (Sawyer Borduin, 2011). Moreover, long-term followups showed that favorable outcomes extended to the siblings of the juvenile offenders in the MST condition and produced considerable cost savings (Dopp, Borduin, Wagner, & Sawyer, 2014; Klietz, Borduin, & Scha- effer, 2010; Wagner, Borduin, Sawyer, & Dopp, Effectiveness Studies The first MST effectiveness study (Henggeler, Melton, & Smith, 1992) was conducted in collaboration with a community mental health center and local juvenile justice authorities. With a sample of violent and chronic juvenile offenders at imminent risk of incarceration, MST improved family relations and peer relations at posttreatment, and decreased recidivism by 43% and out-of-home placement by 64% at a week followup. Encouraged by these findings, Henggeler, Melton, Brondino, Scherer, and Hanley (1997) conducted a multi-site randomized clinical trial (RCT) with violent and chronic juvenile offenders and their families to determine whether similar outcomes could be achieved absent a key component of the MST quality assurance system —weekly consultation from an MST expert. Results were not as consistently favorable as in prior MST research that included strong clinical support for the therapists, but one important finding emerged. Therapist treatment adherence to MST was inversely associated with youth rearrest. This finding highlighted the importance of assessing and promoting intervention fidelity and, as noted subsequently, has been replicated in several studies. International replications Several groups of European researchers have conducted MST RCTs with youth presenting serious antisocial behavior and their families. Ina multi-site Norwegian study, Ogden and colleagues (Ogden & Hagen, 2006; Ogden & Halliday-Boykins, 2004) found that MST decreased youth externalizing and internalizing symptoms as well as out-of-home placements while increasing social competence and some of these outcomes were sustained through a month followup. Moreover, these investigators observed that certain sites performed more effectively than others, demonstrating that site effects are critical to examine in multi-site clinical trials. Favorable MST outcomes were also reported by British and Dutch researchers. Butler, Baruch, Hickley, and Fonagy (2011) observed that MST improved parenting and reduced youth offenses and out-of-home placements for British juvenile offenders, and such reductions in crime were associated with cost savings (Cary, Butler, Baruch, Hickey, & Byford, 2013). Similarly, Asscher, Dekovic, Manders, van der Laan, and Prins (2013) found that MST improved parenting and youth peer relations, and decreased youth antisocial behavior among Dutch youth with severe and violent antisocial behavior at posttreatment. Several of these outcomes were sustained at year followup, but effects on recidivism were not observed at year follow-up. An important RCT with Swedish youth with conduct disorder and their families (Sun- dell et al., 2008), however, failed to replicate favorable MST outcomes. Treatment fidelity was very low in this study and, similar to Henggeler et al. (1997), was associated inversely with youth arrest. Subsequently, these investigators (Lofholm, Eichas, & Sundell, examined the performance of Swedish MST teams from 2003 to 2009 and reported findings that shed light on the aforementioned failure to replicate and have important implications for the broader evaluation literature. They found that therapist fidelity and corresponding youth outcomes were lowest during the time of the RCT and steadily improved as therapists and teams gained experience. The investigators concluded that clinical trials should not begin until practitioners and programs have demonstrated satisfactory adherence to intervention protocols. www.FamilyProcess.org 6 / FAMILY PROCESS
American replications Favorable MST outcomes for youth with serious antisocial behavior have been obtained in independent American studies as well. Timmons-Mitchell, Bender, Kishna, and Mitchell) found that MST improved youth functioning across several domains and decreased rearrests for juvenile offenders at imminent risk of out-of-home placement. Similarly, decreased symptoms, improved functioning, and decreased out-of-home placements were observed by Stambaugh et al. (2007) in their study comparing MST with Wraparound for youth with serious emotional disturbance and antisocial behavior. Painter) found MST to be more effective than case management for youth with externalizing disorders across several life domains, including juvenile justice involvement and favorable outcomes for behavior problems, parenting, and school attendance were also reported by Weiss et al. (2013) for adolescents with serious conduct problems in self-contained classrooms. Substance abusing juvenile offenders Several studies (Henggeler et al., 1991; Letourneau et al., 2009; Timmons-Mitchell et al., 2006) have shown that MST can reduce substance use in heterogeneous samples of adolescents presenting serious antisocial behavior (i.e., including likely substance abusing youth and youth who are not abusing substances. Two additional studies focused on juvenile offenders with diagnosed substance use disorders. Henggeler, Pickrel, and Brondino (1999) showed that MST was more effective than usual substance use treatment at decreasing substance use and days in out-of-home placement at an month follow-up. Favorable outcomes for violent crime and marijuana use extended through a year followup (Henggeler, Clingempeel, Brondino, & Pickrel, 2002). In the context of an evaluation of juvenile drug court, Henggeler et al. (2006) showed that MST enhanced favorable substance use outcomes achieved by juvenile drug court. Moreover, MST was associated with decreased substance use among the siblings of the juvenile offenders (Rowland, Chapman, & Henggeler, Juvenile sex offenders An early efficacy/effectiveness hybrid (Borduin, Henggeler, Blaske, & Stein, 1990) with a small sample of juvenile sex offenders and their families produced very promising findings for MST at a year followup (i.e., 93% reduction in sexual offending. A larger efficacy effectiveness hybrid (Borduin, Schaeffer, & Heiblum, 2009) subsequently showed numerous favorable outcomes at posttreatment (e.g., improved family relations and academic performance) as well as large decreases in sex offense recidivism, recidivism for other crimes, and days incarcerated for youth in the MST condition in comparison with counterparts provided usual community services at a year followup. Indeed, an economic analysis (Borduin & Dopp, 2015) revealed a cost benefit of $343,455 per MST participant. In addition, MST effectiveness research with juvenile sex offenders (Letourneau et al., 2009) found that youths in the MST condition evidenced decreased sexual behavior problems, delinquency, substance use, externalizing symptoms, and out-of-home placements at 12 months postrecruitment. At year followup (Letourneau, Henggeler, et al., 2013), favorable MST results were sustained for problem sexual behavior, self-reported delinquency, and out-of-home placements, but treatment effects were not observed for criminal recidivism. Adaptations to standard MST Building on the established effectiveness of MST in treating serious antisocial behavior and as noted in the clinical section of this article, several investigators have developed Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 7
and tested adaptations of MST to address other serious and costly clinical problems experienced by youth and their families. The first substantive adaptation was aimed at youth with serious emotional disturbance (Henggeler, Schoenwald, Rowland, & Cunningham. This adaptation (i.e., MST-Psychiatric), developed under the leadership of Rowland, was evaluated in an RTC of MST as an alternative to emergency psychiatric hospitalization of youth presenting with psychotic behavior or high risk of suicide or homicide (Henggeler, Rowland, et al., 1999). At posttreatment, youths in the MST-Psychiatric condition, in comparison with youth admitted to the inpatient unit and receiving aftercare, had decreased externalizing problems, improved family relations, increased school attendance, and higher consumer satisfaction. Large between-groups differences were also observed for days hospitalized and days in other out-of-home placements (Schoenwald, Ward, Henggeler, & Rowland, 2000). Similarly favorable findings were observed in an RCT comparing MST-Psychiatric with Hawaii’s intensive Continuum of Care for youth with serious emotional and behavioral disturbances and their families (Rowland et al., 2005). Recalling that one of the earliest MST studies was an RCT with maltreating families (Brunk et al., 1987), Swenson and Schaeffer further validated the effectiveness of MST with this important population. In an effectiveness RTC, Swenson, Schaeffer, Henggeler, Faldowski, and Mayhew (2010) compared MST adapted for child abuse and neglect (MST- CAN) with parent training and enhanced outpatient treatment. MST-CAN was more effective at decreasing youth and caregiver symptoms, improving parenting, increasing social support, and decreasing out-of-home placements. Similarly favorable results have been obtained for an enhancement of MST-CAN in an evaluation with families with co- occurring parental substance abuse and child maltreatment (Schaeffer, Swenson, Tuerk, & Henggeler, Ellis and Naar-King have developed and evaluated innovative adaptions of MST to address chronic and costly healthcare problems experienced by youth (MST-HC). In three separate studies, MST-HC has proven more effective than alternative interventions at improving diabetes adherence and metabolic control as well as decreasing hospital admissions (Ellis et al., 2004, 2005, 2012) for youth with Type 1 diabetes under poor metabolic control. MST-HC has also produced favorable disease-specific outcomes for inner-city adolescents with primary obesity (Naar-King et al., 2009), HIV infected youth with medication adherence problems (Letourneau, Ellis, et al., 2013), and adolescents with poorly controlled asthma (Naar-King et al., 2014). More than a dozen research articles have been published from these studies, and more extensive descriptions of their research methods and findings can be viewed at http://mstservices.com/files/outcomestudies.pdf. Mediators and Moderators of MST Outcomes The MST model of change, noted earlier, describes the hypothesized mediators of MST effectiveness. In general, therapists collaborate with the family to enhance caregivers parenting competencies and these, in turn, are considered central to helping youth reduce antisocial behavioral tendencies and build prosocial competencies with peers, school, and the community. This model has been supported by several quantitative and qualitative studies. In the quantitative studies, mediational analyses were conducted in the context of RCTs. Based on data from two MST studies with juvenile offenders (Henggeler et al., 1997; Henggeler, Pickrel, and Brondino 1999), Huey, Henggeler, Brondino, and Pickrel (2000) found that favorable changes in family relations and deviant peer associations mediated the relationship between treatment adherence and decreased delinquency. Similarly, mediation studies with juvenile sex offenders (Henggeler, Letourneau, et al. (2009)) www.FamilyProcess.org 8 / FAMILY PROCESS
and Dutch youth with serious antisocial behavior (Dekovic, Asscher, Manders, Prins, van der Laan, 2012) support the view that improved parenting is central to decreased antisocial behavior by the adolescents. The MST theory of change has also been supported by several uncontrolled studies. Tiernan, Foster, Cunningham, Brennan, and Whitmore (2015) found that high parental monitoring and low association with deviant peers was associated with decreased antisocial behavior for adolescents in MST programs. In separate qualitative studies conducted in Britain, respondents emphasized the impact of enhanced parenting skills and family relations on improved functioning of juvenile offenders (Tighe, Pistrang, Casdagli, Baruch Butler, 2012); and youths (Paradisopoulos, Pote, Fox, & Kaur, 2015) and caregivers (Kaur, Pote, Fox, & Paradisopoulos, 2015) attributed their sustained positive change to the therapeutic alliance, improved family functioning, and removing negative peer influences. These latter investigators also emphasized the bidirectionality of the MST model of change. Many MST RCTs have examined race, gender, socioeconomic status, and age as moderators of MST outcomes. In the vast majority of cases, these variables have not been associated with differential outcomes. This finding is likely due to the fundamental and central capacity of MST to adjust clinical interactions and procedures to the identified strengths and weaknesses of each family’s unique context. On the other hand, certain clinical characteristics at the peer, family, and youth levels have begun to emerge as moderators of youth outcomes. Regarding peer relations, Boxer Boxer, Kubik, Ostermann, & Veysey, 2015) observed that negative peer involvement, especially gang affiliation, was associated with treatment failure. Similarly, high deviant peer affiliation has predicted less of a decline in aggression and delinquency during treatment (Ryan et al., 2013). Interestingly, regarding family relations, Weiss, Han, Tran, Gallop, and Ngo (2015) found that MST effects were most favorable in families where parenting practices were ineffective, but positive family relations and parental mental health were high. Such suggests, consistent with the recent MST meta-analysis (Van der Stouwe et al., 2014), that improved parenting practices are central to achieving favorable outcomes. Finally, at the individual youth level, high narcissism and callous traits were associated with less favorable outcomes for Dutch youth with severe antisocial behavior (Manders, Dekovic, Asscher, van der Laan, & Prins, 2013). It should be noted, however, that MST has been shown to reduce psychopathic symptoms significantly (Butler et al., Implementation Research The widespread transport of MST programs to community settings has enabled researchers to evaluate the functioning of the MST quality assurance/quality improvement system as well as factors that influence the adoption and performance of MST programs. Initial multisite implementation studies (e.g., Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002; Schoenwald, Sheidow, Letourneau, & Liao, 2003) evaluated hypothesized linkages between key components of the quality assurance system consultant adherence, supervisor adherence, therapist adherence, and youth and family outcomes. As summarized by Schoenwald (in press, consultant adherence to the MST consultation protocol and supervisor adherence to the MST supervisory protocol predicted therapist treatment fidelity and more favorable youth outcomes. Likewise, and consistent with several RCTs (e.g., Ellis, Naar-King, Templin, Frey, & Cunningham, 2007; Hengge- ler et al., 1997), high therapist adherence was associated with better youth outcomes. More recent implementation research has supported the overall importance of the MST quality assurance system in promoting favorable youth outcomes. Smith-Boydston, Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 9
Holtzman, and Roberts (2014) observed that removal of ongoing clinical and organizational consultation from MST Services for existing MST programs resulted in program drift (e.g., 50% reduction in family contacts) and deteriorated juvenile justice outcomes for youth. Similarly, as noted previously, Lofholm et al. (2014) found that therapist adherence and youth outcomes improved with continued and ongoing quality assurance. Likewise, Brunk, Chapman, and Schoenwald (2014) showed that a composite index of overall program fidelity (e.g., treatment completion rates, program capacity, stakeholder relationships) was associated with fewer youth arrests and team closures. Thus, considerable research has supported the significance of the key components of the MST quality assurance system as well as the system in its entirety. Several other MST-related implementation studies have been recently published, and these have broad implications for the field of evidence-based practice. Stout and Holleran (2013) showed that adding MST and functional family therapy (FFT) programs to a state’s system of care were associated with reduced out-of-home placement and an estimated in annual savings to the state. Dutch investigators (Hendriks, Lange, Boon- stoppel-Boender, & van der Rijken, 2014) found that community stakeholders, consistent with MST referral protocols, were more likely to refer higher risk youths to MST programs than to other evidence-based treatment. And, Ogden et al. (2012) observed that relative to another evidence-based treatment, MST programs in Norway evidenced superior recruitment, supervision, performance assessment, data systems, administrative support, and systems interventions and leadership. Finally, recent studies in the United States (Welsh Greenwood, 2015) and Chile (Pantoja, 2015) indicate that a common set of conditions are necessary to promote the successful and large-scale adoption of MST and other evidence- based treatments. These include the structured involvement of all stakeholders (e.g., fun- ders, providers, referral sources, persistent leadership by effective champions, special funding for pilot testing, and ongoing technical assistance for adopters. Together, the implementation research reviewed here suggests that the adoption, sustainability, and effective functioning of evidence-based treatment programs requires well-validated quality assurance systems as well as ongoing program support from a variety of key stakeholders. CONCLUSION Multisystemic therapy is a well-specified family based treatment that includes a well- validated quality assurance system that promotes treatment adherence, program fidelity, and youth outcomes. The MST theory of change posits that improved family functioning is critical to achieving favorable youth outcomes, and this theory has been supported by several quantitative and qualitative studies. Numerous studies, including 25 published RCTs conducted mostly by independent investigators, support the effectiveness of MST in treating very challenging clinical problems including violence, substance abuse, serious emotional disturbance, child maltreatment, and chronic healthcare conditions. Moreover, findings from MST-related implementation research are demonstrating the conditions needed for evidence-based interventions to be transported effectively and sustained in community settings. REFERENCES Asscher, J. J, Dekovic, M, Manders, WA, van der Laan, PH Prins, P. J. MA randomized controlled trial of the effectiveness of multisystemic therapy in the Netherlands Post-treatment changes and moderator effects. Journal of Experimental Criminology, 9, 169 –187. Borduin, CM Dopp, AR. Economic impact of multisystemic therapy with juvenile sexual offenders. Journal of Family Psychology, 29, 687 –696. www.FamilyProcess.org 10 / FAMILY PROCESS
Borduin, CM, Henggeler, SW, Blaske, D. M, & Stein, R. (1990). Multisystemic treatment of adolescent sexual offenders. International Journal of Offender Therapy and Comparative Criminology, 35, 105 –114. Borduin, CM, Mann, B. J, Cone, LT, Henggeler, SW, Fucci, BR, Blaske, D. Metal. Multisystemic treatment of serious juvenile offenders Long-term prevention of criminality and violence. Journal of Consulting and Clinical Psychology, 63, 569 –578. Borduin, CM, Schaeffer, CM Heiblum, NA randomized clinical trial of multisystemic therapy with juvenile sexual offenders Effects on youth social ecology and criminal activity. Journal of Consulting and Clinical Psychology, 77, Boxer, P. (2011). Negative peer involvement in multisystemic therapy for the treatment of youth problem behavior Exploring outcome and process variables in “real-world” practice. Journal of Clinical Child & Adolescent Psychology, 40, Boxer, P, Kubik, J, Ostermann, M, & Veysey, B. (2015). Gang involvement moderates the effectiveness of evi- dence-based intervention for justice-involved youth. Children and Youth Services Review, 52, 26 –33. Bronfenbrenner, U. (1979). The ecology of human development Experiments by design and nature. Cambridge, MA: Harvard University Press. Brunk, MA, Chapman, J. E, & Schoenwald, SK. Defining and evaluating fidelity at the program level in psychosocial treatments. Zeitschrift fur Psychologie, 222, 22 –29. Brunk, M, Henggeler, SW Whelan, JP. A comparison of multisystemic therapy and parent training in the brief treatment of child abuse and neglect. Journal of Consulting and Clinical Psychology, 55, Butler, S, Baruch, G, Hickley, N, & Fonagy, PA randomized controlled trial of MST and a statutory therapeutic intervention for young offenders. Journal of the American Academy of Child & Adolescent Psychiatry, Cary, M, Butler, S, Baruch, G, Hickey, N, & Byford, S. (2013). Economic evaluation of multisystemic therapy for young people at risk for continuing criminal activity in the UK. PLoS One, 8(4), e61070. Child Welfare Information Gateway (2003). A coordinated response to child abuse and neglect The foundation for practice. Washington, DC US. Department of Health and Human Services, Children’s Bureau. Curtis, NM, Ronan, KR Borduin, CM. Multisystemic treatment A meta-analysis of outcome studies. Journal of Family Psychology, 18, 411 –419. Deater-Deckard, K, Dodge, K. A, Bates, J. E, & Pettit, GS. Multiple risk factors in the development of externalizing behavior problems Group and individual differences. Development and Psychopathology, 10, 469 –493. Dekovic, M, Asscher, J. J, Manders, WA, Prins, P. J. M, & van der Laan, P. (2012). Within-intervention change Mediators of intervention effects during multisystemic therapy. Journal of Consulting and Clinical Psychology, 80, 574 –587. Dopp, AR, Borduin, CM, Wagner, D. V, & Sawyer, AM. The economic impact of multisystemic therapy through midlife A cost-benefit analysis with serious juvenile offenders and their siblings. Journal of Consulting and Clinical Psychology, 82, Elliott, D. S. (1994). Serious violent offenders Onset, developmental course, and termination The American Society of Criminology 1993 Presidential Address. Criminology, 32, Ellis, DA, Frey, MA, Naar-King, S, Templin, T, Cunningham, PB Cakan, N. (2005). Use of multisys- temic therapy to improve regimen adherence among adolescents with type 1 diabetes in chronic poor metabolic control A randomized controlled trial. Diabetes Care, 28, Ellis, DA, Naar-King, S, Chen, X, Moltz, K, Cunningham, PB Idalski-Carcone, A. (2012). Multisystemic therapy compared to telephone support for youth with poorly controlled diabetes Findings from a randomized controlled trial. Annals of Behavioral Medicine, 44, Ellis, DA, Naar-King, S, Frey, MA, Templin, T, Rowland, M, & Greger, N. (2004). Use of multisystemic therapy to improve regimen adherence among adolescents with type 1 diabetes in poor metabolic control A pilot study. Journal of Clinical Psychology in Medical Settings, 11, Ellis, DA, Naar-King, S, Templin, T, Frey, MA Cunningham, PB. Improving health outcomes among youth with poorly controlled type 1 diabetes The role of treatment fidelity in a randomized clinical trial of multisystemic therapy. Journal of Family Psychology, 21, 363 –371. Hendriks, MED, Lange, AM. C, Boonstoppel-Boender, M, & van der Rijken, RE. A. (2014). Functional family therapy and multisystemic therapy A comparison of target populations. Orthopedagogiek: Onderzock en Praktijk, 53, 355 –366. Henggeler, SW. Efficacy studies to large-scale transport The development and validation of MST programs. Annual Review of Clinical Psychology, 7, 351 –381. Henggeler, SW, Borduin, CM, Melton, GB, Mann, B. J, Smith, L, Hall, J. A. et al. (1991). Effects of multi- systemic therapy on drug use and abuse in serious juvenile offenders A progress report from two outcome studies. Family Dynamics of Addiction Quarterly, 1, 40 –51. Henggeler, SW, Clingempeel, W. G, Brondino, M. J, & Pickrel, S. G. (2002). Four-year followup of multisys- temic therapy with substance abusing and dependent juvenile offenders. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 868 –874. Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 11
Henggeler, SW, Halliday-Boykins, CA, Cunningham, PB, Randall, J, Shapiro, SB Chapman, J. E. Juvenile drug court Enhancing outcomes by integrating evidence-based treatments. Journal of Consulting and Clinical Psychology, 74, 42 –54. Henggeler, SW, Letourneau, E. J, Chapman, J. E, Borduin, CM, Schewe, PA McCart, MR. (Mediators of change for multisystemic therapy with juvenile sexual offenders. Journal of Consulting and Clinical Psychology, 77, 451 –462. Henggeler, SW, Melton, GB, Brondino, M. J, Scherer, D. G, & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821 –833. Henggeler, SW, Melton, GB Smith, LA. Family preservation using multisystemic therapy An effective alternative to incarcerating serious juvenile offenders. Journal of Consulting and Clinical Psychology, 953 –961. Henggeler, SW, Pickrel, S. G, & Brondino, M. J. (1999). Multisystemic treatment of substance abusing and dependent delinquents Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 171 –184. Henggeler, SW, Rodick, JD, Borduin, CM, Hanson, CL, Watson, SM Urey, JR. Multisystemic treatment of juvenile offenders Effects on adolescent behavior and family interactions. Developmental Psychology, 132 –141. Henggeler, SW, Rowland, MR, Randall, J, Ward, D, Pickrel, S. G, Cunningham, PB. et al. (1999). Home- based multisystemic therapy as an alternative to the hospitalization of youth in psychiatric crisis Clinical outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1331 –1339. Henggeler, SW, Schoenwald, SK, Borduin, CM, Rowland, MD Cunningham, PB. Multisystemic therapy for antisocial behavior in children and adolescents, 2nd ed. New York Guilford Press. Henggeler, SW, Schoenwald, SK, Liao, JG, Letourneau, E. J, & Edwards, D. L. (2002). Transporting efficacious treatments to field settings The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child Psychology, 31, 155 –167. Henggeler, SW, Schoenwald, SK, Rowland, MD Cunningham, PB. Serious emotional disturbance in children and adolescents Multisystemic therapy. New York Guilford Press. Huey, SJ, Henggeler, SW, Brondino, M. J, & Pickrel, S. G. (2000). Mechanisms of change in multisystemic therapy Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting and Clinical Psychology, 68, 451 –467. Kaur, P, Pote, H, Fox, S, & Paradisopoulos, DA. Sustaining change following multisystemic therapy: Caregiver’s perspectives. Journal of Family Therapy, Advance online publication. doi: 10.1111/1467- 6427.12093. Klietz, SJ, Borduin, CM Schaeffer, CM. Cost-benefit analysis of multisystemic therapy with serious and violent juvenile offenders. Journal of Family Psychology, 24, 657 –666. Letourneau, E. J, Ellis, DA, Naar-King, S, Chapman, J. E, Cunningham, PB Fowler, S. (2013). Multisys- temic therapy for poorly adherent youth with HIV Results from a pilot randomized controlled trial. AIDS Care, 25, 507 –514. Letourneau, E. J, Henggeler, SW, Borduin, CM, Schewe, PA, McCart, MR, Chapman, J. E. et al. (2009). Multisystemic therapy for juvenile sexual offenders year results from a randomized effectiveness trial. Journal of Family Psychology, 23, 89 –102. Letourneau, E. J, Henggeler, SW, McCart, MR, Borduin, CM, Schewe, PA Armstrong, KS. (2013). Two-year followup of a randomized effectiveness trial evaluating MST for juveniles who sexually offend. Journal of Family Psychology, 27, 978 –985. Lofholm, CA, Eichas, K, & Sundell, K. (2014). The Swedish implementation of multisystemic therapy for adolescents Does treatment experience predict treatment adherence. Journal of Clinical Child & Adolescent Psychology, 643 –655. Manders, WA, Dekovic, M, Asscher, J. J, van der Laan, PH Prins, P. J. M. (2013). Psychopathy as predictor and moderator of multisystemic therapy outcomes among adolescents treated for antisocial behavior. Journal of Abnormal Child Psychology, 41, 1121 –1132. Naar-King, S, Ellis, D, King, PS, Cunningham, P, Secord, E, Bruzzese, J.-M. et al. (2014). Multisystemic therapy for high-risk African American adolescents with asthma A randomized clinical trial. Journal of Consulting and Clinical Psychology, 82, 536 –545. Naar-King, S, Ellis, D, Kolmodin, K, Cunningham, P, Jen, KL. C, Saelens, Bet al. (2009). A randomized pilot study of multisystemic therapy targeting obesity in African-American adolescents. Journal of Adolescent Health, 45, Ogden, T, Bjornebekk, G, Kjobli, J, Patras, J, Christiansen, T, Taraldsen, K. et al. (2012). Measurement of implementation components ten years after a nationwide introduction of empirically supported programs a pilot study. Implementation Science, 7, 49. www.FamilyProcess.org 12 / FAMILY PROCESS
Ogden, T, & Hagen, K. A. (2006). Multisystemic therapy of serious behaviour problems in youth Sustainability of therapy effectiveness two years after intake. Journal of Child and Adolescent Mental Health, 11, Ogden, T, & Halliday-Boykins, CA. Multisystemic treatment of antisocial adolescents in Norway Replication of clinical outcomes outside of the US. Child and Adolescent Mental Health, 9(2), Painter, K. (2009). Multisystemic therapy as community-based treatment for youth with severe emotional disturbance. Research on Social Work Practice, 19, 314 –324. Pantoja, R. (2015). Multisystemic therapy in Chile A public sector innovation case study. Psychosocial Intervention, 97 –103. Paradisopoulos, D, Pote, H, Fox, S, & Kaur, P. (2015). Developing a model of sustained change following multi- systemic therapy Young people’s perspectives. Journal of Family Therapy, 37, Rowland, MD, Chapman, J. E, & Henggeler, SW. Sibling outcomes from a randomized trial of evi- dence-based treatments with substance abusing juvenile offenders. Journal of Child & Adolescent Substance Abuse, 17, Rowland, MR, Halliday-Boykins, CA, Henggeler, SW, Cunningham, PB, Lee, T. G, Kruesi, M. JP. et al. (2005). A randomized trial of multisystemic therapy with Hawaii’s Felix Class youths. Journal of Emotional and Behavioral Disorders, 13, Ryan, SR, Brennan, PA, Cunningham, PB, Foster, S. L, Brock, R. L, & Whitmore, E. (2013). Biosocial processes predicting multisystemic therapy treatment response. Biological Psychology, 92, Sawyer, AM Borduin, CM. Effects of MST through midlife A year followup to a randomized clinical trial with serious and violent juvenile offenders. Journal of Consulting and Clinical Psychology, 79, 643 –652. Schaeffer, CM Borduin, CM. Long-term followup to a randomized clinical trial of multisystemic therapy with serious and violent juvenile offenders. Journal of Consulting and Clinical Psychology, 73, 445 – 453. Schaeffer, CM, Swenson, CC, Tuerk, EH Henggeler, SW. Comprehensive treatment for co-occur- ring child maltreatment and parental substance abuse Outcomes from a month pilot study of the MST- Building Stronger Families program. Child Abuse and Neglect, 37, 596 –607. Schoenwald, SK. (in press. The multisystemic therapy â quality assurance/quality improvement system. In W. O’Donahue (Ed, Quality improvement in behavioral health. New York Springer. Schoenwald, SK, Sheidow, A. J, Letourneau, E. J, & Liao, JG. Transportability of multisystemic therapy Evidence for multilevel influences. Mental Health Services Research, 5, 223 –239. Schoenwald, SK, Ward, D. M, Henggeler, SW Rowland, MD. MST vs. hospitalization for crisis stabilization of youth Placement outcomes 4 months post-referral. Mental Health Services Research, 2, 3 –12. Smith-Boydston, J. M, Holtzman, R. J, & Roberts, MC. Transportability of multisystemic therapy to community settings Can a program sustain outcomes without MST Services oversight Child & Youth Care Forum, 43, 593 –605. Stambaugh, L. F, Mustillo, SA, Burns, B. J, Stephens, R. L, Baxter, B, Edwards, D. et al. (2007). Outcomes from wraparound and multisystemic therapy in a center for mental health services system-of-care demonstration site. Journal of Emotional and Behavioral Disorders, 15, Stout, B. D, & Holleran, D. (2013). The impact of evidence-based practices on requests for out-of-home placements in the context of system reform. Journal of Child and Family Studies, 22, 311 –321. Sundell, K, Hansson, K, Lofholm, CA, Olsson, T, Gustle, L. H, & Kadesjo, C. (2008). The transportability of MST to Sweden Short-term results from a randomized trial of conduct disordered youth. Journal of Family Psychology, 22, 550 –560. Swenson, CC, Schaeffer, C, Henggeler, SW, Faldowski, R, & Mayhew, AM. Multisystemic therapy for child abuse and neglect A randomized effectiveness trial. Journal of Family Psychology, 24, 497 –507. Thornberry, T. P, & Krohn, MD. (Eds) (2003). Taking stock of delinquency An overview of findings from contemporary longitudinal studies. New York Kluwer/Plenum. Tiernan, K, Foster, S. L, Cunningham, PB, Brennan, P, & Whitmore, E. (2015). Predicting early positive change in multisystemic therapy with youth exhibiting antisocial behaviors. Psychotherapy, 52, 93 –102. Tighe, A, Pistrang, N, Casdagli, L, Baruch, G, & Butler, S. (2012). Multisystemic therapy for young offenders: Families’ experiences of therapeutic processes and outcomes. Journal of Family Psychology, 26, 187 –197. Timmons-Mitchell, J, Bender, MB, Kishna, MA Mitchell, CC. An independent effectiveness trial of multisystemic therapy with juvenile justice youth. Journal of Clinical Child and Adolescent Psychology, Van der Stouwe, T, Asscher, J. J, Stams, G. J. J. M, Dekovic, M, & van der Laan, PH. The effectiveness of multisystemic therapy (MST A meta-analysis. Clinical Psychology Review, 34, Wagner, D. V, Borduin, CM, Sawyer, AM Dopp, AR. Long-term prevention of criminality in siblings of serious and violent juvenile offenders A year followup to a randomized clinical trial of multisys- temic therapy. Journal of Consulting and Clinical Psychology, 82, 492 –499. Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 13
Weiss, B, Han, S, Harris, V, Catron, T, Ngo, V. K, Caron, A. et al. (2013). An independent randomized clinical trial of multisystemic therapy with non-court-referred adolescents with serious conduct problems. Journal of Consulting and Clinical Psychology, 81, Weiss, B, Han, SS, Tran, NT, Gallop, R, & Ngo, V. K. (2015). Test of facilitation vs. proximal process moderator models for the effects of multisystemic therapy on adolescents with severe conduct problems. Journal of Abnormal Child Psychology, 43, Welsh, BC Greenwood, PW. Making it happen State progress in implementing evidence-based programs for delinquent youth. Youth Violence and Juvenile Justice, 13, APPENDIX MST NINE TREATMENT PRINCIPLES PRINCIPLE 1: FINDING THE FIT An assessment is made to understand the fit between identified problems and how they play out and make sense in the entire context of the family’s environment. Assessing the fit of youth and parent successes also helps guide the treatment process. PRINCIPLE 2: FOCUSING ON POSITIVES AND STRENGTHS Multisystemic therapy therapists emphasize the positives they find and use strengths as levers for positive change. Focusing on family strengths has numerous advantages, such as building on strategies the family already use, instilling hope, identifying protective factors, decreasing frustration, and enhancing caregivers confidence. PRINCIPLE 3: INCREASING RESPONSIBILITY Interventions are designed to promote responsible behavior and decrease irresponsible actions by all family members. PRINCIPLE 4: PRESENT-FOCUSED, ACTION-ORIENTED, AND WELL-DEFINED Interventions deal with what’s happening now in the family’s life. Therapists look for action that can betaken immediately, targeting specific and well-defined problems. Family members are expected to work actively toward goals by focusing on present-oriented solutions, rather than gaining insight or focusing on the past. When the clear goals are met, the treatment can end. PRINCIPLE 5: TARGETING SEQUENCES Interventions target sequences of behavior within and between the various interacting systems —family, peers, teachers, home, school, and community—that sustain the identified problems. PRINCIPLE 6: DEVELOPMENTALLY APPROPRIATE Interventions are setup to be appropriate to the youth’s age and fit his or her developmental needs. www.FamilyProcess.org 14 / FAMILY PROCESS
PRINCIPLE 7: CONTINUOUS EFFORT Interventions require daily or weekly effort by family members so that the youth and family have frequent opportunities to demonstrate their commitment and practice skills. Advantages of intensive regular efforts to change include more rapid problem resolution, earlier identification of when interventions need fine-tuning, continuous evaluation of outcomes, more frequent corrective interventions, and more opportunities for family members to experience success. PRINCIPLE 8: EVALUATION AND ACCOUNTABILITY Intervention effectiveness is evaluated continuously from multiple perspectives, with MST team members being held accountable for overcoming barriers to successful outcomes. MST does not label families as resistant, not ready for change or unmotivated.” This approach avoids blaming the family and places the responsibility for positive treatment outcomes on the MST team. PRINCIPLE 9: GENERALIZATION Interventions are designed to invest the caregivers with the ability to address the family s needs after the intervention is over. The caregiver is viewed as the key to long-term success. Family members drive the change process in collaboration with the MST thera- pist. Fam. Proc., Vol. x, xxxx, 2016 HENGGELER & SCHAEFFER / 15
Share with your friends: |