The Research, Policy, and Practice Context for Delivery of Evidence-Based Mental Health Treatments for Adolescents: A Systems Perspective
Research discoveries in the past 20 years have led to major breakthroughs in identifying treatments for adolescents that are most likely to improve their mental health functioning. Problems of attention, depression, and various forms of anxiety, including those related to traumatic abuse and obsessive-compulsive disorders, can now be treated and treated effectively through cognitive-behavioral, interpersonal, and medication therapies and, in some cases, a combination of these therapies (Lonigan, Elbert, & Johnson, 1998; March, Amaya, Jackson, Murray, & Schulte, 1998; March and the TADS Consortium, MTA Cooperative Group, 1999a; Mufson, Dorta, Olfson, Weissman, & Hoagwood, 2004; Silverman, Kurtines, & Hoagwood, 2004). A range of community-based services to support youth and families in accessing or continuing with treatments have also been examined rigorously. Some of this work is demonstrating how engagement techniques, particularly forms of intensive case management, in-home therapeutic services, and functional family therapies can help parents or caregivers care for their children (Alexander & Sexton, 2002; Burns, Costello, Angold, et al., 1995; Chamberlain & Reid, 1991; Evans, Banks, Huz, McNulty, 1994; Farmer, Dorsey & Mustillo, 2004; Henggeler & Schoenwald, 1998; McKay & Lynn, in press). Research progress in these areas has been of unprecedented proportions, in part because there was a tripling of funding at the National Institute of Mental Health (NIMH) between 1989 and 2001 for studies of children's mental health in general (National Advisory Mental Health Council, 2001). Although in general research on child and adolescent mental health has lagged far behind studies of adult mental illness, research on the efficacy of specific treatment and service models for youth is now in an era of expansion.
The nexus for this progress has arisen because of the growing popularity of evidence-based practices (EBPs). At least 26 federal Web sites use this term to refer to their practices. A MEDLINE search from 1995 to 2002 found over 5,400 citations that included the terms evidence-based medicine, evidence-based treatment, or evidence-based practice. Between 1900 and 1995 there were only 70 such citations. The term evidence-based practice has captured the public imagination in part because it provides what appears to be a scientific imprimatur upon a body of work whose application in real-world clinical practice is presumed to lead to improvements in children's emotional or behavioral functioning. Because the state of current mental health service de-livery has been widely criticized as being fragmented, ineffective, and insufficient (Bickman, 1996c; Stroul & Friedman, 1986), policymakers' hopes for improvements in service delivery and practices are now largely linked to delivery of EBPs. Logically, this is a reasonable assumption.
However, the growth in empirical knowledge and focus on community-based care has drawn attention to the “different worlds of research and practice” (Ringeisen, Henderson, & Hoagwood, 2003). A series of influential reports, including the Institute of Medicine Report (1998), the Surgeon General's Report on Mental Health (U.S. Department of Health and Human Services [DHHS], 1999), and the Surgeon General's National Action Plan on Children's Mental Health (U.S. Public Health Service [USPHS] 2001b), have uniformly voiced a single theme: the gap between research and practice must be closed.
Yet, despite this progress, there exist numerous challenges to providing quality care for youth and their families within the systems that serve these populations. These challenges are largely systemic and, unfortunately, have become endemic to the current structure of youth mental health care in this country. As Flynn points out in a recent special issue on EBP, “Many parents are frankly doubtful that grafting evidence based practices onto a failed and fragmented system will succeed” (Flynn, in press).
Among the challenges to embedding effective treatments within the current system of mental health care for youth are a range of both research-based knowledge gaps and systemic barriers. Gaps in research knowledge exist in five major areas: treatment development for specific disorders; the comorbidities that exist among these disorders and psychiatric classification more generally; ways of bridging the gap between research and practice; the categorizations of EBPs; and means of implementing or disseminating effective practices. Systemic barriers exist in five areas: service fragmentation; access and use of services, which are especially relevant to ethnic and cultural variations; lack of sustained family involvement; regulatory practices; and fiscal disincentives.
The purpose of this chapter is to outline these challenges to improving delivery of effective treatments for adolescents and their families and to define a set of research and policy directions that may help improve delivery. In particular, a number of major multisite studies supported by several foundations and federal agencies are currently under way to examine contextual influences on delivery of youth mental health services. These studies will very likely lead to new ways of thinking about implementation and dissemination of effective practices.
Gaps in Treatment Development
A recent report from the National Alliance for Mental Illness (NAMI, 2004) indicated that research on serious mental illness has been underfunded, compared to other chronic, disabling illnesses, and consequently is insufficiently prioritized. As a result, little is known about the safety, efficacy, or effectiveness of treatments for bipolar disorders, eating disorders, depression, and trauma. The placebo effect in medication research, particularly in studies of adolescent depression, can be higher than 50% (Emslie, Walkup, Pliska & Ernst, 1999), thus calling into serious question the mechanisms whereby treatment affects outcomes. Furthermore, when families of youth with these kinds of problems are seeking treatment, they are unlikely to be able to identify any providers sufficiently well informed to be able to diagnose and treat these disorders. In addition, despite progress in identifying the efficacy of certain medication treatments for certain conditions (e.g., attention-deficit hyperactivity disorder [ADHD], obsessive-compulsive disorder [OCD], aggression), no studies have yet been completed on the long-term safety and efficacy of these medications. Thus families do not know what the long-term effects may be of treating their children with medication therapies.
Gaps in Comorbidity and Issues in Psychiatric Diagnosis
The strength of the evidence in research-based knowledge centers largely on discrete treatments for discrete disorders (Weisz et al., 1995a; Weisz, Donenberg, Han, & Weiss, 1995b). Unfortunately, many children present with multiple, chronic, and severe problems. The strength of the evidence about mental health care for these youth is weak. This is a major problem because comorbidity of disorders among youth is relatively common. The Great Smoky Mountain Study (GSMS; Costello, Farmer, Angold, Burns, & Erkanli (1997) Costello et al., (1996a,b), conducted in North Carolina, found that while the prevalence of psychiatric disorders in an epidemiological sample among youth was 20%, a full third of the youth had more than one diagnosis. In addition, studies by Weisz and colleagues conducted within public mental health clinics in Los Angeles found that the modal number of diagnoses among youth presenting to these clinics was 5, and that youth with single disorders were extremely rare (Weisz et al., 1995a, 1995b). As a consequence, the lack of a developed research base on the effectiveness of treatments for youth with more than one problem presents serious problems for delivery of services within complex service systems that routinely are responsible for multiproblem youth.
A more fundamental problem exists with respect to psychiatric classification itself, especially in the field of children's mental health. Diagnostic vicissitudes are the norm rather than the exception. This has given rise to a variety of perspectives, not to mention tensions, on what constitutes mental health, mental illness, or psychiatric impairments among children and on their etiology (Jensen & Hoagwood, 1997). The epistemological issues surrounding diagnostic classification and its nomenclature have been described elsewhere. Diagnostic criteria for children rely primarily on observational markers subject to clinical bias (Bickman, 1999), cultural inferences, and these problems are compounded by lack of reliability between research-based diagnoses and community-based diagnoses (Lewczyk, Garland, Hurlburt, Gearity, & Haugh, 2001). These issues raise questions about the na ture of psychiatric knowledge itself, and underscore the importance of inclusion and collaboration of family members, teachers, and other persons knowledgeable about the behavior of children prior to assigning a diagnosis. In short, the complexities surrounding psychiatric diagnosis for children push the notion of collaboration into a place of conceptual prominence with respect to children's mental health.
Meta-analyses of psychosocial treatments for youth have indicated that psychosocial treatment appears to work equally well for internalizing conditions (which include depression and anxiety disorders) and externalizing disorders (those directed more outwardly, including disruptive behavior problems such as conduct disorder). However, most of these studies have been conducted in university or laboratory settings rather than in community clinics. Thus the degree of adaptation (either cultural, practical, or organizational) needed to translate a protocol from a laboratory setting into a community clinic is unknown and is not built into the design or methods for developing treatments in the first place, even though it is a critical ingredient for the ultimate fit of a treatment to a clinical context (Hoagwood, Jensen, Roper, Arnold, Odbert et al., in press).
Studies by Weisz and colleagues have demonstrated that mental health interventions used to treat youth in everyday clinical practice are not only different from those studied in academic settings but also potentially less effective (Weisz et al., 1992; 1995b). Weisz et al. (1992) found that the vast majority of studies supporting the effectiveness of these models were conducted in either university, school, or laboratory settings. Under these conditions, the interventions improve the outcomes for the children. However, the dozen studies that investigated outcomes of these treatments in naturalistic (clinic) settings demonstrated a negative effect. The implications of these different outcomes are that the conditions of routine clinic care are vastly different from the conditions under which most studies of treatment effectiveness have been conducted.
Weisz and colleagues (Weisz, 2000; Weisz et al., 1992, 1995a, 1995b) also identified some possible explanations for the disparity between results in laboratory and clinic settings. One is that laboratory settings may be more conducive to therapeutic gain, simply because they have more resources. Providers may be better trained and have more modern equipment and intensive supervision, making it a better setting in which to deliver services. Another explanation is that psychosocial treatment provided in laboratory settings may result in better outcomes because it is more likely to use behavioral treatment methods, which are well established. Studies are currently under way to examine how and whether research-based treatment models can be delivered in clinic settings and whether these models actually improve clinical outcomes beyond treatments usually delivered within these settings (MacArthur Foundation Network on Youth Mental Health, 2004; J. Weisz, personal communication, March 2004). This study is described in greater detail below.
Limitations and Discrepancies in the Categorization of Evidence-Based Practices
Kazdin (in press-a) suggests that an emphasis on the distinctions between evidence-based and not evidence-based is misguided and limits the potential of research studies to further the goal of improving practices. Instead, he suggests broadening the continuum of evidence to include a range of categories for differentiating studies about treatments. The categories he suggests are (1) not evaluated; (2) evaluated, but unclear, no, or possibly negative effects at this time; (3) promising (some evidence); (4) well-established (parallel to well-established in conventional schemes); and (5) better/best treatments (treatments shown to be more effective than other evidence-based treatments). He also points out that an exclusive focus on outcomes rather than on the reasons that treatments work—i.e., on the mechanisms or processes of change within therapies—is leading to a proliferation of treatments, many of which are likely to be similar. Attention to mechanisms of change can create better efficiencies of effort and improve treatments more generally (Kazdin, in press).
In addition to these limitations, there is currently no consensus on how to define evidence-based, nor on when the evidence base, however it is defined, is ready to be deployed, moved out, or used in community settings. While the term is generally used to demarcate research-based, generally structured and manualized practices that have been examined with randomized trial designs from less rigorous or well-tested practices, no currently agreed-upon definition exists, and it is becoming a popular phrase with which to capture public attention and public funds (Tannenbaum, 2003).
In this chapter I use the term evidence-based practices to refer to a set of research-based treatments, preventive interventions, services, or clinical practices (e.g., assessment, screening, referral). Evidence-based policies refer to local, state, or federal mandates or initiatives promoting the use or adoption of evidence-based practices.
In fact, numerous and discrepant criteria are being used by professional associations and by the scientific community to differentiate evidence based from non–evidence based. The varying definitions make it difficult for policymakers or practitioners to decide which among the practices to adopt in any given circumstance. Currently foundations and federal agencies are attempting to create agreed-upon criteria and an archive of research-based practices that can be updated to assist field practitioners and the scientific community in evaluating the quality, strength, and fit of evidence for specific clinical practice.
In addition, the standards for entry into the lists of EBPs vary widely. Operational criteria have been proposed by the Division of Clinical Psychology of the American Psychological Association (APA; Lonigan et al., 1998) and applied to studies of specific psychosocial treatments for childhood disorders. A similar process has been developed for evaluating the evidence for pharmacological treatments (Jensen, Bhatara, et al., 1999), preventive programs (Greenberg, Domitrovich, & Bumbarger et al., 2001), and school-based mental health services (Rones & Hoagwood, 2000). Yet, because inclusion criteria vary, the ways in which delivery variables have been taken into account also vary. For example, the APA standards apply criteria that ignore parameters of effectiveness that would be likely to yield answers to questions about the readiness of a treatment model to be implemented. Such factors are often excluded from clinical trials of specific treatments for the very reason that they may create “noise” around the interpretation of treatment effects. Yet it is precisely these “nuisance” factors that are essential to understand if implementation of a treatment is likely to be successful. So the relevance of the applicability of most EBPs' integration into routine services is largely unknown (Mufson et al., in press). These different definitions make it difficult for policymakers, families, treatment developers, and consumer organizations to speak in a uniform voice or to learn from each other's lessons in the complex task of implementing these models.
Gaps in Knowledge About Implementation and Dissemination
Knowledge about ways in which to integrate evidence-based practices within complex, dynamic service systems is currently lacking. Challenges remain in “scaling up” interventions from local sites to states. The use of the term evidence-based practice is thus far associated with specific models of child or family interventions; it has not yet been applied to system-or organization-level interventions (e.g., quality assurance methods) (Chambers, Ringeisen, & Hickman, in press). Studies from outside of mental health (in AIDS or cancer trials, for example) have found that characteristics of the environment within which an intervention is placed influence the delivery of the model and the outcomes associated with it. Consequently, studies that examine how to build an infrastructure to support delivery of quality clinical care are greatly needed.
As Ringeisen and colleagues (2003) point out, both content-specific and contextual factors influence the ability to disseminate services and treatments. For example, Kendall et al. (1997) examined specific mechanisms of action within a cognitive-behavioral intervention for childhood anxiety disorders and found that exposure ac counted for the response. This component of the intervention—a feature of its content—is critical to reducing reported childhood fears and anxiety responses. Other factors that reside outside of the content of specific EBPs also influence outcomes. Such factors might include degree or intensity of clinical supervision, or an organization's support for employees. These factors of context are beginning to be examined. For example, Henggeler, Schoenwald, Liao, Letourneau, and Edwards (2002) describe the positive relationship between model-consistent clinical supervision and a therapist's fidelity to multisystemic therapy (MST). Supervision consistent with the MST approach is also positively related to clinical improvement. New studies just beginning to be undertaken in the field of child and adolescent mental health services are examining how systemic features, such as organizational leadership, work attitudes, and climate, may explain variance in the quality of provided services (e.g., Glisson & James, 2002).
Another type of context factor that limits the ability to disseminate EBP models is the small number of studies on the cost-effectiveness of these models. In fact, in mental health, those empirically based interventions that currently have the widest dissemination are also those that have had cost-effectiveness data to support their impact relative to usual care prior to wide-scale promotion (e.g., functional family therapy, Alexander & Parsons, 1973; multisystemic therapy, Henggeler, Schoenwald, Rowland, & Cunningham, 2002; nurse visitation model, Kitzman, Olds, Sidora, Henderson, Hanks et al., 2000). The paucity of studies on the cost-effectiveness of service or treatment models seriously limits the adoption by policymakers of these interventions. These studies are labor-intensive and costly and require careful delineation of the service or treatment components.
Another kind of problem in the spread of empirically based practices occurs because of the rapidity with which knowledge about effective practices is being deployed, coupled with the dearth of knowledge about how best to implement them. This problem has become a kind of catch-22 for the field (Hoagwood et al., in press). The problem arises when treatment developers or organizations not affiliated with the development and testing of a particular treatment model design strategies to take that model to scale. Considerable resources are required to design strategies for overcoming the obstacles enumerated above—for example, in strengthening the capacity of clinicians to implement a model with fidelity. Yet resources must be expended in advance in order to garner research funding for assessing the impact of those strategies (Schoenwald & Henggeler, 2002; Torrey et al., 2001). Thus, although several groups of treatment and service developers have produced similar multilevel approaches to the problem of taking an effective model to scale (Schoenwald & Henggeler, 2002), the methods used to do so have been idiosyncratic, bootstrapped, and informed as much by field experience as by theory and research on the diffusion of innovation, technology transfer, and organizational behavior.
In the mid-1980s a series of federal and state initiatives focused on strengthening the community-based service system for children and adolescents. Called the Child and Adolescent Service System Program (CASSP), a group of state grants were awarded by the NIMH to create youth and family bureaus within state systems. This initiative was given principled footing through the development of an influential model, called the System of Care (Stroul and Friedman, 1986). This model articulated a series of values, centered around maintaining children within their communities, coordinating services, involving families integrally in delivery and planning of treatments and services, and attending to the cultural relevance of services. The most recent iteration of the model has been under an important federal initiative designed to support local community-based services for children and adolescents. Called the Comprehensive Community Mental Health Services for Children and Families (CMHSC) and supported by the Center for Mental Health Services (CMHS) of the Substance Abuse and Mental Health Services Administration (SAMHSA), this program consti tutes the single largest federal program supporting mental health services for youth with serious emotional or behavioral problems. The program is currently financed at close to $100 million per year. One major focus of this program in recent years has been the inclusion of EBPs into the local community-based programs.
Despite the importance of this program, it exists only in a small percentage of communities in the United States. As a consequence, most families seeking mental health care do not have access to these federally funded programs and instead face significant system barriers in accessing effective treatment models. Some of these barriers are outlined below.
Many sectors are involved in delivering services for children, adolescents, and their families, a characteristic that makes them cumbersome to study. In general, at least six separate sectors or administrative structures may be involved in serving youth with mental health problems: specialty mental health; primary health care; child welfare; education; juvenile justice; and substance abuse (Stroul & Friedman, 1986). Across most of the United States, these sectors are administratively, fiscally, and structurally distinct. As a consequence, the responsibility for providing services for a youth with mental health problems or for his or her family is divided among different agencies, each with a unique set of regulations, intake procedures, rules for inclusion and exclusion, and service options. This fragmentation creates enormous problems for families whose children often need different kinds of services for different kinds of problems (Burns, 2003; Burns et al., 1995; Friedman, 2003).
Studies of the impact of coordinated models for delivering services to youth with mental health problems have found that integrated models improve access to care, reduce restrictiveness of placements (e.g., reduce in-patient stays), and improve family satisfaction (Bickman, 1996a, 1996b). However, these same studies have found that these integrated models do nothing in the way of improving clinical outcomes, and concerted calls for improving the clinical care that can be embedded within coordinated systems models have been made repeatedly (Burns, 2003; Henggeler, Schoenwald, & Munger, 1996; Hoagwood, Hibbs, Brent, & Jensen, 1995; Hoagwood, Jensen, Petti, & Burns, 1996).
Underrecognition of youth mental disorders is well documented across all ethnic and racial groups (Costello et al., 1996; DHHS, 1999; Horwitz, Leaf, & Leventhal, 1998; Horwitz, Leaf, Leventhal, Forsyth, & Speechley, 1992; Lavigne et al., 1993; National Advisory Mental Health Council, 2001; Attkisson & Rosenblatt, 1998; Fisher, Dulcan, Davies, Piacentini et al., 1996; Roberts, 1998, Shaffer, 1999; USPHS, 2001a, 2001b). Current evidence suggests that these difficulties disproportionately affect minority youth and families because they are also less likely to have access to general health-care services (DHHS, 1999; USPHS, 2001b). The recent supplement to the Surgeon General's Report on Mental Health, Culture, Race, and Ethnicity (USPHS, 2001c) notes that the disparities affecting mental health services for racial and ethnic minorities have several key characteristics, namely that minorities (1) have more barriers and less access to mental health services, (2) are less likely to receive needed mental health services, (3) often receive a poorer quality of services, and (4) are underrepresented in studies of mental health. Minority parents are at further disadvantage because they and their children are exposed to proportionately greater levels of strains and risks under normal day-to-day conditions, all of which are increased even further when they must deal with a child with emotional or behavioral disabilities (Leaf et al., 1996).
The cumulative impact of lack of access to mental health services, higher levels of risk factors, and historical and current societal discriminatory attitudes place these children on a dangerous trajectory that can lead to more serious emotional or behavioral problems later in life (Loeber & Farrington, 1998; USPHS, 2000). Fur thermore, studies have documented that less than one third of children with suicide ideation, conduct disorder, and substance abuse or dependency seek or receive help for these problems, and that less than half of youth with major depression seek or receive help for depression (Stiffman, Earls, Robins, & Jung, 1988). By necessity, access to mental health services for youth usually depends on the recognition and actions of key adults (Pescosolido, 1992; Stiffman, Chen, Elze, Dore, & Cheng, 1997), some of whom themselves may suffer from mental health problems. Other barriers such as stigma, lack of availability of services, lack of specialists, long waiting lists, and lack of insurance also impede use of services.
According to numerous recent reports and studies (Kataoke et al., 2002; USPHS 2001a; NIMH, 2001; Ringel & Sturm, 2001; Stiffman et al., 1997), the majority of all children—both minority and nonminority—with mental health problems do not receive any mental health service. The recently released report of the President's New Freedom Commission on Mental Health (2003) has identified access to services among children as a top priority. In addition, a report released by the NIMH (2001) indicated that approximately three quarters of children with mental health needs do not receive any type of mental health service, and the disparity between need and use of services is highest among minority youth. These rates are identical to those reported in the mid-1980s by the U.S. Congress, Office of Technology Assessment (1991), which shows that the level of need for services remains unchanged despite decades of scientific progress in developing evidence-based assessments, treatments, and services for these children.
Underidentification is of particular concern in the gatekeeping systems such as schools and primary care settings, where almost all children are seen at some point and where early identification is especially feasible (Costello et al., 1988a, 1988b; Horwitz et al., 1992, 1998; Lavigne et al., 1993; USPHS, 2001a). In fact, in a prospective cohort study, Horwitz et al. (1998) examined the effect of families' attitudes about the appropriateness of discussing psychosocial concerns on pediatric providers' identification of psychosocial problems. The authors found considerable discrepancy between what parents reported as appropriate to do and what they actually did when they recognized mental health problems in their children. Most (81.1%) believed it was appropriate to discuss four or more of the six hypothetical situations with their children's physician, whereas only 40.9% actually did discuss any of these problems with a physician when a problem occurred.
Directly connected to lack of access are significant problems in capacity: most communities in the United States have few if any child psychiatrists to treat children or adolescents (IOM, 2000). Furthermore, the training of other professionals who are most likely to come in contact with children or adolescents (e.g., family physicians or pediatricians, school counselors, psychologists, social workers) is unlikely to have included any content on empirically based practices (American Academy of Pediatrics [AAP], 2000; National Association of Social Workers). Consequently, the mental health system's ability to accommodate the needs of children and adolescents with mental health problems is acutely compromised.
Cultural Barriers to Access
Additional barriers to access deter racial and ethnic minorities, including mistrust of treatment, discrimination, and differences in communication (McKay & Lynn, in press). Mental health care disparities may also stem from minorities' historical and current struggles with racism and discrimination, which may affect their mental health or contribute to lower economic, social, and political status (USPHS, 2001b). The cumulative weight of all barriers to care, not any single one alone, is likely responsible for mental health disparities.
In addition, studies by Takeuchi, Leaf, and Kuo (1988) have demonstrated that different ethnic groups gain access to services through different routes or pathways. For example, Takeuchi found that African-American families entered community mental health care through referrals from social agencies more often than Caucasian families did, and that Mexican-American fami lies entered more often than Caucasian families through school referrals. The patterns of service use also vary across different racial and ethnic groups (McCabe, Yeh, Hough, Landsverk, Hurlburt et al., 1999). These findings have implications for the ways service systems should be organized to serve those who need them and to increase access to care.
Lack of Attention to Family-Driven Practices
Treatments that are inaccessible to those who may benefit from them may be effective but are essentially useless if not accessed. However, a series of important studies have found that successful efforts can be made to enhance a family's service engagement and decrease rates of premature treatment termination. Using brief telephone interviews prior to service entry, a range of studies have been undertaken to troubleshoot barriers to service engagement and increase minority families' participation in services (McKay & Lynn, in press; Santisteban et al., 1996; Szapocznik et al., 1988). Unfortunately, attempts to integrate evidence-based and family-driven engagement strategies with evidence-based clinical treatments are rare. A notable exception to this is a series of studies in New York State, linking implementation of EBPs for child trauma to family-driven practices (Murray, Rodriguez, Hoagwood, & Jensen, in press).
Among the values that have become intrinsic to community-based services have been those that stipulate that parents, guardians, or consumers must be integrally involved in treatment planning and delivery if the quality of care for children is to improve. The federal government through SAMHSA has supported the development of an infrastructure within state mental health agencies to support consumer involvement in service planning, and most states have consumer or recipient offices to strengthen this involvement. Numerous family advocacy organizations now exist to support the needs of families with children who have emotional or behavioral problems more generally, and for those with specific psychiatric disorders (e.g., ADHD, bipolar disorders, depressive disorders, etc.). Simultaneously, there have been several major initiatives in primary care (through Institutes for Healthcare Improvement, for example) to reform health-care services nationally by positioning consumers centrally in treatment planning so that they are empowered to make decisions about their own health care. These initiatives within both general health care and mental health care are leading to innovations in delivery, such as providing families with vouchers to function as case managers for their child's care. The movement away from office-based practice and toward empowerment of consumers is likely to increase significantly over the next decade.
A major issue affecting the acceptability of EBPs to many families is the concern that use of EBPs may interfere with the individualization of services. For many families, the single most important criterion of acceptability is the extent to which services can be individually tailored (Flynn, in press). Evidence-based practices are often seen as prescriptive and constraining. Thus efforts are being made to meld family-based preferences for individualization of care with EBPs, such as the development of parent empowerment programs taught by parent advocate and professional teams to improve knowledge about EBPs (Hoagwood & Johnson, 2003; Jensen, Hoagwood, & Trickett, 1999). Family advocacy groups, such as the Federation of Families, are also developing family-friendly guides to describe in detail the expectations and levels of involvement that specific EBPs require for their implementation (T. Osher, personal communica-tion, April 2004).
Across the range of what are called EBPs (for psychosocial treatments, medications, services, or preventive programs) there are differences in the ways in which deployment occurs. For example, in pharmaceutical medicine, evidence-based approaches have been built into the regulatory standards developed by the U.S. Food and Drug Administration (FDA) to review scientific evidence and identify effective medications. The strength of the evidence for pharmacological treatments is regulated by the FDA, and an industry for their distribution has grown up around this. In contrast, psychosocial treatments, services, and preventive interventions do not have regulatory backing, and their distribution depends largely on the resourcefulness of individuals who developed the treatments or services (Hoagwood et al., 2001)
A series of financing policies are shaping the reimbursement mechanisms for mental health services for children and their families and consequently the possibilities for delivering new effective clinical treatment models. The most significant of the federal policies are those targeting the financing of children's mental health services. These have included significant increases in funding from sources outside of traditional mental health block grants, including expanded funding for Medicaid, State Children's Health Insurance Program (SCHIP), certain educational programs (e.g., Head Start, Safe and Drug-Free Schools), and privately insured pharmacy benefits, among other sources. Policies focusing on the application of behavioral managed care in the public and private sectors and on expansion of health insurance coverage under SCHIP for uninsured children are reducing the role of mental health agencies to effect policy reforms single-handedly. A related set of financing issues has included expansion of the parity for mental health and general health benefits. These are outlined below.
Medicaid Expansion and State Children's Health Insurance Program
The policies involving expansions of public health insurance, including Medicaid expansions in the late 1980s and the introduction of SCHIP in the late 1990s, changed the insurance distribution for youth who use mental health services. The share of all children, and of children using services, who were uninsured fell (Glied & Cuellar, 2003) because public insurance covered children who would otherwise have been uninsured. Public insurance coverage has also displaced private insurance. In 1998 about 25% of children with a diagnosed mental health problem were publicly insured (Glied & Cuellar, 2003). Among poor minority children, those eligible for Medicaid are likely to have higher rates of mental health problems than privately insured children (Glied & Cuellar, 2003). These policy shifts in health insurance coverage mean that state Medicaid directors must now consider both seriously and less-seriously ill children in making decisions about service distribution.
Behavioral Managed Care
Medicaid programs in most states have shifted their mental health coverage into behavioral managed care. By 1999, for example, 42 states operated some form of managed behavioral health care, and this was triple the number in 1996 (Glied & Cuellar, 2003). Behavioral managed care for youth mental health care has generated substantial savings, mainly through marked reductions in the use of inpatient services and a consequent rise in outpatient services. It is not clear whether these reductions have generated negative outcomes for children, because no studies have yet examined the overall effects of behavioral managed care on mental health outcomes for youth. Cost savings have occurred. The movement toward managed care creates a risk for decreased service, cost shifting, and disenrollment. A study of Medicaid managed care in Pennsylvania found a higher rate of disenrollment for children with psychiatric admissions than for those with nonpsychiatric admissions (Scholle, Kelleher, Childs, Mendeloff, & Gardner, 1997). There is also a potential for this sudden rise of managed care to shift the burden of mental health care to public systems, such as juvenile justice, special education, or child welfare (Hutchinson & Foster, 2003).
Over the past 8 years, some legislative efforts have targeted parity of mental health and general health benefits. In 1996, the federal government passed the Mental Health Parity Act (which was implemented in 1998), requiring parity in benefits if mental health coverage was also offered. Although it applied only to lifetime and annual dollar limits (and did not require parity in copayments, deductibles, or limits on days or visits), it represented a major policy advance by offering for the first time mental health coverage. The impact of this legislation on service use and costs are being examined (Goldman & Azrin, 2003).
RECENT INITIATIVES ADDRESSING RESEARCH AND SYSTEM FAILURES
There are several significant initiatives currently under way and supported by foundations and federal agencies to reform some of the system failures referenced above.
National Registry of Effective Practices
The Substance Abuse and Mental Health Ser-vices Administration (SAMHSA) has recently launched a new expansion of its National Registry on Effective Practices (NREP) to include mental health. This registry creates a standard set of criteria with which to establish an archive of research-based practices that can be updated to provide assistance to field practitioners on the quality and strength of effective mental health practices. This registry marks a major effort to bring order to the plethora of usages of the term evidence-based practice.
MacArthur Foundation Youth Mental Health Network
A major new 4-year initiative from the MacArthur Foundation, entitled the Youth Mental Health Initiative, under the leadership of John Weisz, is further extending the reach of EBPs by reviewing the evidence for therapies targeted at the most common childhood disorders. A set of studies will address gaps between research-based treatments and their delivery within clinic systems by examining how best to bring effective treatment practices to youth in mental health service settings. Studies of technology transfer in medicine, nutrition, agriculture, and other fields have shown that effective dissemination requires (a) adaptation of the technologies to fit their intended users, and (b) an understanding of organizational and system barriers to change. This initiative includes two components. A Clinic Treatment Project will test two alternative methods of delivering EBPs within public community-based mental health clinics, using training and supervision procedures designed for the settings and users. A Clinic Systems Project will investigate the organizational, system, and payment issues that influence the ability of providers and clinics to use EBPs. The findings from these two projects are likely to yield answers to significant questions about the readiness of research-based treatments for integration into community practices and the readiness of community practices for adoption of new clinical strategies.
Annie E. Casey Blue Sky
Another new initiative of the Annie E. Casey Foundation is also focusing on disseminating EBPs by melding training and supervisory models across different sets of interventions. The foundation, through its BlueSky Project, has enlisted the involvement of the developers of three evidence-based interventions for youth with disruptive behavior problems who are likely to be involved in the juvenile justice system (Multisystemic Therapy [FFT], Treatment Foster Care, and Functional Family Therapy) to create an integrated training and supervision model. An implementation demonstration program is being planned with two to three states beginning in September 2004.
A group of states are undertaking major efforts to create state-level strategies for disseminating single or, in some cases, multiple EBPs. In several of the states, these efforts also include providing comprehensive training, supervision, or regulatory activities to support the implementation of these EBPs. A brief description of some of the state efforts is provided below.
New York is implementing a range of EBPs and is engaged in a major effort to evaluate the impact of the implementation processes. For example, FFT, a research-based treatment for youth with antisocial behavior problems and delinquency, is being implemented in approximately 12 sites statewide, following an active process of community involvement that sought stakeholder input about needs and options. Another large-scale implementation effort is being undertaken to carry out and evaluate a set of evidence-based cognitive-behavioral trauma treatments for approximately 600 youth affected by the September 11th World Trade Center disaster. New York State has also created a research bureau specifically focused on EBPs for children and adolescents, to track the implementation of research-based services and to include assessment tools, engagement strategies (McKay, Pennington, Lynn, & McCadam, 2001), and outcomes monitoring (Bickman, Smith, Lambert, & Andrade, 2003). In addition, New York has been awarded one of the NIMH-SAMHSA State Planning Grants to develop a set of tools and methodological approaches for assessing the fit between specific EBP models and organizational and contextual factors within mental health clinics and schools statewide. The focus of this effort is to improve understanding about differences among families, clinicians, administrators, and treatment developers in their perspectives on organizational issues relevant to the adoption of new clinical practices.
A different strategy has been undertaken in Texas, where the state is formulating a benefit design package of selected EBPs, supported through training, supervision, and monitoring. In 2003, a Consensus Conference was convened to assist state policymakers in designing the new benefit package. The Consensus Conference included family advocates, policymakers, clinical practitioners, and treatment and service model developers. The recommended benefit package includes both diagnostic-specific psychotherapies and comprehensive community-based services to address the full range of mental health needs of youth and families. This benefit design was implemented in 2004.
In Michigan, a state planning process led to the identification of specific interventions to address the most common clinical problems and a statewide initiative to embed a standardized measure of functioning (CAFAS) into clinical practice. A plan was developed to train practitioners to provide cognitive behavior therapy for internalizing disorders, with assistance from UCLA, and parent management training to address externalizing disorders, with assistance from the Oregon Social Learning Center.
Center for Mental Health Services System of Care and EBPs in Kentucky, Ohio, and California
The Center for Mental Health Services (CMHS) is funding implementation initiatives of specific EBP models in several states, including Kentucky, Ohio, and California. In conjunction with Kentucky's Children's Services Grant Program, CMHS is supporting formal implementation of parent–child interaction treatment (PCIT) in a system-of-care site in eastern Kentucky. Although many randomized clinical trials of PCIT have been conducted, this is the first one conducted outside of academia. The PCIT is being randomized to schools and will also be implemented in a large county in Oregon where mental health centers will be the locus of treatment. Through the Center on Innovative Practices, Ohio is implementing Multisystemic Therapy, intensive home-based services, and wraparound services. Through the California Institute of Mental Health, the state of California is implementing Treatment Foster Care, FFT, and Webster-Stratton's Incredible Years.
An ambitious dissemination approach has been undertaken in Hawaii, called the Hawaii Experi ment (Chorpita, Yim, Donkervoet, Arensdorf, Amundsen et al., 2002). The combination of a 14-month review of the research literature on psychosocial treatments for youth by a group of stakeholders, including families, policymakers, researchers, and practitioners, identified a set of treatments that have been systematically deployed into Hawaii schools. A partnership between academia and the state led to the implementation effort. A process of distillation of EBPs into core practice components, has enabled more clinician flexibility in selecting treatments that fit with parent, child, and clinical needs. This statewide initiative is being carefully evaluated and a set of clinical practice guidelines and tools are being developed (Chorpita, Daleiden, & Weisz, in press).
RESEARCH ON ORGANIZATIONAL CONTEXT AND ITS IMPACT ON SERVICE DELIVERY
During the past decade, a series of studies originally conducted within business and industry has been applied to human service agencies, most recently including mental health (Glisson, 2002; Glisson & James, 2002; Schoenwald, Sheidow, Letourneu, & Liao, 2003). This application has arisen because it has become apparent that theory and constructs about organizational behavior have much to offer as interpretive frameworks within which to understand the capacity of mental health systems to change, adapt, or adopt new technologies, including EBPs. In the mental health field, organizations or organizational systems refer to the range of service delivery settings where treatments or mental health services are delivered (e.g., clinics, schools, group homes, or other work environments).
Much of the new thinking about the application of organizational theory to mental health service delivery is derived from the diffusion of innovation literature, generally ascribed to Everett Rogers (1995). Rogers delineated different stages of adoption that characterize the uptake of new technologies and identified characteristics of the diffusion process that lead to their sustainability over time (Van de Ven, Polly, Garud, & Ventkataraman, 1999). For example, Rogers (1995) noted that the characteristics and motivations of adopters tend to differ among those adopting at the beginning, middle, and end periods of an innovation, and he identified attributes of innovations that can be used to design new technologies for their later successful diffusion. These characteristics include the extent to which an innovation is believed to be better than the current model of care, the degree to which an innovation is perceived to be consistent with existing values, the ease with which an innovation can be used, and the extent to which results of an innovation are visible to others.
Empirical studies from the diffusion of innovation literature indicate that organizations and systems are not equally innovative and that some are much more open than others to adopting and implementing new practices. The factors that affect the capacity of an organizational system to change, adapt, or take up new innovations have been examined in a series of studies mostly outside of the human services field. These studies have identified a core group of dimensions along which work environments or service systems vary. In recent work, most notably by Glisson (2002) and Schoenwald et al. (2003), key factors within mental health agencies have been identified that affect delivery of services.
Some of the key constructs within the organizational literature found to influence the behavior of providers are summarized below.
Organizational Culture and Climate
The constructs of organizational culture and climate have been measured in tandem and recently found to be discrete (Glisson & James 2002; Verbeke, Volgering, & Hessels, 1998). In general, organizational theorists define climate as the way people perceive their work environment, and culture as the way things are done in an organization (Verbeke et al., 1998). In other words, climate is defined as a property of the individual and culture as a property of the organization.
Climate is further subdivided into psychological climate and organizational climate (Glisson & James, 2002; James & James, 1989; Jones, James, & Bruni, 1975; James, James, & Ashe, 1990). Psychological climate is defined as an individual's perception of the psychological impact of their work environment on their well-being (James & James, 1989). Organizational climate occurs when employees in the same organizational unit have the same perceptions about their work environment (Jones & James, 1979; Joyce & Slocum, 1984). Some of the core components of the psychological climate of a work environment include emotional exhaustion, depersonalization, and role conflict. A general psychological climate factor (PCg), reflecting the workers' overall perception of the positive or negative valence within the organization, is believed to constitute the core features underlying this construct (James & James, 1989; James et al., 1990; Brown & Leigh, 1996; Glisson & Hemmelgarn, 1998; Glisson & James, 2002).
Culture is described as a “deep” construct that reflects the normative beliefs and shared behavioral expectations in an organization or work unit (Cooke & Szumal, 1993). These beliefs and expectations provide a guiding framework by which the priorities of the organization are made explicit. These priorities or values include conformity, consensus, and motivation. According to Glisson (2000), organizational culture is often described as layered, with behavioral expectations and norms representing an outer layer and values and assumptions representing an inner layer (Rousseau, 1990). Hofstede (1998) described behavior as the visible part of culture and values as the invisible part.
Structure refers to the formal organization of an agency or unit. This construct has the longest history and has been studied in greater depth and detail than the other organizational constructs. Core components of organizational structure include the centralization of power and the formalization of responsibilities and position in an organization. Structural functions such as participation in decision making, hierarchy of authority, ways in which roles are divided, and the procedural specifications that guide the division of labor all comprise elements of an organization's structure (Glisson, 2002).
Job satisfaction and organizational commitment have been studied since 1976 and appear to comprise features associated with work attitudes (Glisson & Durick, 1988). The distinction between satisfaction and commitment inheres in the difference between attachment to an organization and positive acceptance of one's duties within it (Mowday, Porter, and Steers, 1982; Williams & Hazer, 1986).
The characteristics of effective leadership have been identified largely through studies of staff perceptions about this quality. In these studies, three types of skills have been identified (Glisson & Durick, 1988): the extent to which a leader is perceived as willing to make key decisions and comfortable with this responsibility; the extent to which a leader is perceived as using the power to make decisions without authoritarianism; and the perceptions of the leader as intelligent. These dimensions are believed to comprise a single factor reflecting overall staff perceptions of a leader's capabilities (Glisson & Durick, 1998).
Organizational Readiness to Change
The construct of readiness to change is a recent addition to the dialogue within the organizational literature. It is a construct that has arisen within the substance abuse literature and includes aspects of culture, climate, structure, leadership, and work attitudes (Simpson, 2002). It has been examined with reference to the question as to why some organizations are more apt to adopt innovations than others (Simpson, 2002). Debate currently exists as to whether readiness to change is a separate dimension from the other constructs or whether it constitutes a whole subsumed within which are the other dimensions of organizational behavior. One of the research projects being undertaken in New York state is targeted at constructing a typology for or ganizational readiness, which will reflect multiple stakeholder perspectives, including, importantly, families and consumers. This typology will be used to assess for policy-planning purposes the readiness for uptake of a set of specific EBPs for youth (Hoagwood et al., in press).
Studies of Organizational Effectiveness
Recent work by Glisson and colleagues suggests that considerable variation exists among organizations in their capacity to accept innovations and to implement new technologies. These studies have demonstrated that an organization's structure and leadership affect the extent to which work environments allow experimentation with innovations or the adoption of new technologies. Studies from literatures within business and industrial organizational research suggest that work environments vary considerably along the dimension of innovativeness and openness to adoption. Structure and leadership are known to be important dimensions of organizations, but other variables—notably organizational culture, climate, and work attitudes—are also factors that predict innovation. These studies suggest that flexible structures, strong leadership, constructive cultures, nonrestrictive climates, and positive work attitudes contribute to innovation in organizations and the adoption of cutting-edge technologies (Glisson & James, 2002).
In addition, Glisson has identified four organizational requirements for effective work environments (2002): use of assessments and treatment interventions that are appropriate, valid, and effective for the populations targeted by the service system; assurance that adherence to the protocols is obtained; positive alliances between the clinicians and the clients; and finally, provision of services that are available, responsive, and characterized by continuity. Glisson has found that these characteristics can be modeled and that they improve the culture and climate of child welfare and juvenile justice systems (Glisson, 2002). Whether this will hold true for mental health–care agencies remains to be determined.
Summary of Organizational Research Findings
Characteristics of organizations that promote adoption, adherence, alliance, and service responsivity have been identified in relatively global ways. Yet a specific understanding of the complex relationships that coexist among culture, climate, structure, leadership, attitudes, and the complex therapeutic process of alliance, adherence, and fidelity has yet to be delineated. According to Glisson, constructive and non-defensive organizational cultures, less centralized and formalized organizational structures, safe organizational climates, and positive work attitudes promote these attributes of service delivery. In identifying ways to improve health-care delivery and quality, it is likely that attention to issues of the specific fit among organizational elements and therapeutic processes will need to be identified if EBPs are to be sustained.
SUMMARY AND RECOMMENDATIONS
Create a Context for Constant Empirical Inquiry in Routine Practice
Current approaches to the implementation of EBPs within state and local service systems are largely characterized as unidirectional: research-based models are taken off the academic shelf and put into place within routine practice settings. An alternative approach is to encourage routine practice settings to become the seat of empirical inquiry—to become empirically driven centers for both delivering and examining practices and their link to outcomes (Kazdin, 2004). Such normalization of research-based approaches to practice would demystify the scientific enterprise and create services that could be constantly reevaluated, refined, and improved. This kind of revolution in thinking could lead to the creation of service clinics that construct locally relevant evidence. This approach could be used to create a context for empiricism within routine service settings, leading ultimately to improvements in quality.
Ensure That all Studies of Treatment Development and Delivery Include Perspectives of Families and Providers
If the goal is to enhance the generalizability and uptake of research findings into practice, then from the outset research models should incorporate the perspectives of families, providers, and other stakeholders into the design of new treatments, preventive strategies, and services. Only by doing so can issues relating to the relevance of the intervention for stakeholders, the cost-effectiveness of the intervention, and the extent to which it reflects the values and traditions of families and community leaders be addressed. These issues are ultimately essential for the evidence base to be of any practical utility.
Put into Action Clinic and Community Intervention Development and Deployment Models
The time lag between creation of an EBP and its acceptance into routine practice is estimated to be 20 years (IOM, 2001). To accelerate the pace of development of EBPs and their deployment into routine practice, new models for their creation have been proposed. Building on the deployment focused model (DFM) of Weisz (2003, 2005), Hoagwood, Burns, and Weisz (2002) developed the clinic–community intervention model (CID) to extend the DFM by attending to those context variables such as characteristics of the practice setting (e.g., practitioner behaviors, organizational variables, community characteristics) and involvement of families and community from piloting, manualization, and dissemination that are essential to the ultimate acceptability of new services. These models are proposed as a way to ensure strong scientifically based practices and to accelerate the pace of the uptake of research findings into practice.
It does little good to know what treatments are effective if those treatments cannot reach the children and families who need them. Likewise, it does little good to have effective treatment protocols sitting idly on academic shelves.
Significant state and national policy initiatives are currently focused on more closely aligning science and practice. These initiatives present unique opportunities for linking scientific developments on effective clinical care to organizational system and policy reform. The availability of a growing research base on effective clinical treatments and practices offers an opportunity to tap into a reservoir of scientifically based strategies. Testing their applicability within locally based service systems is the key question for the next generation of services research. However, limitations in the evidence base as well as limitations in understanding the fit of EBPs to service contexts pose considerable challenges to treatment developers, policymakers, administrators, clinicians, and families. New models for crossing the boundaries between research and practice and accelerating delivery of quality care need to be applied. Creating family-driven and empirically based services and supporting these services through policies that provide fiscal incentives for delivery of outcome-based practices requires commitment to empirical inquiry and to the “obstinate questioning” (Wordsworth, Intimations of Immortality: An Ode, 1819) that alone sustains creative change.