Generalizability in two clinical trials of Lyme disease

Objective To examine the generalizability of two National Institutes of Health (NIH)-funded double-blind randomized placebo-controlled clinical trials in patients with chronic Lyme disease and to determine whether selection factors resulted in the unfavorable outcomes. Design Epidemiologic review of the generalizability of two trials conducted by Klempner et al. This paper considers whether the study group was representative of the general chronic Lyme disease population. Results In their article in The New England Journal of Medicine, Klempner et al. failed to discuss the limitations of their clinical trials. This epidemiologic review argues that their results are not generalizable to the overall Lyme disease population. The treatment failure reported by the authors may be the result of enrolling patients who remained ill after an average of 4.7 years and an average of 3 previous courses of treatment. The poor outcome cited in these trials may be explained by having selected patients who had undergone delayed treatment or multiple treatments unsuccessfully. These selection factors were not addressed by the studies' authors, nor have they been discussed by reviewers. The trials have been over-interpreted by the NIH and widely publicized in a press release. The results have been extrapolated to other groups of Lyme disease patients by commentators, by a case discussant in an influential medical journal, and by health insurance companies to deny antibiotic treatment. Conclusion The Klempner et al. trials are assumed to be internally valid based on a Randomized Control Trial (RCT) design. However, this review argues that the trials have limited generalizability beyond the select group of patients with characteristics like those in the trial. Applying the findings to target populations with characteristics that differ from those included in these trials is inappropriate and may limit options for chronic Lyme disease patients who might benefit from antibiotic treatment.


Background
According to the Centers for Disease Control and Prevention (CDC), Lyme disease is the fastest growing vectorborne disease in the United States with over 40,000 cases reported during 2001-2002, representing a 40% annual increase in incidence [1]. Studies from the 1990's suggest that the actual number of cases may be as much as twelve times higher when factoring in underreporting [1]. Enzootic cycles of Lyme disease can be maintained in a wide range of ecologic conditions [2,3]. Forty-nine of 50 states and the District of Columbia in the USA had reported Lyme disease cases by 1998 [4] The areas of the country endemic for Lyme disease have expanded [5]. In New York, over an 11 year period, cases have spread throughout the state from the original southeastern focus [6]. Increasing numbers have also been reported in the United Kingdom, France, and Sweden [3].
Lyme disease presents formidable challenges because of the high percentage of cases that become chronic in the absence of early treatment [7,8] and the complexity and cost of managing the long-term use of antibiotics in treating chronic disease [9]. The number of Lyme disease cases that become chronic has been reported to be as low as 0.5% to 13% per year for patients treated at the time of an erythema migrans rash [10][11][12][13]. However, as many as 34% to 62% developed the chronic form of the disease in two studies [7,8]. A third of Lyme disease patients in one population-based retrospective cohort treated in the late 1980's were found to have chronic Lyme disease an average of 6.2 years after treatment [7]. Two thirds of 215 Lyme disease patients diagnosed in Westchester County, New York, USA remained ill an average of 3.2 years after treatment [8].

Analysis
The Klempner et al. trials that appeared in the New England Journal of Medicine examined the benefits of treating chronic Lyme disease patients with one month of intravenous ceftriaxone followed by eight weeks of oral doxycycline [14]. Chronic Lyme disease patients who received antibiotics for 90 days were no more likely to improve than those receiving placebo. The high treatment failure rate of 60% was significantly greater than the 0-37% failure rates documented in previous studies of chronic Lyme disease [15][16][17][18][19][20] (Table 1). For this reason, it seemed prudent to examine the validity of the authors' conclusions regarding the treatment, in general, of chronic Lyme disease patients.

Internal validity of the Lyme disease trials conducted by Klempner et al
Blinded, randomized controlled trials (RCT) are seen as the most reliable evidence in medicine if internal and external validity can be assumed [21,22]. Klempner et al. enrolled a homogeneous patient population, used a randomized design, standardized treatment, placebo-controls, blinding, a validated quality of life outcome measure, and intent-to-treat analysis. The study suggested that treatment for 3 months was no better than placebo for a select population who remained ill an average of 4.7 years after an average of more than three courses of treatment [14]. Assessing the internal validity of these trials for estimating efficacy in the study population is beyond the scope of this paper; for present purposes, results of the trials are assumed to be internally valid, based on the RCT design.
However, achieving internal validity does not imply generalizability.

Generalizability of the Lyme disease trials conducted by Klempner et al
Generalizability can be assessed by considering factors that may influence the outcome of an intervention across varied medical settings with diverse patient populations [23]. The authors of the trials did not discuss generalizability to the everyday medical setting.
Investigators of other conditions have demonstrated the consequences when a study fails to address the broader group seen in everyday clinical practice. Jüni et al. cites the lack of effectiveness of fibrinolytic therapy for suspected acute myocardial infection when generalizing the results from a younger group to the elderly and when generalizing timely fibrinolytic therapy to patients presenting more than 12 hours after symptom onset [22].
This review examines whether the Lyme disease patients enrolled in the Klempner et al. trials represent those commonly seen in everyday practice. Subjects were eligible if they were at least 18 years old, had a history of Lyme disease acquired in the United States, and had at least one of the following: a history of a single or multiple erythema skin lesion, early neurologic or cardiac symptoms attributed to Lyme disease, radiculoneuropathy, or Lyme arthritis. Documentation by a physician of previous treatment of acute Lyme disease with a recommended antibiotic regimen was also required. At the time of enrollment, all patients had one or more of the following symptoms that interfered with their function: widespread musculoskeletal pain, cognitive impairment, radicular pain, paresthesias, or dysesthesias. Profound fatigue often accompanied one of these symptoms. The chronic symptoms had to have begun within 6 months after the initial infection with B. burgdorferi and had to have persisted for at least 6 months but less than 12 years.
Patients were excluded if they had hypersensitivity to the study medication, had previously received parenteral antibiotic therapy for 60 days or more for their current symptoms, had active inflammatory synovitis, had a coexisting condition that could have accounted for their symptoms, or were unable to discontinue medication that could interfere with the evaluation of their response to the treatment regimen (e.g., narcotic analgesics or prednisone in a dose of 10 mg per day or more). Patients with a positive polymerase-chain-reaction (PCR) test for B. burgdorferi DNA in plasma or cerebrospinal fluid at baseline were also excluded.
The results of the trials are not generalizable to patients receiving treatment for chronic Lyme disease within 6 months of their initial presentation, because the investigators excluded these patients from study. Chronic Lyme disease has been reported as early as 2-4 weeks after onset of acute disease [18,24]. The impact of treatment delay is poorly understood. Asch et al. describe a retrospective cohort of 215 subjects with an average 6-week delay in getting treatment [8]. Sixty-two percent were ill an average of 3.2 years after initial treatment. The Klempner study did not consider the impact of treatment delay on long-term treatment failure. In the published report, the authors did not make it very clear that participants had already been ill with Lyme disease an average of 4.7 years at the time of their enrollment in the study [14]. Information about this potential selection factor is found only in the tablesrather than in the results section or the abstract, where it should have been in order to avoid misinterpretation of the results [14]. The Klempner report's failure to take average duration of study participants' illness into account when interpreting the results gives readers the potentially misleading impression that the Klempner et al. study can be generalized to the overall population of patients that present with persistent symptoms and a history of Lyme disease.
Furthermore, the results of the trials may not be generalizable to chronic Lyme disease patients presenting for a first or second retreatment. The authors did not discuss the significance of the study participants' average of 3 previous courses of antibiotics [14]. Again, this potential selection factor is only described in the tables, rather than, more appropriately, in the results section or the abstract [14]. Initial retreatment is both commonplace and successful in previous studies [15][16][17][18][19][20][25][26][27]. By failing to enroll a sufficient number of patients who had received fewer than three previous courses of treatment, the researchers may have excluded the subset of Lyme disease patients most likely to benefit from retreatment.
The investigators can only draw conclusions about the 3month combination of oral and intravenous antibiotic treatment that was chosen for the study and not about longer treatments or simultaneous administration of multiple antibiotics. They dismissed the potential benefit of longer treatment or other combinations of antibiotics by saying "Experience with other chronic infectious diseases caused by persistent bacteria (e.g., syphilis, tuberculosis, and helicobacter infection) suggests that it is unlikely that more prolonged antibiotic therapy or a different combination of antibiotics would result in greater improvement than was observed in this study" [ [14], p.89]. The authors did not provide references to support this statement; prolonged antibiotic use or simultaneous administration of multiple antibiotics have been effective for tuberculosis [28] and helicobacter infection [29].
The two trials cannot be generalized to chronic Lyme disease patients who have never received treatment, since these patients were excluded from study. Up to one third of patients never present with the classic erythema migrans rash, Bell's palsy, meningitis, heart block, and/or arthritis, which are indications of early Lyme disease [17,18,20,27]. It is not clear that the same treatment that is effective for early Lyme will be equally effective for untreated chronic Lyme. Prolonged antibiotic treatment may be called for in cases of untreated chronic Lyme.  [13] note that "There is no scientific evidence to justify prolonged antibiotic therapy for patients with any manifestation of Lyme disease, and our study and that of others [34] should further help to discourage such practice. In addition, antibiotics are no better than placebo in treating patients who carry the label of 'chronic Lyme disease,' probably because evidence indicates that this entity is not infectious" [ [35], p.577]. The authors make this statement without citing evidence that supports the notion that chronic Lyme disease is not infectious, other than the Klempner trials.
In another published paper citing the Klempner et al. trials as evidence that post-Lyme syndrome is distinct from Lyme disease, patients were said to have "developed a syndrome of diffuse arthralgia, myalgia, fatigue, and subjective cognitive difficulty during or soon after LD" [ [36], p.385]. However, these so-called "post-Lyme" symptoms are also typical of Lyme disease itself. The authors did not present clear evidence that "post-Lyme syndrome" was a distinct illness in patients who are demonstrably no longer infected with Lyme spirochetes.  [38] discussed the poor sensitivities of these tests for chronic Lyme disease [15]. For example, only one of a series of twenty-seven cases of neurologic Lyme disease presented with a CSF pleocytosis and that case had only 7 cells [15]. Furthermore, PCR and culture tests may only be useful for subjects with early Lyme disease who have never been treated with antibiotics [39]. Neither increased white blood count nor erythrocyte sedimentation rate is elevated in acute or chronic Lyme disease [40].
A discussant in a clinician's corner published in JAMA [41]

Conclusion
Klempner et al. did not adequately critique the generalizability of their trials. This review argues that the study participants were not representative of the overall population of chronic Lyme disease patients that present with persistent symptoms and a history of disease. Limited generalizability has been a problem of other randomized trials [21] and it remains one here.
This review argues that the poor treatment response in the Klempner et al. trials may be explained by having selected patients who had undergone delayed treatment or multiple treatments unsuccessfully. The quality of life of subjects enrolling in the Klempner et al. trials was worse than that of the average type II diabetic or patient recovering from a heart attack, and as poor as that of subjects suffering from congestive heart failure [14]. In other words, it may be an example of offering patients "too little too late." Klempner himself described his concerns about the study population to the editors of Science: "After a year of advertising, only 57 subjects had been enrolled. The goal is to get 260 by the time the study ends in 2 years. More than 1200 people have expressed interest, and 700 have come in for screening. But only 1 in 10 who appear in the clinic fits the study's strict criteria" [45, p.1431]. Neither in the Science interview [45], nor when reporting the trials' results in the New England Journal of Medicine [14], did Klempner discuss whether the strict criteria was a factor leading to the average 4.7 year onset of illness of subjects enrolled.
Two additional randomized trials and one case series have been published since the 2001 Klempner et al. trials [25][26][27]. A fourth trial by Krupp supported antibiotic treatment for a subset of chronic Lyme disease patients with fatigue [46]. One month of intravenous ceftriaxone was effective at reducing the primary fatigue endpoint but not the secondary endpoints of cognitive function or OspA antigen [46]. These additional trials of chronic Lyme disease [25,26,27,46] continue to suggest that treatment may be beneficial for some subgroups of patients who were not well represented in the Klempner et al. trials.
In summary, this review exposes the limited generalizability of the findings of Klempner et al., and the overreaching impact these trials have had on influencing policies that affect unrepresented patient groups. In interpreting the results of these trials, physicians should consider the select group of patients that were chosen for study and whether the patients in their care might respond differently to treatment.