Bridging Differences in Outcomes of Pharmacoepidemiological Studies: Design and First Results of the PROTECT Project

Background: Observational pharmacoepidemiological (PE) studies on drug safety have produced discrepant results that may be due to differences in design, conduct and analysis. Purpose: The pharmacoepidemiology work-package (WP2) of the Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium (PROTECT) project aims at developing, testing and disseminating methodological standards for design, conduct and analysis of pharmacoepidemiological studies applicable to different safety issues using different databases across European countries. This article describes the selection of the safety issues and the description of the databases to be systematically studied. Methods: Based on two consensus meetings and a literature search, we selected five drug-adverse event (AE) pairs to be evaluated in different databases. This selection was done according to pre-defined criteria such as regulatory and public health impact, and the potential to investigate a broad range of methodological issues. Results: The selected drug-AE pairs are: 1) inhaled long-acting beta-2 agonists and acute myocardial infarction; 2) antimicrobials and acute liver injury; 3) antidepressants and/or benzodiazepines and hip fracture; 4) anticonvulsants and suicide/suicide attempts; and 5) calcium channel blockers and malignancies. Six European databases, that will be used to evaluate the drug-AE pairs retrospectively, are also described. Conclusion: The selected drug-AE pairs will be evaluated in PE studies using common protocols. Based on consistencies and discrepancies of these studies, a framework for guiding methodological choices will be developed. This will increase the usefulness and reliability of PE studies for benefit-risk assessment and decision-making.


INTRODUCTION
Randomised Clinical Trials (RCT) of drug adverse events do not optimally reflect real life situations: small sample sizes, highly selected populations and short duration of exposures [1]. During the past decades, it has been realized that adverse drug-events (AE) need to be further evaluated in pharmacoepidemiological (PE) studies [2]. PE methods were, however, still in development and therefore had the potential for reporting biased results. An example is the falsely reported relationship of breast cancer to use of the blood pressure lowering drug reserpine [3]. The growing availability of large routine electronic health record databases has made it possible to study less frequent and less severe AEs. An example is the risk of deep venous thrombosis in users of third generation oral contraceptives [4]. Alhough (pharmaco)-epidemiological methods have progressed, the challenge of studies of low absolute and relative risks associated with medications may have pushed pharmacoepidemiology to the borders of what can reliably be detected beyond the level of background noise [5]. Furthermore, efforts focusing on evaluation of type A AEs (those with dose dependent and predictably augmented pharmacological effects) and intended effects of drugs have increased the potential for bias [6].
Study conduct and design choices are one of the factors contributing to the diversity and discrepancy of study results. For instance, using the same database (the UK Clinical Practice Research Datalink [7] and including a large number of patients, two studies that were independently conducted reached very different conclusions [8,9]. Within the same source study population, discrepant results between studies can be explained by small differences in study design such as different definitions of exposure time windows, confounder selection and age matching [9,10]. Moreover, exposuretime-dependent hazard functions can substantially affect comparisons between different studies of the same drug [11]. The use of different statistical methods to adjust for confounding is another explanation for dissimilar study results [12]. For instance, in a database study and in simulation studies, systematic differences were found in effect estimates when propensity scores were used compared to logistic regression or Cox-proportional hazards regression [13,14]. Immortal time bias has been suggested as another important source of variability in results between observational studies on drug effects [15]. Furthermore, several studies that have evaluated the same data source have drawn different conclusions about the plausibility of a pharmacological explanation of an observed association. Among these are: use of inhaled corticosteroids and risk of hip fracture [16,17], use of beta-blocker and risk of hip fracture [18,19]; use of oral bisphosphonates and risk of cancer of oesophagus [20,21]; and more recently, use of proton pump inhibitors and risk of hip fracture [22][23][24][25]. The influence of methodological variation should be minimized and quantified, in order to interpret differences in associations between drugs and AEs that arise between types of data sources and healthcare systems in the different countries. A clear interpretation of differences in results between studies performed in the same database, and between different databases, is currently not completely feasible due to these methodological differences. This situation poses difficulties for all stakeholders, such as regulatory agencies, industry, healthcare professionals and patients. Difficulties in interpreting individual and/or groups of observational studies limit their usefulness for decision making on the benefit-risk balance of drugs. These experiences highlight the need to increase understanding of the implications of different methodological choices by investigators and for a framework on PE methodology across different data sources. To understand and subsequently validate differences caused by methodological and nonmethodological (data related) factors we have selected five different drug-AE pairs, to be analysed in five different European databases based on a common protocol that includes extensive sensitivity analyses.
The Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium (PROTECT) study is a collaborative European project that addresses limitations of current methods in the field of pharmacoepidemiology and pharmacovigilance [26]. PROTECT is a multinational consortium of 29 partners including academics, regulators, small and medium enterprises (SMEs) and European Federation of Pharmaceuticals Industries and Associations (EFPIA) companies, coordinated by the European Medicines Agency (EMA) with GlaxoSmithKline (GSK) as deputy coordinator. The "Framework for pharmacoepidemiology studies" work-package (WP2) of PROTECT, co-led by Utrecht University and Pfizer, aims at developing, testing and disseminating methodological standards for the design, conduct, and analysis of PE studies applicable to different safety issues using different data sources. This article presents the rationale, design and the first results of the WP2 of PROTECT initiative.

Selection of Drug-AE Pairs
Criteria for the selection of key AEs to evaluate in different databases included: 1) the AE selected having resulted in (major) regulatory decisions such as drug withdrawal or major summary of product characteristics (SmPC) changes; 2) public health impact aspects including seriousness of the event (prioritise more serious events); having variable incidence rates (both rare and common events); and prevalence of drug exposure (commonly used drugs and infrequently used drugs); 3) possibility to investigate a broad range of relevant methodological issues including feasibility to ascertain events in electronic healthcare databases (events both easy and difficult to ascertain); hazard functions (acute and long-term effects, delayed/transient effects); setting of drug use (in-/outpatient use); type of use (short/long-term, as needed); and different indications of use. All drug-AE pairs needed to fulfil these criteria. Furthermore, at least one drug-AE pair was selected taking into account those chosen by the public-private US initiative Observational Medical Outcomes Partnership (OMOP) in order to facilitate comparison with this initiative [27].
An initial inventory of potential drug-AE pairs was compiled, based on recommendations from public and private partner experts in the field of epidemiology and pharmacovigilance (European and national medicines agencies, pharmaceutical industry and academia). All partners were asked to nominate 10 drug-AE pairs that would fulfil the previously defined criteria for selection. This resulted in an initial list of 55 AEs and >55 individual drugs and drug classes. A first consensus meeting produced five AEs and a limited number ( 3) of drugs per AE with high priority. Supported by extensive research of the scientific literature and publicly available information sources, including PubMed, EMA and the US Food and Drug Administration (FDA) websites, each of the criteria for the selected drug-AE pairs was assessed. Subsequent to this assessment, the selection of five drug-AE pairs was finalized in a second consensus meeting.

Databases
All PROTECT partners who manage or have access to electronic healthcare or reimbursement databases were asked to describe characteristics of these databases. Databases incorporated medical and registry-based data sources, such as the Danish national registries, the Dutch Mondriaan project, the British CPRD and The Health Improvement Network (THIN) databases, the Spanish BIFAP project and the German Bavarian claims database. In addition, the French PGRx case-referent system will be made available to investigate and/or confirm some of the drug-AE pairs. All partners were sent a questionnaire in order to systematically collect the information. Parameters included information on period of data collection, coding systems, accessibility procedures and an extensive list of specific categories for longitudinally collected data such as drug prescribing/ dispensing, clinical data, laboratory test data and life style parameters. The databases from the Netherlands, Spain, Denmark, and UK are based on primary care (GP and/or Pharmacy) covering all prescription drugs regardless of reimbursement.

Analytical Approach
Common study protocols to study each of the drug-AE pairs have been developed and comply with the ENCePP methodological standards (including the ENCePP checklist) and were submitted to the ENCePP registry of studies [28]. These protocols include different study designs such as cohort, case-control, and case-cross-over design. All studies are retrospective, based on existing data from the databases described above. We will use data from the period 2001-2009. Inclusion for entry in the cohort studies is that subjects would have to have at least 1 recorded prescription or dispensing of the drug of interest. This approach reduced confounding by indication and still allows comparing between subjects that are on the drug at a certain time during follow-up versus subjects that are not currently on the drug but used the drug in the past. Operational definitions of exposures and outcomes are harmonized as much as possible and varied in a range that reflects the possibilities and limitations of the available databases. For the outcome of liver injury a automated algorithm has been developed taking into account diagnostic codes and laboratory tests. Detailed code lists are available upon request. Exposure will be analysed time-dependently in all studies and some confounders will also be classified time-dependently if appropriate. Different methods for the selection of and control for confounding variables will be applied. Not all databases have the same level of detail with regard to confounders. We will conduct an analysis for each drug-ae pair that includes a minimum set of confounders that all databases have available. In subsequent sensitivity analyses we will also assess the impact of further adjustment for confounders that are available in some, but not all databases. For all databases we will describe exposure to the drugs of interest and for those databases with sufficient information on diagnoses we will describe the outcomes of interest. For the association studies we have implemented a blinding procedure with central results management. Results for each design will be un-blinded only after all databases have been analysed and produce the adjusted association measures.

The Drug-AE Pairs
The five drug-AE pairs fulfilling the a priori defined criteria are: 1) inhaled long-acting beta-2 agonists and acute myocardial infarction; 2) antimicrobials and acute liver injury; 3) antidepressants and/or benzodiazepines and hip fracture; 4) anticonvulsants (approved for treatment of epilepsy) and suicide/suicide attempts; 5) calcium channel blockers and malignancies. The following information is described for each drug-AE pair: public health impact, drug utilisation, the level of evidence to support a causal association, the proposed pharmacological mechanism(s), and methodological challenges specific for the drug-AE association. Table 1 shows the selected AEs and their characteristics. Table 2 shows the characteristics of the selected drugs. Table 3 displays the drug-AE associations and characteristics such as the range of relative risks, the study designs that have been used to study the association, the main methodological issues, and the suggested hazard function (in relation to onset and offset of the increased risk after initiation or discontinuation of the drug).

The Databases
General features of the databases participating in PROTECT are presented in Table 4. The six databases contain data from patients from five different European nations: the Danish national registries, the Dutch Mondriaan database, the British CPRD and THIN databases, the Spanish BIFAP project, the German Bavarian claims database. The Danish registries have national coverage, while other databases contain regional data or a representative sample of a total population. All the databases were quite representative of their nation. Most of the databases were established more than 10 years ago with regular and expanding data collection and validation history. Routine checks on quality are performed in all databases. The majority of databases include GP data and two (Danish and CPRD) include registries for and linkages to mortality, cancer, and secondary care data. Three (Danish registries, Mondriaan and Bavarian claims) out of six databases include or had linkages to claims data. A particular characteristic of the Bavarian Claims database is the availability of information on prescriptions and diagnoses in quarters of a calendar year. The exact dates of prescribing and diagnoses are not available. Therefore, we decided to use this database for descriptive purposes only and refrained from conducting association studies for which this information is pivotal. For some databases, linkage to other national registries requires additional procedures and financial compensation. Table 4 briefly describes the participating databases.

DISCUSSION AND UPCOMING STUDIES IN PROTECT
We prioritised five drug-AE associations that are highly relevant from the perspective of various stakeholders including regulatory agencies, patients and the pharmaceutical industry. These associations allow investigation of the  [34] For biologicals [41] 5-year fatality rate: 43%-71% [53] Tumour diagnosis cancer registry Moderately Easy AE = adverse event IHD = ischemic heart diseases or CHD= coronary heart diseases both terms include acute myocardial infarction EHR = electronic healthcare records [ ] = number indicating the reference including these data influence of variation in methodology. Furthermore, we characterised seven routine electronic healthcare databases from five European countries that will be used for the evaluation of the selected drug-AE associations.
The work of WP2 of PROTECT is in the front line of currently on-going large (inter-) national initiatives such as the Observational Medical Outcomes Partnership (OMOP), FDA Sentinel Initiative [29] and EU-ADR (EU-Adverse Drug Reactions) project [30]. OMOP is a public-private partnership that conducts experiments to assess value, feasibility, and utility of observational data to identify and evaluate the safety risks and potential benefits of prescription drugs [31]. Furthermore, OMOP tests approaches for creating the infrastructure for accessing and managing the required data. The FDA Sentinel initiative aims at development of a national electronic safety monitoring system in order to strengthen FDA's ability to monitor postmarketing performance of medical products and to enable FDA to access existing automated healthcare data by partnering with data holders. EU-ADR project is focussing on utilizing electronic healthcare data records and biomedical databases for the early detection of AEs. In the EU-ADR project a list of 23 events were judged as important in pharmacovigilance and three AE (acute myocardial infarction, acute liver injury, and suicidal behaviour/attempt) on this list have also been prioritised in our project. The OMOP project has also defined a list of health outcomes of interest (HOI) and drug pairs to be further investigated. As previously mentioned we included two of these pairs (DILI and antimicrobials, hip fracture and benzodiazepines) in our prioritised list of five drug-AE pairs. Although these projects have a different focus than those of WP2 of PROTECT, the overlap in prioritised AEs (and drugs) will facilitate comparisons.
The strengths of our approach include the development of a common study protocol (that includes variation in methodology e.g. different designs) for five drug-AE associations that will be studied in different databases. In addition, some of our findings will be confirmed in specific registries such as PGRx [32]. Our approach will allow us to distinguish between variation in results due to variation in methodology and those due to database differences. Analysing these discrepancies will provide guidance regarding the optimal methodology for certain safety issues and the optimal selection of appropriate data source(s). The experience obtained in the PROTECT database network will improve the possibilities for multinational database studies for various safety issues, including the investigation of rare serious AE. Finally, other research activities of WP2 of PROTECT will further improve the methodological guidance on pharmacoepidemiological studies. These include an evaluation and improvement of methods to control for confounding such as propensity scores and instrumental variables in simulation studies, and drug utilisation research.
A limitation of our approach may be the scope of the drug-AE pairs and selected healthcare databases. Our findings may not be extendable to other safety issues or other databases that we do not study. However, our selection of drug-AE pairs includes common drug safety issues presenting different methodological challenges. The different types of databases (GP, claims, and registries) owned by PROTECT partners, also make extrapolation of our findings to wider ranges of data sources possible. Furthermore, our findings will be validated by testing different drug-AE pairs in the same databases and confirmation of drug-AE association in specific registries that include more detailed information on outcomes and potential confounding factors.
In conclusion, WP2 of PROTECT will assess the influence of methodological parameters on the association between selected AEs and drug class of interest. The selected AEs include resulted in (major) regulatory decisions such as drug withdrawal or SmPC changes or allow the investigation of a broad range of relevant methodological issues. The anticipated results of this project include the creation of a European database network and further development of methodological standards for the conduct of (multi-) national PE studies. Methodological standards will be included when appropriate in the EMA-based ENCePP guidance on methodological standards. Increasing methodological standards and registration of study protocols may decrease discrepancies in results from these studies, increase transparancy and thereby increase the usefulness and reliability of these studies for benefit-risk assessment and decision-making of marketed drugs in Europe and beyond.

Funding
The research leading to these results was conducted as part of the PROTECT consortium (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, www.imi-protect.eu) which is a public-private partnership coordinated by the European Medicines Agency.
The PROTECT project has received support from the Innovative Medicine Initiative Joint Undertaking (www.imi. europa.eu) under Grant Agreement n° 115004, resources of which are composed of financial contribution from the European Union's Seventh Framework Programme (FP7/ 2007-2013) and EFPIA companies' in kind contribution. In the context of the IMI Joint Undertaking (IMI JU), the Department of Pharmacoepidemiology, Utrecht University, also received a direct financial contribution from Pfizer. The views expressed are those of the authors only and not of their respective institution or company.