Inter-rater agreement of the triagesystem RETTS-HEV

OBJECTIVE
The purpose of this study was to evaluate the inter-rater agreement among nurses using the triage system RETTS-HEV (rapid emergency triage and treatment system - hospital unit west) in a Danish emergency department (ED).


BACKGROUND
The use of triage systems in Denmark has been implemented recently together with structural changes in hospital organization. Testing and evaluation is therefore needed. The RETTS-HEV is a five-scale triage system being used in the ED of Herning, Denmark, since May 2010. The ED is semilarge, with 29 000 annual visits.


MATERIALS AND METHODS
Consecutive patients presenting to the ED were assessed by both a duty and a study nurse using RETTS-HEV. Nurses did not receive training before the study. In all, 146 patients were enroled and a blinded, paired and simultaneous triage was conducted independently to evaluate inter-rater agreement using Fleiss κ.


RESULTS
A total of 155 patients were triaged over a 10-day period and complete data were available for 146 patients. We found the overall agreement to be good [Fleiss κ 0.60 (95% confidence interval 0.48; 0.72)]. The κ estimate was higher for the group of patients who required immediate attention [0.83 (95% confidence interval 0.18; 1.47)].


CONCLUSION
The study found good inter-rater agreement between two independent observers not receiving any new triage training before the study.


Introduction
Triage is a tool for categorizing patients and assessing the severity and urgency of their condition. The main purpose of triage is to determine how fast the patient needs medical evaluation and treatment. Several countries have designed different models of triage systems that are currently being used worldwide.
Emergency departments (EDs) in Denmark handle around 1 000 000 patients annually [1]. Within the last couple of years, there have been, and still are, ongoing structural changes in the Danish healthcare system. EDs are becoming fewer in number but larger and more centralized and are consequently expected to handle an increasing number of patients as well as provide longer lasting treatments within the ED [2]. Limited resources, ED overcrowding and introduction of quality standards require prioritization of patients [3].
In Denmark, 15 (71%) of 21 EDs use a triage system but only eight EDs use it systematically on all patients. Seven EDs use an established system adapted to Danish healthcare units (ADAPT, ESI and MTS) and eight EDs use a locally developed system [1].
The RETTS-HEV (rapid emergency triage and treatment system -hospital unit west) is developed from the Swedish RETTS.
The Danish RETTS-HEV is similar to the Swedish system, except for a few changes in the emergency symptoms and signs (ESS) cards according to Danish national guidelines [4].
RETTS is the primary system currently used in most Swedish hospitals (http://www.predicare.se). The RETTS-HEV has been used at the ED in Herning since May 2010. The idea behind RETTS is a triage system that combines vital signs, chief complaints, symptoms and signs. The system has been validated internally according to its ability to predict in-hospital mortality at the Sahlgrenska Universitetssjukhus [5,6]. Both studies conclude that RETTS is a sensitive tool to find patients in need of immediate medical attention [5].
The objective of our study was, in an unselected population of ED patients, to prospectively determine the reliability of the RETTS-HEV, that is, 'Will two ED nurses who triage the same patient independent of each other categorize the patient to the same level of triage?'

Materials and methods
The study was designed as an observational study of two groups of nurses assigning triage levels to patients to assess the reliability of RETTS-HEV.
RETTS-HEV consists of an algorithm for vital signs and 45 ESS algorithms. In Table 1, the algorithm for vital signs is presented and Fig. 1 shows an example of an ESS algorithm.
There are five categories: (1) red (immediate); (2) orange (can wait 10 min); (3) yellow (can wait 60 min); (4) green (can wait 120 min) and (5) blue (can wait 240 min and can be treated by a trained ED nurse). Triage category 5 patients include only smaller orthopaedic injuries referred to the ED from a general physician; they are classified as category 5 on arrival and do not undergo the full process of triage.
The patient's vital signs are typed into a computer program that automatically estimates a triage level. Red patients are announced by paramedics who bring patients to the ED and are evaluated and treated immediately by an emergency team. The triage is conducted and typed into the chart when the patient is in a stable condition.
The final triage level can be altered depending on the chief complaint. The nurse can upgrade the level of triage on the basis of the ESS algorithm but can never downgrade a patient from the level determined by vital signs. There are 45 flowcharts describing different ESS algorithms. More than one can be chosen in the case of several complaints.
The triage is a two-step simultaneous process: first, a registration of vital signs in the computer program related to the patient's electronic chart and second choosing one or more ESS cards. The highest triage level of the two measures equals the final triage level. The triage is normally performed by a single nurse and the process is estimated to last no longer than 10 min [4].
The study was carried out in the ED at the Regional Hospital of Herning, which annually receives around 29 000 visits, of which B45% (13 000) are admitted. Herning is a provincial town in Denmark and the hospital services a population of B300 000 people. The ED receives all patients, except cardiac and paediatric medical patients.
Data were collected over a period of 10 weekdays in 7 h shifts between 8 a.m. and 3 p.m. in October 2012. One of 10 study nurses affiliated with this project and one of 20 ED staff nurses simultaneously, but blinded of each other, performed triage of patients in the ED.
The study nurses were also part of the regular ED staff; only they signed in voluntarily to take part in the study. None of the nurses received new RETTS-HEV training before the study.
Patients were selected without preference but limited to the available time of the study nurse who participated in the process of triage evaluating as many patients as possible during that shift. The study nurse moved from one patient to another by selecting the next incoming patient.
Only patients in triage categories 1-4/red-green were included, leaving out category 5/blue/fast-track patients.
The ED staff nurse conducted the interview with the patient. The study nurses' triage level was based solely on observation of the staff nurses' interview and observation of vital signs from the monitor. Temperature and respiratory frequency obtained by the staff nurse was shared verbally with the study nurse. Afterwards, both nurses chose the ESS algorithms they found most suitable on the basis of the information provided by the patient during the interview. The nurses chose the ESS algorithm independent of each other.
Both levels of triage were registered in the patient's electronic chart.
Furthermore, a questionnaire on job experience and RETTS training was handed out to all the participating nurses.  [7].
The value 1.0 accounts for perfect agreement, whereas 0 equals chance agreement. Values are interpreted as follows: k value less than 0.40 is considered as poor agreement; 0.40-0.75 as good agreement and greater than 0.75 as excellent agreement [8].
The local Ethics Committee found the study to be exempt from formal ethics review as it was considered a quality assurance study not interfering with or altering standard patient care.

Results
The sample size of reliability studies should exceed 50; we included 146 patients [9].
In the 10-day period, 155 patients underwent simultaneous study and staff nurse triage. Nine patients were excluded because the staff nurse did not register them in the electronic chart, leaving 146 double-evaluated patients with 292 processes of triage.  Twenty staff and 10 study nurses participated in the datagathering process. The nurses' had an average working experience of 17.8 years (4-38 years) and an average working experience in the ED of 10.6 years (2 months to 36 years). Twenty of the participating nurses (66%) had received 1 h of formalized training using RETTS-HEV when the system was implemented in 2010 and 10 learned through colleagues.
The interobserver agreement in the triage on the basis of vital signs and ESS, respectively, as well as the final triage score are shown in Table 2.
In the final triage, the nurses agreed on exact triage level in 108 cases (74%) and were within one adjacent triage level in all cases (100%). Among the 38 patients with disagreement, 24 (63%) were between yellow and green, 12 (32%) were between yellow and orange and two (5%) were within red and orange.
In all three assessments, the overall k value and the 95% confidence interval were within the interval for 'good agreement'.
A clear tendency towards high agreement (0.83) was found in patients who needed immediate treatment (red), although the confidence interval is broad.
On comparing the level of triage on the basis of vital parameters with the final level many patients were upgraded after the selection of an ESS algorithm. The number of patients in the red triage level increased from 10 to 12, patients in orange increased from 11 to 50 and patients in yellow increased from 62 to 142, whereas patients in green decreased from 193 to 88.
In 123 patients, the same ESS was chosen at least once and in seven of the remaining 23 patients, disagreement was because of the selection of different ESS algorithms. In the last 16 cases, the ESS algorithm was only available from one of the nurses.

Discussion
Our study showed an overall triage agreement of 0.60 between two ED nurses. Agreement was 0.65 after obtaining vital signs and 0.60 after choosing an ESS algorithm. All results were within the category for good agreement. Agreement was higher within higher levels of triage (red) 0.83.
Other similar studies have found higher values of agreement [10][11][12] using weighted k [13,14], whereas we obtained a lower value, as expected, using Fleiss k [7]. Also, nurses in our study did not receive new triage training before data collection to avoid overestimation of agreement [15].
Our study was designed as recommended in a Swedish review from 2011 with real-life patient cases [16] to ensure that the data were as realistic as possible. Few other studies we reviewed applied this approach to determine reliability [10][11][12]17], and most of the other studies on reliability were carried out using paper case scenarios [10,15,[18][19][20][21][22][23][24][25].
For category 1 (red) patients, in two out of seven cases, there was disagreement between category 1 and 2 ( Table 3). One patient with disagreement had two different ESS algorithms selected, leading to disagreement in the final triage category. The other patient had the same vital signs among both nurses but different triage categories on the basis of vital signs. As vital signs were the same among both nurses, we believe that one nurse made a mistake while typing vital signs into the system.
The study nurse did not perform an independent assessment. She obtained her information partly by  Numbers in bold highlights perfect agreement between the two groups of nurses. RETTS-HEV triage scale: red = immediately; orange = urgent; yellow = semiurgent; green = nonurgent. RETTS-HEV, rapid emergency triage and treatment system -hospital unit west.
overhearing the staff nurse's interview. The advantage of this method is that the patient only provides one set of information, whereas conducting interviews independently ensures complete blinding but has the disadvantage of potentially obtaining different information from the patient.
Testing of RETTS-HEV in other hospital units is needed to determine the external validity of our results and should be considered for further research. It is important that triage systems are tested in their own environment even though evidence exists from different settings. Patients in Danish EDs may not be comparable with those attending EDs in other countries because of differences in emergency care provided in primary healthcare. For the future, we recommend a more standardized training of nurses to ensure a more uniform conduction of the triage process [26] and further evaluation of RETTS-HEV is needed; currently, validation of the system is ongoing.

Conclusion
This was the first study carried out in Denmark to evaluate inter-rater agreement in a triage system and the first to evaluate agreement for RETTS-HEV by conducting simultaneous and blinded triage of real-life patients We have shown that the RETTS-HEV has an overall good inter-rater agreement (k = 0.60). We found high agreement (k = 0.83) between nurses evaluating patients who needed immediate treatment.