Design and internal validation of an obstetric early warning score: secondary analysis of the Intensive Care National Audit and Research Centre Case Mix Programme database
You can respond to this article at http://www.anaesthesiacorrespondence.com
Presented in part at the Association of Anaesthetists of Great Britain and Ireland Annual Congress, Harrogate; September 2010.
This article is accompanied by an Editorial by McGlennan et al., p. 338 of this issue.
Summary
We designed and internally validated an aggregate weighted early warning scoring system specific to the obstetric population that has the potential for use in the ward environment. Direct obstetric admissions from the Intensive Care National Audit and Research Centre's Case Mix Programme Database were randomly allocated to model development (n = 2240) or validation (n = 2200) sets. Physiological variables collected during the first 24 h of critical care admission were analysed. Logistic regression analysis for mortality in the model development set was initially used to create a statistically based early warning score. The statistical score was then modified to create a clinically acceptable early warning score. Important features of this clinical obstetric early warning score are that the variables are weighted according to their statistical importance, a surrogate for the FIO2/PaO2 relationship is included, conscious level is assessed using a simplified alert/not alert variable, and the score, trigger thresholds and response are consistent with the new non-obstetric National Early Warning Score system. The statistical and clinical early warning scores were internally validated using the validation set. The area under the receiver operating characteristic curve was 0.995 (95% CI 0.992–0.998) for the statistical score and 0.957 (95% CI 0.923–0.991) for the clinical score. Pre-existing empirically designed early warning scores were also validated in the same way for comparison. The area under the receiver operating characteristic curve was 0.955 (95% CI 0.922–0.988) for Swanton et al.'s Modified Early Obstetric Warning System, 0.937 (95% CI 0.884–0.991) for the obstetric early warning score suggested in the 2003–2005 Report on Confidential Enquiries into Maternal Deaths in the UK, and 0.973 (95% CI 0.957–0.989) for the non-obstetric National Early Warning Score. This highlights that the new clinical obstetric early warning score has an excellent ability to discriminate survivors from non-survivors in this critical care data set. Further work is needed to validate our new clinical early warning score externally in the obstetric ward environment.
The 2003–2005 Report on Confidential Enquiries into Maternal Deaths in the UK recommended the use of a modified early warning score (EWS) to assist in identifying the obstetric patient at risk of deterioration 1. The rationale is that in many cases, early warning signs of impending maternal collapse go unrecognised. This recommendation was reiterated in the subsequent triennial maternal mortality report 2.
The Royal College of Physicians has recently recommended the use of a standardised National Early Warning Score (NEWS) throughout the National Health Service (NHS) for use in all adult inpatients 3. The report emphasises the importance of standardising physiological monitoring charts and plans across the NHS to facilitate education and training, and ultimately improve patient safety. However, the authors state that NEWS should not be used in women who are pregnant because of a modified physiological response to acute illness in this group.
At present, there is no national or international ‘gold standard’ obstetric early warning score (EWS) in use. An example of an empirically designed obstetric EWS was, however, included in the 2003–2005 report 1. An adaptation of this obstetric EWS has recently been prospectively validated against morbidity in 676 obstetric admissions to a single unit 4. Swanton et al. have also empirically designed a modified early obstetric warning score (MEOWS) following a postal survey of UK maternity units in 2007 5. Their survey found that 30 (19%) maternity units were regularly using an EWS in obstetric patients yet only nine (6%) were using a system modified for parturients. These scoring systems were not consistent in the physiological variables used, the layout of the charts or the trigger thresholds. Considering the principles behind the introduction of a standard national EWS for adult non-obstetric patients 3, it would seem appropriate that a standard national EWS is also developed for the obstetric population.
By using a large dataset of obstetric admissions to critical care units, we aimed to design and then internally validate a statistically based aggregate weighted EWS specific to the obstetric population. This was to be achieved using data collected from obstetric patients after their admission to critical care units participating in the Intensive Care National Audit and Research Centre (ICNARC) Case Mix Programme (CMP).
Methods
A study proposal was submitted and accepted by ICNARC (January 2009). The South Manchester Research Ethics Committee considered the study proposal and stated that as it was a retrospective review of routinely collected audit data and given the section 81 approval of the ICNARC dataset, ethical approval was not required under NHS research governance arrangements (August 2009). Confidentiality agreements were signed (February 2009) and ICNARC provided the requested data (February 2009).
The CMP, which is co-ordinated by ICNARC, is a national audit of patient outcomes from adult general critical care units in England, Wales and Northern Ireland. The physiological data collected by the CMP relates to the most abnormal measurements within the first 24 h following critical care admission. Data were extracted from 604 846 admissions to 181 critical care units in the CMP, covering the period from December 1995 to September 2008 inclusive. The details of data collection and validation have been previously published 6.
Relevant cases were provided by ICNARC using a methodology previously described by Harrison et al. 7. All female admissions, aged between 16 and 50 years inclusive, were selected from the CMP database. Obstetric admissions were identified using the four admission fields: ‘primary reason for admission’, ‘secondary reason for admission’, and two ‘other condition relevant to the admission’ fields. These fields are coded using the ICNARC coding method 8. The free text field of the database was also searched to identify additional cases. The obstetric admissions were divided into direct and indirect. Direct obstetric admissions included all cases where the ‘primary reason for admission field’ or ‘secondary reason for admission’ contained an obstetric condition (Table S1 (online only)). Indirect admissions were cases that did not meet the above criteria for direct obstetric admission and met any of the following three criteria: the ‘other condition relevant to the admission’ fields contained an obstetric condition; the entry in any of the four admission fields was a partially completed code with the site tier recorded as ‘ovary, fallopian, uterus or genitalia (obstetric)’; or the patient was identified as being pregnant or having recently been pregnant by searching the text field for a predefined list of pregnancy-related search terms. The remaining cases (women aged 16–50 who were not identified as a direct or indirect obstetric admission) made up the non-obstetric admissions group.
The definitions used for direct and indirect obstetric admissions are distinct from the definitions of direct and indirect maternal deaths used by the Confidential Enquiries into Maternal Deaths. The latter defines direct maternal deaths as those resulting from conditions or complications or their management that are unique to pregnancy, occurring during the antenatal, intrapartum or postpartum periods. The indirect maternal deaths are those resulting from previously existing disease or disease that develops during pregnancy not as the result of direct obstetric causes, but that is aggravated by physiological effects of pregnancy 1.
Case mix (age, illness severity, surgical status), outcome (unit and hospital survival) and activity (unit and hospital length of stay) data were extracted. Illness severity was measured using the Acute Physiology and Chronic Health Evaluation II (APACHE II) score 9. Surgical status was classified as either non-surgical, elective surgery or emergency surgery 6. Length of stay in the CMP unit was calculated in hours from the dates and times of admission and discharge. Hospital length of stay was calculated in days from the dates of hospital admission until ultimate discharge.
The dataset of female critical care admissions was randomly allocated by computer-generated numbers in a 1:1 ratio into two sets (Set 1 and Set 2). The direct obstetric admissions from Set 1 formed the model development set, which was used to identify variables associated with unit mortality. The validation set, formed by the direct obstetric admissions within Set 2, was reserved for internal validation of the developed model (Fig. 1).
All physiological variables that could be easily recorded within the ward environment were selected from the dataset: highest and lowest heart rate; highest and lowest systolic blood pressure (BP); highest and lowest diastolic BP, highest and lowest respiratory rate separated into ventilated and non-ventilated; highest and lowest temperature separated into central and non-central; lowest total Glasgow Coma Score (GCS); urine output; lowest PaO2; and FIO2 associated with the lowest PaO2. The CMP database, and consequently the model development dataset, records the most abnormal physiological measurements within the first 24 h following critical care admission.
The CMP database provides the lowest total GCS score (or pre-sedation GCS score for admissions sedated or paralysed for the entire first 24 h period on the CMP unit) for the first 24 h of the patient's admission to the critical care unit. As GCS is not commonly measured on general hospital wards, conversion of GCS values into a simplified assessment of conscious level AVPU (alert, responds to voice, responds to pain, unconscious) was necessary. The relationship between GCS and AVPU has previously been compared in the literature 10, 11. We considered every possible GCS combination and calculated the corresponding AVPU score. Using the previously published findings in combination with our own information, the GCS ranges for each AVPU category were assigned as follows: alert – GCS 15, responds to voice – GCS 10–14, responds to pain – GCS 7–9, unresponsive – GCS 3–6. The GCS data were also further simplified into ‘alert’ (GCS 15) or ‘not alert’ (GCS 3–14).
In the CMP database, urine output is recorded as the total volume (ml) passed in the first 24 h after admission. The total volume of urine was divided by 24 to provide a value for the mean hourly urine output.
Within the model development set, each variable was screened for missing values (Table 1). In the CMP database, temperature is recorded as highest core temperature, highest non-core temperature, lowest core temperature and lowest non-core temperature. New variables ‘maximum temperature’ (highest value from core and non-core highest temperature) and ‘minimum temperature’ (lowest value from core and non-core lowest temperature) were created to reduce the number of missing values for the temperature variable. Respiratory rate is recorded as the highest and lowest during controlled ventilation (highest ventilated and lowest ventilated, respectively) and during spontaneous ventilation (highest non-ventilated and lowest non-ventilated, respectively). Missing data are unavoidable in the respiratory rate fields due to the initiation of mechanical ventilation before admission to the CMP unit (non-ventilated data missing) or mechanical ventilation not being required within the first 24 h of admission (ventilated data missing). Missing values were imputed to the group-specific (unit survivors/non-survivors) median value.
| Unit non-survivors (n = 39) | Missing data | Unit survivors (n = 2201) | Missing data | |
|---|---|---|---|---|
| APACHE II score | 21.0 (16.0–26.0 [10.0–32.0]) | 10 (25.6%) | 10.0 (7.0–13.0 [0.0–41.0]) | 117 (5.3%) |
| Lowest heart rate; beats.min−1 | 90.0 (70.0–110.0 [30.0–160.0]) | 0 (0.0%) | 75.0 (65.0–86.0 [10.0–140.0]) | 23 (1.0%) |
| Highest heart rate; beats.min−1 | 136.5 (117.0–150.0 [62.0–200.0]) | 1 (2.6%) | 112.0 (100.0–128.0 [62.0–220.0]) | 26 (1.2%) |
| Lowest systolic BP; mmHg | 85.5 (70.0–98.0 [28.0–125.0]) | 1 (2.6%) | 100.0 (90.0–115.0 [19.0–178.0]) | 27 (1.2%) |
| Lowest diastolic BP (paired value for lowest SBP); mmHg | 52.0 (39.0–60.0 [8.0–80.0]) | 1 (2.6%) | 59.0 (50.0–69.0 [0.0–114.0]) | 28 (1.3%) |
| Highest systolic BP; mmHg | 160.0 (132.0–180.0 [85.0–225.0]) | 2 (5.1%) | 151.0 (136.0–170.0 [87.0–258.0]) | 29 (1.3%) |
| Highest diastolic BP (paired value for highest SBP); mmHg | 88.0 (70.0–103.0 [45.0–140.0]) | 2 (5.1%) | 83.0 (71.0–95.0 [20.0–150.0]) | 29 (1.3%) |
| Lowest ventilated respiratory rate; breaths.min−1 | 12.0 (10.0–16.0 [5.0–22.0]) | 4 (10.3%) | 12.0 (10.0–12.0 [2.0–100.0]) | 1026 (46.6%) |
| Highest ventilated respiratory rate; breaths.min−1 | 17.5 14.0–23.0 [12.0–39.0]) | 11 (28.2%) | 15.0 (13.0–19.0 [10.0–102.0]) | 1217 (55.3%) |
| Lowest non-ventilated respiratory rate; breaths.min−1 | 20.0 (11.0–32.0 [8.0–42.0]) | 28 (71.8%) | 12.0 (10.0–15.0 [0.0–42.0]) | 295 (13.4%) |
| Highest non-ventilated respiratory rate; breaths.min−1 | 34.5 (29.0–39.0 [10.0–52.0]) | 33 (84.6%) | 22.0 (19.0–26.0 [5.0–62.0]) | 342 (15.5%) |
| Lowest core temperature; °C | 35.8 (34.3–36.0 [31.5–38.4]) | 20 (51.3%) | 36.0 (35.5–36.6 [29.7–38.5]) | 1037 (47.1%) |
| Highest core temperature; °C | 37.1 (36.3–39.0 [33.2–39.8]) | 21 (53.8%) | 37.4 (37.0–37.9 [35.3–40.9]) | 1069 (48.6%) |
| Lowest non-core temperature; °C | 35.3 (34.6–36.0 [32.0–39.2]) | 21 (53.8%) | 36.0 (35.5–36.5 [31–39.8]) | 1144 (52.0%) |
| Highest non-core temperature; °C | 36.9 (36.3–38.2 [35.0–40.0]) | 20 (51.3%) | 37.3 (36.9–37.8 [34.4–40.2]) | 1164 (52.9%) |
| Maximum temperature; °C | 37.3 (36.5–38.3 [33.2–40.0]) | 5 (12.8%) | 37.4 (37.0–37.9 [35.0–40.9]) | 71 (3.2%) |
| Minimum temperature; °C | 35.5 (34.3–36.0 [31.5–39.2]) | 4 (10.3%) | 36.0 (35.5–36.5 [29.7–38.5]) | 28 (1.3%) |
| Lowest total GCS | 3.0 (3.0–7.0 [3.0–15.0]) | 16 (41.0%) | 15.0 (15.0–15.0 [3.0–15.0]) | 480 (21.8%) |
| Urine output; ml.h−1 | 37.5 (9.5–95.9 [0.0–312.1]) | 1 (2.6%) | 88.3 (55.6–130.4 [0.0–341.0]) | 61 (2.8%) |
| Lowest PaO2; kPa | 11.45 (8.15–14.50 [3.4–36.3]) | 3 (7.7%) | 11.60 (9.80–14.00 [1.5–55.4]) | 406 (18.4%) |
| FIO2 associated with lowest PaO2 | 0.60 (0.44–1.00 [0.21–1.0]) | 3 (7.7%) | 0.30 (0.25–0.40 [0.21–1.0]) | 413 (18.8%) |
- APACHE, Acute Physiology and Chronic Health Evaluation; GCS, Glasgow Coma Score.
From the model development set data, case mix, outcome and activity were described for the direct and indirect obstetric admissions.
Within the model development set, the effect of the CMP variables on unit mortality was assessed using univariate analysis. Significant variables (p < 0.05) were then entered into a multiple logistic regression model. This method of model development has been previously described 12. The regression coefficients were used to weight each covariate, creating a statistical obstetric EWS. To improve clinical acceptability, additional physiological variables were added to the statistical EWS and a new clinical obstetric EWS developed. The ranges and weightings of each additional variable were selected on an empirical basis and the impact assessed using the area under receiver operating characteristic (ROC) curves. The ROC curves were then used to assess the ability of the new score to discriminate between survivors and non-survivors. The newly designed EWS was then applied prospectively to the validation set. The predicted outcomes were compared with actual outcomes for the individual cases in the validation set using ROC curve analysis.
We compared the statistical EWS and clinical EWS with Swanton et al.'s empirically designed MEOWS 5; the Confidential Enquiries into Maternal Deaths obstetric EWS 1; and the Royal College of Physicians’ non-obstetric NEWS 3.
All analyses were performed using Statistical Package for Social Sciences (SPSS) 19 for Mac 2010 (IBM, Somers, NY, USA) and Number Cruncher Statistical Systems (NCSS) 2007 (Hintze, J.; NCSS, LLC, Kaysville, UT, USA). Mortality rates are presented with Clopper-Pearson 95% CI and the effects of predictor variables as odds ratios with 95% CI. Statistical significance was defined for two-sided p < 0.05.
Results
From the 604 846 admissions in the CMP database, 71 108 were identified as female admissions aged between 16 and 50 years inclusive (Fig. 1). The final total of 4440 direct obstetric admissions represented 0.7% of all CMP admissions and 6.2% of all female admissions aged 16–50 years.
Table S2 (online only) summarises the case mix, outcome and activity data for the direct and indirect obstetric admissions within Set 1. The unit and hospital mortalities of the direct obstetric admissions were 1.7% (95% CI 1.2–2.4%) and 2.1% (95% CI 1.6–2.8%), respectively. These were much lower than the mortalities for the non-obstetric admissions which were 13.0% (95% CI 12.7–13.4%) and 17.5% (95% CI 17.1–17.9%) for unit and hospital mortalities, respectively. The hospital mortality for all obstetric admissions (direct and indirect) in Set 1 was 2.8% (95% CI 2.2–3.5%).
Table S3 (online only) shows the numbers of admissions with each specific ICNARC coding method obstetric condition and the mortality associated with each condition. Peripartum or postpartum haemorrhage was the most common condition (47% of direct obstetric admissions), but with the lowest hospital mortality (0.8%).
Table 1 shows the missing values associated with each variable. Values for the temperature variable were missing in < 13% in the non-survivors subgroup and < 4% for the survivor subgroup. Values of respiratory rate for non-ventilated non-survivors were missing in 85% and ventilated survivors in 55%. Lowest total GCS variable values were missing in 41% of non-survivors and 22% of survivors. All other variables of interest had missing values of < 8% (non-survivors) and < 19% (survivors).
Table S4 (online only) highlights the factors associated with unit mortality as identified by univariate analysis. Although the highest and lowest heart rate are both significant, the direction of change in the variables suggests that it is an increase in heart rate that is associated with worsening mortality. This association is also seen with highest and lowest RR. The lowest systolic and diastolic BP are paired correlated values necessitating the inclusion of only one of the variables.
The remaining significant variables (lowest total GCS, highest heart rate, lowest systolic BP, minimum temperature, highest non-ventilated respiratory rate, hourly urine output and FIO2 associated with the lowest PaO2) were entered into a multiple logistic regression model. We then repeated the multiple logistic regression modelling substituting GCS first for ‘AVPU’ and then for ‘alert/not alert’ (Table 2). The regression equation coefficients were processed to factor in the size of change in the physiological variable. For example, SBP coefficient: 0.018 represents a 1 mmHg change in SBP therefore 0.36 (0.018 × 20) represents a 20 mmHg change (Table 2). The coefficients were then ‘rounded’ and used to weight each covariate and create a statistically based obstetric EWS (Table 3).
| Logistic regression coefficient | Standard error | Wald test statistic | p value | Estimated odds ratio (95% CI) | Covariate weighting derived from regression coefficient | Weightinga | |
|---|---|---|---|---|---|---|---|
| Not alert | 5.06 | 0.831 | 37.1 | < 0.001 | 157 (30.8–800) | 5.06 from alert to not alert | 5 |
| Highest heart rate; beats.min−1 | 0.026 | 0.011 | 6.00 | 0.014 | 1.03 (1.01–1.05) | 0.52 per 20 beats.min−1 change | 0.5 |
| Lowest systolic BP; mmHg | −0.018 | 0.012 | 2.34 | 0.126 | 0.982 (0.959–1.01) | −0.36 per 20 mmHg change | 0.4 |
| Urine output; ml.hr−1 | −0.008 | 0.004 | 5.20 | 0.023 | 0.992 (0.985–0.999) | −0.24 per 30 ml.h−1 change | 0.2 |
| FIO2 associated with lowest PaO2 | 3.80 | 0.928 | 16.8 | < 0.001 | 44.9 (7.28–276) | 0.38 per 10% change in FIO2 | 0.4 |
| Minimum temperature; °C | −0.420 | 0.177 | 5.62 | 0.018 | 0.657 (0.464–0.930) | −0.21 per 0.5 °C change | 0.2 |
| Highest non-ventilated respiratory rate; breaths.min−1 | 0.182 | 0.030 | 36.5 | < 0.001 | 1.2 (1.13–1.27) | 0.91 per 5 breaths.min−1 change | 0.9 |
- a Weighting for score created by rounding to one significant digit.
One of the significant variables was ‘FIO2 associated with lowest PaO2 on arterial blood gas sampling’. From the data in Table S4, it can be seen that the median lowest PaO2 values were similar for the non-survivors and survivors (11.45 kPa and 11.60 kPa, respectively), yet the median FIO2 value was noticeably higher for the non-survivors compared with the survivors (0.60 vs 0.30). The corresponding SpO2 value for a PaO2 equal to 11.5 kPa is 96% 13. As ‘FIO2 associated with lowest PaO2 on arterial blood gas sampling’ is not routinely measured in the ward environment, a surrogate variable of ‘FIO2 to maintain SpO2 ≥ 96%’ was identified (Table 3).
The statistically relevant variables from the logistic regression analysis and additional variables needed to meet clinicians’ expectations were incorporated to create a clinical EWS specific to the obstetric population (Table 4).
Figure 2 shows the ROC curves using the model development set and CMP mortality for the initial statistical EWS, the clinical EWS and the clinical EWS with the RR component excluded. The area under the curve (95% CI) was 0.986 (0.973–0.999), 0.978 (0.967–0.989) and 0.958 (0.938–0.978), respectively.
Data in the validation set were prepared in the same way as for the model development set. Within the validation set, each variable was screened for missing values (Table 5). Missing values were imputed to the group specific median value. Unfortunately this was not possible for the non-ventilated respiratory rate data. As a quirk of the randomisation process, 31 out of 32 (96.9%) values for non-ventilated respiratory rate were missing within the non-survivor subgroup. Non-ventilated respiratory rate missing values were therefore imputed to the model development set median values rather than the validation set values.
| Unit non-survivors (n = 32) | Missing data | Unit survivors (n = 2168) | Missing data | |
|---|---|---|---|---|
| APACHE II score | 17.0 (15.0–27.0 [10.0–40.0]) | 8 (25.0%) | 10.0 (7.0–14.0 [0.0–37.0]) | 111 (5.1%) |
| Lowest heart rate; beats.min−1 | 88.0 (60.0–100.0 [26.0–140.0]) | 3 (9.4%) | 75.0 (67.0–87.0 [38.0–143.0]) | 42 (1.9%) |
| Highest heart rate; beats.min−1 | 130.0 (119.0–148.5 [88.0–168.0]) | 4 (12.5%) | 113.0 (100.0–128.0 [61.0–208.0]) | 49 (2.3%) |
| Lowest systolic BP; mmHg | 75.0 (60.5–91.0 [32.0–130.0]) | 4 (12.5%) | 101.0 (90.0–117.0 [34.0–200.0]) | 43 (2.0%) |
| Lowest diastolic BP (paired value for lowest SBP); mmHg | 47.0 (34.5–54.5 [20.0–85.0]) | 4 (12.5%) | 60.0 (50.0–70.0 [0.0–125.0]) | 43 (2.0%) |
| Highest systolic BP; mmHg | 145.5 (128.5–169.5 [61.0–310.0]) | 4 (12.5%) | 150.0 (135.0–170.0 [85.0–260.0]) | 48 (2.2%) |
| Highest diastolic BP (paired value for highest SBP); mmHg | 82.5 (66.5–100.5 [38.0–128.0]) | 4 (12.5%) | 82.0 (70.5–95.0 [20.0–155.0]) | 48 (2.2%) |
| Lowest ventilated respiratory rate; breaths.min−1 | 12.0 (12.0–14.5 [9.0–20.0]) | 4 (12.5%) | 12.0 (10.0–12.0 [2.0–85.0]) | 1048 (48.3%) |
| Highest ventilated respiratory rate; breaths.min−1 | 16.0 (14.0–20.0 [12.0–22.0]) | 11 (34.4%) | 16.0 (13.0–19.0 [8.0–100.0]) | 1238 (57.1%) |
| Lowest non-ventilated respiratory rate; breaths.min−1 | 20.0 (20.0–20.0 [20.0–20.0]) | 31 (96.9%) | 12.0 (10.0–16.0 [0.0–48]) | 316 (14.6%) |
| Highest non-ventilated respiratory rate; breaths.min−1 | 24.0 (24.0–24.0 [24.0–24.0]) | 31 (96.9%) | 22.0 (19.0–27.0 [0.0–69.0]) | 373 (17.2%) |
| Lowest core temperature; °C | 34.4 (34.0–35.2 [30.2–37.4]) | 18 (56.3%) | 36.1 (35.5–36.6 [21.3–38.4]) | 1046 (48.2%) |
| Highest core temperature; °C | 37.0 (36.1–37.3 [33.0–37.9]) | 18 (56.3%) | 37.4 (37.0–37.9 [35.6–40.5]) | 1078 (49.7%) |
| Lowest non-core temperature; °C | 35.1 (34.0–36.9 [33.5–37.7]) | 18 (56.3%) | 36.0 (35.6–36.5 [30.4–38.4]) | 1122 (51.8%) |
| Highest non-core temperature; °C | 37.2 (36.9–38.4 [35.4–39.6]) | 21 (65.6%) | 37.4 (36.9–37.9 [33.8–40.0]) | 1152 (53.1%) |
| Maximum temperature; °C | 37.1 (36.4–37.5 [33.0–39.6]) | 8 (25.0%) | 37.4 (37.0–37.9 [35.7–40.5]) | 88 (4.1%) |
| Minimum temperature; °C | 34.6 (34.0–35.6 [30.2–37.7]) | 5 (15.6%) | 36.0 (35.5–36.5 [21.3–38.4]) | 46 (2.1%) |
| Lowest total GCS | 3.0 (3.0–3.0 [3.0–9.0]) | 22 (68.8%) | 15.0 (15.0–15.0 [3.0–15.0]) | 516 (23.8%) |
| Urine output; ml.h−1 | 38.9 (9.2–76.4 [0.0–185.0]) | 5 (15.6%) | 84.3 (52.1–125.0 [0.0–459.1]) | 79 (3.6%) |
| Lowest PaO2; kPa | 10.25 (7.30–12.65 [3.1–32.1]) | 4 (12.5%) | 11.70 (9.80–14.50 [2.5–65.6]) | 425 (19.6%) |
| FIO2 associated with lowest PaO2 | 0.56 (0.47–0.99 [0.3–1.0]) | 4 (12.5%) | 0.30 (0.24–0.40 [0.21–1.0) | 436 (20.1%) |
- APACHE, Acute Physiology and Chronic Health Evaluation; GCS, Glasgow Coma Score.
Table S5 (online only) compares the case mix, outcome and activity data for the model development and validation sets. The only obvious anomaly between the two sets was that hospital length of stay was one day shorter in the validation set with median values (IQR [range]) of 9 (6–14 [0–397]) and 8 (6–13 [0–431]) days (p = 0.034) in the development and validation sets, respectively.
Figure 3 shows the ROC curves of predicted outcomes for the statistical EWS, the clinical EWS and the pre-existing EWS to actual outcomes for the individual cases in the validation set. The areas under the ROC curves for the statistical EWS and clinical EWS are 0.995 (95% CI 0.992–0.998) and 0.957 (95% CI 0.923–0.991), respectively. The area under the ROC curves for the pre-existing EWS are also shown: Swanton's empirically designed MEOWS 0.955 (95% CI 0.922–0.988) 5; the obstetric EWS provided as an example in the 2003–2005 Report on Confidential Enquiries into Maternal Deaths 0.937 (95% CI 0.884–0.991) 1; and, although not intended for obstetric patients, the Royal College of Physicians NEWS 0.973 (95% CI 0.957–0.989) 3. In addition, Fig. 3 shows the area under the ROC curve for the clinical obstetric EWS with the respiratory rate component excluded to be 0.936 (95% CI 0.891–0.981).
Discussion
The aim of this study was to design and then internally validate an aggregate weighted scoring system specific to the obstetric population which has the potential for use in the ward environment. We achieved this using a large database of critically ill obstetric patients.
The mortality rate for all obstetric admissions in this study of 2.8% is consistent with previously published UK data of 3.1% from a smaller ICNARC data set 7 and 3.3% from the South West Thames database 14. This study found that 0.9% of UK critical care admissions were related to pregnancy, in keeping with the ICNARC dataset 7, but lower than the South West Thames database 14. The South West Thames data from 14 critical care units identified 1.8% of admissions (210 out of 11 385) as being due to obstetric causes. Harrison et al., commenting on this difference, highlighted that ‘the responsible consultant’ identified a quarter of the obstetric admissions in the South West Thames database 7. This information is not available in the CMP database.
A new obstetric EWS has been developed. Logistic regression analysis was initially used to appropriately weight the statistically relevant variables and create a purely statistical EWS. The area under the ROC curve for the statistical EWS in the validation set (0.995 (95% CI 0.992–0.998)) is extremely high, indicating the statistical score's excellent ability to discriminate between survivors and non-survivors. However, the statistical score is complex, does not fulfil minimum monitoring criteria 15 and does not meet the expectations of the clinicians that would use the score 5. It was therefore necessary to add non-significant variables to the statistical EWS to produce the clinical EWS.
When designing an EWS, it is important that the score meets with the expectations of the clinicians who will be implementing the score. Swanton et al. reported that diastolic BP was included in all nine of the obstetric specific EWS that they reviewed 5, and from this we assume that the majority of clinicians would expect diastolic BP to form part of an obstetric specific EWS. Diastolic BP is also an important variable in the screening for and diagnosis of pre-eclampsia. Hypertension (systolic and diastolic) in pregnancy has been the subject of a recent National Institute for Health and Clinical Excellence (NICE) clinical guideline 15 and the management of hypertension, in particular systolic hypertension, forms one of the top ten recommendations in the latest Confidential Enquiries into Maternal Deaths report 2. High systolic BP and diastolic BP, although not statistically significant variables, have therefore been incorporated into the new clinical score. The systolic and diastolic BP ranges were initially assigned using the definitions for mild (140–149/90–99 mmHg), moderate (150–159/100–109 mmHg) and severe (≥ 160/≥ 110 mmHg) hypertension as outlined in the NICE clinical guideline 15. Highest temperature, another non-significant variable, was also added to the new clinical score to meet clinicians’ expectations as sepsis is highlighted by the latest Confidential Enquiries into Maternal Deaths report 2. The initial cut off of a temperature > 38 °C was selected from the systemic inflammatory response syndrome (SIRS) criteria 16. Finally, for completeness, low HR and low respiratory rate (again not statistically significant variables) were also added to the clinical score. The area under the ROC curve for the new clinical EWS in the validation set (0.957 (95% CI 0.923–0.991)) shows that creating a clinically acceptable score, rather than using only the statistically significant variables, has not resulted in a significant decrease in score discrimination.
A minimum standard for routine monitoring of physiological observations has been set by NICE 17 which includes heart rate, respiratory rate, systolic BP, level of consciousness, arterial oxygen saturation and temperature. The new clinical EWS variables comply with this minimum standard assuming ‘FIO2 required to maintain SpO2 ≥ 96%’ is an acceptable substitute for oxygen saturation. Arterial oxygen saturation is commonly used in early warning scores and has been shown to be a useful variable in predicting the need for ICU admission 18. A comparative cohort study by Cuthbertson et al. considered two groups of surgical high dependency patients, one group requiring and the other not requiring ICU admission. They studied the performance of individual variables in predicting the need for ICU admission and showed that oxygen saturation could differentiate between the two groups 48 h before ICU admission 18. The new clinical EWS incorporates the FIO2 required to maintain SpO2 ≥ 96%. Inclusion of this surrogate marker of the FIO2/PaO2 relationship in an EWS seems intuitively correct. When incorporating our new clinical obstetric EWS into an observation chart, the actual FIO2 and SpO2 values should also be recorded to allow tracking of a patient's oxygenation over time. Other variables included in our clinical EWS are consistent with the previously published work looking at factors predicting clinical deterioration: Goldhill et al. noted the most common abnormalities in pre-admission physiological observations from patients admitted to ICU to be tachypnoea and altered level of consciousness 19, and Kause et al. revealed hypotension and fall in GCS to be the most common antecedents to cardiac arrest, death or emergency ICU admission 20. The new clinical EWS has the additional benefit of, wherever possible, appropriately weighting these variables according to their importance.
Despite the CMP’s providing the largest available dataset pertaining to critically ill obstetric patients, the relatively low frequency of obstetric admissions to critical care units and the overall very low mortality rate for obstetric patients have resulted in significant amounts of missing data. Of concern is that the cases with missing data may be systematically different from other cases, leading to misleading results. The high percentage of missing non-ventilated respiratory rate values (patients’ lungs ventilated on admission to the CMP unit) being imputed to the median value questions the validity of describing respiratory rate as a significant variable. The non-ventilated respiratory rate variable was also absent from 96% of the deaths in the validation set, again necessitating imputation of data. The concern over respiratory rate data imputation was addressed by comparing the area under ROC curve for the clinical EWS (0.957 (95% CI, 0.923–0.991)) with the clinical EWS calculated with the respiratory rate variable excluded (0.936 (95% CI, 0.891–0.981)) in the validation dataset. The latter illustrates that the score retains its discriminatory ability even with the respiratory rate variable excluded. However, respiratory rate is often reported as one of the more sensitive parameters when trying to identify patients at risk of deterioration 20-22 and is one of the physiological observations making up the minimum standard for an EWS as set out by NICE 17. It was therefore inappropriate to consider excluding respiratory rate from the new score.
Accurate measurement of urine output requires bladder catheterisation, and for this reason NICE 17 does not include it as a core parameter for an EWS. The nine obstetric specific EWS considered by Swanton et al. did not agree on the inclusion of urine output as a variable 5. From the logistic regression analysis (Table 2), we can see that the weighting allocated to urine output was very small (25 times less than alert/not alert). Sensitivity analysis revealed that exclusion of urine output did not alter the performance of the new score. After consideration of these points, urine output was not included in the new score. Urine output is, however, useful in certain obstetric conditions and there should be provision for its measurement on an obstetric EWS chart even though it is not formally incorporated into the score.
Score simplicity is another key factor in design of an early warning score: simple scores are more reliable 23, are less prone to human calculation errors 19 and have increased reproducibility 23, 24. In addition, the Royal College of Physicians has emphasised the importance of standardising monitoring charts across the NHS 3. With these factors in mind, empirical approaches were employed to ensure that the new clinical score was as simple as possible. This included the reduction of weighting for the ‘high respiratory rate and ‘not alert’ variables from 5 to 3.
If the new clinical obstetric EWS was simply designed as a prognostic score to predict death on the CMP unit, a cut-off value of 12 gives a sensitivity of 97%, specificity of 87% and total accuracy of the score of 88%. However, the role of an EWS is to predict the patient at risk of deterioration and the patients in this dataset had all already been admitted to a critical care unit. It is therefore necessary to select a much lower threshold value to detect the patient at risk of deterioration.
The National Institute for Health and Clinical Excellence 17 has recommended that the EWS response strategy to patients identified at being at risk of clinical deterioration should be graded and consist of three levels: a low-score group that results in increased frequency of observations and alerting of the nurse in charge; a medium-score group, resulting in simultaneous urgent calls to the medical team caring for the patient and to personnel with competencies for acute illness; and a high-score group resulting in an emergency call to a team with critical care competencies and diagnostic skills. This graded response strategy (low, medium and high clinical risk) has been incorporated into the new clinical obstetric EWS system. The clinical EWS triggers for low, medium and high clinical risk are shown in Table S6 (online only). As NEWS is aimed for national adoption 3, we have selected trigger thresholds that correspond with those used in the NEWS system for ease of use and to maintain a standardised approach.
The exact nature of the response for each level (time frame, frequency of ongoing observations, seniority of attending personnel, appropriate environment for ongoing management) cannot be dictated centrally and needs to be arranged at a hospital level 17. The new clinical EWS system, as with any EWS, also has the provision for the staff caring for a patient to override the prescribed response and request a review of a patient about whom they are clinically concerned.
Colour coding of EWS charts is commonly used to provide a visual cue to aid in the identification of abnormal physiological variables 1, 3, 5, 17. The new clinical EWS system incorporates a colour scheme as well as a numeric measure of illness severity (red = 3; orange = 2; green = 1).
When implementing any EWS, obstetric-specific conditions need to be considered. Barrett and Yentis recommend that a track and trigger model for obstetrics should include underlying pathophysiology such as pre-eclampsia or eclampsia, hepatic disorders of pregnancy and potentially women with significant underlying medical diseases 25. These factors can be considered in conjunction with the new clinical EWS system.
Development of this clinical EWS system has created a score that accurately identifies the obstetric patient at risk of death. The model development set (n = 2240) comprises parameters measured after admission to critical care and those factors associated with unit mortality have been identified and incorporated into the new obstetric scoring system. However, physiological parameters measured within the first 24 h of critical care admission will not necessarily reflect the physiological changes that occur in the period before deterioration. The majority of the admissions within the model development set were following emergency surgery. This means that a degree of resuscitation and stabilisation of abnormal physiology may have occurred before arrival on the CMP unit. It is the parameters in the period before the deterioration that are of most interest when trying to design an EWS. As an estimated 0.07% of maternities 1 or 0.17% of deliveries 14 require critical care, prospective data collection of a similarly sized dataset of critical care obstetric admissions would require participation by all 181 CMP hospitals in the country over a 6-year period. Although aware of the limitations of the CMP database in this setting, we have therefore, designed our clinical obstetric EWS system using the best available dataset.
Another potential limitation results from using mortality as the outcome measure. Using mortality may result in inappropriate significance being given to variables that are actually associated with irreversible pre-terminal events rather than severity of illness. However, it cannot be determined from the dataset whether earlier admission to critical care with a lesser derangement in these physiological variables would result in improved survival. Furthermore, as this study is a secondary analysis of pre-existing data, potentially relevant data (parity, gestation or time since delivery) are missing from the dataset. Although not helpful for the current analysis, the CMP specification has been revised to include additional fields relevant to obstetric admissions.
We also assessed the performance of Swanton et al.'s empirically designed MEOWS 5, the obstetric EWS suggested in the Report on Confidential Enquiries into Maternal Deaths 1 and, although not intended for obstetric patients, the Royal College of Physicians’ NEWS 3 in our validation dataset using ROC analysis. The area under the ROC curves for these three scores, whilst outperformed by the statistical EWS (0.995 (95% CI 0.992–0.998)), are broadly similar to that of our clinical EWS (0.957 (95% CI 0.923–0.991)). Although this is a superficial comparison, it suggests some degree of construct validity to MEOWS 5, the Confidential Enquiries into Maternal Deaths obstetric EWS 1 and NEWS 3. We can therefore comment that our obstetric EWS, which has been developed using critical care data, performs in a similar manner in this analysis to the obstetric and non-obstetric EWS that are based on ward data. This similarity in performance suggests that our clinical EWS may also perform well when applied to ward-based data. However, external validation in a ward setting is needed to confirm this hypothesis.
Although our newly designed clinical obstetric EWS has a higher area under the ROC curve than the pre-existing obstetric EWS we analysed, it is not convincingly so, raising questions as to the benefit of the new score over the pre-existing scores. However, there are several important additional factors that differentiate the new clinical obstetric EWS from the pre-existing obstetric EWS. The new clinical EWS has the benefit of being developed using logistic regression and therefore the variables are weighted, where possible, according to their statistical importance. This contrasts with the pre-existing obstetric EWS, which have been empirically developed. Another significant difference between the new and pre-existing obstetric EWS is that inspired oxygen concentration is included in the new score. The new clinical EWS is not the first EWS to take inspired oxygen into consideration. Within the non-obstetric NEWS system, two points are added to the score if the patient requires supplemental oxygen 3. This means that a patient with SpO2 = 97% maintained by an oxygen mask with a reservoir bag would score the same as a patient with SpO2 = 97% requiring 1 l.min−1 of oxygen via nasal cannulae. The new EWS variable of ‘FIO2 required to maintain SpO2 ≥ 96%’, being a surrogate for the FIO2/PaO2 relationship, allows greater differentiation between patients. Another difference between the new and pre-existing obstetric EWS is the substitution of alert/not alert instead of the more complex AVPU system: this serves to help simplify the new score. In addition, the new clinical EWS system has been designed to standardise the score, triggers and graded response with the proposed National Early Warning System 3. A standardised approach to EWS systems within the NHS would hopefully facilitate education and training and ultimately improve patient safety 3.
The ICNARC CMP database, despite the limitations, is the best available data set of critically ill obstetric patients. We have used logistic regression analysis to estimate the relative weightings of various physiological variables and, in combination with clinical judgment, have developed an internally validated clinical obstetric EWS. Ongoing work will be to refine further and then to validate the new score externally in an obstetric ward population. We hope that our new statistically based obstetric EWS will prove useful in the development of a National Obstetric Early Warning Score.
Acknowledgement
We are grateful to Dr David Harrison, Intensive Care National Audit and Research Centre.
Competing interests
No external funding and no competing interests declared.
