How to Analyze Performance Data
A common feature of a well-run quality improvement project is
that the team plots several measures related to the aim over time. For example,
the horizontal (x) axis can represent time in days,
weeks, or months, and the vertical (y) axis can represent
the performance measure. The analyst annotates the plots with the timing of the
interventions or other significant events, such as the use of a new drug or the beginning
of a new rotation of residents. Although this approach works well for most types
of data, it may be difficult with ordinal data.
Time series consisting of data from small samples collected continually
over time is the standard for quality improvement work for several reasons. They
are less costly to implement than measurements associated with the large, one-time
data collection of controlled trials. Change is inherently a temporal phenomenon.
Study of the patterns of variation in the measures over time ascertains the
performance of the system and its stability. The patterns in time series data contain
important information that other methods that rely on averages or other summary statistics
can hide (e.g., comparing results across the arms of a randomized, controlled trial).
Uncontrolled observational measurement holds higher risk for unrecognized bias and
incorrect conclusions about cause and effect than more rigorous designs. However,
unlike these designs in which measurement ends when the researchers leave, in a health
care delivery setting, the time series measurement continues to determine whether
the perceived improvement is sustainable. Incorrect claims of improvement resulting
from faulty inference become apparent in time.
Periodic measurement can be built into daily work and allows the
clinical team to continually evaluate performance. Any decline in performance is
evident, and a clinical team can quickly respond. If properly planned and coordinated,
such local outcomes tracking, when combined with descriptions of evolved systems
of care, can aggregate learning at the level of clinical teams, clinics, hospitals,
regions, care delivery systems, states, and the nation. This widespread learning
and improvement is consistent with the "call for action" issued in 1997 by Lundberg
and Wennberg.[72]
There is often uncertainty regarding the unit of analysis in the
control chart. Should the unit of time be days, weeks, months, or years? The unit
of analysis should be determined to optimize learning and is influenced by sample
size requirements[73]
and by the frequency with
which the investigator plans to give feedback to caregivers and launch additional
interventions. In general, feedback regarding process measures can be provided more
frequently than for outcome measures.
To determine whether improvement has occurred, baseline data should
be obtained before implementing improvement efforts. There must be a system to collect
this data. To pilot test the data collection system and understand baseline performance,
improvement teams should collect and graph some baseline data before implementing
an intervention. The collection of baseline data should not unduly delay the tests
of changes. In a health care delivery system, it is rare that a particular change
will have a substantial effect in a short period. Soon after the baseline data collection
has begun, small-scale tests of change can be planned and initiated concurrently.
