Developing and testing quality indicators for the Thai Quality and Outcomes Framework

Abstract

Background

Primary care serves as an entry point in the Thai health care system. Whilst effective interventions are provided in the primary care setting, the quality of the services have not been measured or tracked. A number of initiatives were undertaken to improve primary health care quality including the use of financial incentives to reward adherence to performance indicators. However, there were concerns that the current quality indicators had not been developed in a systematic, participatory, and evidence-based manner. Therefore, this study aims to develop new quality indicators for use in subsequent iterations of the program.

Methods

The development of indicators follows a well-designed approach. Reviews of existing documents as well as secondary data analyses were performed and presented to key stakeholders. Disease areas were then prioritised. Recommendations from the Thai clinical practice guidelines on the prioritised areas were then used to formulate statements and templates for each indicator. Finally, the indicators developed were piloted for 3 months in 28 primary care units across the country.

Results

Indicators related to care for diabetes and hypertension, maternal and child health, and rational use of antibiotics received high acceptability, and information was available and collectable in the current administrative database. However, there were problems in implementing indicators for managing cardiovascular risk, care for bedridden patients, and asthma and COPD.

Conclusions

The development of quality indicators using a guideline-based approach is a useful way of generating evidence to support the effective implementation of a program. Indicator piloting is recommended prior to introducing indicators in the health system.

Background

Primary care in Thailand serves as the entry point in the health care system. Beneficiaries of the Universal Coverage Scheme (UCS), the publicly financed health insurance program, cover 75% of the Thai population, who are required to register at the contracting unit for primary care (CUP) in their residential area. In general, a CUP includes one district hospital and several health-promoting hospitals or primary care units (PCUs). District hospitals offer both primary and secondary care services whilst health-promoting hospitals provide ambulatory care and community services. These are staffed by nurse practitioners and public health officers [1].

A key feature of the UC scheme is the Thai Quality and Outcomes Framework (Thai QOF). This pay-for-performance scheme was introduced in 2013 to address variations in the quality and accessibility of primary care. Health care providers are incentivised to improve primary care quality in key predetermined areas expressed as a series of quality indicators. Achievement against each indicator is calculated on an annual basis using national administrative records and given a point value. The total points obtained are then converted into a financial value, which is allocated to the CUPs [2].

Quality indicators are an essential component of this program as they influence both provider effort and subsequent funding allocation [3, 4]. However, early evaluation of the program identified that the quality measures in use were perceived to have poor validity which negatively impacted upon the acceptability of the scheme, with resulting variation in uptake and financing at the local level [5]. In response, an independent research unit in the Ministry of Health-the Health Intervention and Technology Assessment Program (HITAP) was commissioned to develop and pilot potential quality indicators to be implemented in fiscal year 2017. This was done in collaboration with the National Institute for Health and Care Excellence (NICE) National Collaborating Centre for Indicator Development, University of Birmingham, and former NICE International.

Two broad approaches have been used to develop quality indicators: a deductive approach and an inductive approach [6]. A study conducted by Stelfox and Straus found that the majority of quality indicators were developed using a deductive approach [6, 7]. This approach suggests that the quality indicators should be derived from scientific evidence related to important quality-of-care concepts. Studies in the UK suggest that the key attributes of good quality indicators are reliability, validity, feasibility, acceptability, being attributable, and sensitivity to change. Reliability is the extent to which the performance of an indicator is consistently assessed. Validity represents the extent to which evaluators are measuring what they intend to measure. Feasibility considers the extent to which the necessary data are available and collectable. Acceptability refers to whether the indicator is acceptable and fair as viewed by health professionals; it also covers any unintended consequences resulting from the introduction of an indicator. Being attributable considers whether the indicator is in the control of health professionals. Finally, sensitivity to change refers to the ability of an indicator to detect changes in care quality [8, 9].

Since more than 3% of the primary care budget is being spent on the QOF program [10], piloting the indicators is desired in order to ensure efficient health care spending. In addition, indicator piloting provides an opportunity to prove the value and reveal practical issues of the developed indicators before spending a significant amount of time or money on actual implementation [8, 9, 11].

Methods

The development of quality indicators was carried out in two stages: indicator development (May–November 2015) and indicator testing (December 2015–February 2016). The process was overseen by a steering committee, chaired by the Director of the Healthcare Accreditation Institute. The steering committee provided strategic direction including the prioritisation of disease areas, potential indicators in each area, the approval of the indicator testing protocol, and the recommendation about the final set of indicators.

Indicator development

The overall process of quality indicator development and testing is described in Fig. 1. The process began with prioritising areas/issues that needed quality improvement. This was followed by developing the initial set of indicators based on clinical practice guidelines and testing them.

Fig. 1

Flow diagram of the indicator development process

Prioritisation of areas requiring quality improvement

Prioritisation of clinical areas requiring quality improvement was guided by five criteria agreed by the steering committee and was developed with reference to international literature on priority setting. Available evidence on the principles or criteria for setting priorities in primary care services was reviewed. Literature in the USA showed that different principles (e.g. safe, person-centred care, affordability) were considered in setting the national strategy for quality improvement [12]. In Sweden, the criteria included severity of the health condition, patient benefit, and cost-effectiveness [13]. Korea used the burden of condition, seriousness of the quality problem, interest and demand of society, acceptability, and feasibility as criteria for setting national priorities for quality assessment of health care services [14]. Based on this review, five criteria were selected for priority setting in Thailand: (1) burden of condition, (2) variation in quality of care, (3) availability of clinical practice guideline(s), (4) feasibility of quality assessment, and (5) the extent to which the area is in line with national policy. These criteria were ratified during two stakeholder meetings which included policy-makers from the Ministry of Public Health (MOPH) and National Health Security Office (NHSO), academics, regional health managers, and primary health care workers.

Prior to these stakeholder meetings, the research team developed a shortlist of eight areas thought to be suitable for quality improvement because these problems had high levels of burden of disease and variation in quality of care. Stakeholders were then asked to discuss these and to individually select the three areas they perceived as being most suitable for quality measure development. The five most endorsed areas from both groups were presented to the steering committee, which made the final determination of priority areas. These included chronic diseases identification and management (hypertension, diabetes, and cardiovascular diseases), maternal and child health, care of the bedridden patients, rational use of antibiotics, and asthma and chronic obstructive pulmonary disease (COPD).

Selection of clinical practice guideline(s)

Recommendations from clinical practice guidelines were used as a basis for formulating the quality indicators. Available Thai clinical practice guidelines in the selected clinical topics were assessed using the AGREE appraisal tool [15]. The research team also consulted experts in each area to understand the use of guidelines in primary care units and identify related service delivery issues. Guidelines deemed appropriate were subsequently used to formulate the indicators.

Drafting the initial set of indicators

Twenty-six indicators were developed by the research team and discussed with the steering committee. Seven indicators were excluded because they did not add value or they were anticipated to be difficult to assess through the existing information system. However, five new indicators were suggested by the steering committee including the control of blood pressure and blood glucose levels, referral of pregnant women with high risk of pregnancy, visits of a family care team to bedridden patients, and admission rates of patients with an exacerbation of asthma to the emergency department. Altogether, 24 key indicators were recommended for piloting.

The indicators were then assessed with a focus group for content validity in terms of clarity of wording and necessity, defined as whether primary care units adhering to the indicator provided higher quality care/service than those who were not doing so [8]. The group included three experts in selected disease areas, two NHSO staff, four representatives from provincial and district health offices in different regions, and two health care workers from primary care units. The indicator template and questions were sent to participants prior to the meeting. Participants discussed each indicator in turn, and comments and suggestions were reviewed by HITAP. Changes to indicator wording were then made as a result of the discussion.

Indicator testing

Indicators were tested through a 3-month pilot in a purposive sample of three geographically diverse regions in the south, north-east, and central parts of Thailand. These regions were identified based upon the willingness of the provincial health offices to participate in the pilot. The provincial health officers in each province were asked to select three CUPs using the following criteria: population size according to the UC population and location of the CUP (a mix between rural and urban). Then, three primary care units (PCUs) under those selected CUPs were selected based on their willingness to participate in the study as well as the type of PCU (one PCU managed by the hospital and two health-promoting hospitals). The recruitment process of primary care units is illustrated in Fig. 2.

Fig. 2

Recruitment process of primary care units for testing quality indicators. *CUP is abbreviated from Contracting Unit for Primary Care. **PCU is abbreviated from Primary Care Unit

Participating PCUs were informed about the indicators and also given a study handbook and details of the electronic data extraction. The study handbook provided all of the details regarding the indicators, aim of the pilot, timelines, responsibility of PCUs during the pilot, and the process of indicator evaluation after piloting.

Data collection and analysis

Mixed methodologies were used to collect and analyse data including self-administered questionnaires, interviews, and data extraction of patient’s medical records.

Self-administered questionnaires

For every pilot site, descriptive information was collected on population characteristics, opening hours, and staffing using a self-administered structured questionnaire. Descriptive statistics were used to describe and summarise the data. In addition, all staff involved in the pilot were asked to complete a workload questionnaire at the end of the pilot. The questionnaire included date and time of activity and any equipment used to provide services. The questionnaire aimed to quantify the time (in minutes) that was spent on each activity including work with patients and administrative work.

Semi-structured interviews

Semi-structured interviews with open-ended questions (see Appendix 1) were undertaken with PCU staff to obtain information regarding acceptability, feasibility, potential barriers, and unintended consequences of the introduction of the indicators. Interviews were coded and analysed thematically. Regarding acceptability, participants were asked whether they would accept the indicators for the Thai QOF program. The acceptability rate for each indicator was grouped.

Administrative database

QOF achievement was calculated using patient-level information held in a national database managed by the MOPH. This database contains information about the patient, their use of ambulatory services, and inpatient services of health facilities under the federal government [16]. For an indicator to be feasible, the national database must hold information about the specified care activity. The availability of information was categorised based on the specified care activity at the individual indicator level into three groups: (1) necessary information is available in the national database and extraction rules can be applied; (2) necessary information might be available in the national database but staff did not enter the necessary data; and (3) necessary information is not collected in the national database.

Post-pilot selection of potential indicators for the Thai QOF program

Two consultation meetings were organised in March 2016. The aims of these meetings were for the research team to present the preliminary findings of the pilot. The first meeting involved 15 key stakeholders at the national level including policy-makers from the MOPH and researchers from academic institutes, as well as representatives from the NHSO and its regional offices. At the second meeting, 14 participants attended including representatives from provincial and district health offices and health facilities. Comments and suggestions on the pilot findings were summarised and presented to the steering committee.

Results

Assessment of clinical practice guidelines

The result of guideline assessment is provided in Table 1. These guidelines deemed appropriate to use for indicator development.

Table 1 The score of the quality of clinical practice guidelines for developing quality indicators

Indicator development

In total, 24 indicators were suggested for piloting. The list of indicators is presented in Table 2.

Table 2 Developed indicators for the QOF program

Indicator testing

Twenty-eight PCUs participated in this pilot study including 5 PCUs with less than 3000 people, 13 PCUs with 3001–7000 people, 5 PCUs with 7001–10,000 people, and 5 PCUs with over 10,000 people. Staffing was different according to the types and size of PCUs. PCUs under district hospitals were staffed by approximately 2–4 physicians, 1–3 nurses, 1–2 dentists, 2–3 public health officers, and 1 staff who managed data. Health-promoting hospitals were staffed by 1–2 nurse practitioners, 1–2 nurses, and 1–4 public health officers.

Acceptability and implementation

Informants included 25 PCU directors, all 91 health care workers who deliver services related to the tested indicators, and 15 data managers during the pilot. The acceptability of indicators is illustrated in Table 3.

Table 3 Availability and acceptability of quality indicators

Indicators related to care for diabetes, hypertension, maternal and child health, and bedridden patients were highly acceptable. The reasons for this were that the indicators were perceived to be beneficial both for people and health providers; PCUs were perceived to have the capacity to provide services in terms of staffing, technology, and skills; non-communicable diseases (NCDs) were priorities for quality improvement due to the associated burden; and indicators were viewed as a good guide for health providers to know the priority areas of health services and how good quality services should be provided.

Some of the results are illustrated in the comments from participants

“The indicators on hypertension and diabetes screening should be included in the QOF program because it provides information on the standard of the care for hypertension and diabetes - which are important health problems in our area. If we are able to screen more people, we might detect more patients in the early stages of the diseases and this will result in better prognosis. Previously, when we did not have such indicators, people who are in the early stages of the diseases were hidden in the community and did not get access to the care”—nurse practitioner in a health centre.

Implementation concerns related to some indicators raised by participants included the lack of support systems from other agencies (e.g. training, databases, feedback system), lack of knowledge of heath care workers to provide services, lack of equipment, lack of human resources, and some indicator criteria that were out of the providers’ control such as lifestyle modification-related issues. For example, the indicator on screening for cardiovascular diseases using the Thai global risk score was really appreciated by policy-makers and was perceived as necessary by experts; however, health care workers did not accept this indicator because they did not have the capacity to carry out related activities in terms of equipment, knowledge, and skills. Moreover, data related to this indicator is not currently available in the routine database.

Some of the results are illustrated in the comments from participants

“Our health center has just been informed about the policy of cardiovascular diseases screening with the Thai Cardiovascular (CV) risk score. We have not prepared for its implementation yet. We have not trained our staff to be able to delivery this service, or provide them with the needed materials. If we need to screen the general population aged between 35-70 years, it will increase our workload a lot. We are not ready for this indicator”—nurse practitioner in a health centre.

Data availability

Examination of the care activities collected as part of the national health database revealed some issues with data availability. These results are presented in Table 3. When considering the completeness of data, it showed that the rates of missing data were exceptionally high for variables related to referral services and smoking status (51–98%). In addition, we found that staff incorrectly entered the International Classification of Diseases 10th Revision (ICD-10) code for hypertension during pregnancy by entering the code for hypertension in the general population.

Workload

The analysis of workload information revealed that most of the activities related to the piloted indicators were carried out by practice nurses. Figure 3 shows that health care workers spent the most time on care for bedridden patients in the community, followed by registering bedridden patients and providing first antenatal care for pregnant women. Assessing lung capacity by peak flow expiratory meter and providing asthmatic control test took the least time.

Fig. 3

Workload associated with developed indicators

Overall recommendations

Based on the summarised information gathered from the pilot and two consultation meetings, recommendations were made on which indicators the NHSO should implement in the QOF program. Ten indicators were recommended for implementation in the next QOF program because the indicators were accepted by health care workers, feasible for implementation in the primary care setting, and data related to these indicators were available in routine databases. The list of recommended indicators is presented in Table 4.

Table 4 Selected indicators for the future QOF program

Discussion

This paper describes the process of quality indicator development and piloting for the Quality and Outcomes Framework in Thailand. To our knowledge, this is the first time that Thailand has developed quality indicators for primary care following systematic and transparent processes, incorporating evidence for prioritising areas, and using recommendations in clinical guidelines to ensure that quality indicators attribute to better health outcomes. In addition, the developed indicators were assessed using real-world data. Indicator piloting allowed us to identify a number of implementation issues and concerns of health care providers specific to the pilot indicators. It also enabled us to explore data availability in order to assess technical feasibility of the indicators.

This study confirms that piloting should be a prerequisite for policy-makers before they introduce indicators in the health system [9]. Our study showed that although indicators were systematically developed according to a well-designed approach, their implementation in practice was not straightforward. The indicator on screening for cardiovascular diseases using the Thai global risk score was a good example. As a result, this indicator was not recommended unless health care workers were trained, equipment was provided, and the national health database was amended to capture the care activity.

Another valuable aspect of testing indicators is that it allows problems to be identified and solutions to be prepared prior to implementation. For example, the ICD-10 code for hypertension during pregnancy was incorrectly entered. Therefore, we recommended that the NHSO provides guidance for PCUs on how data related to each indicator should be entered. Similarly, we found a high degree of missing data for referral indicators. By verifying this finding with some PCUs, we found that the missing data was not a reflection of the care not having been delivered but the data was not entered in the database. Applying a financial incentive could result in better, more consistent data entry. We confirmed that the availability and comprehensiveness of clinical data are crucial for the P4P program as suggested by other studies [8]. The health information system was not created specifically for the QOF program, resulting in incompleteness of data to assess performance based on some indicators. We therefore recommend the MOPH to improve the national database to be more useful for future QOF and other initiatives on primary care. Using administrative data for measuring quality of care could provide advantages because these secondary data are readily available and are relatively inexpensive. The USA used hospital claims data and out-patient claims data to measure the quality of medical care provided to Medicare beneficiaries [17].

The process of indicator development and piloting as described in this study might be applied in other resource-constrained settings. This work followed a systematic approach to indicator development and testing, drawing upon protocols used to support indicator development in the UK and elsewhere [8, 18] and adapting culturally to ensure sensitivity and feasibility in the Thai context. Some attributes (cost-effectiveness, sensitivity to change, and reliability) were not feasible for assessment in our context given the high PCU workload and short pilot time frame. However, we believe that our pilot provides enough information for policy-makers as the tested key attributes are similar to key attributes described in the QUALIFY protocol for testing quality indicators in Germany [19].

This study provides recommendations on 10 indicators that should be implemented for the fiscal year 2017. It also highlights 14 indicators that should be postponed until the health database is ready or health care staff is able to provide services. In our context where indicators are part of our health care system, this is the best available evidence so far that can be used to convince organisations about the value of indicator piloting. Policy-makers and program implementers at both the national and international levels can benefit from this study for ensuring the development of acceptable, feasible, and evidence-based indicators.

The study has several limitations. Firstly, we did not assess the representativeness of stakeholders participating in the prioritisation process. Although we invited key policy-makers from different departments in the MOPH, not all invited departments joined the meeting. We were also unable to include patient representatives and lay people in this process despite knowing their crucial roles. Secondly, we considered adopting a RAND/UCLA Appropriateness Method [20] to rate the clarity and necessity of each indicator but ended up with a focus group discussion as participants did not rate the indicators prior to the meeting and did not agree to rate it during the meeting. This unsuccessful activity might be explained by the following: (1) the number of indicators to be assessed was too high for participants to be able to rate in the given timeframe; (2) the time given for participants (15 days) was too short; and (3) participants were not familiar with this method and might not appreciate its benefits. Thirdly, health providers failed to keep workload diaries because they found it too burdensome in addition to their usual work. Therefore, workload was measured at the end of the pilot. The reported workload might over- or underestimate the real value as it was measured at a single point in time and did not account for variations across the piloting period. Also, the workload is calculated based on PCUs that provided these activities, so a low workload might be due to non-participation. Finally, due to tight deadlines in the overall process, the piloting time of 3 months might be too short to assess indicators comprehensively compared to 6 months in the UK QOF [8] or 12 months in the US HEDIS [18]. For example, the analysis of sensitivity to change was not performed as data after piloting could not be extracted from the database. Similarly, implementation issues or unintended consequences might not have been captured as staff would need more time to engage with the indicators.

Conclusions

This systematic, evidence-based indicator development for the QOF or other pay-per-performance initiatives is necessary and feasible in resource-constrained settings. The study also confirmed that indicator piloting is a prerequisite for policy-makers prior to introducing indicators in the health system since it ensures acceptability, feasibility, relevance, and effectiveness of the indicators.

Appendix 1: Example of interview guide

Questions for indicator HT 1: Percentage of individuals aged 35 and above who received a screening for hypertension in the last 3 years (staff that provided care during the pilot period)

• 1. Before having this indicator, what were your care activities related to this indicator?

• 2. During the pilot of this indicator, how did you adapt your care activities in order to achieve the indicator target?

• 3. How did you provide screening to elderly people, bedridden patients, or the disabled?

• 4. What were the barriers you faced while piloting this indicator? How did you deal with these barriers?

• 5. What were the negative consequences from providing care activities related to this indicator?

• 6. What recommendations would you give for improving the indicator?

• 7. Do you think this indicator should be included in the QOF program? Why?