• Summary
• Introduction
• Method 
• Results
  • Details of returned questionnaires
  • Waiting times for clinics
  • Delays in communication
  • Grade of clinician signing letter
  • Adequacy of information
  • "Comments"
• Discussion 
• Appendix 
  • About EQUIP 
  • The questionnaire 

The quality of information sent to GPs following their patients' attendance at Sheffield's gynaecology out-patient departments was evaluated using a short questionnaire. Questionnaires were sent to twenty general practices in the EQUIP project, and referred to the first letter received at each practice regarding the clinic attendance from Jessops Hospital for Women or the Northern General Hospital, for a new referral. The results permitted the following areas to be quantified:

* The average length of time between date of referral and date of clinic;

* Where and to what extent delays occurred between a patient's attendance at the clinic and the receipt of the follow-up letter at the practice;

* The grade of clinician signing the letter;

* GP opinion on the adequacy of the information in the letter regarding;

i) diagnosis

ii) investigations

iii) future management (hospital)

iv) future management (GP)

v) medication.

The results showed that information in the letters was considered adequate by GPs in the vast majority of cases. Waiting times and delays in communication could be explored only to a limited degree because of missing information, either on the hospital letters or because of administrative omissions at the practices. This report examines both the results themselves, and the effectiveness of the questionnaire in gathering potentially useful, and useable, information.

The purpose of this exercise was to evaluate the timeliness and adequacy of information sent to GPs following their patients' first appointment at Sheffield gynaecology out-patient clinics. The evaluation was one of a rolling programme of evaluations in the EQUIP project (see Appendix 1). It aimed to gather useful information on the quality of the communications which might indicate areas where improvements could be made, as well as identifying aspects of good practice. A simple questionnaire, designed to make minimum demands on the practices involved, was used in the evaluation. An important part of the exercise was to assess the effectiveness of this questionnaire in gathering the necessary information in a valid and practical way.

In addition to examining the main features of timeliness and adequacy of information, the exercise also addressed other areas of interest, including the average wait between referral and clinic, and the grade of clinician signing the letters. The intention is that the questionnaire designed for this evaluation can also be used to monitor changes over time in all the above factors.

The evaluation used a short questionnaire which asked both for factual information, such as referral details and relevant dates, and GP views on the adequacy of the information in the first letter from hospital following an out-patient clinic. Practices were asked to identify all new referrals to any gynaecology clinic at either Jessops Hospital for Women or the Northern General Hospital during the month of December 1991. For each patient referred, GPs were asked to complete a questionnaire with reference to the first letter received from the hospital regarding the referral. The evaluation was carried out during June 1992.

Of the twenty two practices currently participating in the EQUIP project, it was known at the outset of this evaluation that two of them were not able to identify the patient population of interest because of limitations in their computer software. One of these practices did, however, helpfully furnish us with the collective views of the partners on many of the issues covered in the questionnaire. Of the twenty practices who were sent questionnaires, seventeen returned them completed. The remaining three did not complete questionaires, since they either had made no referrals to gynaecology over the specified period, or were unable to respond because of workload constraints through staffing difficulties.

The 17 practices returned a total of 88 questionnaires. From the combined list size of these practices (109,568 patients) this represents a referral rate to Sheffield gynaecology clinics of 0.80 patients per 1000 per month. Fourteen of these referrals, as indicated on the questionnaire, were made to a colposcopy clinic. Referrals were made to 10 consultants in all. The following tables show how the referrals, by hospital and consultant, were distributed, excluding referrals to colposcopy clinics:

                 Jessops                                             Northern General

Referrals to colposcopy clinics were 6 and 8 to Jessops and Northern General respectively.

Of the 70 questionnaires included for further analysis, 11 referred to colposcopy clinics.

The average waiting time for appointments at each hospital was determined from the difference between the date referral was made and the date of the clinic stated on the letter. It was possible to calculate this in only half the cases, though, since the date of the clinic was commonly omitted from the letters. Details are presented in the following table:

(N= number of letters on which 'date of clinic' was specified.)

In order to determine the average waiting time, by consultant to whom the patient was referred, the 'date on letter' was used as a proxy measure because of the large number of cases where the 'date of clinic' was not specified. This can be only an approximate measure because of the delays between the date of the clinic and the date on the letter. However, using this proxy measure, average waiting times per consultant excluding colposcopies are shown in the following table. The average waiting times calculated this way are shown next to the 'Maximum waiting times for new non-urgent out-patient appointments' published by the DHA , as at December 1991, for comparison.

(Letters not received = 3)

The average waiting times for colposcopy clinics at Jessops and the Northern General, using the proxy measure, were 77 days (n=5) and 97 days (n=6) respectively (one letter from each clinic had not yet been received). The DHA stated maximum waiting time figures for colposcopy clinics at Jessops and the Northern General were 63 days and 175 days respectively.

The average length of time between the date a patient attended an out-patient clinic and the receipt at the practice of the first letter concerning that visit, consisted of the following delays;

a) the delay between the date of the clinic and the date on the letter;

b) the delay between the date on the letter and its receipt at the practice.

Unfortunately, the average delays could only be determined from a small number of cases - either the 'date of clinic' was omitted from the letter, or the 'date letter was received at practice' was not specified on the questionnaires, since some practices were not, at the time, routinely date stamping their incoming mail. Consequently, an even smaller number of cases could be used to calculate the total delay, which depended on knowing both pieces of information. The following table shows where and to what extent the delays occurred, based on the information that was available:

(N= number of questionnaires which contained the necessary information.)

The following table shows the grade of the clinician who signed the first letter to the practice following an out-patient clinic:

(Other=Clinical Medical Officer, Associate Specialist)

The number of letters where information on the five criteria was rated 'adequate' as opposed to 'not adequate' is expressed below as a percentage. The figures relate to the number of letters where the information was considered relevant:

Table showing proportion of letters where relevant information was rated as 'adequate'.

Percentage of 'adequate' letters

The charts overleaf show pictorially the proportions of letters where information was considered 'adequate' in relation to 'not adequate' or 'not relevant' information:

Of the initial eighty eight questionnaires returned, twenty two had comments in the free text space at the end. These ranged from the general to the detailed and specific.

Four of the comments referred to the fact that no letter had yet been received from the hospital following the clinic. Four comments related to unsatisfactory information (missing or unclear) regarding diagnosis, investigations medication and future management (GP). One stated that information was often inadequate with reference to patients who require annual smears, where future management is delegated to GPs. It was suggested that clearer guidelines would be hepful as to how many years annual smears will be required, and when the frequency can be reduced back to three yearly.

Two comments referred to cases where in-patient discharge summaries had been sent but no letter following the out-patient clinic. In one case this caused no difficulty since the dates of clinic and admission were close together, but the long delay in communication in the other case was considered unsatisfactory.

Some GPs referred to the long waiting times for appointments, one practice commenting that the long wait for a colposcopy clinic can be very distressing for patients informed that they have an "abnormal smear".

Other comments related to incidences such as patients' failure to attend, cancellations, or transfers. Several GPs commented on the helpfulness of some of the communications, eg. "brief, sensible, and to the point".

The most specific and detailed comment, representing the subjective views of a number of partners, was where they had ;

"...expressed concern regarding hospital management of patients with menorrhagia - a common referral condition. Firstly, treatment with NSAID and/or Progestagens which had 'failed' prior to referral were often repeated. The procrastination over the management of these patients often caused long delays in making decisions regarding hysterectomy. It was suggested the grade of clinician seen by the patient at the first appointment may be contributing to this."

This exercise served two main purposes:

i) to gather information about gynaecology out-patient communications to General Practices;

ii) to test the effectiveness of the particular questionnaire used in gathering information that, once analysed, is both helpful and useable.

i) The results.

One of the main, and encouraging, findings of this evaluation was the very high percentage of letters rated as adequate by GPs on the five criteria specified. In only a minority of cases did GPs comment on how the quality of information on these criteria might have been improved.

The finding that half the letters received from the hospitals did not specify the date of the clinic is worth noting for two reasons. First of all, confusion could arise for GPs over which particular clinic attendance the letter refers to. Secondly, the exact waiting time between referral and clinic cannot be assessed from such correspondence without this information. The proxy measure used here to calculate waiting times, using the date on the letter instead of the date of the clinic, provides only a rough estimate given the wide range of delays between these two dates (in those cases where this could be determined).

Monitoring the delays between the clinic date and the date the letter is received at the practice may be of help in identifying which stage of the process to focus attention. It is only possible to monitor and provide potentially useful information on this area if the date of clinic is specified in the letters, and/or practices routinely date-stamp incoming mail.

With reference to the grade of clinician signing the letters, no attempt was made at this stage to seek comments on who should have signed in each individual case. However, the information presented here may be useful to either purchasers or providers in considering the appropriateness of the grade of clinician, eg. in the case of patients with menorrhagia described in the above quote.

On a more general point, although we were very encouraged that 17 of the practices returned completed questionnaires, we were hoping for a slightly larger sample size so that more could have been performed on and extracted from the results. The lower than anticipated number of returns was due to an overestimation on our part of the referral rate to gynaecology clinics during December last year. For

evaluations which refer to in-patient services, this problem will not arise since, with the assistance of the DHA, we will know more precisely the numbers of patients admitted during any particular period of time to any specialty by every EQUIP practice. This will aid us greatly in our planning, and hopefully will be extended to out-patient services before too long.

ii) The questionnaire.

With all EQUIP evaluations, the intention is to provide valid and useful information on the quality of provider services whilst making minimum demands on the practices. This evaluation did provide information on a number of issues regarding out-patient communications by using a simple questionnaire. It demonstrated that, were standards set for some of the areas identified here, the questionnaire would be an efficient and effective way to measure the extent to which the standard was being met. The questionnaire could also monitor changes over time in the areas covered,

as well as detect areas of GP dissatisfaction, provide some detail as to the source of the dissatisfaction, and relay suggestions for improvement.

A major consideration for the EQUIP team is that the information we gather, and the results we produce with it, should not be merely 'interesting'. They should also have an inherent capacity to inform the change process, where the need for change is indicated, and monitor the effects of changes once implemented. To do this successfully requires an on-going process of consultation, negotiation, and collaboration with both purchasers and providers who have a commitment to improve the quality of services. The time and effort that the participating practices invest in the project has to be rewarded by our ensuring, as far as we can, that the evaluations have an optimum likelihood of receiving serious attention from those in positions to effect changes. From such dialogue with purchasers and providers, which will increase as the project evolves, it is hoped that the evaluations will become increasingly pertinent and useful.

This particular evaluation has been instrumental in raising many issues relevant to the above discussion. For example, how specific does the focus of an evaluation have to be? What depth of viewpoint is it necessary to gain? Where can generalisations be made, and where are some matters just too case-dependent to be prescriptive about? What aspects of services should just be reported on, and on which aspects should we seek the judgement and opinion of GPs regarding their quality? These and other issues will become clearer as the project unfolds. We would welcome views either on this particular evaluation, or on the general points raised above.

Finally, we would like to thank all those GPs and practice staff who contributed to this exercise.

APPENDIX 1

There are currently twenty-two General Practices participating in the EQUIP Project. The practices, geographically dispersed throughout Sheffield, range in size from a single-handed practice with a patient list size of 2200 to a six partner practice with a patient list size of 11500. In all, the project involved 77 GPs with a total list size of approximately 137,000 patients at the time of this survey. This patient base represents about a quarter of Sheffield's population.

Aims:

To provide routine and valid evaluation of provider unit services, in relation to:

a) patient care (both process and clinical outcome), from patient and general practitioner perspectives:

b) communication between secondary and primary services from GP and consultant perspectives.

This information to inform quality specifications in the negotiation of Service Agreements between purchaser and provider units.

To develop a toolkit of methodologies of use to other groups of GPs in service evaluation.

1. To ensure all participating practices have adequate data collection systems.
2. To establish a data base of participating practices' referrals.
3. Define questions of interest regarding quality; consulting with the General Practice teams, the LMC, the Department of General Practice and DHA planners.
4. To run a rolling programme of evaluations, of discharge and out-patient communications.
5. To evaluate at least one out-patient clinic, from the patients' perspective, using semi-structured interview and postal questionnare techniques to compare methods.
6. To complete at least one in-patient survey of patient experience (post discharge) eg. Obstetrics, also using semi-structured interview and postal questionnaire techniques.
7. Develop a mechanism to ensure that results are used to inform purchasers in their negotiation of Service Agreements.
8. Continue to ensure that all relevant bodies are kept informed of project progress, eg. TCP Management Group, participating practices, MAAG, the LMC, the FHSA, the Hospital Audit Committees etc.

1. To carry out a more comprehensive survey of patients' experiences, as both out and in-patients.
2. To continue the rolling programme of evaluation of discharge and out-patient communications, refining the methodologies.
3. Audit the quality of referral letters from primary to secondary care, developing standards that are mutually acceptable to both GPs and hospital consultants.
4. To explore the possibility of extending the evaluation to examine quality of services given by the Community Unit; following patients from out-patient appointments to in-patient stay through to discharge and community care.
5. Publish and promote a toolkit of methodologies to be adopted by other groups of GPs.