Is progress a level playing field?

David Bailey FCCT, 28 Apr 2025

Acknowledgement: Data for the Chritchlow-Rogers analysis provided by Stephen Gregory

The difference a good teacher can make...

It must have been more than 20 years ago when I participated in a training course on 'teacher quality'. The facilitator said something that took me aback.

They said, 'A good teacher can make as much as a 30% difference to pupils'. At the time, the surprising element of this comment was that the difference quoted didn't seem particularly large. Intuitively, it felt that teachers should be able to make much more of a difference than 30%.

The research the facilitator based this figure upon has faded in the mists of time. However, a more recent analysis by Denoël et al (2017) (1), based on the PISA survey (2015) (2) puts the predictive power of the teacher at around 15% of the performance of the pupil, so perhaps even less of an impact than suggested all of those years ago.

This compares with:

• 21% due to general pupil mindset

• 8% due to subject orientation mindset

• 8% due to student behaviours

• 20% due to school factors

• 9% other factors

Source: Denoël et. al (2017) (1)

This analysis highlights that although the teacher and the school can influence outcomes directly (and perhaps more so indirectly over time by developing pupil mindset), many other factors also influence overall performance.

One notable factor is economic disadvantage. The gap in attainment between economically advantaged and economically disadvantaged learners is highlighted in the EEF report 'The Attainment Gap' (EEF, 2017) (3). Despite the optimism that schools can compensate for economic disadvantage, it has been reported by the Education Policy Institute that COVID has had a significant detrimental impact, meaning 'since 2015, progress in narrowing the gap had stalled, and following the pandemic, has gone into reverse' (EPI, 2022) (4).

Another factor is the additional educational needs of pupils. Many reports, such as the one for the Institute for Fiscal Studies by Sibieta & Snape (2024) (5), identify that additional needs have increased significantly in recent years, both in terms of the amount and severity of needs of pupils. They also highlight an unequal distribution of the need and support across the system.

These are factors that I'll discuss more about another time, but there is another factor that is relevant - the Critchlow-Rogers effect.

The Critchlow-Rogers effect...

Skip ahead a few years from that initial training session to the mid-2000s, I was in a meeting with a North Yorkshire Science Adviser, John Crossland, someone for whom I had a great deal of respect. He was the first person I heard mention the concept behind the 'Critchlow-Rogers Effect'.

Discovered back in the 2000s, the Critchlow-Rogers effect is named after two teachers from North Yorkshire, who each independently concluded that it was easier for children with higher prior attainment to make more progress in tests.

During a discussion about departmental self-evaluation, John referred to this effect and pointed out that I should consider this when I was evaluating the performance of teachers in my department. I should expect that pupils in the upper-ability groups were far more likely to make more progress than those in the middle, and those in the lower-ability groups were likely to make less.

This was another one of those moments that challenged my thinking. I had expected that pupils who had attained well previously might attain well in the future, however, I had thought up until this moment that pupil progress was far more of a level playing field. Yet here I was, being told that the gap would be highly likely to continue to widen.

He said that, for a school around the national average, the effect could be as much as one additional grade of progress more for the top prior attainers and up to one grade worse for the lower prior attainers. Put simply, he said that the reasons that children had made more progress in the past meant that they were more likely to make more progress in the future. Therefore, additional progress for this group might not be due to the teacher at all.

This effect is described concisely in a blog by the Centre for Evaluation and Monitoring (CEM) from 2017 (6). In this blog, they highlight that although it might be easy to assume that pupil progress and teacher effectiveness are closely linked, it is far less straightforward. They point out that the Chritchlow-Rogers effect should be carefully considered when evaluating data. This helps avoid incorrect inferences about the quality of teaching.

Since considering this, I have been mindful of not rushing to draw conclusions about the effectiveness of teachers from pupil data from ability groups. This cautionary note is one of the messages that I've included in the data analysis part of the EnlightenEd Leadership programme, endeavouring to pass on the lessons learned to other leaders.

A year's progress in a year...

Jumping ahead in time to another conference in March 2023, this time with Professor John Hattie. During that he talked about pupils making 'a year's progress in a year' as something that you might reasonably expect teachers to do. There is quite a bit of debate about his work on effect sizes in Visible Learning that I won't discuss here. However, I will note that there is an 'expectation' in the current UK school system that pupils making average progress (i.e. the same progress achieved by an average pupil) is a benchmark on which schools are compared.

What about Chritchlow-Rogers at a school level?

Having been very cautious about drawing conclusions from data at a pupil level and class level, I recently considered how this might apply across schools.

Gorard (2018) (7) explores the connection between KS4 attainment and progress in his book Education Policy: Evidence of Equity and Effectiveness. This also shows data at a school level for Yorkshire schools with 2004 data and there is a clear, near linear, correlation shown (+0.96): The higher the attainment, the higher the progress. He also points out that the effect appears year on year in England's primary and secondary schools.

The Department for Education publishes school-level outcome and progress data, so I wondered whether this trend applied. Working with Stephen Gregory, we analysed the national 2024 dataset (8). We pulled out the key stage 4 Progress-8 score for each school. We then looked at the prior attainment for each school, filtering the data so that we were considering mainstream state-funded schools.

This is shown below in Figure 1 (below).

Adding a trendline shows a pattern. This is almost exactly what had been quoted by John Crossland - the schools with the highest prior attainers had made about 1 grade better progress than average and the lowest prior attainers about 1 grade worse progress.

Strikingly, almost every school with an intake average points score of 108 or above had a positive Progress 8 score. This indicates that for schools with high prior attainers, national average or above progress almost always happens.

There is some obvious variation in the data, particularly below 108 points, which prompted further reflection as to what the strength of the correlation is.

When analysed further the whole dataset shows a weak correlation of +0.2, however this is based on all state-funded mainstream schools and there is huge variation between the characteristics of these schools. Many of these other factors that are likely to make a difference to progress. It was important to understand these more fully.

This is something I'll discuss further at a later date, however I'll add here that repeating the analysis by choosing more similar types of schools within the dataset tends to strengthen the correlation, in some cases even just filtering by one factor can increase the strength of this correlation quite noticeably to at least a moderate strength correlation.

In conclusion

The Chritchlow-Rogers effect is one of many reasons that a 'year's progress in a year' is only a reasonable assumption for some schools. If it is a low prior attaining cohort, national average progreess is far more ambitious. Conversely, for schools with a high prior attaining cohort, national average progress is far less ambitious.

Importantly, I'm not saying that we should lower our expectations for low prior attainers, or indeed any other vulnerable groups - far from it. I know from many years in the classroom that showing belief in your pupils and having higher expectations can lead to very positive outcomes. Teacher self-belief and belief in their pupils has been reported to have a positive impact on pupil outcomes (e.g. Hattie) (9). Although this is disputed (e.g. Jerrim, Sims & Oliver, 2022) (10), it also appears to do no harm.

However, when it comes to comparing progress across schools, the Chritchlow-Rogers effect suggests that it really isn't a level playing field at all.

References:

1. Denoël, E., Dorn, E., Goodman, A., Hiltunen, J., Krawitz, M., & Mourshed, M. (2017). Drivers of Student Performance: Insights from Europe, McKinsey & Company. Available at: https://www.mckinsey.com/industries/education/our-insights/drivers-of-student-performance-insights-from-europe [last accessed 27/04/2025]

2. OECD (2016), PISA 2015 Results, PISA, OECD Publishing: Paris

3. Education Endowment Foundation (2017). The Attainment Gap. London: EEF. Available at: https://d2tic4wvo1iusb.cloudfront.net/production/documents/support-for-schools/bitesize-support/EEF_Attainment_Gap_Report_2018.pdf?v=1742133598 [last accessed 27/04/2025]

4. Tuckett, S., Hunt, E., Robinson, D., Cruickshanks, R. (2022) COVID-19 and Disadvantage Gaps in England 2021. Education Policy Institute. Available at: https://epi.org.uk/publications-and-research/covid-19-and-disadvantage-gaps-in-england-2021/ [last accessed 27/04/2025]

5. Sibieta & Snape (2024). Spending on special educational needs in England: something has to change. Institute for Fiscal Studies. Available at: https://ifs.org.uk/publications/spending-special-educational-needs-england-something-has-change [last accessed 27/04/2025]

6. CEM (2017) Is student progress the same as teacher effectiveness? Centre for Evaluation and Monitoring. Available at: https://www.cem.org/blog/is-student-progress-the-same-as-teacher-effectiveness [last accessed 27/04/2025]

7. Gorard (2018) explores the connection between KS4 attainment and progress in his book Education Policy: Evidence of Equity and Effectiveness p85

8. DfE (no date) Compare School Performance. Available at: https://www.compare-school-performance.service.gov.uk/download-data [last accessed 27/04/2025]

9. Hattie Ranking: 252 Influences And Effect Sizes Related To Student Achievement: Available at: https://visible-learning.org/hattie-ranking-influences-effect-sizes-learning-achievement/ [last accessed 27/04/2025]

10. Jerrim, J., Sims, S., & Oliver, M. (2023). Teacher self-efficacy and pupil achievement: much ado about nothing? International evidence from TIMSS. Teachers and Teaching, 29(2), 220–240. Available at: https://doi.org/10.1080/13540602.2022.2159365https://www.tandfonline.com/doi/full/10.1080/13540602.2022.2159365#abstract [last accessed 27/04/2025]