Do You Know How To Measure The Impact Of Your SATs Interventions? Here’s How We Do It

As part of our online one-to-one maths interventions we work with thousands of pupils every week to raise attainment and boost confidence in maths. The majority of the schools we work with use the intervention to prepare their Year 6 pupils for SATs.

During the spring term, our primary school tutoring programme to a SATs tutoring programme that is tightly focused on reinforcing and consolidating previous learning to help pupils approach the SATs with confidence.

Naturally the schools we work with are always interested in being able to see what pupils have learnt and what progress they are making, which is why we include detailed pupil and cohort reports as part of the intervention programme.

But (just like the rest of the programme) these are a constantly evolving feature. Here’s how we’ve made our reporting system even clearer and more useful than ever before. To illustrate our discussion we’ll be using anonymised data from a small school with 4 pupils enrolled on our SATs intervention programme.

The Progress Overview Page: What The Cohort Has Covered & How They’ve Found It

The progress overview page is your one-stop shop for how pupils have been performing on the SATs programme as a whole. It aggregates and averages out pupil progression in a clear, plain-English format so you can tell at a glance how much impact the programme is having, with individual pupil reports available below.

Let’s dive a little further into these context statements. Learning objectives covers what has been learnt in the lessons, specifically those objectives that tutors have marked as either ‘Knowledge Reinforced’ or ‘New Content Learnt’ (more on those later). In this case pupils have only just started their sessions, but they’ve managed to cover 39 objectives between the four of them already!

Because we also provide access to individual pupil reports, you can use those to identify which objectives and topics have been covered in our sessions, and use that to inform and adapt maths revision plans for your KS2 classes.

SATs-style questions refers to the problem solving questions pupils encounter in each session (there are at least 4 per session) and practise with tutor assistance. Independent questions are the Post Session Questions (PSQs) that pupils can choose to attempt, and serve as a good test of how well pupils have understood the topic closely following the lesson.

These pupils have attempted 8 PSQs, details of which can be found in their individual session reports.

So far we’ve looked at some objective impact data, the kind that is great to show SLT as evidence of maths progress and attainment, but just as important to the learning experience is whether pupils are finding their sessions enjoyable and useful. That’s what the ‘What your pupils are saying‘ section is for.

We ask pupils for feedback after every lesson, and usually they’re more than happy to tell us how they found learning via Third Space! We take this feedback (available to see for each pupil separately in their session reports) and average it out to provide the two figures you see above.

As you can see, all 4 of this school’s pupils enjoyed their sessions, and on average they seem to find those sessions very useful.

All of these numbers will grow and change as sessions continue, so regularly checking the progress overview will help you establish not just how your pupils are doing in the moment but the full learning journey they’ve been going through.

Pupil Reports: In-Depth Data On Each Pupil’s Performance

The second part of the Progress Overview page is a table of more detailed information about how all of your pupils have been performing, separated pupil-by-pupil.

S/E/PP denotes pupils with SEN, EAL or Pupil Premium funding. If you have a large number of pupils enrolled and want to see how a particular demographic has been doing, this category lets you find them all the quicker.

PSQ shows not just the post-session questions answered, but also how many of these they answered correctly. Since different pupils will have different numbers of PSQs answered, you can sort by percentage answered correctly rather than purely number.

In our example school for instance, Pupil A may have answered the most questions, but Pupil D has answered the greatest percentage correctly (i.e. 100%).

Focus is a subjective metric based on tutor discretion; they can note how well a pupil was performing during their sessions. Despite being a subjective measure it still has its benefits: it provides context to the other data points. For example, Pupil A may have answered some of their PSQs incorrectly because they were distracted during the lesson (as the tutor noted) and didn’t pay attention to some key information.

If you’re ever confused about any part of this page, the ‘Key’ button opens up a pop-up window that explains each metric in detail.

Every field in this table is sortable to save you some trouble. We know how time consuming it can be to try and sort a large volume of data to find the specific point you’re looking for, so we’ve made it as easy as possible to find them!

A Pupil Report In Detail

Clicking on any of the ‘Report’ icons in the summary table will take you to an individual pupil’s progress report. Let’s jump into Pupil A’s.

Right away, as with the Progress Overview page, we provide you with an at-a-glance set of impact data. Pupil A has attended 2 of a possible 3 sessions, covered 16 learning objectives and been secure in 94% of them. They’ve enjoyed all of their sessions, and feel that they were as useful as it is possible to be.

We provide equal weighting to these latter metacognitive measurements for a multitude of reasons: they’ve been proven to be effective (see the EEF toolkit), they allow pupils to actively reflect on their learning, and they provide another dimension to the ‘evidence-of-impact’ question.

Beneath these you’ll find the real meat of the Pupil Progress Report – a detailed analysis of their learning journey.

When pupils begin on any of our online one-to-one intervention programmes they take an initial diagnostic assessment (that we refer to as Mission Zero). In the SATs programme, this assessment is used to establish a pupil’s level of knowledge in maths topics before beginning their sessions.

We then compare this with our SATs revision plan (details of which you can find out about in our blog ‘How To Create A KS2 Maths Revision Programme So That All Your Year 6 Achieve 100 in SATs 2021’) to help identify which topics should be covered in earlier sessions to maximise their progress and SATs results.

One of the most interesting pieces of feedback we’ve received about these reports has been that the old version of this analysis was too confusing; it contained too many details and didn’t follow the learning journey as the children experienced it.

So, we made it simpler, cleaner and more reflective of the learning experience. The new report clearly lays out where a child was before starting the programme, whether they’ve been taught the material (we estimate it takes one to two sessions to fully cover a topic), how long it took them to secure their knowledge, and where they were both immediately after a session and after the full programme.

Aligning with the learning journey in this manner also makes it easier for you to keep track of which topics have been covered during intervention sessions. As part of our SATs programme class teachers can choose the order of intervention lessons; this means you can decide which topics you’ll want to cover in class yourself.

In our example, Pupil A’s teacher has opted to front-load many of the subjects where Pupil A is weak. Their plan may be to have Third Space Learning help Pupil A ‘catch up’ where they are falling behind, while covering those subjects Pupil A is already comfortable with in class. Their tutor will also be aware of this, and can adjust their lesson plans accordingly.

Individual Session Reports: Exactly What Was Learnt, And How Well

The most useful information in this report is the full list of learning objectives for the lesson taught, along with whether this was new content, knowledge being reinforced, or something that still needs more work. Again, this could be used to inform your own lesson plans, or just to keep track of what’s being covered session to session.

Along with these come a more detailed analysis of any post-session questions the pupil has attempted. Here you can see speed of answer and whether the question has been completed correctly, and look at the questions themselves.

In this case, Pupil A has attempted two questions (which we knew from her progress report), but answered both of these too quickly and incorrectly. This might suggest that her confidence has improved, but her ability has not yet caught up. Or else she may have made a calculation error, or even attempted them only ‘for show’.

Our Reporting Is A Work-In-Progress And ABOUT Work In Progress

We’ve been changing and improving our reporting system since we first implemented it, and the work continues every day. Whether it’s from teacher feedback, our own analysis and reflection, or even tutor feedback we’re constantly aiming for clearer, more useful reporting documents.

But it’s worth noting that what we do – and therefore what we report on – isn’t a summative be-all end-all for pupil progress. We provide reports on our sessions as part of the formative process; we are working with (and reporting on) pupils in the process of making progress. Our interventions are intended to be supplementary to class work with the understanding that there may not be time to cover absolutely everything that every child needs.

Of course, if you’re looking for more summative assessments of progress we can provide those too. Our online Maths Hub includes several practice SATs papers, and a number of these are free to download.

We’re proud of the fact that our interventions can help pupils make the progress they need to succeed at SATs, and that we provide a wealth of data about this to save teachers time and confusion about whether any impact is being made.