The KS1 and KS2 SATs can be a cause of consternation for teachers. The process of creating a SATs test is surprisingly opaque. So just how are the KS2 SATs Papers created, and why are they so “hard”?
Are papers just written by someone at the DfE before they are unleashed on unsuspecting schools? Where do some of the questions come from (chicken, anyone?) And what is the “science” behind the SATs exams?
To find out the answer to questions like these, and more, we went undercover and talked to an individual involved in the test development process. We think that, like us, you’ll be surprised by some of the answers. Which is why we’ve put together the 35 most surprising insights about how SATs papers are developed.
For instance, did you know that the SATs papers aren’t actually put together by the DfE? Instead, they’re created by the Standards and Testing Agency (STA). The STA is actually managed separately from the DfE and has its own budget for this very reason.
For 34 more great facts about SATs exam papers, just read on!…
N.B – Some names of individuals and companies have remained anonymous in order to maintain the security and privacy of the SATs examination process.
The Secret Ingredients to SATs: Test Development Frameworks & Content Domains
Behind every good SATs paper, there is a robust test development framework and well defined content domains. After all, if we don’t know exactly what is being tested, then what is the point of the test?
2. Did you know that the papers that pupils sit during for their national assessments are actually put together according to the test development framework not the National Curriculum?
3. The test development framework, which is publicly accessible and should be essential reading for Year 6 Teachers, sets out all the objectives from across KS2 that can be tested within the KS2 SATs papers. This is what is referred to as the content domain.
4. The test framework also sets out the proportion of questions that should be tested from different content areas. For example, 83-93 marks each year will be from Number (including calculations and fractions), Ratio or Algebra content areas.
5. Also, of all the content domain references available, each year only around 40% covered in SATs are from Year 6 objectives. This means that it is theoretically possible for children to achieve expected standard, without being proficient in any of the Year 6 content.
We’ve been saying for a while that KS2 SATs aren’t just about Year 6, so we’re pleased to hear it from other experts, too!
The “Science” Behind Testing: Why are SATs questions so hard?
Just like good teaching, good testing requires a strong theoretical understanding of learning and, especially in the case of SATs, how to objectively assess learning. With all this theory, it’s no wonder the SATs questions can seem so hard! Cue the four cognitive domains…
6. Alongside content domains, the test framework also sets out the four different cognitive domains that are crucial to how the government structures KS2 SATs.
In short, cognitive domains are the type of cognitive skills and abilities that each question is assessed against. And, interestingly, they are an often neglected part of the test framework when people create mock or sample papers, but they are an essential element of the test development process.
Get ready for 2018 KS2 assessments with this free set of 3 Maths SATs practice papers
The four cognitive domains are as following:
- The first cognitive domain is Depth of Understanding. This strand is used to assess the demand associated with recalling facts and using procedures to solve problems.
- The second cognitive domain is Computational Complexity, which is used to assess the computational demand of a question.
- The third cognitive domain, Spatial Reasoning and Data Interpretation, is used to assess the demand associated with the representation of geometrical problems involving shapes and position and movement. This strand is also used to assess the demand associated with interpreting data presented in tables, charts and graphs.
- The fourth cognitive domain is Response Strategy, which describes the demand associated with constructing a response to a question.
7. In the SATs papers, each question is scored against a 1-4 scale for each cognitive domain that it tests. In which one represents low (or none) and 4 represents high. A definition of what represents a low, intermediate and high level of ‘demand’ for each cognitive domain is provided in the test framework.
8. Questions which involve a higher level of content (i.e. from Year 5 or 6) often have cognitive domains that sit towards the lower end of each scale, or only have a high level of cognitive demand in one or two of the domains.
Usually, there are only a very small number of high content and high-cognitive domain questions (around 4-6 marks per year).
The Minds Behind the Maths: Who Writes SATs Questions
You might wonder how some SATs questions end up in the papers, but the road to creating a good SATs question is actually a long and intensive one!
9. As teachers, SATs can be an understandable source of frustration. When individual questions, or the mark schemes can feel bewilderingly unfair, it might feel like they were put together all too quickly.
— Debra Kidd (@debrakidd) July 11, 2017
An example of an “unfair” question, picked up by teachers lamenting the 2017 #SATsShambles
But actually each SATs paper starts its journey at least two years before it arrives at your school. It’s far from a simple “write a paper and release it” process.
10. Questions (or “items” in test development speak) are written by a number of “item writing agencies”, who use the test framework to write individual questions. These agencies include well-known names within the UK educational scene, as well as those which are better known in other countries.
11. Once received into the Standards and Testing Agency, these “items” are reviewed by a diligent team of civil servants who oversee the test development process.
(These civil servants often have an education background themselves, and have specialised in test development)
12. All questions are then entered onto a central database of “items”. This means that – if any are not used in the current test development process, they can still be called upon in future years.
13. The “items” that are selected as possible candidates for the papers are then taken to a series of expert review groups. There are several separate “groups”; including specialist teachers, academic experts and inclusion experts from various subjects.
The SATs Secret Society Behind the Test Development Process
Great assessment isn’t just about the strength of individual questions. Putting together a series of structured questions that pupils are required to answer in a set time, is a process in and of itself.
Stage 1 – SATs Question Level Review
14. The first meeting of the Expert Review Group for a given paper takes place over two years before the date of the assessment.
It goes without saying that these meeting are very strictly confidential. No notes can leave the room, and there is high security surrounding the meeting.
This also means that the test development process for at least two separate years of assessments takes place concurrently.
15. In this meeting, all of the items that have been selected for possible inclusion in the year are discussed over a two day period.
These are analysed in depth by the members of the group, and honest and open feedback given.
At least 150 reasoning paper items are presented at these initial meetings – more than three times the number needed.
16. The expert review groups can suggest many things, such as whether or not questions are suitable, or whether amendments to wording, contexts, illustrations etc, need to be made. They even pour over which words should be in bold!
There is often discussion and debate amongst the different group members. A discussion on any particular question can last between five to 30 minutes. So you can imagine how long it takes!
17. The inclusion group focus on the inclusivity of each question, considering a whole range of factors. They also discuss any adaptations needed for the modified versions of the papers.
18. Then all groups consider the context of any questions, ensuring that contexts are as realistic and recognisable to pupils as possible.
This goes somewhere to explain the sometimes limited range of contexts used, as there are surprisingly few contexts that are universally “real” to most 11 year olds.
19. Finally, the feedback from each different expert review group is collated and used by the STA staff to make any amendments to the questions, or to refer questions back to the item writing agency.
Stage 2 – SATs Sample Papers
20. Once the questions (or “items”) have been decided, they are formed to create a large number of sample papers.
These are put together in a way that ensures the test framework is rigorously followed and the appropriate mix of content and cognitive domains are covered across a “suite” of papers (i.e. Paper 1, 2 & 3).
21. Once the sample suite is assembled, they are given to Year 6 children to sample in schools across the UK.
Any UK state school could be selected to sample SATs papers and – if a school is selected – their Year 6 will be given at least one sample paper to be administered by an external agency, usually in June or July.
Schools cannot refuse involvement in this process and, accordingly, results from sampling are not given back to schools (nor linked to schools or pupils in any way!) Although teachers are usually asked to provide an assessment of those who sit the paper, as a baseline.
Note – There is no risk of your Year 6 actually sitting a sample question in their real SATs, as they are only exposed to questions that might be used two years later.
22. Interestingly, the sample papers are designed with a whole host of different variations. This means that things like the effect of question order can be tested and analysed.
23. After Year 6 have sat the sample papers, they are marked by a specialist team of markers. During this process the mark scheme and guidance is also reviewed and created where needed.
24. The markers flag any unexpected responses. For example, is a question “leading” children to give a certain incorrect answer? Incorrect answers like these are “coded” to describe the type of error.
25. Once a wide range of sample papers have been sat and marked, the results are statistically analysed in immense detail.
While the papers are analysed, many aspects of the data are weighed up and considered at this point, such as:
• Potential gender bias (more boys answering correctly than girls, or vice versa)
• The number of children getting the question correct; the impact of question placement
• The performance of high, medium and low ability children (as identified by their teachers)
26. During this process, the types of errors children have made on each question are also analysed heavily. To ensure that questions aren’t universally causing a certain error.
27. In questions with two or more marks, the number of children gaining partial marks (i.e. one mark from a two mark question) is also examined. This is to ensure that it’s possible both one and two marks. Not just the one partial mark.
28. Finally, once the in-depth analysis is complete, the questions are taken back to the second round of expert review groups.
Here, the amended questions (along with sampling results) are reviewed, and more suggestions are made.
We are now around one year away from the actual test date.
29. Finally, all “items” are amended. Any questions that are still not quite ready are deferred to a later year to be part of another sampling round.
The final countdown – the last stage of SATs paper creation
30. The six month countdown to the test date is now underway. Using the combined insight from sampling, expert review groups, and other stages in the test development process, the STA put together a final suite of SATs papers.
31. There is a lot of statistical analysis that goes into creating the final suites of papers. This is to make sure that they:
• Assess the full range of the test framework
• Have the correct breakdown of content areas (as given in the test framework)
• That the demand from the different cognitive domains are fair
32. Rather than creating three Papers, the STA actually create six (so two suites). One suite is kept as a backup, in case of leaks, issues in the testing process, or other emergencies. For each test year, 2 complete suites of papers are created.
33. These completed papers are then taken to a smaller, combined expert review group. Security around this group is, understandably, even tighter. Here, final tweaks and amendments are made to the tests and mark schemes.
Then the complex final sign off process is begins!
34. Once papers have had final sign off, they are securely printed, stored and then when the time comes, are sent out to schools. The backup set is also printed.
35. If there are any leaks, issues or concerns during the lead up to the test date, then the second set of papers would be sent to schools and used. However, this hasn’t happened in a very long time.
N.B to reader: Due to the need for security surrounding how National Assessment SATs Papers are created, we have omitted some information about this process.
So, there you have it. You now know about the secret life of SATs Papers. You can also see the hugely complex journey they have gone on before they get to you (and a bit more about what is done to ensure they’re as fair as possible!)
Interested in more reading on SATs, or free assessment resources? Click here for our list of the 15 Best Free KS2 SATs Papers & Revision Resources to Use in 2018.