Smarter Resolutions programme: Randomised Control Trial B (Workstreams 1 and 2: pre-notification content and form)

Background

The Advisory, Conciliation and Arbitration Service (Acas) provides free and impartial information and advice to employers and employees on all aspects of workplace relations and employment law.  

Employment tribunals make decisions about employment disputes. The voluntary early conciliation (EC) process has been designed so that Acas can provide an impartial conciliation service with the aim of parties reaching a settlement and avoiding the potentially time consuming, costly, and stressful employment tribunal process.

In 2021, the Ministry of Justice (MoJ) and the Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a 2-year period (2021 to 2023) from the HM Treasury (HMT) Shared Outcomes Fund. The funds were to work with Acas to test a series of early interventions in the employment dispute resolution system (the 'Smarter Resolutions' programme). The Smarter Resolutions programme comprises 4 operational workstreams that focus on improving different parts of the early conciliation (EC) process: 

•    workstream 1: content strategy 

•    workstream 2: assisted notification 

•    workstream 3: optimised distribution 

•    workstream 4: large group claims 

Each workstream explores how early interventions can help Acas to improve the service offered to customers, and to resolve disputes more quickly and effectively. This can prevent the escalation of issues, improve the experience of individuals and reduce the flow of cases into the employment tribunal (ET) system. It is expected that this will deliver cost savings across government and arm's length bodies such as Acas.  

This protocol details how we will assess the impact of 2 parts of the 4 workstreams: workstream 1 – pre-notification content and workstream 2 – assisted notification, using a Randomised Control Trial (RCT) approach.

A separate RCT, RCT A, which has no impact on this evaluation, has tested elements of workstream 1. RCT B immediately follows RCT A so the results for RCT A are not yet available. An evaluation of workstreams 3 and 4 will happen after this RCT (RCT B) is completed.

Population

The population for the trial is all claimants and representatives with an employment dispute who visit the www.acas.org.uk/tell webpage and click the 'make an individual claim' button between 24 October and 25 December. We estimate this to be around 12,500 cases. 

Claimants are individuals who make a claim to an employment tribunal if they think someone has treated them unlawfully, such as an employer, potential employer or a trade union. A representative is someone who acts on behalf of another individual who wishes to make a claim. Claimant representatives can be 'professionals' (for example, a union official, lawyer or someone from a law centre) or more informal, for example a friend or family member.

Intervention

New content that claimants and representatives can access before they make a claim ('pre-notification content') will provide information and context to help claimants and representatives have a better understanding of the next steps in the EC process, the time limit and what will be required from them after they agree to EC. This content includes new automated emails, information pages and explanatory videos.  

A new notification form ('assisted notification'), which claimants use to notify Acas that they intend to submit an employment tribunal claim, will minimise errors and incomplete responses and will ensure necessary information is collected from a claimant to determine whether a valid claim is captured. 

The new pre-notification content and assisted notification form will always appear to claimants and representatives together. For example, a claimant seeing new content would always see a new notification form.  

Acas expects that by providing people with the right information, in the right format and at the right time, they will reach better resolutions.

Objectives

The aim of RCT B is to test whether new pre-notification content, and the new assisted notification form, leads to improvements in the EC process, including a better understanding of the process, increased satisfaction with the process and more claims settled during EC. 

Method and design

The impact study is an experimental design with claimants and representatives allocated to intervention and control groups at random on a 1:1 basis, without any stratification or clustering in the randomisation. 

Acas will programme its webpage to automatically randomise claimants and representatives who click 'make an individual claim' on the www.acas.org.uk/tell webpage to either treatment pre-notification pages or control pre-notification pages. The treatment arm will experience the new pre-notification pages and the new assisted notification form. The control arm will experience the control pre-notification pages and the control notification form.  

RCT B will run for a period of 9 weeks, between 24 October 2022 and 25 December 2022. We estimate around 6,250 cases will be assigned to each arm of the trial. 

Outcomes are recorded by Dynamics 365 (Acas's case management system), Google Analytics, and 3 online surveys:  

• an exit survey completed by claimants and representatives when they leave the pre-notification pages 
• a survey when they submit their notification form 
• a survey 6 weeks after claimants notify  

If a claimant completes the full EC journey, they would fill in 3 separate surveys. 

The primary outcomes are: 

• improved application experience 
• more cases settled during the 6 weeks of EC 
• improved data quality to inform EC service design and delivery 

A process evaluation will complement the trial. We will conduct qualitative interviews with 56 individuals in the treatment group to explore the way in which the new content and form was used. We will also explore perceptions of how the new content and form impacted on their EC experience and decision-making.  

We will also conduct 2 focus groups with early conciliation support officers (ECSOs) to discuss their experiences of triaging submitted notification forms. The intervention implementation, mechanisms for change and assumptions will be tested through this analysis. See section 4 for full details of the process evaluation design. 

An evaluation report will be published in summer 2023, bringing together evidence from all 4 Smarter Resolution programme workstreams. 

About Acas

Acas provides free and impartial information and advice to employers and employees on all aspects of workplace relations and employment law.  

Employment tribunals make decisions about employment disputes. Nearly all legal cases about employment are heard in employment tribunals. This includes cases about unfair dismissal, redundancy and discrimination. There are also many other sorts of claim that can be brought, such as non-payment of wages.  

Acas has a long-standing statutory duty to promote the resolution of claims before the employment tribunal (ET) to avoid recourse to a full hearing, by means of its conciliation service. In April 2014, a new legal requirement was introduced that means individuals must notify Acas of their intention to make an ET claim.  

Since this change in the law, Acas has offered the opportunity for both parties involved in the dispute (the individual and the employer) to resolve the issue through a new voluntary early conciliation (EC) process. EC notifications can be made by an individual against their employer or on behalf of a group against their employer. 

The EC process has been designed so that Acas can provide an impartial conciliation service with the aim of parties reaching a settlement and avoiding the potentially time consuming, costly and stressful employment tribunal process. Cases progressing through EC will be assigned an impartial conciliator who may: 

• explain the conciliation process 
• discuss the issues with both sides 
• give an overview of the law 
• discuss how employment tribunals have considered similar cases 
• help both sides explore the strengths and weaknesses of the case 
• discuss possible options without making any recommendations 
• remain independent of the dispute – they will not take sides or tell either party what to do  
• outline the employment tribunal process 
• record a binding settlement of the legal dispute in the form of a COT3 agreement so that the case does not need to proceed to a hearing 

If the offer of EC is refused or fails to reach agreement, the individual can still pursue an ET claim. In this case, the respondent usually has to reply to a claim in writing within 28 days of getting the claim form. Once they have replied, the tribunal will decide whether there will be a full hearing.  

These cases are normally held at the employment tribunal office closest to where the claimant lives. All tribunal hearings will be heard by a judge and the respondent, and depending on the type of case, there may be 2 other tribunal members.

Background to the Smarter Resolutions programme

In 2021, the Ministry of Justice (MoJ) and the Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a 2-year period (2021 to 2023) from the HMT Shared Outcomes Fund, to work with Acas to test a series of early interventions in the employment dispute resolution system (the 'Smarter Resolutions' programme). 

The Shared Outcomes Fund incentivises and funds innovative pilots, with departments working collaboratively to improve cross-governmental outcomes. It places a high value on robust, high quality impact evaluation and sharing results. More information about the Shared Outcomes Fund is available in section 5. 

The Smarter Resolutions programme comprises 4 operational workstreams that focus on different parts of the EC service, which is Acas's service for resolving individual disputes. Each workstream explores how early interventions can help Acas to improve the service offered to customers, and to resolve disputes more quickly and effectively. This can prevent the escalation of issues, improve the experience of individuals and reduce the flow of cases into the employment tribunal (ET) system.  

It is expected that this will deliver cost savings across government and arm's length bodies such as Acas. The 4 workstreams and workstream visions are:  

• workstream 1: content strategy. Users grasp basic principles of employment law related to their dispute, the most appropriate way to resolve it, and have clear expectations of where Acas can help
• workstream 2: assisted notification. Users are guided to supply information that conciliators need to provide the best possible service at the earliest point
• workstream 3: optimised distribution. Cases are directed to the most appropriate and available conciliator to quickly and efficiently resolve their dispute
• workstream 4: large group claims. Users who belong to the same workplace dispute are quickly brought to the attention of a team with the expertise, tools and resources to resolve their dispute efficiently

The overall programme vision is that people involved in a workplace dispute have a clearer understanding of how they can reach a quicker, more cost-effective resolution at the earliest point possible.

Figure 1: Smarter Resolutions programme vision

This trial protocol relates to assessing the impact of workstreams 1 and 2.  

Workstream 1 is developing several new forms of content, to provide potential claimants or their representatives with information that helps them better understand the dispute resolution process, as it applies to them at various points in their journey. The underlying principle is that users need to see 'the right content delivered at the right time'. The content aims to ensure claimants, or their representatives, have a holistic understanding of the conciliation journey, in order for them to make informed choices about their next steps and have realistic expectations about the dispute resolution process. 

Workstream 1 content is wide ranging and will be embedded across 4 points of the user journey. Three of these points are before a claim is made and one point is after a claim has been submitted. The 4 stages in order and grouped by time point are: 

Pre-notification content (information given before a claim is made): 

• "Start to find help" content, delivered across the Acas website, to aid users when they are at the early information-gathering phase of their dispute journey. This includes the: 1) 'emotional support' page, 2) 'time limits' page. This is included in this trial. 
• "Intend to notify start page" content, embedded in the www.acas.org.uk/tell webpage. The content includes information about the purpose of the service, instructions on how to return to the form once it is started, information on what information the user will need to share to start a claim, information on what the user will need to share if they want to try EC and other ways to notify Acas (for example, helpline and by post).  This content is designed for claimants to engage with at the point of initially arriving on the form, with the intention of notifying Acas about their ET claim. It is centralised on this page (formerly, it was spread out across multiple webpages), and is a simplified version of previously used content. This is included in this trial.
• "Complete the form" content embedded throughout the notification form itself and content sent to users after their notification has been received, outlining what to expect next. This involves an automated 'you've notified' email once the form has been submitted and an automated 'introduction to claimant or respondent' email once a case has been allocated to a conciliator, which includes a link to an 'intro to conciliation' video. This is included in this trial. 

Post-notification content (information given to claimants after they have submitted their claim): 

• "Conciliate" content, also known as 'post-notification content', provided by the conciliator to users once they have reached the stage of EC. Conciliators will have access to a series of materials they can use at their discretion, including template emails and links to wider resources. The wider resources include the: 1) 'emotional support' page, 2) 'time limits' page, and 3) 'wages' video, mentioned above. This is not included in this trial because it was tested in RCT A. 

The workstream 1 content was initially developed during late 2021 and early 2022 by an external content strategist. The "conciliate", post-notification content was tested in RCT A in July 2022 and is therefore excluded from RCT B.

RCT B immediately follows RCT A, so the results of RCT A are not yet available. However, the post-notification content that was tested in RCT A will be implemented and become the new 'business as usual' for post-notification content in RCT B.  

The remaining content is going through beta testing from the end of August to September 2022, before RCT B begins.  

Workstream 2 focuses on improving the notification form to minimise errors and incomplete responses; and to ensure necessary information is captured to determine whether it is a valid claim. Incomplete data is an issue for Acas because it requires Acas to manually adjust case information before a case is ready to be allocated to the most appropriate conciliator. This can delay the start of conciliation by a week or more in some circumstances.  

There are 2 existing 'legacy' forms. Form 1 is the information mandatory for a notification (and some limited voluntary contact information). Form 2 contains the optional information on the background of the claim. 

RCT B will trial a new single form known as the 'assisted notification' form. This form will have the following new features: 

• simple explanations and signposting to relevant resources within the form 
• progressive disclosure and channelling – progressive disclosure is an interaction design pattern that sequences information and actions across several screens (for example, a step-by-step signup flow). The new format allows users to click on a question and takes them to a new page containing the question and associated help text, instead of seeing long single pages asking for multiple inputs of information. It is used to make applications less error-prone. Channelling refers to the process of only showing notification form questions or sections that are relevant to the claimant or representative, based on their responses to previous questions or sections 
• ability to save and return to the form 
• the ability to share information at more points to help give conciliators relevant case information 

RCT B will focus on workstream 1 pre-notification content, and the workstream 2 assisted notification form. The pre-notification content includes the 'emotional support' and 'time limits' pages.  

RCT B starts when visitors to the www.acas.org.uk/tell page are randomly assigned to either a control or treatment group. The pre-notification journey for the control group is to visit 5 pages, each about EC. If they decide to start a claim – the notification journey – they either complete legacy form 1 or legacy form 1 and 2. Their post-notification journey involves submitting the form and receiving an automated email with video content about conciliation. RCT B ends at the point of this email.  

The pre-notification journey for the treatment group is to possibly visit 3 pages – one about emotional support, one about time limits and one with a video about wages. Or, they can go straight to the 'start claim' page, which has been simplified, with information centralised to the one page compared to the control group. The notification journey is after they start the claim and begin the new assisted notification form. The form includes the new features outlined above. 

Their post-notification journey is the same as the control group: it involves submitting the form and receiving an automated email with video content about conciliation. RCT B ends at the point of this email. The diagram below provides a summary as to where RCT B sits in the user journey.

Figure 2: RCT B user journey

Figure 3 below is the RCT B logic model. The model captures our understanding of the workstream 1 and 2 activities relevant for RCT B, including our assumptions on how the programme will operate and how claimants and representatives will experience it. It illustrates the mechanisms for change and how programme activities are translated into impacts.  

The rationale for workstreams 1 and 2 is that if employees and their representatives: 

• receive information about the early conciliation (EC) process earlier, and relevant to their stage in the conciliation journey, they are more likely to submit a valid claim and be more satisfied with the notification process 
• are digitally supported through the notification form they will submit complete and accurate information, and valid cases will require less input from early conciliation support officers (ECSOs) to triage  

The assumptions underpinning the RCT B logic model are: 

• most users start their EC journey at www.acas.org.uk/tell landing page ('Find a solution to an employment dispute'). Analysis data from Google analytics for March 2022 shows 35,220 users entered the service on this page (Google Analytics TEST Tell Acas live website Goals Goal funnel)  
• notification form changes do not introduce new issues to the completion of the form or to the conciliation process 
• most users have the capacity to grasp basic principles of employment law, the best way to resolve their dispute and have clear expectations of where Acas can help 
• most users have the capabilities to engage with the digitally-hosted information and notification form 
• most claims close by 6 weeks after notification is submitted 

Acas's workstream 1 and 2 teams worked with contractors, including content strategists, user designers and service designers (inputs). Together, they identified 4 user journeys that start at the www.acas.org.uk/tell webpage and conducted disabled user testing to achieve a disability access certificate (activities).  

The result of those activities was the development of a suite of outputs: 

• 2 new www.acas.org.uk/tell webpages, about emotional support and time limits for EC 
• a video discussing what wages include or do not include, and the legal reasons an employer may take money from an employee's wages 
• an automated email that confirms people have submitted a claim and signposts them to an 'Introduction to conciliation' video 

Acas's workstream 2 team and conciliators worked with contractors, including Deloitte, and user and service designers (inputs). Together, they identified that more information was required for the notification form to capture whether the claim was valid and identified claim issues preventing form acceptance or conciliation (activities).  

The result of these activities was the development of a new assisted notification form with the following key features (outputs):  

• simple explanations and signposting to relevant resources within the new assisted notification form 
• progressive disclosure and channelling 
• the ability to save and return to the form 
• more space for claimants or representatives to provide more detailed case information  

The above combined outputs of both workstream 1 and 2 teams, and their contractors, are expected to lead to short-term outcomes – at the point of submitting a claim – for claimants and representatives. These short-term outcomes are: 

• claimants and representatives understand the EC timetable 
• claimants and representatives understand if the claim is valid or invalid 
• claimants and representatives have exhausted the internal appeal processes 
• claimants and representatives feel less stressed by the EC process 
• claimants and representatives can access the right information at the right time 

Acas hypothesises that if these short-term outcomes are achieved, claimants and representatives will benefit in other ways, within 6 weeks of submitting a claim. These mid-term outcomes are: 

• claimants and representatives feel better informed about the EC process 
• claimants and representatives submit fewer invalid claims 
• claimants and representatives feel satisfied with the notification form process 

Ultimately, and by 6 weeks of submitting a claim, claimants and representatives are expected to have an improved application experience. 

In parallel to the above work undertaken by Acas workstream teams and contractors, Acas's digital team (input) designed data infrastructure for case allocation processing (activities). This activity is intended to lead to changes in the infrastructure in which the notification form data is held and used for case allocation processing.  

The assisted notification form, alongside the form data infrastructure, is expected to lead to the following short-term outcomes for the Acas EC service: 

• improved digital take-up of the form 
• notification form information is complete and accurate 

If these short-term outcomes are realised then less time is required from EC support officers to triage cases, and less time is required between notification and first contact with a conciliator.  

Ultimately, and by 6 weeks of submitting a claim, the EC service is expected to see more cases settled in the early conciliation window, and improved data quality to inform its service design and delivery.

Figure 3 RCT B content and notification form logic model

Note: the mechanisms refer to mechanisms of change. These are the point of connection between specific activities and outcomes that Acas believes will bring about the outcomes. The hypothesis is that these will bring about specific outcomes and the absence of them, or different mechanisms, will not bring about the intended outcomes. In the case of RCT B, the mechanism for change is the features of the notification form:  

• simple explanations and signposting to relevant resources enable users to understand the EC timetable and if their claim is valid  
• progressive disclosure and channelling enable users to input complete and accurate information, complete the form digitally (instead of by paper) and to access the right information at the right time 
• the ability to save and return to the form enables users to input complete and accurate information, complete the form digitally (instead of by paper) and to access the right information at the right time 
• more space for more detailed case information enables users to input complete and accurate information

RCT B objectives

The primary research question to be addressed in the RCT B impact evaluation is: 

What is the difference between the average number of cases settled within 6 weeks of notification submission and of the number of invalid notifications (outcome), for claimants and representatives accessing the form (population), randomised to receive the new pre-notification content and assisted notification form (intervention), compared with claimants and representatives randomised to the ‘legacy’ pre-notification content and notification form (business as usual; control)?  

The secondary research questions are: 

1. What is the difference in the average perceived satisfaction with the pre-notification content, and notification form process (outcome) reported by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to the 'legacy' pre-notification content and notification form (business as usual; control)? 

2. What is the difference in the average accuracy and completeness of notification forms (outcome), for claimants and representatives completing the form (population), randomised to receive the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to the 'legacy' pre-notification content and notification form (business as usual; control)? 

3. What is the difference in the perceived knowledge of the EC timetable (outcome) reported by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

4. What is the difference in the perceived knowledge of whether their claim is valid or invalid (outcome) reported by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

5. What is the difference in the belief that internal appeal processes have been exhausted (outcome) reported by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

6. What is the difference in the levels of stress about the EC process (outcome) reported by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

7. What is the difference in the belief that the right information about the EC process was accessed at the right time (outcome) reported by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

8. What is the difference in digital take-up of the notification form (outcome) by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

9. What is the difference in the average completeness of information in the notification form (outcome) by claimants and representatives (population), and that are randomised to the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)? 

10. What is the difference in the average time spent by ECSOs after forms are submitted to triage cases before being allocated to a conciliator (outcome), between claimants and representatives accessing the notification form (population), randomised to receive the new pre-notification content and assisted notification form (intervention), compared to claimants and representatives randomised to control conditions (business as usual; control)?  

The trial will be complemented by a parallel-running, process evaluation. This is discussed in more detail in section 4. 

Trial design

Randomised Control Trial B (RCT B) focuses on measuring the impact of new pre-notification content and an assisted notification form on claimants making a notification, including represented and non-represented claimants, and professional and non-professional representatives. 

Acas will use Google Analytics to set up an A/B test whereby people who access the www.acas.org.uk/tell webpages will automatically be directed to either treatment pre-notification pages or control pre-notification pages. The treatment arm will experience the new pre-notification pages and the new assisted notification form. The control arm will experience control pre-notification pages and the control notification form. Each arm will access the same post-notification content which has been tested in RCT A and become the new 'business as usual' by RCT B.  

Outcomes are recorded by Dynamics 365 (Acas's case management system), online exit surveys completed by claimants and representatives when they leave the pre-notification pages, when they submit their notification form or an online survey 6 weeks after the claimants notify; and Google Analytics.  

The impact study is an experimental design with claimants and representatives allocated to intervention and control at random on a 1:1 basis, without any stratification or clustering in the randomisation.

Trial period

RCT B will run for a period of 9 weeks, running between 24 October 2022 and 25 December 2022. This is the longest it could be run within the parameters of the Smarter Resolutions programme implementation plan (this timeframe allowed the notification form to be tested, re-prescribed by the Government Digital Service then refined, before being tested through the RCT) and the evaluation timescales, and to allow for sequential evaluations of other programme elements (discussed in accompanying trial protocols). 

Participants

The participants in this trial are claimants and representatives who intend to notify Acas of an employment tribunal (ET) claim. All individuals that enter the www.acas.org.uk/tell subdomain and click the 'make an individual claim' button during the trial period will be in scope to be randomised into the treatment or control group. Some individuals may enter the www.acas.org.uk/tell subdomain directly from links on the main Acas website; some will enter from external links and signposting – they will not form part of the trial.  

Respondents – employers that claims are made against – are a relevant audience for the trial because any outcome related to the new form affecting claimants will likely have a knock-on effect to respondents. Respondents are not a primary audience the interventions hope to bring about change to, or the trial, so are not included in the design.

Outcomes

Acas expects new pre-notification content and the assisted notification form will lead to the short-term outcomes, which in turn will result in longer-term impacts. These outcomes and impacts, and the associated indicators for measuring them are summarised in the evaluation framework below.  

We will be able to link the trial arms to both the case management system (CMS) and online survey data. 

Table 1: RCT B evaluation framework

Sample size

In this section, we estimate the precision of inferences that we expect to achieve with different sample sizes. The sample size calculation requires assumptions that typically cannot be rigorously tested until the data has been collected. However, using assumptions based on past experience is a crucial part of design, establishing the level of confidence that could be associated with empirical work. 

The sampling frame for RCT B are the claimants and representatives who visit the www.acas.org.uk/tell landing page between 24 October and 25 December 2022. 

On average, Acas receives 5,500 notifications from employees about employment disputes that are appropriate to go to conciliators each month. Each of these 5,500 disputes may relate to multiple individuals. The trial will run for 9 weeks, thus the estimated sample size is around 12,500 cases, of which: 

• 6,250 will be placed in the control group 
• 6,250 will be placed in the treatment group 

One of the primary outcomes for RCT B is the proportion of cases settled within 6 weeks of notification submission. According to Ipsos MORI (2020), in the evaluation of Acas Individual Conciliation 2019 conciliation had taken place in around half of all cases, and this is the assumed control-group rate for power calculations. 

Power analyses are useful in giving a sense of the size of the effects one could reasonably expect to demonstrate with a study of a given size. We use a 2-sample proportion test with the assumptions of significance level of 5%, power of 80%, and balance or equal-allocation design (same number of individuals in the control and experimental group). The test also assumes that the experimental-group proportion is larger than the specified control-group value. We used the large-sample Pearson's and likelihood-ratio tests commonly used to test hypotheses about 2 independent proportions. Additionally, we calculate the Fisher's exact test commonly used to compare the two proportions in small samples. We have performed all the calculations with the 3 tests and the results are consistent. 

Table 2 presents the effect size for different sample sizes using the control group proportion of 50%; delta gives us the difference between proportions, that is the minimum detectable effect. Thus, if we run RCT B for 9 weeks to obtain 80% power for a 95% confidence interval, the true effect size must be at least 2.52%.  

Table 2: Power calculations for RCT B

2-sample proportion test: Pearson's chi-squared test

Leaving pre-notification pages pop-up online survey

The pop-up exit survey relies on people agreeing to take part in the research. This is an implied consent approach. When a person leaves either the control or treatment pre-notification pages they will see a pop-up window inviting them to respond to 6 closed questions. For respondents on the treatment pre-notification pages, they will also be asked if they would be willing to take part in a more in-depth follow-up interview as part of the evaluation. The consent to recontact is also asked of respondents in the treatment arm who submit a claim.  

Line of questioning: 

1. To what extent do you agree or disagree with the statement: "I understand the timetable for accessing the early conciliation process" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

2. To what extent do you agree or disagree with the statement: "I know what to expect from the early conciliation process" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

3. Have you completed your organisation's disciplinary or grievance process? – Yes, No, I am currently completing it, do not know 

4. Do you intend to submit a claim? – Yes, No, I do not know 

a. If no, choose the reason why – I realised my claim was invalid; I will try my employer's disciplinary or grievance process first 

5. What is the nature of the claim? Note – this is because we may find there is naturally occurring (unrelated to the intervention) attrition in different jurisdictions or the new material in the treatment improves due to the nature of the claim. 

6. Would you be willing to do a follow-up interview with one of our trained researchers? This will help us understand more about your experience and what improvements could help people like you. – Yes, No 

a. If yes – What is the best phone number to call you on to schedule this interview? 

In line with pop up surveys from Acas's Digital Advice Evaluation 2021, the pop-up will be managed using Hotjar and implemented on the relevant webpages by the Acas Digital team and displayed as a full-screen survey. This removes one barrier to completion: individuals do not have to click through to a survey hosted on another page – the questions are embedded in the pop-up. This screen approach also helps ensure we meet accessibility guidelines (lessons learned from Acas's Digital Advice Evaluation 2021 study). The pop-ups will appear when they leave a page.  

Based on the 2021 evaluation mentioned above, we expect a 1% completion rate. In the 2021 evaluation, it was estimated that around 52,000 users visit Acas advice pages each week – in total around 468,000 over a 9-week period. Based on this experience, our expected response rate for the treatment and control groups is around 2,340 each. 

The data collected by the pop-up survey will be securely transferred by Acas to IFF in batches for analysis, and for contacting those individuals who have opted in to the follow-up interview.

Notification form submission online exit survey

The notification form exit survey relies on people agreeing to take part in the research. When a person (both treatment and control) submits their notification form and exits the form page they will receive a pop-up survey.  

Line of questioning: 

1. To what extent do you agree or disagree with the statement: "I am confident my claim will be accepted for EC" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

2. To what extent do you agree or disagree with the statement: "I know what to expect from the early conciliation process" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

3. During the early conciliation process, did you see the advice on the Acas website about 'getting emotional support'?  

4. During the early conciliation process, did you see the advice on the Acas website about 'time limits'? 

5. To what extent do you agree or disagree with the statement: "I got the right information at the right time when completing the notification form" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

6. To what extent do you agree or disagree with the statement: "I found the notification form easy to complete" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

a. If 'Strongly disagree', or 'tend to disagree', what reasons: progressive disclosure or channelling, save and return, embedded information, other 

7. What is the nature of the claim? Note – this is because we may find there is a systemic attrition in different jurisdictions or the new material in the treatment improves

On average, Acas receives 5,500 notifications from employees about employment disputes that are appropriate to go to conciliators each month. Each of these 5,500 disputes may relate to multiple individuals. The trial will run for 9 weeks thus the estimated sample size is around 12,500 cases. Based on the 2021 evaluation described in the previous section, we expect a 1% completion rate. Our expected response rate for each treatment and control is around 1,250. 

Claimant and representative follow-up online survey

The claimant and representative survey relies on people agreeing to take part in the research. After the 6-week period (after submitting the notification form) is reached we will send an email invite and up to 3 reminders to maximise response to the online survey. This will be followed by a period of telephone calling to further boost response. 

Line of questioning:

1. Have you completed the early conciliation process? – yes/no  

2. To what extent do you agree or disagree with the statement: "I am confident my claim will progress to early conciliation" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

3. To what extent do you agree or disagree with the statement: "I understand the stages involved in the early conciliation process" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

4. During the early conciliation process, did you see the advice on the Acas website about 'getting emotional support'?  

a. If yes – at what point did you view this? 

b. How useful did you find this? 1– not useful at all – 5 very useful 

5. During the early conciliation process, did you see the advice on the Acas website about 'time limits'? 

a. If yes – at what point did you view this? 

b. How useful did you find this? 1– not useful at all – 5 very useful 

6. Before you submitted your notification form, had you engaged with your company's disciplinary and grievances process first? – yes/no  

7. To what extent do you agree or disagree with the statement: "I got the right information at the right time when completing the notification form" – Strongly agree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know  

8. To what extent do you agree or disagree with the statement: "I found the notification form easy to complete." – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

a. If 'Strongly disagree', or 'tend to disagree', what reasons: progressive disclosure or channelling, save and return, embedded information, other 

9. Before you started to complete your notification form, did you have further questions about the early conciliation process that were not answered? – yes/no 

a. What further questions did you have about the early conciliation process? 

10. Before you began the notification form, to what extent did you feel stressed and worried about the early conciliation process (1– Not worried and stressed at all – 5 very worried and stressed) 

11. After you submitted the notification form, to what extent did you feel stressed and worried about the early conciliation process (1– Not worried and stressed at all – 5 very worried and stressed) 

12. After you submitted your notification form, did you have further questions about the early conciliation process that were not answered – yes/no 

a. What further questions did you have about the early conciliation process? 

13. After you submitted your form, did you receive an email with the subject line 'You've notified' with a link to a video called 'Intro to conciliation'? 

a. If yes – Did you watch the video?  

b. How useful did you find this – 1 – not useful at all – 5 very useful 

14. Thinking about your experience overall, how satisfied or dissatisfied are you with Acas's early conciliation service? – 1 very dissatisfied – 5 very satisfied 

15. To what extent do you agree or disagree with the statement: "I got the right information at the right time when completing the notification form" – Strongly disagree, tend to disagree, neither agree nor disagree, tend to agree, strongly agree, do not know 

We are anticipating an overall response rate of 30% (19% achieved from an initial email invites and reminders, and a further 15% achieved through telephone chasing). Our assumptions are based on a survey conducted in 2021 evaluating Acas's digital advice (46% response rate), a previous evaluation of EC users (33% response rate) and current fieldwork on RCT A (30%). This includes an assumption that attrition is random and balanced across both arms. The survey response rate assumptions are set out in the table below. 

Table 3: Survey response rate assumptions

An achieved sample of 1,884 at both control and intervention groups means we will need to see a difference of between 1.4 and 3.2 percentage points between the 2 groups (at the 95% confidence level) for the difference to be statistically significant. The confidence intervals would be higher and the difference we would need to see for a finding to be significant would need to be higher if the response rate, and achieved base size, was lower. 

As claimants and representatives are equally likely to be randomised to either treatment or control group, the non-response to the survey should be the same in both arms and should not introduce bias. However, we will investigate the need for non-response weights after a review of overall response. 

We will also check that the mode of completion and date since EC had begun do not have an impact on responses (for example, some people may complete the survey between 6 and 9 weeks after notifying Acas). 

Randomisation

The randomisation will occur at the www.acas.org.uk/tell landing page at the point of an individual clicking the 'make individual claim' button because an algorithm allocates the individual (claimants and representatives), randomly dividing into 2 subsamples. All members of the first subsample will be assigned to the treatment group and all the members of the second subsample will be assigned to the control group. Randomisation will be tested before the trial goes live to ensure the algorithm to randomise the population is working as intended. 

The randomisation will be based on the landing on this page, and not seek to take account of any sub-groups within those landing. This is because – at this stage – there is no data about the nature of the claim or claimant, and there is no specific methodological reason to use a more sophisticated randomisation process. 

We are interested in the average treatment effect (ATE) of the intervention, considering that we will know which customers view the reworked www.acas.org.uk/tell pages and assisted notification form, and which customers view the current www.acas.org.uk/tell pages and legacy notification form. 

Attrition is the absence of data from participants, and it can present a grave threat to the internal validity of the experiment because missing outcomes effectively 'un-randomize' the assignment of subjects. In the case of the data collected by the web stats or CMS, the risk of losing data is very low because the information is captured by the system.  

The attrition in data collected outside automated systems, such as surveys, is unlikely to have the different rates across the 2 arms of the RCT, or at least any bias in the attrition (non-response to surveys) will affect subgroups similarly across arms. Data collection will allow some testing of this based on completed surveys.

Sequence

RCT B will be a first stage RCT and there will not have been any prior allocation stage by this RCT.

Case allocation

The processes after the randomisation are automated with the claimant or representative interacting online with the 2 (treatment and control) www.acas.org.uk/tell pages. This means – once allocation to the treatment or control group has occurred – there is no further Acas operational input into the allocation of cases to trial and control group for the duration of this RCT.  

After allocation, the interaction of the claimant or representative with the website will be tracked by web stats but anonymously. The claim type – whether in the treatment or control group – will only be attached to the record at the time the notification form is submitted. There will only be data on claim type for those claimants and representatives leaving the webpages without making a notification if they choose to respond to pop-up exit surveys. 

There is an important complication to case allocation. In a small number of cases, users could find their way to the notification form without being randomised. Workstream 1 'emotional support' and 'time limits' webpages will be available to all the claimants and representatives. Acas analysis of service user pathways to the notification form suggests 76% of users enter the service through the 'intent to notify' page (https://tell.acas.org.uk/check-user-wants-conciliation).  

So, the vast majority of claimants will use the www.acas.org.uk/tell route to make a notification and can be randomised at this point. However, there is a possibility that – because some users will have visited the site before the trial – they will have and use direct links to the 'time limits' page and 'emotional support' page, bypassing the randomisation. This is expected to be most likely for representatives, who would have used these pages in interactions with the new content developed prior to this trial and – in their use of this material – have bookmarked it, have cookies aiding their web use, or have a browser that will 'remember' the links. 

We are taking steps to ensure that this route into the treatment group bypassing the randomisation can be reduced: 

• Acas will make sure the new pages are not searchable on the Acas website or through search engines 
• for the duration of the trial, any visitors who go directly to the 'time limits' and 'emotional support' pages, will be automatically redirected to the 'intend to notify' page, for randomisation  
• Acas will take steps in the post-RCT data collection period to alert us to those that have received treatment but have not been randomised into treatment – for example, in the follow-up survey, we will ask users if they went directly to those pages at the start of their journey, and we will explore this in qualitative interviews with users in the process evaluation

Implementation

Before the trial we will check: 

• sample selection and recruitment 
• randomisation 
• secure data capture procedures 

Appropriate ethical framework 

For the ethical framework, at this stage, the National Statistics Data Ethics guidance is used as an overarching framework. There is a public good to the study, in that evaluating the effectiveness of improvements to the conciliation processes will ensure better service delivery and value for money. That benefit will be to a wide set of the UK population, in that the Acas remit of all employees and employers does not target a sub-population.  

Data handling and processing will be undertaken so that the data subject's identity (whether person or organisation) is protected, information is kept confidential and secure, and the issue of consent is considered appropriately.  

The development of this RCT has carefully-assessed risks and limits of new technologies. The automation associated with the online activities is also anonymised, in that the platform will not track the individual's use of the site in an identifiable way. The analytics associated with the site would provide statistics about use and the pathways of users in the site. This provides a degree of privacy about the way an individual interacts with the site, having been allocated to treatment and control. The fact of treatment or control is the only case-specific data that will be received from the site. 

This treatment data will then be linked to the surveys and the transactional data about the case (for example, time taken to settle, nature of case). The data used is personal but not highly sensitive, focusing mostly on the process of conciliation in terms of its efficiency and participant views. The publication of the RCT protocol, alongside other research dissemination, will ensure that the views of the public are considered proportionally in light of the data used and the perceived benefits of the research. 

During the trial we will check effective compliance and attrition reduction procedures. 

The study is grounded in the administrative data associated with Acas processes. Overall, administration data linked to surveys provides a robust evidence base for impact analysis because attrition is not present in the administrative data; and the linked dataset provides many opportunities to identify and correct for attrition in the surveys, where considered a problem.

Blinding

The allocation rule will be blind for the claimant or representative but the fact of the intervention itself will be known to the claimant or representative. Representatives involved in more than one case are unlikely to receive both treatment and control arm because we will use cookies to ensure they are in the same group after the initial randomisation.  

We can also check for this among submitted claims, in the CMS. There is a risk that independent participants might interact and have received a different treatment, but this should be small. This contamination could be greater in the large claims. Delivering the intervention will be blind, in that it is through the Acas website online and entirely automated. 

Statistical methods

In the context of an experimental estimation, whether to include control variables or covariates depends on the success of the randomisation.  

If the randomisation is successful, those receiving the treatment differ from those that did not only in the fact of receiving the treatment. Randomisation should mean all other factors (both observed and unobserved) are identical. This means the first analysis of results would be a simple difference-in-means estimation, asking whether the treated reached better outcomes than the those in the control. 

Some remaining imbalances in observed and unobserved variables are possible even after randomisation: the cases may not prove to be in exactly the same mix of jurisdictions, type of claimant or employer type across the 2 arms of the trial. It is then possible that the control variables explain a lot of the outcome, and their inclusion could reduce residual variance and raise precision (reducing standard errors) of estimated treatment effects.  

In the case of RCT B, data collection will focus on the reasons for entering the www.acas.org.uk/tell area without pursuing a claim, that is the attrition. This is the focus because the randomisation should provide a balanced sample for each arm, treatment and control, bar this attrition. Evidence will be gathered at this exit point, but this will be through a short survey and so data may be limited. A second analysis will focus on the cases that do notify, considering the customer (claimant or representative), nature of claim, region where the company is located, sector of the company, the length of service to the company and age of the individual. 

There are then some uses for these covariates in ameliorating the attrition issue: 

1. Analyse attrition as an outcome, that is the treatment affects the chance of making a claim and this differentially affects different types of claimants, claims and whether a representative is used. 

2. Adjust other outcome analyses to then ensure treatment and control comparisons acknowledge differential attrition. 

3. If the randomisation is compromised in some other way, adjusting for covariate differences may remove any resulting biases.

Risks

Table 4: Risks and mitigating actions

The process evaluation will assess how the intervention has been implemented and whether the elements of the theory of change underpinning the programme worked as intended. It will also explore perceptions of outcomes among claimants and representatives through in-depth interviews, and among ECSOs through focus groups.  

Research questions include: 

• Have claimants and representatives seen pre-notification, notification and post-notification content as intended?  
• How have claimants and representatives experienced the pre-notification, notification and post-notification process? 
• What difference is the new process perceived to make to claimants and representatives, and why? 
• What difference is the new process perceived to make to ECSOs, and why? 
• When and for whom is the process perceived to be most effective? 

Sample

All claimants and representatives in the treatment arm who both decide to notify and who decide not to notify Acas of an employment tribunal (ET) claim between 24 October 2022 and 25 December 2022, and who consent to be recontacted, will be eligible for qualitative research. Table 5 illustrates our purposive qualitative sample frame. 

Purposive sampling is a kind of technique for developing a sampling frame for non-probability methods of data collection, like the qualitative research used here. Purposive samples are chosen based on specific criteria; instead of randomly selecting them from a group. The goal is for the achieved sample to represent the widest range of views and experiences, rather than to quantify the views. The criteria we’ve chosen (summarised in table 5) are what we think are the main influencers on claimant and representatives’ experiences of the pre-notification and notification content.

Table 5: Treatment arm claimant and representative qualitative interview sample frame

Across interviews, monitor: 

• whether they have completed a notification before
• track type
• whether they saw 'emotional support' page, 'time limits' page and 'wages' video

We will also conduct 2 focus groups with ECSOs, with up to 6 staff per group. 

Recruitment

Sample for claimants and representatives will come from a 'consent to recontact' question in the pop-up and follow-up surveys conducted as part of the RCT. IFF's in-house recruitment team will contact participants by email or phone, inviting them to take part in an interview. 

They will ask participants to complete the screener questionnaire, answer any questions and where participants consent to take part, schedule an interview. The screener questionnaire ensures we include the right mix of participants.  

For example, it will include: 

• checking we are speaking to the right person in the sample 
• core questions to ensure a spread of participants as captured in the purposive sample frame above (Table 5)  
• secondary questions to monitor for a spread of things we think may influence experiences – for example, whether they have completed a notification before, their track type, whether they saw the 'emotional support' page or 'time limits' page  
• access questions, for example, what would help them to take part in an interview  

After an interview is scheduled, the recruiter will circulate a confirmation email with an information leaflet attached and link to the privacy notice. 

Sample for the 2 ECSO focus groups will come from Acas.

Fieldwork

Interviews with claimants and representatives will be conducted online using Zoom or Microsoft Teams, or by telephone, depending on the individual’s preference. Interviews will last up to 45 minutes and participants will receive a £40 voucher after the interview.  

Researchers will facilitate the discussions, supported by semi-structured topic guides and stimulus, and using journey mapping techniques to aid recall. The discussions will broadly cover (tailored by audience): 

• background and context – warm-up, whether they have used EC before or been to a tribunal, confidence with digital services, expectations of EC process 
• overall experience of the notification process 

Using a journey mapping approach, explore the following for each of pre-notification, notification, and post-notification stages, including how they felt throughout: 

• ease, accessibility, time taken, steps taken (for example, if they go away or come back), whether or what support was accessed (for example, from Acas, friend or family), effectiveness of content or elements to deciding whether to notify and supporting accurate or successful completion for those who do notify 
• barriers to engagement – for example, features of pages and form, personal circumstances and capabilities, any gaps in form undermining documenting complete picture of claimants' circumstances 
• overcoming barriers – for example, steps considered or completed; what works well or less well for overcoming barriers; suggestions for improving content and form to make a decision or capture of quality evidence 
• how taking part in the process affected them – for example, description of effect (outcome), quality of effect, relevance to their needs at the time, importance of effect to the participant, what they think brought about that effect (listening out for features of the process and services) 
• whether expectations were met, whether they would recommend the process, what they wish they had known or done, looking back 

Through the above approach, we will explore the theory of change, including the assumptions, context and mechanisms for change to bring about the intended outcomes of the pre-notification content changes. For example, by exploring: 

• the full customer journey, including where they start, we can test the assumption that most users start their EC journey at the www.acas.org.uk/tell page 
• ease and accessibility, we can test the assumption that most users have capabilities to engage with digitally-hosted information and forms, and capture detailed evidence on whether or how the outputs and outcomes related to the notification form changes are operating as intended 
• how the process affected users, we are capturing insight on whether and how outcomes have emerged, including whether there are any unintended outcomes  

Through analysis, we will map the evidence back to the logic model, including evidence captured elsewhere in the evaluation, to test whether and how the programme theory has been realised in practice.  

Two focus groups with up to 6 ECSOs per group will be conducted online using Teams and last up to 75 minutes. ECSOs will not receive an incentive because they are taking part in the research in the capacity of their job. The groups will be facilitated by a researcher, supported by a semi-structured topic guide and stimulus.  

The groups will explore their understanding of experiences of support after notification but before allocation to a conciliator (to explore differences in amount and type of support provided in completing the supplementary form), and how they and the claimants and representatives they support were affected by the pre-notification, notification and post-notification process.

Analysis

Our analytical approach for the qualitative research will be both deductive and inductive.  

It will be deductive because we will be testing the theory of change and logic model using the qualitative evidence (as well as other evaluation evidence). It will be inductive because we will also build upwards from the views of participants – incorporating elements of 'grounded theory' analysis. For example, the thematic review and continual analysis of hypothesis from participants’ dialogue and researchers' impressions of the discussion (for example, pauses, tone). The latter ensures we do not miss new or different explanations or experiences that we had not anticipated at the start of the evaluation.  

We will also take an iterative and summative approach to analysis. Analysis will begin informally during fieldwork itself; as our research team work closely together throughout the fieldwork period, feeding back headline findings to each other as discussions are conducted, and continually updating our approach and thinking as we amass data. All interviews are written up in detail, including verbatim quotes, in an analytical framework in Excel.  

The framework will be structured around the logic model and research questions, and include key sample data, to allow for comparison of findings by different characteristics. When fieldwork is complete, the data will then be analysed to search for themes and trends, both present and absent. Once qualitative analysis is complete, we will then compare those findings with other evidence to challenge and address gaps. Director-led analysis sessions will bring this thinking together, encourage challenge of assumptions and identify areas for further, targeted analysis.

Funding

The pilot and evaluation are being funded through HM Treasury's Shared Outcomes Fund, which was announced at Spending Review 2020. 

The Shared Outcomes Fund has been designed to fund pilot projects to test innovative ways of working across the public sector, with an emphasis on thorough plans for evaluation. 

Spending Review 2020 announced £200 million, made available for a second round of the Shared Outcomes Fund between 2021 to 2022 and 2023 to 2024.  

The funding for this pilot is £5m. 

The stakeholders are: 

• Ministry of Justice (MoJ) 

• Department for Business, Energy and Industrial Strategy (BEIS) 

• Advisory, Conciliation and Arbitration Service (Acas)