Smarter Resolutions evaluation 2024

This report presents an overview of findings from the evaluation of the Smarter Resolutions programme of work. In 2021, the Ministry of Justice (MOJ) and the then Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a 2-year period (2021 to 2023) from the HM Treasury (HMT) Shared Outcomes Fund, to work with Acas to test a series of changes (or 'interventions') to the early stages of the employment dispute resolution system. This is known as the Smarter Resolutions programme.  

The programme was composed of 4 operational workstreams that focused on improving different parts of the early conciliation service. A summary of the 4 workstreams are listed below.

Workstream 1 – Content strategy 

• This workstream focused on delivering several new types of online content (including videos and information pages), to provide potential claimants or their representatives with the information to help them better understand the dispute resolution process.  

Workstream 2 – Assisted notification  

• This workstream focused on improving the notification form, which claimant-side service users use to notify Acas that they wish to make a claim. 

Workstream 3 – Optimised distribution  

• This workstream focused on improving the system for distributing cases to conciliators through the introduction of automated case allocation and an enhanced system for capturing and logging conciliator availability. 

Workstream 4 – Large group claims  

• This workstream focused on improving the handling of large groups cases (involving 10 or more individuals belonging to the same workplace dispute), through automated case allocation and new processes to identify large group cases.

The central aim of the evaluation was to understand the attributable impact of each of the Smarter Resolutions workstreams and the overall programme on the effectiveness, efficiency and user experience of the early conciliation service. Due to tight timescales, however, the impact of Workstream 3 could not be explored in isolation. Findings in Section 6 of this report therefore detail the overall impact of Workstreams 1, 2 and 3.

To ensure the evaluation was robust, Acas sought views from government and wider experts including Ministry of Justice (MOJ), Department for Business, Energy and Industrial Strategy (BEIS), the Evaluation Taskforce and the Evaluation Advice Panel. Workstreams 1 and 2 were evaluated through randomised control trials (referred to as RCT A and RCT B in this report). The remaining strands used a quasi-experimental, pre- and post-implementation approach. Evaluation of all aspects of the programme involved analysis of data from Acas's case management system (CMS), comparing outcomes for cases that went through the new processes (intervention or post-implementation groups) to those that went through the legacy processes (control or pre-implementation groups). For 3 of the strands, surveys were also administered to claimants and claimant representatives. Across all evaluations, case management system and survey data was supplemented with qualitative fieldwork with Acas staff and claimants or claimant representatives.  

Acas's early conciliation process has a statutory, standard 6-week period during which a conciliator will help both parties to resolve the issue. After this point, it is up to the discretion of Acas conciliators if they feel early conciliation is worth continuing. To ensure comparability of outcomes across individuals and evaluation strands, all end of early conciliation surveys were administered at the 6-week point, and all analysis of outcomes from case management system data is at the 6-week point. This cut-off point was necessary due to the short evaluation timeline; however, it is important to note that a high proportion of settlements occur outside of this statutory 6-week period. Settlement rates reported at this point are not therefore directly comparable to Acas's overall settlement rates, as quoted in Acas's annual report or statistical bulletin figures. Differences between arms of the trials are directly comparable, however.  

This report provides a high-level summary of the evaluation findings. Further detail on background, methodology and findings can be found in the full technical report and trial protocols.

Summary of programme-level findings 

Overall, the Smarter Resolutions programme brought benefits to the early conciliation process. In terms of effectiveness, the programme had a positive impact on one of its key intended programme outcomes: increasing the proportion of cases reaching a definite outcome in the 6-week early conciliation period. A case reaches a 'definite outcome' when it can be taken no further by Acas. It encompasses a number of outcomes including:

• settling  
• reaching an impasse 
• the claimant or respondent declining early conciliation  
• the claimant being uncontactable

Across most of the 4 workstreams, this was more likely following the introduction of the new processes. Under Workstream 2, a definite outcome was reached in 15% of cases in the intervention group, compared to 14% of the control group. Under Workstream 1 to 3, this was 66% of cases in the post-implementation group, compared to 55% in the pre-implementation group. Under Workstream 4, the proportion of cases that reached a definite outcome increased from 37% to 94%.  

An increase in the proportion of cases settled (reaching a COT3 agreement – a legally binding agreement between claimant and employer, required for a case to be settled) was also visible in the evaluation of Workstream 2. Among users who reached a definite outcome, 15% settled in the intervention group compared to 9% of the control group. 

In terms of efficiencies, the impact was largely positive. Following Workstreams 1 to 3 and under Workstream 4, speed of case allocation improved (by 0.91 days and 2.3 days respectively). While there were improvements in the proportion of cases reaching a definite outcome, the impact on the time taken to reach an outcome was mixed. This improved under Workstream 4, with cases reaching an outcome 4.5 days sooner. However, it took longer for cases to reach a definite outcome after Workstream 1 (by 1.5 days) and Workstream 2 (by 6 days). These results indicate that while the overall intended programme outcomes were achieved, there may have been some trade-offs in terms of efficiencies. 

Summary of workstream findings  

The workstreams also improved their respective areas of focus. The 'post-notification' content, developed under Workstream 1, was designed to improve understanding of the early conciliation process among claimant-side service users after they notify Acas of their intention to make a claim. The new content accomplished this with users feeling more informed across a range of measures.  

The 'pre-notification' content (Workstream 1) and changes to the notification form (Workstream 2) aimed to improve understanding among claimant-side service users about whether a claim was valid and improve the quality of information they submitted through the form. The impact on this was mixed. The changes had no significant impact on the proportion of invalid claims or understanding of a valid claim. On quality of collected information, the evaluation showed a reduction in notification forms with incomplete information by 1 percentage point, however, there was also an increase in cases requiring assistance from early conciliation support officers by 4 percentage points. That said, those experiencing the intervention were more likely to have their cases reach a definite outcome (increase of 1 percentage point) and settle (7 percentage points), so the increased need for triage did not have a detrimental influence on the programme-level intended outcomes.  

Automated case allocation, developed under Workstream 3, sought to allocate cases to the appropriate conciliator, faster. While this workstream could not be tested in isolation, the evaluation of Workstreams 1 to 3 showed that this aim was achieved: there was a reduction in time between case notification and allocation by nearly a day (0.91) and the number of times a case was reallocated reduced by 0.31 allocations. The proportion of cases reaching a definite outcome also increased by 11 percentage points. The evaluation showed improvements in service user experience across a wide range of measures; however, claimant representatives gave fewer positive responses than claimants themselves.

Automated case allocation of group cases to the group cases team, initiated under Workstream 4, aimed to improve the handling of large groups cases (involving 10 or more individuals belonging to the same workplace dispute) and make the process faster. The evaluation showed that this aim was achieved: the time taken between case notification and allocation reduced by 2.3 days. Further to this, the time taken to reach a definite outcome reduced by 4.5 days and the time taken to contact employer-side respondents reduced by 5.6 days. There were further positive findings in relation to the programme-level aims: an increase in cases reaching a definite outcome from 37% to 94%. One negative finding was that the number of times a case was reallocated increased from 2.2 to 2.3 post-implementation. 

Interviews and focus groups showed that staff generally felt the changes made as part of the Smarter Resolutions programme had either a neutral or positive impact on their experience of performing their job role. Sometimes, their account and experience of the changes did not reflect the positive impacts shown by the case management system data but this most likely reflects the fact that the impacts were relatively small.  

It is important to note that the evaluation was a snapshot of an evolving programme based on agile and user-centred development approaches. It was anticipated that the service changes would require further fine tuning to make them work effectively based on learning from this evaluation. As part of the longer-term process of iteration, a number of the issues that were identified have since been resolved.

The Advisory, Conciliation and Arbitration Service (Acas) has a statutory duty to promote the resolution of employment dispute claims before they reach employment tribunal (ET). In April 2014, a new legal requirement was introduced that means individuals must notify Acas of their intention to make an employment tribunal claim. Acas then has a statutory, standard 6-week period in which it offers the opportunity for both parties involved in the dispute (that is, the individual claimant and the employer, or their representatives) to resolve the claim through a voluntary early conciliation (EC) process. If the offer of early conciliation is refused or fails to reach agreement, the individual can still pursue an employment tribunal claim. The early conciliation process has been designed so that Acas can provide an impartial conciliation service with the aim of parties reaching a settlement and avoiding the potentially time consuming and costly employment tribunal process.

Background to the Smarter Resolutions programme 

In 2021, the Ministry of Justice (MOJ) and the then Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a 2-year period (2021 to 2023) from the HM Treasury (HMT) Shared Outcomes Fund, to work with Acas to test a series of changes (or 'interventions') to the early stages of the employment dispute resolution system. This is known as the Smarter Resolutions programme. 

The Smarter Resolutions programme is composed of 4 operational workstreams, that focus on different parts of the early conciliation service. The 4 workstreams and workstream visions are: 

• Workstream 1 (WS1), content strategy: content introduced through a claimant-side service users' journey, supports them to grasp the basic principles of employment law related to their dispute, and have clear expectations of how Acas can help. 
• Workstream 2 (WS2), assisted notification: claimant-side service users are guided to supply accurate information that conciliators need, to ensure Acas provide the best possible service at the earliest point to users. 
• Workstream 3 (WS3), optimised distribution: cases are quickly directed to the most appropriate and available conciliator to efficiently resolve their dispute. 
• Workstream 4 (WS4), large group claims: cases that belong to the same workplace dispute are quickly brought to the attention of a team with the expertise, tools and resources to resolve them efficiently.

At the start of 2022, Acas commissioned an evaluation of the 4 workstreams that make up the Smarter Resolutions programme. This evaluation was funded by the HM Treasury (HMT) Shared Outcomes Fund. The evaluation was delivered in partnership between IFF Research and Belmana.

The central aim of this evaluation was to understand the attributable impact of each of the Smarter Resolutions 4 workstreams and the overall programme on the efficiency, effectiveness, and user experience of the early conciliation service. The evaluation also sought to produce insights into how and why intended outcomes have or have not come about, and considerations for the service moving forward. Due to tight timescales, however, the impact of Workstream 3 could not be explored in isolation. Individual findings are not therefore reported for this workstream and Section 6 of this report details the overall impact of Workstreams 1, 2 and 3.  

Acas's early conciliation process has a statutory, standard 6-week period during which a conciliator will help both parties to resolve the issue. After this point, it is up to the discretion of Acas conciliators if they feel early conciliation is worth continuing. To ensure comparability, all analysis of outcomes from case management system data is at this 6-week point. This cut-off point was necessary due to the short evaluation timeline; however, it is important to note that a high proportion of settlements occur outside of this statutory 6-week period. Settlement rates reported at this point are not therefore directly comparable to Acas's overall settlement rates, as quoted in Acas's annual report or statistical bulletin figures. Differences between arms of the trials are directly comparable, however.  

The evaluation was delivered through 4 strands; the purpose, methodology, and key findings of each are explored in turn below. Programme-level outcomes are presented in the first instance, then followed by sections dedicated to each strand.  

This report provides a high-level summary of the evaluation findings. Further details on background, methodology and findings can be found in the full technical report and trial protocols. Logic models, mapping the rationale, assumptions, inputs, activities, and key outcomes of each of the workstreams can also be found in the appendix of the technical report.

We have used a 95% confidence level for reporting differences (differences are only reported where there is a 95% chance that a true difference is present). Likelihood of reaching this level of sensitivity in difference was constrained by a short evaluation timeframe and sample volumes. To illustrate the broader positive trend, the technical report also provides details of findings that reached a 90% confidence level (see overview of evaluation outcomes in each workstream section) and non-significant findings (see Appendix).

Throughout the report reference is made to 'claimant-side' and 'employer-side' service users. 'Claimant-side' service users are the total sample of claimants and their representatives, combined unless otherwise stated. 

Claimant representatives, particularly those who are representatives in a professional capacity, are likely to have a better baseline of understanding than direct claimants due to previous experience of the process. We might therefore expect the impact on knowledge and understanding will be greater for claimants. We therefore report findings on the experience of claimant-side service users both combined and separated as claimants and representatives, where there is a significant finding. 

Evaluation limitations and considerations 

The limitations of the evaluations are outlined below. These should be considered when reviewing the findings.  

During the randomised control trials, only those in the intervention group should have been exposed to the new materials being tested. In both trials, however, those in the control groups reported that they had seen the new materials. If true, this could impact the robustness of the trials. We hypothesise that findings for the control group may be based on engagement with older materials, which would not impact the robustness of the trial.  

There was a further issue regarding randomisation which impacted RCT B (the evaluation of Workstream 2). A large group of cases, with the same characteristics and outcomes, bypassed randomisation during the trial and were automatically allocated to the control group. This was due to a claimant representative using an automation to make numerous individual notifications (1,779) against the same respondent through a pre-saved link to the Acas notification form. We conducted analysis to understand the impact of this. There was evidence that these cases did differ from the average so have the potential to bias results. The results of Workstream 2 which are presented in the body of the report should therefore be treated with some caution. 

For further detail on the impact of these cases see Section 5 of this report and the Appendix of the full technical report (A.3.5).

The overall programme vision, as shown in Figure 1, is that 'people in a workplace dispute have a clearer understanding of how they can reach a quicker, more cost-effective resolution at the earliest point possible'. 

The overall outcomes are that: 

• parties spend less time in the ET system 
• fewer cases require judicial time (including final determination) 
• conciliators are focused on resolution and not administration 
• parties engage in discussion with conciliators which is focused on resolution 

These lead into the programme's 4 workstreams. These are: 

• workstream 1: content strategy 
• workstream 2: assisted notification 
• workstream 3: optimised distribution 
• workstream 4: large group claims 

We hypothesised that improving efficiencies within each of the 4 workstreams would lead to the realisation of each of the aims, with a more efficient process creating more time for conciliators to focus on resolution and reach an outcome within the 6-week early conciliation window.

Time spent in the employment tribunal system, and the proportion of cases requiring judicial time could not be measured within the timeframe of the evaluation. Shorter-term outcomes were assessed including the proportion of cases settled (reaching a COT3 outcome) and the proportion of cases reaching any definite early conciliation outcome within the 6-week early conciliation window, with the assumption that improvements against these would result in fewer cases requiring judicial time and reaching an employment tribunal.  

A case is settled when a COT3 agreement is signed. This is a formal document which sets out a legally binding settlement agreement reached between the claimant and the employer. A case is settled on acceptance of the terms for agreement and the COT3 document records the agreement for the parties' future reference. This represents an end to the dispute at the early conciliation stage and generally prevents the need for an employment tribunal hearing.  

A case reaches a definite outcome when it cannot be taken any further by Acas. This case status encompasses a number of outcomes including settling, reaching an impasse, the claimant or respondent declining early conciliation and the claimant being uncontactable. 

Intended outcomes 

As such, the primary research questions at a programme level were: 

• What is the difference in the average number of cases that reached a definite outcome within 6 weeks of notification between claimant-side service users in the intervention group compared to the control group? 
• What is the difference in the average number of cases that settled (reached a COT3) within 6 weeks of notification between claimant-side service users in the intervention group compared to the control group? 

Within each evaluation strand, evidence was collected to assess the impact on overall aims, and the contribution of each workstream to the above programme outcomes. For some strands, the primary outcomes aligned with the overall programme outcomes; however, each evaluation strand also had specific outcomes they sought to measure relevant to the specific interventions in that workstream. Some of these would inevitably contribute to the programme outcomes, for example a more efficient early conciliation process should help more cases to reach a definite outcome. Other strand-specific primary outcomes tended to relate more to the service user experience.

Below is a summary of programme-level findings. Further detail on these findings and individual workstream outcomes and findings are detailed in the rest of the report.

Summary of findings  

The following table (Table 1) shows the impact for each workstream against the intended outcome of a higher proportion of cases reaching a definite outcome at early conciliation. The indicator for this is the percentage of cases reaching a definite outcome at early conciliation.

The following table (Table 2) shows the impact for each workstream against the intended outcome of a higher proportion of cases settling at early conciliation. The indicator for this is the percentage of cases reaching a COT3 at early conciliation.

Workstream 1 of the Smarter Resolutions programme focused on developing several new forms of content, to help potential claimant-side service users (claimants and representatives acting on behalf of claimants) understand the dispute resolution process. Content was wide ranging and was intended for use across various points of the user journey.  

This section focuses on the evaluation of the 'post-notification' content which was aimed at claimant-side service users after they submitted a notification to Acas that they wish to make a claim.  

The content included 2 automated emails sent to claimant-side service users:  

• An introductory email when a conciliator had been assigned, containing a video outlining the early conciliation process 
• An early conciliation certificate email (issued where relevant), with a video explaining the certificate's purpose and use  

3 additional pieces of content were developed that were up to the discretion of conciliators to send to claimant-side service users:  

• A webpage on time limits within the early conciliation process  
• An emotional support webpage  
• A video on wage disputes  

This content was assessed through a randomised control trial (RCT A).

Intended outcomes: Workstream 1 

The aim of the 'post-notification' content was to equip claimant-side service users with the right information at the right time and, as a result, improve their understanding of the early conciliation process and improve efficiency. This was expected to lead to an increase in the proportion of cases reaching a definite outcome and settling in the 6-week early conciliation period (programme-level outcomes). 

The primary research questions were: 

• What is the difference in the average number of cases that reached a definite outcome and that settled within 6 weeks of notification between claimant-side service users in the intervention group compared to the control group?  
• What is the difference in understanding and knowledge reported by claimant-side service users? 
• What is the difference in the efficiency of the early conciliation process? 

In addition to the primary outcomes, the evaluation measured a number of other outcomes related to service user experience. Secondary outcomes are only reported in the body of the report where findings are statistically significant at a 95% confidence level or above. All outcomes measured for Workstream 1 are reported in the 'Summary of findings' table in this section.

Methodology: Workstream 1 

Outcomes were assessed through a cluster randomised control trial (RCT A), with conciliators split into 'intervention' or 'control' groups. Conciliators in the intervention group had access to and were encouraged to use the new content in their early conciliation process. Conciliators in the control group did not have access to the new content and made use of legacy processes and materials in their early conciliation process. Upon notification, claimant-side service users were then randomly allocated to a conciliator in either group, making this a cluster randomised control trial. The trial ran from 25 July to 30 August 2022. 

Differences in outcomes and experiences for the 2 groups were then assessed through: analysis of Acas's case management system data, and an end of early conciliation survey of claimant-side service users. Analysis of each focused on the early conciliation process and achieved outcomes within the 6-week early conciliation period. Findings from this analysis were supplemented by in-depth interviews with 10 conciliators and 20 in-depth interviews with claimant-side service users. All individual conciliators and claimant-side service users were from the intervention strand of the trial. 

As noted, this trial focuses solely on the 'post-notification' content. A second trial tested the other elements of Workstream 1 alongside changes from Workstream 2 of the Smarter Resolutions programme. These findings are presented in Section 5 ('pre-notification content and assisted notification') of this report.

Summary of findings: Workstream 1 

The following tables give an overview of the Workstream 1 outcomes and indicators. Unless '(CMS)' is specified, outcomes were measured in survey data.

Key to Tables 1 to 7: 'Positive impact' shows a positive finding at 95%; 'Negative', shows a negative finding at 95%; 'No sig. difference' shows no significant difference between the intervention and control group. Base (Intervention/Control). End of early conciliation survey: All (756/731); Claimants (619/603); Claimant Representatives (137/128). Case management system data: All (1,439/1,429).

Findings: Workstream 1

Overall, there is evidence that the 'post-notification content' brought benefits to the early conciliation process, particularly in relation to its main intended workstream outcome: improving understanding among claimant-side service users. Claimant-side service users in the intervention group were more likely to feel that the information provided by Acas was accurate. When analysing claimants only (excluding representatives), there were also improvements in the proportion of those feeling informed and feeling they received the right information at the right time. In relation to more general service user experience, there was a reduction in stress levels among claimants in the intervention group.  

The new content had no impact on the proportion of cases reaching a definite outcome or settling within the 6-week early conciliation period, although this is somewhat unsurprising compared to the other workstreams given Workstream 1 targeted the early stages of a case, ensuring parties are well-informed about the process so the case could progress.  

There was a negative impact on one measure of efficiency: there was an increase in the time taken to reach a definite outcome.  

Case reaching definite outcome and settlement  

• No significant impact on cases reaching a definite outcome.  
• No significant impact on case settlement. 

Efficiency of early conciliation 

• More time taken to reach a definite outcome: Cases from the intervention group took longer to reach an early conciliation outcome within the 6-week early conciliation period, with a difference of 1.5 days. 

Service user understanding and knowledge 

• Information provided by Acas is accurate: Claimant-side service users in the intervention group were more likely to agree that the information provided by Acas was accurate (75% compared to 71% in the control group). 
• Service users get the right information at the right time: When looking at claimants only, those in the intervention group were more positive: 70% agreed they had received the right information at the right time, compared to 64% of control group claimants. 
• Service users are better informed about early conciliation: Claimants in the intervention group also felt better informed through the early conciliation process (64% compared to 57% in the control group). 

Service user experience  

• Lowered stress levels: Claimants in the intervention group were more likely to have reduced worry following initial contact with their conciliator (41% compared to 35%).  
• Fewer case outcomes matched claimant expectations: Claimant representatives in the intervention group were more likely than those in the control group to disagree that the early conciliation outcome matched their claimants' expectations (30% compared to 19%). 

Views on the new 'post-notification' content 

Broadly, the new content was well received by both conciliators and claimant-side service users. Some conciliators felt that their explanations of the process were streamlined and made more efficient with the introduction of the new content, allowing more time to focus on resolution. Others highlighted that this was reliant on claimant-side service users engaging with the content, which did not always happen. Conciliators and claimant-side service users both felt that the information itself was suitable, but introducing text to accompany videos, and more generally thinking about accessibility and variability in mode preferences could improve engagement by claimant and claimant representatives.

This section explores the impact of the 'pre-notification' content (developed under Workstream 1) and the assisted notification form (developed under Workstream 2), which were evaluated together through a randomised control trial (RCT B).

Workstream 1 focused on developing several new forms of online content, to help potential claimant-side service users (claimants and representatives acting on behalf of claimants) understand the dispute resolution process as it applies to them at various points in their user journey. The content included an emotional support page, a time limits page, and a video about wages.

During RCT A (outlined in the previous section), conciliators in the intervention group had access to this content and, if relevant, could send it to claimant-side service users after they made a claim (post-notification). As part of RCT B (outlined in this section), this content was embedded across the Acas website and therefore available for claimant-side service users to access before making a claim (pre-notification). While some of the materials are the same, the 2 trials tested them at different points in the user journey. 

Under Workstream 2, a new assisted notification form was developed for claimant-side service users to notify Acas that they intend to submit an employment tribunal claim. The new assisted notification form included features such as:

• Progressive disclosure, where the user is taken step-by-step through each page with a new question and associated help text, instead of seeing long single pages asking for multiple inputs of information
• Channelling, so only the questions or sections that are relevant are shown, based on previous responses
• The ability to save and return
• New embedded, informative content and signposting throughout

Intended outcomes: Workstream 1 and 2

The main aim of the new 'pre-notification' content and assisted notification form were to:

• improve data quality coming through on the notification forms 
• improve claimant-side understanding on whether they had a valid claim in order to reduce the number of invalid claims

This was expected to create efficiencies in the early conciliation process and increase the proportion of cases reaching a definite outcome or settling in the 6-week early conciliation period (programme-level outcomes).

The primary research questions were:

• What is the difference in the average number of cases that reached any definite outcome and that settled within 6 weeks of notification between claimant-side service users in the intervention group compared to the control group?
• What is the difference in data quality (accuracy and completeness of information in the notification form)?
• What is the difference in the proportion of invalid notifications between claimant-side service users in the intervention group compared to the control group? 
• What is the difference in the perceived knowledge of whether their claim is valid or invalid?

In addition to the primary outcomes, the evaluation measured a number of other outcomes related to efficiency and service user experience. Secondary outcomes are only reported where they are statistically significant at a 95% confidence level or above. All outcomes measured for Workstream 1 and 2 are reported in the 'Summary of findings' table in this section.

Methodology: Workstream 1 and 2

Outcomes were assessed through a randomised control trial (RCT B). Half of the claimant-side service users who entered the 'notify Acas' website were randomly allocated through a website algorithm to the intervention group (directed to pages containing the new content and assisted notification form) and the other half to a control group (directed to the legacy notification webpages and form). The trial ran from 24 October to 25 December 2022. 

Differences in outcomes and experiences for the 2 groups were then assessed through: analysis of Acas's case management system data, and an 'end of early conciliation' survey to claimant-side service users. Pop-up surveys were also administered to users exiting the notification process (both exiting early or on completing the form). Findings from this analysis were supplemented by focus groups with 9 Early Conciliation Support Officers who finalise and triage cases before they are allocated to a conciliator. In-depth interviews with 42 claimant-side service users from the intervention strand of the trial were also conducted.

Summary of findings: Workstream 1 and 2

The following tables give an overview of the Workstream 1 and 2 outcomes and indicators. Unless '(CMS)' is specified, outcomes were measured in survey data.

Key to Tables 8 to 15: 'Positive impact' shows a positive finding at 95%; 'Negative impact,' shows a negative finding at 95%; 'No sig. difference' shows no significant difference between the intervention and control group. Base: End of early conciliation survey (Intervention/Control): All (1,011/1,266); Claimants (815/1056); Claimant Representatives (196/210). Base: Leavers: All intervention (459); All control (377). Base Completers: All intervention (1479); All control (1450). Base case management system data: All intervention (8,668); All control (12,821)

Findings: Workstream 1 and 2

Overall, there is evidence of the 'pre-notification' content and assisted notification form bringing some improvements to the early conciliation process. The programme level intended outcomes were achieved: both the proportion of cases that settled and that reached a definite outcome was higher amongst the intervention group. However, these cases also took longer to reach an outcome. 

While data quality improved, this did not always translate into further efficiencies or a decrease in invalid claims. There was a reduction in incomplete information on forms, showing improvements, however there was an increase in cases requiring triage (support from early conciliation officers) and the time taken to allocate a case, and a reduction in cases going straight to a conciliator. 

Three of the negative findings are related: an increase in time taken to reach a definite outcome; an increase in cases requiring triage; and a reduction in cases going straight to a conciliator. The Smarter Resolutions project team believed these findings were related to the structure of the notification form where users were offered an opportunity to choose how to interact with Acas. The presentation of these options has since been changed, leading to an increase in the proportion of users submitting both telephone and email details, which should reduce the need for triage, limiting delays to allocation.

There were no significant differences in terms of perceived knowledge of whether a claim was valid.

Case reaching definite outcome and settlement

• Cases more likely to be settled: Cases in the intervention group were more likely to be settled at the early conciliation stage than those in the control group, as indicated by a COT3 outcome (15% compared to 9%).
• Cases more likely to reach a definite outcome: Cases in the intervention group were more likely to succeed in reaching a definite outcome within the 6-week early conciliation period (15% compared to 14%).

Case efficiency

• More time taken to reach a definite outcome: The intervention group took longer to reach a definite early conciliation outcome during the 6-week early conciliation period, on average 32 days, compared to 26 days for the control group.
• More time taken to allocate a case to a conciliator: The intervention group took 2.4 days longer to have their case allocated to a conciliator, on average 10.7 days compared to 8.3 days in the control group. 
• Less cases going straight to a conciliator: 18% of intervention group case were sent straight to a conciliator (compared 21% in the control group

Data quality and invalid claims

• Incomplete information on form: The intervention group were less likely to have incomplete form fields (0%) than the control group (1%).
• More cases requiring triage: The intervention cases were more likely to require Early Conciliation Support Officers to triage (22%) compared to the control group (18%). 
• Invalid claims: There were no significant differences in terms of the proportion of invalid claims.

Service user understanding and knowledge

• Knowledge of invalid claims: There were no significant differences in terms of service users' perceived knowledge of whether a claim was valid
• Outstanding questions about the early conciliation process: Claimant representatives in the intervention group were less likely to have questions about early conciliation both prior to starting the form and after submitting, compared to the control group claimant representatives (86% and 83% compared to 93% and 91%, respectively).

Views on the new pre-notification content and data quality

During qualitative interviews with the intervention group, claimant representatives reported that the new form was faster to complete as only relevant information was gathered.

Some Early Conciliation Support Officers reported more relevant and complete information was gathered in the assisted notification form compared to the legacy form. This improvement helped early conciliation support officers to process cases more efficiently. 

Some claimant-side service users were dissatisfied with the new content, as they felt there was too much information to digest on the website that caused them to feel overwhelmed.

Considerations: Workstream 1 and 2

The findings outlined above should be treated with a degree of caution. During the trial, there was an issue regarding randomisation which impacts the case management system findings. A large number of cases, which had similar characteristics and outcomes, bypassed randomisation and were automatically allocated to the control group. This was due to a single claimant representative using an automation to make numerous individual notifications (1,779) against the same respondent. This automation avoided the randomisation, and, resultingly, these cases only entered the system through the legacy route and form. 

To control for this effect, different options were considered, primarily removing the affected cases or using statistical matching to rebalance the control group. In consultation with the evaluation working group (which governs the Smarter Resolutions evaluation), the decision was made to conduct further analysis removing the affected cases. Removing batches was primarily undertaken by removing the cases identical in terms of the employer, jurisdiction and whether a representative was used. The approach removed, as expected, significantly more cases from the control than the intervention group.

This analysis shows that the cases that bypassed randomisation did differ from the average so have the potential to bias results. When the affected cases were removed, the impact was modest on the outcomes related to efficiency and timings: the speed of allocation, time taken to reach a definite outcome and proportion of cases going straight to a conciliator were still poorer in the intervention group and the findings remain statistically significant. However, the findings related to all other indicators outlined above become statistically insignificant when the affected cases are removed. Excluding these cases also impacts the validity of the trial and so results should be treated with some caution. 

Tables 16 to 21 compare the difference between the intervention and control groups with the affected cases included and the difference when these cases are removed.

Base: All intervention (8,668); All control (12,821); reduced to All intervention (8,411); All control (10,291) when batches removed.

Base: All intervention (8,668); All control (12,821); reduced to All intervention (8,411); All control (10,291) when batches removed.

When the affected cases are removed, cases in the intervention group take 0.9 days longer than cases in the control group to be allocated to a conciliator, compared to 2.9 days when they are included. Similarly, cases in the intervention group take 1.7 days longer to reach a definite outcome than those in the control when the affected cases are removed. When included, the intervention cases take 5.8 days longer. 

With the exception of cases that go straight to a conciliator, all other findings become insignificant when the affected cases are excluded, as shown in table 18. 

Base: All intervention (8,668); All control (12,821); reduced to All intervention (8,411); All control (10,291) when batches removed.

Base: All intervention (8,668); All control (12,821); reduced to All intervention (8,411); All control (10,291) when batches removed.

Base: All intervention (8,668); All control (12,821); reduced to All intervention (8,411); All control (10,291) when batches removed.

Base: All intervention (8,668); All control (12,821); reduced to All intervention (8,411); All control (10,291) when batches removed.

Workstream 3 focused on improving the system for distributing cases to conciliators. To do this, enhancements were made to the system for capturing and logging conciliator availability, location and work profiles, for example expertise, preferences and any reasonable adjustments. The primary feature of the workstream was the development of an algorithm which automatically organises and distributes cases to the right conciliator or team according to this information. Before this, most case distribution was done manually by the case allocation team (CAT), a team of Acas staff who are responsible for manually assigning claimant cases to conciliators. 

Given there was no clean 'time gap' between the rollout of all Workstream 1 and Workstream 2 service-wide, and the implementation of Workstream 3 changes, the impact of Workstream 3 could not be explored in isolation. Quantitative findings in this section therefore relate to the overall impact of the programme (Workstreams 1, 2 and 3). 

Intended outcomes: Workstream 3

The main aim of Workstream 1 to 3 was for cases to be allocated to an appropriate conciliator, faster. Ultimately, this should create more time for conciliation and result in more cases settling or reaching a definite outcome within the 6-week period (programme-level outcomes).

As such, the primary research questions for this evaluation strand were:

1. What is the difference in the average number of cases that reached a definite outcome and that settled within the 6-week early conciliation period, pre- and post-implementation?
2. What is the difference in the time elapsed from case notification to initial allocation to an individual conciliator pre and post-implementation? 

In addition to the primary outcomes, the evaluation measured several other outcomes related to service user experience. Secondary outcomes are only reported where they are statistically significant at a 95% confidence level or above. All outcomes measured for Workstream 3 are reported in the 'Summary of findings' table in this section. 

Methodology: Workstream 3

A comparison of cases pre and post-implementation of Workstream 1 to 3 was carried out to assess improvements in efficiencies and outcomes, using case management system data. Propensity score matching was used to match the post-implementation cases to a set of cases that were as comparable as possible in the pre-implementation period. 

Case management system analysis was supplemented by end of early conciliation survey data. The pre-implementation period (that is, before any Workstream 1 to 3 changes were implemented) for case management system analysis was 1 January 2022 to 31 May 2022. The post-implementation period ran from 27 February to 17 March 2023. Two focus groups were conducted: one with the case allocation team (5 participants) and one with individual conciliators (7 participants). Twelve interviews with claimant-side service users in the post-implementation period were also carried out.

Summary of findings: Workstream 3

The following tables give an overview of the Workstream 3 outcomes and indicators. Unless '(CMS)' is specified, outcomes were measured in survey data.

Key to Tables 22 to 26: 'Positive impact' shows a positive finding at 95% significance, 'Negative impact' shows a negative finding at 95% significance, 'No sig. difference' shows no significant difference between the post-implementation and pre-implementation group. Base (Post-implementation / Pre-implementation). End of early conciliation survey: All (1,275/731); Claimants (1,003/603); Claimant Representatives (272/128). Case management system data: All (4,844/4,836

Findings: Workstream 3

Overall, findings on the impact of Workstreams 1 to 3 were positive. There were improvements in the speed of case allocation to a conciliator, the main intended outcome of automated case allocation. The evaluation also showed a positive impact on cases reaching a definite outcome, a primary outcome for the programme as a whole, although the proportion of cases that settled decreased.

Efficiency improved with reductions in both the reallocation count and the time taken to allocate a case to a conciliator. Claimant-side service user experience also improved across numerous measures, although when focusing on claimant representatives only, those in the post-implementation group were more negative. 

Case reaching definite outcome and settlement

• Cases settled: Cases were less likely to settle (reach a COT3 outcome) in the post-implementation group: less than 1% compared to 7% in the pre-implementation cases.     

Note this is not comparable to Acas's final settlement figures as reported in the annual report due to the timeframes of the data cuts. In 2022 to 23 Acas's final COT3 settlement rate across all cases was 11%.

• Cases reaching a definite outcome: A definite outcome had been reached within 6 weeks of notification in proportionately more cases in the post-implementation group (66% compared to 55%). 

Efficiency of early conciliation

• Time taken to allocate a case to a conciliator: The time between notification and allocation reduced by nearly a day (0.91), from 6.1 days pre-implementation to 5.2 days post-implementation.
• Allocation count (average number of allocations before reaching the final conciliator): reduced by 0.31 (2.4 allocations on average, post-implementation, compared to 2.7 pre-implementation). Similarly, the proportion of cases that were reallocated reduced from 20% to 7% post-implementation. 

Service user experience

Post-implementation claimant-side service users were more likely than pre-implementation users to agree that:

• The process helped them understand the strengths and weaknesses of their case (51% compared to 45%)
• The early conciliation process felt fair (66% compared to 61%) 
• The early conciliation process reflected expectations (60% compared to 54%)
• The early conciliation outcome reflected expectations (47% compared to 37%)
• They understood the conciliator's role (80% compared to 72%)
• Initial contact meant early conciliation got off to a good start (70% compared to 66%)
• The conciliator had the right skills and knowledge (68% compared to 61%)
• Early conciliation could resolve the dispute before reaching employment tribunal (88% compared to 82%)
• They had reduced worry following initial contact with their conciliator (39% compared to 34%)
• Their case had been handled efficiently (69% agreed compared to 64%)

When looking at claimants only (excluding claimant representatives), those in the post-implementation period were more likely to agree that:

• The early conciliation process helped them feel better informed (63% agreed compared to 57%)
• However, when focusing on claimant representatives, the post-implementation group had more negative views on a range of measures: 
• The process helped them feel better informed (only 47% of representatives in the post-implementation group agreed, compared to 53% in the pre-implementation group)
• Their claimants were satisfied with the early conciliation outcome (only 31% of representatives in the post-implementation group reported satisfaction among their claimants, compared to 52% in the pre-implementation group) 
• Ease of the early conciliation process (9% in the post-implementation group said the process was difficult compared to 6% in the pre-implementation group)
• The early conciliation process felt fair (only 71% agreed compared to 76% in the pre-implementation group)
• Understanding conciliators could help resolve the dispute before it reached employment tribunal (only 77% agreed compared to 81% in the pre-implementation group)

Views on the optimised distribution processes

In qualitative discussions, conciliators acknowledged that cases were getting to them faster, but felt efficiency in this was sometimes perceived to be lost as they were spending more time doing administrative tasks such as gaining contact details, clarifying case details, or correcting spelling errors, that used to sit outside of their remit.

Positively, in qualitative interviews, claimant-side service users were impressed with the speed in which they received contact from a conciliator; one claimant representative noted that this was considerably quicker than previous cases due to the removal of the early conciliation support officer step.

Workstream 4 focused on improving the handling of large groups cases, involving 10 or more individuals belonging to the same workplace dispute, and quickly bringing them to the attention of the dedicated group cases team (GCT) with the expertise, tools, and resourcing to resolve a case efficiently.

Changes under Workstream 4 included: 

• Automatic allocation of group cases to the group cases team rather than a manual process; 
• Changes to the group notification form, to enable claimant representatives to add additional information and give permission for conciliators to speak to respondent(s) first; and 
• A respondent allocation report which helps the group cases team spot cases with 'same facts same respondent'.

Intended outcomes: Workstream 4

The aim of Workstream 4 was to increase efficiencies within the group cases team process (namely time taken between stages within early conciliation), and ultimately increase the proportion of cases that reached a definite outcome or settled within the 6-week period (programme-level outcomes).

As such, the primary research questions were: 

• What is the difference in the average number of cases that reached a definite outcome and that settled before and after Workstream 4 changes were implemented?
• What is the difference in the efficiency of the process for handling large group cases before and after Workstream 4 changes were implemented?

In addition to the primary outcomes, the evaluation measured several other outcomes related to service user experience. Secondary outcomes are only reported where they are statistically significant at a 95% confidence level or above. All outcomes measured for Workstream 4 are reported in the 'Summary of findings' table in this section.

Methodology: Workstream 4

A comparison of large group cases pre and post- Workstream 4 implementation was carried out using case management system data to assess improvements in efficiencies and outcomes. The pre-implementation period was between 3 January and 29 April 2022 and the post-implementation period was between 3 October and 30 December 2022. These findings were supplemented with discussions with the group cases team manager, a focus group with group cases team conciliators, and 20 interviews with claimant representatives (many of whom had experienced the service pre and post-implementation).

Summary of findings: Workstream 4

The following tables give an overview of the Workstream 4 outcomes and indicators. Unless '(CMS)' is specified, outcomes were measured in survey data.

Key to Tables 27 and 28: 'Positive impact' shows a positive finding at 95%, 'Negative impact' shows a negative finding at 95%; 'No sig. difference' shows no significant difference between the post-implementation and pre-implementation group. Base: Pre-implementation (944 cases of which 349 reached an outcome in 6-week window); Post-implementation (1090 cases of which 871 with outcomes within the 6-week period).

Findings: Workstream 4

Overall, there is evidence of the new processes improving the handling of large group cases. Automated case allocation led to an increase in the proportion of large cases reaching a definite outcome, one of the primary programme outcomes. It also led to a reduction in the time taken to reach a definite outcome, to allocate a case to a conciliator and to contact the respondent, pointing to an overall more efficient process. There was, however, one negative finding: The average allocation count increased.

Case reaching definite outcome and settlement 

Cases reaching a definite outcome: The proportion of cases reaching a definite outcome within the 6-week early conciliation period increased from 37% to 94%, although the proportion of cases being settled was no different.

Efficiency of early conciliation

• Time taken to reach a definite outcome: decreased by 4.5 days (from 11.4 days pre-implementation to 7.0 days post-implementation) 
• Time taken to allocate case to a conciliator: The time between notification and allocation reduced by 2.3 days (3 days post-implementation compared to 5.3 days in pre-implementation period)
• Time between case allocation to contacting the employer-side respondent decreased by 5.6 days, from 6.5 days pre-implementation to 0.9 post-implementation 
• Allocation count: The average allocation count (that is. how many times a case changes hands before it reached its final conciliator) increased from 2.2 to 2.3 post-implementation

Views on the new processes for handling of large group cases

In qualitative interviews, the group cases team identified a few areas impacting efficiencies, which could be addressed: 

• Lead cases (usually the first case notified within a group case) being disconnected from linked cases in auto-allocation
• Auto-allocation of individual cases (through Workstream 3) disrupting identification of group cases.

The group cases team felt the new respondent allocation report was reducing some of the manual burden of identifying 'same respondent, same facts' cases.

To the group cases team and claimant representatives, changes to the new group form (permission to contact the respondent, and option to provide additional information) were considered positive, albeit not too impactful in terms of efficiencies. 

It was quite rare for claimant representatives to have noticed a change in the efficiency of large group case handling, but those that had noticed a difference were positive (one cited a noticeable improvement in the handling of these cases).