Independent evaluation of the Global Digital Exemplar programme

An independent evaluation of the Global Digital Exemplar (GDE) Programme has been published by Edinburgh University. Read an executive summary of the report below. View the full report on the university's website.

Beginning a joint digitally enabled transformation and learning journey in the England National Health Service.

Please note: the evaluation team continues to produce outputs, so these findings need to be viewed as evolving.

An ambitious and transformative initiative

The Global Digital Exemplar (GDE) Programme is an ambitious first-of-type national initiative seeking to:

1. advance digitally enabled (service) transformation in selected exemplar NHS England provider organisations already characterised by relatively high levels of digital maturity and bring them up to an international level, and
2. create a national learning ecosystem to spread the knowledge acquired.

The programme was launched after the 2016 Wachter Review proposed a phased approach to digitising the English NHS as the scale of investment required to bring all provider organisations to digital maturity greatly exceeded available resources. The Wachter Review recommended creating a cohort of digitally advanced exemplar provider organisations (hereafter GDEs) who would pass on their learning to less digitally-mature ‘Fast Follower’ provider organisations and thereby catalyse large-scale digitally enabled transformation of the wider English NHS.

Provider organisations that were shortlisted were invited to propose ambitious portfolios of digital change, to be implemented over two to 3.5 years. Provider organisations were selected in several tranches from December 2016. The GDE programme involved 51 provider organisations including 3 ambulance provider organisations, 33 acute provider organisations and 15 mental health (MH) provider organisations. Of the 48 acute and MH provider organisations, 23 were GDEs and 25 were Fast Followers (FF), paired up to share knowledge.

GDE acute provider organisations each received £10 million and Fast Followers received £5 million. Mental health GDE organisations received £5 million, mental health Fast Followers received £3 million, and three ambulance organisations shared £5 million. Central investment across all 51 provider organisations in the GDE programme came to £302 million. Although significant, this represents a relatively small share of the total NHS digital transformation budget between 2016-17 and 2020-21 of £4.7 billion. All organisations were required to secure internal matched funding.

The internationally recognised Healthcare Information and Management Systems Society (HIMSS) Electronic Medical Record Adoption Model (EMRAM) was chosen as one of the guiding benchmarking criteria for the GDE Programme, with the expectation that GDEs and Fast Followers would respectively achieve HIMSS Level 6 (with a plan for 7) and HIMSS Level 5 or equivalent accreditation by the end of the programme. Limitations of this model and its applicability to the NHS were recognised, for example, by setting a lower EMRAM target (Level 5) for GDE mental health providers.

In addition to supporting digitally enabled transformation within selected provider organisations, the GDE Programme offered central support for the establishment of programme governance and delivery assurance arrangements as well as supporting various mechanisms for sharing learning, including the establishment of GDE-Fast Follower partnerships, blueprinting (documents summarising implementation experience) and various learning networks to capture and share implementation experiences.

A learning programme

The GDE Programme learnt from earlier programmes. It followed several years after the National Programme for IT (NPfIT), in which centrally procured frontline systems did not achieve optimal or anticipated local ownership and adoption thus limiting the value for money achieved. Learning from this and subsequent initiatives such as the Safer Hospitals Safer Wards, Nursing Technology Fund and the Integrated Digital Care Fund, the GDE Programme was a national initiative designed to optimise local ownership and accountability by allowing provider organisations flexibility on routes to digital maturity, within a structured framework of accountability (such as funding gates, progress monitoring and targets) to achieve digitally enabled transformation of services rather than merely to fund technology adoption.

In keeping with the recommendations of the Wachter Review, an independent evaluation was commissioned. This evaluation has explored programme processes and outcomes in real-time. Programme leadership incorporated feedback and recommendations from this evaluation and from other stakeholders throughout the Programme. Fifteen provider organisations were not included in the agreed scope of the evaluation (nine Fast Follower provider organisations that joined the programme later, three that merged with other provider organisations in the course of the programme, and three ambulance provider organisations).

The independent evaluation was undertaken by a team from the University of Edinburgh, University College London, and NHS Arden & Greater East Midlands Commissioning Support Unit. Its work was overseen by an Independent Steering Group chaired by Professor Anthony Avery from the University of Nottingham. The relationship between the evaluation team and programme leaders was productive, open and honest, facilitating productive working relationships that helped to shape the programme throughout.

This report summarises processes, outcomes and lessons learnt to date. It is important to bear in mind that the outcomes of major digitally enabled transformation frequently occur in unanticipated ways and only become fully apparent in the long-term. In addition, the COVID-19 pandemic has affected the final year of the programme as both an agent for acceleration of progress and a disrupter.

Findings

Beginning the journey to digitally enabled transformation

Overall, our work shows that the GDE Programme largely achieved what it set out to do, namely stimulating digitally enabled transformation and the sharing of knowledge between participating provider organisations and with programme managers. It also helped organisations to develop a roadmap of digitally-enabled transformation (and achieve international standards of excellence), engage clinical users, and establish baselines of digital maturity against which they could assess progress (defined by nationally developed measurements including the Definition-of-Done). The programme was also seen to have demonstrated the safe delivery of digitally-enabled outcome-driven transformation at pace and lower cost. It achieved this through promoting an ethos of learning and sharing and through an adaptive programme strategy, facilitated by a range of knowledge sharing mechanisms that worked together to promote programme aims.

The GDE Programme also demonstrated the value of broadening the traditional conception of technology programmes from merely IT deployment towards digitally enabled transformation, changing care and work practices to efficiently deliver safe high quality care.

Provider organisations successfully implemented ambitious programmes of digitally enabled transformation

The provider organisations implemented ambitious programmes of digitally enabled transformation, in most cases revolving around major upgrades in core information infrastructures such as Electronic Health Record (EHR) implementations. Sites implemented multiple projects, in many instances as part of a bigger integrated care agenda, with relatively modest additional external funding. Provider organisations with an existing long-term digitisation strategy were able to use the GDE initiative to accelerate digitally enabled transformation substantially. The programme enabled GDEs and Fast Followers beginning their digital maturity journey to develop a digital vision and a roadmap towards achieving digital maturity and substantially advance digitally enabled transformation. Participating provider organisations achieved significant advances and most are on track to achieve their planned delivery as determined by international benchmarks. A wide range of benefits are materialising and feeding through into improved care delivery and outcomes including significant unanticipated benefits – most strikingly an enhanced capacity to respond to the challenges posed by COVID-19.

Critical success factors contributing to the success in achieving digitally enabled transformation

Several inter-related features of the GDE Programme have contributed to its success in achieving digitally enabled transformation. These included:

• high level ownership at board and divisional levels and clinical ownership across the organisation
• dedicated multi-year external funding, released as plans were fulfilled, and the requirement for matched internal funding ensuring organisational commitment to the digital journey
• governance structures around digitally enabled transformation - this included securing executive commitment and the creation of internal structures to manage the implementation of a programme of digitally enabled change
• status and reputational benefits associated with succeeding in the competition to join this prestigious programme which motivated organisation members and provided greater leverage with vendors
• the growth of digital transformation expertise (exemplified by the appointment of chief information officers, chief clinical information officers and chief nursing information officers)
• Development of an internal vision and strategy for digitally enabled transformation and targets linked to accreditation.

These features also contributed to the development of expertise and experience among clinicians, and expanded IT teams.

Establishing the foundations of a learning ecosystem

The GDE Programme also helped to establish the foundations for a learning ecosystem through:

Establishing formal knowledge transfer

Establishing formal knowledge transfer through GDE and Fast Follower partnerships and mechanisms around the creation and circulation of blueprints (documents capturing technical and implementation knowledge) and some funding for formal GDE Learning Networks.

Additional informal networks

Additional informal networks, motivated by the mutual (learning and reputational) benefits of knowledge sharing, greatly strengthened the effects of formal networking structures. Knowledge sharing took place through many channels including: email exchanges, site visits, teleconferences and staff secondments. A distributed web of stakeholders acted as knowledge bridges, translating their implementation experience to would-be adopters in other organisations. Effective exchange was promoted by common technological platforms, geographical proximity (notwithstanding caveats against regional monopolies), shared clinical services and pathways, and institutional and cultural alignment, which enhanced learning benefits and reduced networking costs.

Knowledge transfer

Knowledge transfer, promoted through national initiatives, has helped to strengthen both formal and informal skills and capabilities in change management and hybrid clinical informatics skills as well as technical capabilities. The broader process of professionalisation of digitally enabled transformation expertise contributed to these developments (see below).

A culture of learning, partnerships and information-sharing emerged between provider organisations and with programme managers across the GDE Programme. There was also openness and learning through engaging with the evaluation work, which enabled the evaluation team to gain insights and share these with programme leadership.

The GDE Programme was reinforced by and contributed to developments in the wider environment

Concurrent initiatives including other programmes such as the Local Health and Care Record and Health System Led Investment programme, the NHS Digital Academy, Learning Networks, and the growth of professional networks (for example for health chief information officers and chief clinical information officers) helped to facilitate and reinforce the aims of the GDE Programme.

Common challenges faced in the COVID-19 crisis reinforced online working (particularly locally) and helped digitally mature sites to utilise digital infrastructures in agile ways to respond to new challenges. The enhanced response of digitally mature provider organisations to the unprecedented challenge of COVID-19 demonstrated the benefits of digitally enabled transformation, bringing the latent benefits of digital maturity to the fore.

The emerging development of Integrated Care Systems (ICSs) during the programme promoted regional collaboration and the collaboration established through the programme facilitated the development of ICSs.

Major transformation programmes inevitably face complex challenges and tensions

Challenges and tensions will inevitably emerge in large, complex digitally enabled transformation programmes like the GDE Programme due to the need to manage often opposing or competing stakeholder needs and priorities; tackle unanticipated impacts and issues; and manage risks and dependencies. The key recommendations (below) highlight ways these challenges might be managed and risks mitigated in future programmes and initiatives. Key areas requiring negotiation and flexibility, within a programme management framework during the GDE Programme included:

Digital maturity targets

Participating provider organisations committed to achieving technological functionality objectives as part of the accreditation process at the end of the programme. Most stakeholders recognised the value of digital maturity measures, such as the HIMSS EMRAM, as they helped them map the journey towards digital excellence. However, mandating particular technological functionality targets had complex consequences as these did not necessarily align with local priorities and patient outcomes. Changes in benchmarking criteria and measurement tools over the course of the programme were perceived by some as disruptive. Whilst the HIMSS model had perceived value, there is an ongoing debate about scaling this approach across the NHS.

Progress and outcome monitoring and benefits realisation

There was broad overall agreement on the ethos of demonstrating the achievement of milestones to show due diligence and to develop an evidence base. Requirements to identify planned benefits encouraged organisations to pursue service enhancements, rather than merely implementing IT. There was some contention however in relation to benefit management tools and methods. The resources needed to collect benefits and outcome data locally were high, and fell upon organisational members who did not experience benefits from their use. The content and timing of information collected differed from existing organisational reporting systems, despite attempts to standardise reporting. Some organisations re-used data collected to report to their boards and to encourage clinical engagement. Some organisations also recognised the value of the benefit management approach in implementing change and pursuing quality improvement. The existing literature suggests that it is very difficult to attribute short-term outcomes to digital interventions. This is especially relevant in large infrastructure upgrades such as Electronic Health Records, which take a long time to implement and progressively optimise. As a result, there is a long lag before cost savings or health outcomes can be detected. Towards the end of the Programme, and in responding to COVID-19 demands, organisations were able to leverage existing infrastructure investments and benefits were immediately apparent.

Long-term planning

The short-term nature of annual funding cycles and relatively short duration of programmes (often three years or less) are at odds with the long timeframes required for the digitally enabled transformation journey. The challenges of digital transformation, such as the financial planning of long-term investments in core EHR upgrading, require a longer-term strategy, between 5 to 10 years. These difficulties in resourcing the digital journey are compounded by a policymaking environment that favours creating new programmes over continuing existing programmes. GDE Programme managers and provider organisations had to devote considerable effort to mitigating the instability of the policy and funding environment in order to pursue long-term digitisation strategies.

Market management

This was a recognised area of importance during the programme. Associated initiatives developed during this period have helped to promote a system-wide approach to encouraging diversity and quality of technology supply. The GDE Programme helped strengthen engagements between adopters and vendors collectively, by promoting user groups, as well as individually. There was a shift away from devolved procurement (adopted in reaction to the difficulties experienced with central procurement under the NPfIT) towards concerted procurement with stronger collective engagement between vendors and provider organisations. However, these important developments are still at the early stages of an inevitably gradual transformation. Transforming this market will inevitably be a long-term process that needs to be supported by long-term signalling, shared intelligence and strategic deployment of the procurement power exercised by provider organisations. The slow adoption of health information technology standards for using codified data and limited development of protocols for interoperability between different systems has hampered progress in procurement, implementation and optimisation of systems throughout the Programme and remains an area requiring greater focus and prioritisation.

The GDE Programme has clearly influenced the future of digitally-enabled transformation in NHS England and strengthened the position and understanding of the value of digital technologies in delivering and developing NHS services. GDE and Fast Follower partnerships were seen to have allowed provider organisations to achieve major changes and deliver quality improvements reliably, at greater pace and lower cost. To ensure the progress achieved by the GDE Programme does not dissipate, future initiatives should ensure they build upon the programme to maintain momentum to support the learning ecosystem that has been created, achieve longer-term impact in participating provider organisations, and carry forward learning across a wider range of provider organisations not participating in the programme (as envisaged in the Wachter Review). This will require a degree of central coordination to keep focus on the overarching vision, and support and buy-in from emerging local structures. It needs to be reinforced through longer-term knowledge sharing for continuous improvement, and by maximising the impact of various interrelated knowledge sharing mechanisms identified in this evaluation.

What next? Lessons to carry forward

The progress achieved and lessons learned from the GDE Programme need to be carried forward to inform the development of the broader NHS ecosystem:

Building and maintaining national organisational memory

To ensure that the learning achieved under the GDE Programme is taken forward, it is important to build long-term organisational memory around large-scale digitally enabled transformation initiatives. This includes consideration of how to retain, sustain and best utilise the capabilities and experiences that have been accumulated within national and local organisations during the programme. Clear national recognition of what the sites have achieved in the programme is needed, accompanied by an outline of how the NHS will draw on this learning to inform future programmes.

Addressing the digital divide

Lessons learned from the GDE Programme should inform the development of the broader NHS learning ecosystem and ongoing initiatives to address the existing digital divide across organisations. Although some experiences may not easily transfer to organisations with lower digital maturity, others will.

Early involvement of participating provider organisations and cumulative development of programme management tools

Programme management tools need to be iteratively refined and streamlined, with stakeholder input, to simplify and reduce the burden associated with a multiplicity of programme management and reporting tools. A shared understanding of and capability in planning and using these tools is essential as an intrinsic aspect of digitally enabled transformation. Benefits realisation tools need to be developed jointly across user groups and applied from the outset to plan changes. The learning that widespread engagement delivers transformation therefore also applies to the co-development of appropriately rigorous programme governance arrangements.

Retaining and developing transformation expertise

Developing, retaining and re-using digitally enabled transformation and programme management expertise is important to enable strengthening and sustaining and wider utilisation of valuable, expensively acquired, experience-based learning. There is an opportunity here to look at the role GDE and FF staff can play in wider networking and buddying to support other organisations to mature or link to the Digital Academy and a growing digital alumni network.

Institutional design

Current proposals to shift programme management roles to regions will bring benefits from greater proximity between managers and providers, but may risk dispersing valuable national capacity. There is also a risk that regional actors will not have sufficient intensity of engagement needed to establish specialist expertise. Institutional design needs to consider trade-offs between central and local deployment. Some specialist functions may best be undertaken centrally (such as oversight of markets). Some expertise may best be maintained by a system-wide division of labour (such as procurement) but could be deployed through a matrix of regionally located stakeholders coordinated through ICSs and Integrated Care Partnerships (ICPs).

Establish a visible national function to support market management

The GDE Programme has contributed to establishing a national function to manage the market. Managing the market is a long-term project impinging on all digital programmes. This function now needs to be made visible at local level through expansion and formalisation taking into account long-term investment into the market (to attract newcomers and increase competition), while setting interoperability standards and priorities to help nudge the market toward a more agile, platform-based approach to EHR. It also needs to facilitate and support collaborations between provider organisations within existing user groups.

Long-term vision, strategic support and consistent senior leadership to sustain digital transformation

Vision and senior leadership support is required both in provider organisations with senior digital transformation leadership represented at board level, and nationally, to ensure local organisations can follow a stable overall vision of digital health system transformation. The extension of the NHS Digital Academy is likely to accelerate this. Strategic decision makers need to consider how to ensure the momentum established by the GDE Programme and related initiatives can be sustained i.e. how to establish a self-sustaining ecosystem. There is a need for a long and thin funding stream to establish infrastructures (particularly in less digitally mature organisations), maintain momentum and reinforce the legacy of the GDE Programme. Resources and incentives are needed to support this and the regions may be able to facilitate these developments.

Ensuring that digital becomes mainstream, operationally and in terms of health and care strategy and policy

This includes:

Alignment with other existing change programmes and digitally enabled transformation initiatives: This includes digital transformation funding streams, skills development and networking activities but also, positive action to ensure organisations and systems consider where digital solutions can support sustainability and outcomes in broader service improvement and transformation.

Including digital capability in regulatory and assurance structures: such as assessing and monitoring digital maturity of organisations and local health economies needs to become the norm.

Digital capabilities in institutional operating environments: top level governance support, new digital transformation and skills capabilities, informatics expertise, and clinical engagement.

There are real advantages for pace and scale of progress from ensuring that digital transformation priorities align with wider organisation and system priorities, allowing organisations to align different funding streams and change programmes to optimise impact around a clear shared vision.

Maximising the value of formative evaluations: Traditional summative evaluation methods, based upon discrete changes, do not effectively capture and guide complex, digitally-enabled transformation developments. This is because digital transformation involves extended chains of interaction around infrastructural changes that exceed reporting timeframes and create attribution problems. Formative evaluation approaches exploring processual outcomes (such as this one) feeding back emergent changes and helping to mitigate risks are key going forward.