What AI Implementation Looks Like in a Large Organization With Legacy Systems

Why does a successful pilot not mean you have implemented AI?

Pilots are easy to start and hard to carry. In a survey of 1,993 respondents in 105 nations, McKinsey reported in 2025 that nearly two-thirds say their organizations have not yet begun scaling AI across the enterprise. The same survey found that larger companies, by revenue and by headcount, are more likely to have reached the scaling phase. It does not say why.

For a CIO or CTO with aging infrastructure, the takeaway, in our view, is that the pilot is the cheap part. Implementation is the work of connecting AI to the systems your business already runs on, with owners, controls and a way to tell whether it helped.

What does a phased rollout look like?

Picture a hotel group whose reservations, loyalty and property systems were built over two decades by different vendors. Nobody there wants to rebuild all of it to try AI. A phased rollout gives them a way to start with one workflow and earn the next.

Phase	What happens	Question to answer before moving on
1. Pick one workflow	Choose a single, high-volume workflow with a named owner and a number you can measure.	Can we state the expected value in one sentence?
2. Connect at the edges	Read from and write to existing systems through their interfaces. Leave the core alone.	Can we switch the AI layer off without breaking the system of record?
3. Run with guardrails	Start with a limited group of users, human review and logged decisions.	Can we see what the AI did and who approved it?
4. Measure against a baseline	Compare results to the numbers you captured before launch.	Did it earn the right to scale?
5. Reuse and repeat	Carry the integration and governance work into the next workflow.	What did we learn that changes the next plan?

The narrow start matters. In a 2024 survey of 1,000 executives, BCG described the AI leaders it identified as companies that invest strategically in a few high-priority opportunities to scale and maximize AI’s value.

How do you add AI without ripping out what you have?

Treat each legacy system as a source of truth with a boundary. Your job is to find the seams: the APIs, message queues, database views and file exports that already move data in and out. AI connects there.

This is our view, and the specifics depend on your estate. The sources we cite do not compare integration architectures, so treat the following as engineering judgment, not research findings:

• Keep the system of record authoritative. AI suggests or drafts, and the existing system stores the result.
• Put a thin layer between AI and each legacy system so you can change either side without rewriting the other.
• Log every call in and out, so you can reconstruct what happened.
• Plan the off switch before the launch. If the AI layer fails, the workflow should fall back to how it ran before.

When should governance start?

Earlier than most teams plan. Board.org reported in 2025 that 31% of data leaders at billion-dollar organizations, as reported by respondents, said they were still in the early stages of defining AI governance policies. It also reported that a full third of surveyed leaders named embedding governance into data workflows as their top modernization priority.

Read together, those findings point to the same design choice. In our view, governance belongs in the delivery workflow: access rules, review steps, audit logs and clear ownership of each AI-assisted decision. A governance document written after launch describes a system people already depend on and cannot easily change.

Where do staff and training fit?

Probably in more of the plan than you expect. BCG’s 2024 survey reported that around 70% of the challenges companies face implementing AI stem from people and process issues, 20% from technology problems and 10% from AI algorithms. Those are the publisher’s categories, and BCG did not present them as an independent root-cause audit. Even so, they suggest the model is rarely the only thing that has to change.

Plan training around the actual workflow. Show people what the AI does, where it is allowed to act, when they must review its output, and who to tell when it gets something wrong. Name an owner for the process and for the tool.

How do you show the value is real?

Capture the baseline before launch. Handling time, error rate, backlog size or rework: pick one or two measures the business owner already trusts. Then compare during the limited run and decide with numbers.

BCG reported in 2024 that 74% of companies had yet to show tangible value from their use of AI. That figure comes from a survey of executives, so treat it as a prompt to define value before you build.

We do strategy, design, engineering and AI in one team.

If you are planning your first AI workflow on top of systems you cannot replace, start a conversation with our team.