By Dr. Kevin Shepherdson, DBA (GenAI), Founder and CEO, Straits Interactive, and Harish Pillay, Advisor, Straits Interactive.
Regulators, industry bodies, and consultants increasingly encourage organisations to begin their AI adoption journey in a sandbox environment.
Singapore, for example, has introduced initiatives such as the Generative AI Evaluation Sandbox and Global AI Assurance Sandbox to support responsible AI testing and assurance. The EU AI Act similarly requires Member States to establish regulatory sandboxes where innovative AI systems may be developed, trained, tested, and validated within a controlled environment before being placed on the market or put into service (European Parliament & Council of the European Union, 2024).
This is good practise. A sandbox allows organisations to experiment with AI within controlled boundaries before exposing employees, customers, operations, or the public to unnecessary risk. Singapore, for example, has introduced initiatives such as the Generative AI Evaluation Sandbox and Global AI Assurance Sandbox to support responsible testing and deployment. The EU AI Act similarly provides for regulatory sandboxes where innovative AI systems can be developed, tested, and validated for a limited period before being placed on the market or put into service.
All of this is good, but we would be concerned if an organisation deployed an AI system without first testing it in a sandbox or another controlled environment.
However, the sandbox must be understood as the beginning of the assurance journey—not the final gateway to deployment.
The problem arises when a successful sandbox exercise is treated as evidence that an AI system is ready for the real world.
It is not.
A sandbox can demonstrate that a system performs acceptably under selected conditions. It cannot prove that the system will remain safe, fair, reliable, or governable when it encounters real users, operational pressure, live organisational data, interconnected systems, malicious actors, and situations the project team never anticipated.
This is what we call the Sandbox Illusion:
The belief that because an AI system behaved safely in a controlled environment, it will behave safely once deployed into an uncontrolled one. The irony is that the very characteristics that make a sandbox safe also make it less representative of reality.
What a Sandbox Can—and Cannot—Prove
A well-designed sandbox can help determine whether:
1. the application performs its intended task;
2. obvious hallucinations, bias, privacy, or security weaknesses are present;
3. users can bypass its safeguards;
4. data restrictions operate correctly; and
5. the system fails safely under known test conditions.
These questions are necessary, but they are not sufficient.
A sandbox primarily shows how AI behaves when its users, data, permissions, workflows, and operating environment are controlled. The real world removes many of those controls.
Test users are generally cooperative. Real users may misunderstand the system, ignore warnings, over-trust recommendations, enter inappropriate information, or deliberately manipulate it.
Sandbox data is commonly curated, cleaned, anonymised, or limited. Production data may be incomplete, inconsistent, confidential, biased, duplicated, outdated, or incorrectly labelled.
Test workflows are usually simplified. Production workflows involve legacy systems, changing responsibilities, third-party platforms, external APIs, and people working under pressure.
Finally, sandbox consequences are contained. Production mistakes can affect a real employee, customer, patient, student, citizen, or organisation.
An AI system may therefore pass every planned test and still fail when it encounters a situation that was never included in the test plan.
This does not necessarily mean that the testing was poor. It means that no controlled environment can fully reproduce the complexity and unpredictability of real-world use.
Why AI Risk Changes Outside the Sandbox
AI systems are not merely technical products. They are sociotechnical systems.
This means their behaviour and risk are shaped not only by the underlying model, but also by the people, data, organisational processes, rules, incentives, and technologies surrounding it.
Risk emerges from the interaction between:
1. the AI model;
2. the data it processes;
3. the people using or affected by it;
4. the workflow in which it is embedded;
5. the authority granted to it; and
6. the organisation’s incentives and operating pressures.
Inside the sandbox, these factors are constrained. Outside it, they interact in ways that may be difficult to anticipate.
Consider an AI assistant that drafts responses to customer complaints. In the sandbox, the team may test whether its answers are accurate, polite, and consistent with company policy.
Once deployed, however, a staff member under time pressure may send a response without checking it. The system may receive a complaint written in an unusual dialect. It may retrieve an outdated policy or infer sensitive information from the customer’s message. It may produce an answer that is technically plausible but inappropriate for a vulnerable person.
The failure may not lie solely in the model. It may arise from the user’s reliance on it, the design of the workflow, poor source material, the absence of an escalation process, or pressure to handle more complaints with fewer employees.
This is why AI governance cannot stop at technical testing.
What 500 AI Incidents Reveal
In researching 500 publicly documented AI ethical breaches for my book, AI Ethical Breaches to Beware Of (Shepherdson, 2026), We found that failures were heavily concentrated at the later stages of the AI lifecycle.
Approximately 35.2% of incidents were traced to the Decide stage, where AI influenced or shaped decisions. A further 49.6% became visible at the Distribute stage, where outputs reached users or the public. Together, these two stages accounted for 84.8% of the incidents analysed.
This finding is significant.
It suggests that AI systems do not necessarily fail while they are being developed or tested. They often fail when they are embedded into workflows, interpreted by people, connected to other systems, and used at scale.
The risks become visible only after the AI leaves the controlled environment. The incidents also show that harm is rarely caused by one technical defect. It commonly results from several weaknesses converging:
1. incomplete testing;
2. poor data governance;
3. excessive user trust;
4. unclear accountability;
5. weak human oversight;
6. inadequate monitoring;
7. ineffective escalation mechanisms; and
8. use in a context not considered during development.
A sandbox may detect some of these weaknesses. Others emerge only during actual use.
The Problem with Edge Cases
Project teams frequently describe unusual scenarios as “edge cases.”
This language can be misleading.
An edge case may be statistically rare from the developer’s perspective. But to the individual affected, it is not peripheral. It is their actual case.
A hiring system may perform accurately for most candidates but unfairly reject someone with an uncommon employment history.
A healthcare chatbot may provide useful general guidance but fail when symptoms are described using non-standard language.
A translation system may perform well in major languages but distort meaning for smaller linguistic communities.
An AI agent may execute hundreds of routine actions correctly but delete critical information when two unusual conditions occur simultaneously.
At scale, even low-frequency failures can affect many people. An error rate of 0.1% may appear insignificant in a pilot involving 100 transactions. Across one million transactions, it represents 1,000 potentially affected cases.
A successful pilot should therefore not be judged only by average performance. Organisations must also consider the severity, reversibility, and distribution of failures.
The question is not merely: How often does the AI fail? It is also: Who is affected when it fails, how serious is the consequence, and can the harm be reversed?
Verification Is Not Validation
A useful distinction is the difference between verification and validation.
Verification asks: Did we build the system correctly?
It assesses whether the application performs according to its specifications and technical requirements.
Validation asks: Did we build the right system for the real environment?
It assesses whether the system remains appropriate when used by actual people, in actual workflows, under actual operating conditions.
Sandboxes are generally stronger at verification. They can demonstrate that functions work, permissions apply, safeguards respond, and outputs meet expected standards under test conditions.
Real-world validation is harder.
It asks whether users understand the system’s limitations, whether human oversight remains meaningful under pressure, whether low-frequency harms can be detected, and whether affected individuals can question or appeal an outcome.
An AI system may therefore be technically correct but operationally unsafe.
When Sandbox Success Creates False Confidence
The Sandbox Illusion becomes dangerous when a successful test generates more confidence than the evidence justifies.
Project teams may conclude that the system is safe, safeguards are effective, oversight is adequate, and the AI is ready to scale.
But the evidence may show only that the system behaved acceptably:
1. with selected users;
2. using selected datasets;
3. for a limited period;
4. under close supervision;
5. with restricted access; and
6. without full production integration.
That is not proof of operational resilience. A sandbox should therefore be treated as the first layer of assurance—not the final governance gate.
From Sandbox to Deployment: A Staged Assurance Framework
Organisations should not move directly from sandbox testing to full deployment. Additional stages are needed to introduce real-world complexity, exposure, and consequences progressively.
Stage 1: Sandbox Testing
Assess technical feasibility and identify obvious failure modes.
Testing should cover functionality, privacy, security, bias, hallucination, prompt injection, misuse, and safeguard effectiveness. The environment remains controlled and consequences contained.
Stage 2: Controlled Pilot
Test the system with a limited but representative group of users.
This stage examines workflow fit, usability, misunderstanding, overreliance, and unanticipated behaviour. It should include non-technical users and people who may interact with the system differently from its designers.
Representative data and realistic scenarios should be introduced where legally and ethically appropriate.
Stage 3: Limited Production Deployment
Deploy the AI within a genuine operating environment but restrict its scope. This may involve one department, a defined customer group, limited transaction volumes, or lower-risk use cases.
Monitoring should be intensified. Significant errors, overrides, complaints, near misses, and unexpected behaviours should be reviewed.
At this stage, the organisation is no longer testing only the AI. It is testing whether its governance controls work under real conditions.
Stage 4: Progressive Scaling
Expand deployment only when evidence supports it. As usage increases, reassess model performance, user behaviour, data quality, security, fairness, human oversight, workload impact, incidents, and near misses.
Additional autonomy, system access, or decision-making authority should be granted gradually. The principle is simple: AI should earn the right to scale.
Stage 5: Full Deployment
Full deployment should occur only after the organisation has demonstrated that the system can operate responsibly under representative production conditions.
There should be clear ownership, documented controls, user training, escalation procedures, incident-response mechanisms, and accountability for outcomes.
Full deployment is not the end of governance. It is the beginning of ongoing operational responsibility.
Stage 6: Continuous Monitoring and Assurance
AI systems and their operating environments continue to evolve after deployment.
Users change. Data changes. Workflows evolve. Vendors update their models. New vulnerabilities emerge. Teams may apply the system to uses that were never originally approved.
Continuous monitoring should therefore examine:
1. model and data drift;
2. emerging misuse;
3. complaints and appeals;
4. safeguard bypasses;
5. changes in output quality;
6. integration and dependency risks;
7. regulatory developments; and
8. whether the system remains fit for purpose.
Monitoring must include clear triggers for review, suspension, rollback, or redesign.
Everyone Is a Guardian of Trust
AI governance is not the sole responsibility of technical teams, compliance departments, or the Chief AI Officer. Everyone involved in designing, approving, deploying, using, or monitoring an AI system has a role.
Leaders must ensure that commercial urgency does not override assurance.
Technical teams must test beyond expected use and communicate limitations honestly.
GRC, legal, privacy, and security professionals must convert principles and obligations into operational controls.
Business owners must ensure the system remains suitable for the workflow and that accountability is never delegated to the machine.
Users must question outputs, protect data, report anomalies, and escalate concerns.
Risk, audit, and assurance teams must verify that controls operate in practice—not merely exist in documentation.
Together, they are the guardians of trust.
Before moving towards deployment, every organisation should ask:
1. What controlled conditions will disappear outside the sandbox?
2. Which users, environments, and scenarios were not represented?
3. What happens when the system is wrong?
4. Who can stop or override it?
5. How can affected people challenge an outcome?
6. What evidence will justify moving to the next stage?
7. How will new risks be detected after deployment?
Conclusion
Sandboxes are an essential starting point for responsible AI adoption. They allow organisations to experiment, learn, identify weaknesses, and test safeguards without immediately placing people or operations at risk.
But they must be recognised for what they are: controlled learning environments, not certificates of safety.
The real test begins when AI encounters imperfect data, unpredictable users, operational pressure, connected systems, and situations its designers did not anticipate.
A successful sandbox can show that the system works under selected conditions. Responsible governance requires evidence that the system continues to work when those conditions no longer exist.
This is also why, in our Apps Design and Prompt Engineering course, I repeatedly reinforce one message throughout the AI system lifecycle:
Test. Test. Test.
Test the intended use. Test the unintended use. Test with different users, data, languages, workflows, and operating conditions. Test the safeguards, human oversight, escalation procedures, and the system’s ability to fail safely. Then continue testing as the application progresses from the sandbox to pilot, limited production, full deployment, and continuous monitoring.
Testing should not be treated as a one-off technical activity conducted before launch. It is an ongoing governance responsibility.
The question is therefore not merely whether the AI is ready to leave the sandbox. It is whether the organisation is ready to govern what happens next.
