← Back to blog

How secure is code generated with AI by someone without technical training

The enthusiasm around generative Artificial Intelligence has turned everyone into a potential "software creator". Today, with a few plainly written prompts, a sales manager, a marketer or a business owner can build their own website, a customer management module or an internal app. Without knowing a single line of code.

Behind the spectacular speed of execution, however, lies a major problem that the business world is already starting to feel: code generated without context and without technical oversight creates huge technical debt and massive security holes.

Gartner warns that by 2028, prompt-to-app approaches adopted by so-called "citizen developers" (users without a technical background) will increase software defects by 2,500%, triggering a quality and reliability crisis.

Here is what the industry figures show, and why apps generated with AI by people without a technical background can become your company's biggest vulnerability.

How does AI code generation work, and where do the risks come from?

When a person without technical training uses AI to create an app or a website, a phenomenon called automation bias kicks in: they tend to trust blindly whatever the screen shows them.

If the interface looks good and the button works, the assumption is that everything is fine.

In reality, AI models don't "think" about security; they only predict the next piece of code based on patterns in the data they were trained on. Without an engineer who understands the architecture, AI generates code with no context: it doesn't know how the data is protected, who has access to it or how the system behaves under attack.

The figures from recent studies on the security of AI-generated code confirm this risk:

  • 45% of tasks introduce vulnerabilities: The Veracode GenAI Code Security Report, which tested 80 programming tasks on more than 100 AI models, shows that in 45% of cases the generated code introduces a vulnerability from the OWASP Top 10 list. Java proved to be the riskiest language, with a failure rate of over 70%.
  • 62% design flaws: Research published by the Cloud Security Alliance shows that 62% of AI-generated code solutions contain design flaws or known vulnerabilities, even when the latest models are used. The researchers' explanation: the model doesn't know the risks specific to your application, your internal standards or the context it runs in.
  • Non-existent software packages: A study presented at the USENIX Security 2025 conference analyzed 2.23 million code samples generated by 16 models and found that 19.7% of the recommended packages don't actually exist, with 205,474 completely made-up names. The part that turns the mistake into a vulnerability: 43% of the made-up names reappear identically every time the same request is repeated. Attackers can find out which they are, register them in public repositories and load them with malicious code, which the app then downloads on its own.
  • A rise in registered vulnerabilities (CVEs): The Vibe Security Radar project, run by Georgia Tech's software security lab, tracks public vulnerabilities caused directly by AI-generated code. By the end of March 2026 it had confirmed 74 cases, 14 of them critical, on a trajectory of 6 in January, 15 in February and 35 in March. March alone surpassed the whole of 2025. The researchers point out that 74 is a lower bound, since most AI-generated code leaves no traces that would allow attribution.
  • Successive tweaks make the problem worse: The Security Degradation in Iterative AI Code Generation experiment, presented at IEEE-ISTAS, tested 400 code samples over 40 rounds of successive improvements. After just five rounds in which the model was asked to refine the code, critical vulnerabilities had increased by 37.6%. While fixing the visible error, it introduced others at the architecture level.

Where are the biggest security risks?

In apps built without a predefined security architecture, the major risks usually show up in the following areas:

  • Personal data leaks (GDPR) and exposed databases: the app works, but anyone who changes a link can see other customers' data, because nobody asked for authorization rules and the model doesn't add them on its own initiative.
  • SQL injection and unprotected forms: the contact or order form sends data straight into the database without checking it. An attacker can take control of the site by entering malicious code directly into the "Name" or "Message" field.
  • Admin account takeover: the lack of security limits (guardrails) means passwords or access keys (API keys) are left in plain sight in the site's source code, easy to extract for the bots that scan the internet.

The consequences don't stay technical. An exposed database containing customer data falls under GDPR, no matter who wrote the code, and the liability stays with the company, not with the tool used to generate the app.

How do you use AI in your business without putting your data at risk?

AI is a tool of undeniable value for speed and prototyping, but it cannot replace the rigor of software engineering.

If you want the benefits of automation and fast apps without creating a security crisis in your company, apply these 4 simple rules:

  • Use AI for prototypes, not for production: it's great to use AI to quickly test an idea or a visual flow. But before putting the app in front of customers or connecting it to real data, the code must be reviewed by a specialist.
  • Don't automate without human inspection (human-in-the-loop): no automatically generated line of code should be pushed to a server without a vulnerability review process.
  • Set strict rules and access limits: define clearly which data an app is allowed to access and where automatic task execution must stop.
  • Check periodically, not just at launch: the data above shows that vulnerabilities surface over time, not on the day the app goes live. An app that has moved past the prototype stage and works with company data needs repeated checks.

Instead of a conclusion…

Being able to build apps and automate processes by simply describing, in your own words, what you want is without a doubt one of the most spectacular technological developments of recent years. It's impressive how quickly you can go from an idea to a working prototype today, and how much access to technology has been democratized.

AI gives teams unprecedented testing speed and freedom to innovate. The point is not to discourage this initiative, but to complement it with the necessary rigor. For a quickly built tool to become a safe and lasting resource for the company, the agility AI offers must be backed by a sound technical architecture and clear security rules.

The competitive advantage comes when the speed AI brings is supported by solid infrastructure that protects your data, your customers and the continuity of your business.