The Website You Built With AI Might Be Leaking Customer Data

The demo works. That doesn't mean it's safe.

When you build with AI — or hire someone off a marketplace who builds with AI — you get something that passes the only test most people run: does it do the thing? Click the button, the booking appears. Great. Done.

What that test never shows you is what's happening underneath. Where is the customer data actually being stored? Who can reach it? Is it locked, or is it sitting in a database that's open to anyone who knows the address? Are the passwords protected, or saved as plain readable text? Are the secret keys that connect everything together tucked away safely — or written directly into the code where anyone can copy them?

AI-generated code has a well-earned reputation for getting these exact things wrong. It'll happily leave a database wide open, hard-code a password, or skip the checks that stop one customer from seeing another customer's details — all while the demo runs perfectly. It's not being malicious. It's giving you what you asked for ("make a booking form") without the part you didn't know to ask for ("...that doesn't leak").

None of this is visible from the outside. That's the trap. A leaky app and a solid one look identical to you, to your customers, and to everyone — right up until someone who is looking for the gap finds it.

What a breach actually looks like for a small business

Forget the movie version of a hacker — the hooded figure in a dark room, the dramatic countdown. That's not what happens to a Christchurch café. Here's what actually happens.

The open database. A café builds a loyalty app. It collects names, emails and phone numbers — a few thousand of them over a year. The database it all lives in was set up with its security switched off by default, and nobody turned it on. For eight months, anyone with the right web address can download the lot. Nobody notices, because nothing looks wrong. Then a customer emails: "Why did I get a scam text that knew my name and that I drink flat whites at your place?"

The freelancer's shortcut. A builder hires someone cheap online to make a quote-request form. To get it working fast, the freelancer writes the secret keys — the ones that connect to the email account and the AI service paying by usage — directly into the public code. Months later, someone finds them, and either runs up thousands of dollars in usage on the builder's account overnight, or quietly reads every enquiry that ever came through the form.

The intake form that holds too much. A physio's AI-built booking system asks new clients for injury details and medical history. That's sensitive information under NZ law — a higher bar. It's being stored in a simple spreadsheet-style database with no real access controls, sitting one weak password away from being everyone's business.

Notice what these have in common. No genius attacker. No sophisticated break-in. Just an ordinary business that collected real people's data and — through no fault of their own understanding — stored it somewhere it shouldn't have been. That's the overwhelming majority of what goes wrong. It's not exciting. It's just expensive.

The law caught up — and it applies to you

Here's the part a lot of owners don't realise: this stopped being only a technical problem in December 2020, when the Privacy Act 2020 came into force. If your business holds personal information about customers — and if you take a name and an email, you do — the law now has expectations of you, no matter how small you are.

Two things matter most in plain terms.

You're required to keep it reasonably safe. One of the Act's core principles says you must protect the personal information you hold with security safeguards that are reasonable in the circumstances. "We used an AI tool and didn't check it" is not a defence anyone wants to be leaning on.

If it leaks, you often have to tell people. This is the big change. If you have a privacy breach that has caused, or is likely to cause, serious harm, you are legally required to notify the Office of the Privacy Commissioner and the affected people as soon as you can. Failing to notify the Commissioner when you should have is an offence, with a fine of up to $10,000. And there's a second consideration: sending customer data offshore — which many AI tools and cloud services do by default — comes with its own rules about making sure it's protected once it leaves the country.

But the $10,000 is rarely the real cost. The real cost is the email you have to send every customer explaining that their details got out. It's the trust that took you ten years to build, gone in the length of one apology. For a local business that runs on reputation and word of mouth, that is the number that should get your attention — not the fine.

Five blunt questions to ask about anything that holds customer data

You don't need to read the code. You need to ask the questions — of yourself, or of whoever built it for you — and not accept a shrug. Whether it's a booking form, an app, or an old website you forgot you had:

1. Where does the customer data actually live, and who can reach it? If nobody can give you a clear answer, that's the answer.
2. Is it locked down, or open by default? Databases and storage often ship wide open. Someone needs to have deliberately closed the door — not assumed it was shut.
3. How are passwords stored, and is there two-factor login on the admin? Passwords should be scrambled, never saved as readable text. The account that controls everything should need more than one password to get in.
4. Are the secret keys kept out of the code? The keys that connect your systems together should never be written into anything public. If they've ever been exposed, they need replacing.
5. Are we collecting more than we need — especially sensitive stuff? Every piece of data you hold is something you have to protect. Health details, dates of birth, anything sensitive: only collect it if you genuinely use it.

If you built it yourself with AI, ask the AI these questions directly and make it show you — it'll often own up to the gaps when you name them. If someone else built it, send them this list. The good ones will answer happily. The answer you get, and how fast, tells you most of what you need to know.

You don't need to become a security expert

Let's be clear, because it's easy to read all this and decide the safe move is to build nothing. That's the wrong lesson. Building fast with AI is a genuine advantage for a small business — the answer isn't to stop, it's to not ship customer data into the world without one honest check first.

Think of it like electrical work. You can absolutely put up your own light fitting. But when it's wired into the walls of a building the public walks through, you get a qualified person to confirm it's safe before you switch it on. Nobody thinks less of you for it. It's just what a responsible operator does. A website or app that collects your customers' details is the same — it's wired into your business, and other people are trusting it.

So build the thing. Move fast. Then, before real customer data starts flowing through it, get someone who knows what they're looking at to run their eye over those five questions. An hour of the right attention now is a tiny price against the email you never want to have to send. Fast and safe isn't a contradiction — it's just the version worth shipping.