The Day Every Talk Was the Same Talk

Welcome back everyone 👋

This week's Automation Playbook covers:

🎤 Why we audited ourselves first

🔍 The unknown unknowns of AI-built products

🗺️ The afterthought people actually pay for

Let's get into it 👇

On 20 October 2023, I went to TechMids Conf at the ICC in Birmingham with Kurt.

A full day.

Different speakers.

Different technology stacks.

Different war stories.

By about 3 in the afternoon, I realised they had all given the same talk.

None of them were really talking about code.

They were talking about everything that sits around the code.

Versioning.

Testing.

Documentation.

Performance.

Accessibility.

Security.

The disciplines that determine whether software is genuinely ready for the real world.

We went away and built those disciplines into how we work.

That part went fine.

Then we hit a problem I had not expected.

We run bespoke projects at wildly different scales across wildly different sectors.

And I had no way of answering a very simple question:

Are we actually doing this well, or do we just think we are?

There was no scoreboard.

Why we audited ourselves first

We had spent two years getting good at something we could not measure.

Discipline without a scoreboard is just a belief about yourself.

So we turned the whole thing on ourselves.

We wrote nine agents to find the flaws in our own implementation of the frameworks we had adopted.

Not to check our clients.

To check us.

There is a Sentinel who worries about security.

An Optimiser who cannot let a slow query go.

A Librarian who is quietly furious about your README.

A Guardian watching the wider product.

The first thing they did was mark our homework.

That mattered more than I realised at the time.

We are not asking clients to submit to something we have not survived ourselves.

That is the credibility of the offer.

Those audits now run monthly across our client base.

They produce a report showing the real state of each product, not the state everyone hopes it is in.

It started with nine agents.

There are ten now.

Nugget #1: Auditing ourselves first is what made the service sellable. If you want people to trust your standard, prove you are willing to be measured against it too.

The unknown unknowns of AI-built products

Then vibe coding arrived, and the whole thing became a product.

AI is genuinely excellent at planning and building.

But it has no regard whatsoever for everything surrounding the build unless somebody explicitly asks.

Versioning.

Testing.

Documentation.

Performance.

Accessibility.

Security.

The things that do not appear in the prompt because the person writing it does not know they need to ask.

That is the unknown unknowns problem.

You cannot fix a problem your tools have never mentioned to you.

We kept meeting people who had built something impressive and then simply stopped.

They had a prototype.

But they had not launched it.

Or they did not know how.

The build existed, but the path from built to live, safe, and scalable did not.

That is where the agents matter.

They do not just inspect what was built.

They look at everything the build process forgot to mention.

Nugget #2: AI does not just have blind spots. Outside the build, it can be blind to the entire operating environment. Quality comes from checking what the prompt left out.

The afterthought people actually pay for

At first, the audit was the product.

It found the gaps, graded the product, and showed what was wrong.

Useful.

But not enough.

Because a list of problems can leave someone in exactly the same place as before.

They know more, but they still do not know what to do next.

So we added a gap analysis and a roadmap.

What needs fixing before launch?

What can wait?

What is the fastest route to getting live?

What needs to change before the product can scale?

That was almost an afterthought.

It turned out to be the part people valued most.

The report explains the state of the product.

The roadmap creates a route forward.

There is a broader lesson in that.

The thing you build first is not always the thing people are buying.

Sometimes they are not paying to understand the problem.

They are paying to know what happens next.

Nugget #3: The part of the product we nearly did not build became the reason people say yes. Diagnosis creates clarity. A roadmap creates momentum.

What you can do this week

🔷 List the disciplines surrounding your build that nobody is currently measuring

🔷 Audit one of your own products before asking a client to do the same

🔷 Turn the findings into a prioritised route from where the product is now to live and scalable

Every developer knows the phrase: price, speed, quality, pick two.

People repeat it as if it is physics.

It is not physics.

It is a description of a business that has never automated its quality control.

We are now taking all three corners.

It took a room full of speakers giving the same talk for me to notice what they were really saying.

P.S. Whenever you’re ready, here’s how I can help:

• You can check out the latest episode of the Ctrl Alt Dev podcast, where I break down what's working right now in detail with my co-host Sean Sale: Apple | Spotify | YouTube

• Need a fresh perspective? I’m here to help. Book a free audit call with me, and we’ll figure it out together.