Short answer

In the experiments, using an AI agent under the architectural oversight of an experienced engineer produced more predictable results than fully autonomous development.

Breaking tasks into logical stages, providing clear instructions and regularly reviewing intermediate results helped reduce errors and unnecessary code. AI agents working without sufficient oversight, however, led to unwanted features, additional costs and a build-up of technical debt.

Effectively integrating AI into the development of an existing software product requires up-to-date documentation, clear rules for working with AI agents, engineering oversight and a prior assessment of risks to system stability, security and ongoing maintenance.

Development with an AI Agent: Pair Programming vs. “Autopilot”

Hello everyone. My name is Igor, and I’m one of the co-founders of the DevCats development team. I have more than 20 years of experience in web and mobile development, including extensive experience in the GameDev industry. For the past year and a half, I’ve been using AI in software development workflows and periodically conducting research in this area.

Today, I want to revisit the topic of AI-assisted software development. This time, I’ve come with several conclusions based on efficiency measurements and my own observations.

 

So, let’s get started.

For our tests, we used two different approaches: pair programming with the Codex agent, and development based entirely on a predefined plan using a swarm of agents (with testing also performed by agents).

Although the tasks were different in nature, significant differences in productivity and efficiency became obvious within the first two days.

Codex with the Sol 5.6 model was tested in pair-programming mode in two scenarios related to the gaming industry, using the C programming language. We were developing a plugin (a functionality extension) for a very popular PC game.

The first approach involved me acting like an ordinary user assigning tasks. Everything was done within a single thread and a single branch, using short comments that provided enough information to understand the task without adding unnecessary complexity. From time to time, Codex and I also discussed potential solutions together.

As a result, we quickly reached 7,000+ lines of code. Toward the end, token consumption began growing exponentially, and by the time we were close to completing the task, we had returned to the same recurring problem several times, even though it was relatively trivial when approached with the right architecture.

We also periodically had to fix bugs related to source code compilation errors on the game server.

The next day, I decided to repeat the experiment using a different approach, where I acted as the architect and divided the work into separate threads (individual agent branches in Codex).

I defined the architecture from the very beginning and worked toward the final objective through separate logical stages, gradually expanding the functionality.

At almost every stage, the AI agent and I discussed possible approaches before integrating the code. I carefully reviewed its suggestions and, in roughly 15% of cases, made adjustments, rejected parts of the proposed solutions, or suggested alternative approaches.

As a result, I ended up with 6,000 lines of code — 12% less than in the previous experiment — zero compiler errors, and cleaner final documentation.

When we encountered the same problem again — the one that had required several different “workarounds” in the first approach — this time it was solved in a single attempt.

There was also only one section of code responsible for this vulnerable area, compared with two in the previous approach.

Considering that we were solving a scaling problem — changing the size of objects in the game world in real time — this logic should ideally have been centralized in one place. That way, the entire process could be controlled from a single section of the program, and if new issues appeared with other objects, we could introduce object-specific conditions exclusively in that one place.

In this case, the approach I controlled while acting as the process architect proved entirely justified, even based on this example alone.

Another interesting observation is that the more detailed your task descriptions become, the more unnecessary components and checks you tend to get in the code. By over-specifying the task, you give the model less freedom to suggest alternative approaches that may actually be perfectly reasonable.

At the same time, the less you write and the less attention you pay to the conclusions and output produced by the AI model, the more technical debt you accumulate during development.

Technical debt means potential problems that you will eventually have to deal with later.

My conclusion here is obvious to me: you need to find a balance between short prompts and large, highly detailed sets of requirements. At the same time, a human-driven architectural approach is essential.

 

Now let’s move on to the conclusions regarding agentic programming in “autopilot” mode.

With this approach, we had a predefined list of tasks consisting of short bullet points, while all work related to writing code, modifying tasks, testing, and accepting the results was delegated to Codex AI agents using the Sol 5.6 model. We also tested the Astra model separately.

In this case, roughly 30% of the functionality we received turned out to be unnecessary.

Yes, the code worked, but the final result forced us to make a pivot — changes to the task that required reworking the result we had already received.

I also want to specifically highlight token consumption.

For Sol 5.6, the cost was approximately $100 over three days.

With Astra, things were even more entertaining: $100 in two hours.

This experiment was conducted by my colleague and partner in the GameDev field, who has an extensive technical background as an architect.

According to his assessment, most of the money was spent on checks performed by the agents. They spent too much time circling around each task and making numerous corrections.

My conclusion is this: with this approach, you give agents freedom, but you pay for that freedom with significant financial costs and receive an unpredictable result.

And the latter is the key issue here.

Spending $100 over two days using Sol 5.6 is not a significant amount compared with developers’ salaries. But receiving results that differ from what you expected is an entirely different matter.

It forces you to return to the task, create a new list of corrective tasks, run the process again, and then come back once more to review the resulting output.

With this approach, the developer is also forced to go in circles, just as in the earlier example where technical problems had to be solved while working with AI without an architectural approach and step-by-step supervision.

Another thing that concerns me is the substantial amount of technical debt that accumulates almost invisibly.

The amount of code keeps growing, the documentation grows along with it, and nobody can reliably predict how much the financial cost of AI model tokens and developer time will increase after several months of working this way.

There is a very high chance that such a scenario will lead to unpredictable consequences.

However, I’m not saying this is a bad approach. It simply has significant disadvantages.

And the problem is not the quality of AI models or HARNESS tools such as Codex, Claude Code, and others. They are capable of “thinking,” but these very “thought processes” are part of the problem.

It works much like it does with humans: you give someone a task, and they decide to add a lot of things you never asked for. Meanwhile, what you actually asked them to do gets implemented in their own way rather than the way the client expected.

 

Conclusions

Today, in my opinion, any new software project should be developed using AI agents.

Which HARNESS to choose — Codex or Claude Code — and which model to use are matters of preference and technical considerations that should ultimately be left to the developer. These are their tools, and they should make that decision.

However, it is important to note that, from my perspective, the question is no longer whether to write everything yourself, work together with an AI agent, or delegate all tasks to AI agents.

For me, the option where “only a human writes the code” has been completely removed from that list.

What about large existing projects?

This is where things become more complicated.

If the goal is to move everything onto “new rails,” you need to invest money in organizing the process properly. This includes creating extensive documentation, building a list of skills — documented skills and approaches that the AI agent should apply — and writing business documentation covering all business cases and descriptions of user interaction interfaces within the web system.

Without these steps, integrating an AI agent into the development process is essentially an experiment on the business with unpredictable consequences.

Ultimately, if you choose an architectural approach involving an engineer, the final results will depend heavily on two factors: the developer’s ability to work effectively with AI agents, and whether that developer genuinely has strong software architecture skills.

And there is another important aspect: it is not enough to simply know how to design an architecture. The developer must also know how to modify that architecture while accounting for a new participant in the process — the AI agent.

Applying a swarm of agents to large existing projects — even when everything is documented in detail and a human partially intervenes in the process — is, in my view, a pure experiment with unpredictable results.

And I think this is an entirely normal situation.

A technological revolution has taken place, and it requires developers to acquire new skills for new realities and new projects.

The approach to maintaining and extending existing business solutions should, wherever possible, be adjusted based on an assessment of all potential risks associated with introducing new tools.

The objective should be to preserve stability throughout the entire software lifecycle within an LTS framework.

Updated:

Was this article useful?

0 likes
← Back to all articles