Someone at work has built an application using an AI coding tool. It solves a problem the team has been living with for months. The demonstration goes well, and other staff want to start using it.

The manager now has a decision to make. The application was built outside the company's usual software purchasing or development process. Nobody has established whether it is suitable for company data, whether staff can depend on it, or who will maintain it.

The employee may have produced something genuinely useful. Before putting it into normal use, the business needs to know what it is taking responsibility for.

When a prototype needs a review

An experiment using fictional data can be tested with relatively little exposure. The risks change when the application starts handling customer records, accessing company systems or influencing business decisions.

Consider an internal tool that prepares customer quotations. It might work perfectly with the examples used in development, but what happens if two staff members edit the same quotation? Can one employee access another customer's records? What happens if the connection to the accounting system fails halfway through an update?

The interface may give no indication that these cases haven't been tested.

AI-generated software deserves the same scrutiny as other software. Research has identified security weaknesses in AI-assisted programming, although the findings cannot be used to estimate the risk of any particular application.

For example, Perry and colleagues found that participants using an earlier AI coding assistant produced less secure solutions to several programming tasks and were more likely to believe their solutions were secure. A separate study by Fu and colleagues found potential security weaknesses in AI-generated code snippets collected from public GitHub projects.

Neither study tells us whether a particular internal application is safe. That requires examining the application and how the business intends to use it.

What to check first

A review should begin with the proposed use. A staff-only scheduling tool presents different risks from an application that stores customer payment details.

The business needs to explain who will use the application, what data it will handle and what would happen if it produced an incorrect result or became unavailable.

The technical review can then concentrate on the areas most likely to cause trouble.

Accounts and ownership

Who controls the source code, hosting account, database and connected services?

An application may run under the employee's personal accounts. Giving the business a password does not necessarily give it control of billing, account recovery or the right to transfer the service.

The review should identify every service the application depends on and who can administer it.

Access to information

A working login screen does not establish that users can access only the information they should see.

For an application with separate customer records, a basic test might involve two ordinary user accounts and fictional data. Can one account retrieve or change records belonging to the other?

The review should also follow the data through databases, logs, backups, email services and any AI providers used by the application.

Failure and recovery

Demonstrations usually follow a successful path. Normal use includes interrupted requests, repeated submissions, unavailable services and unexpected input.

A quotation tool might send the same quotation twice. A booking system might confirm a booking without saving it. An integration might update one system and leave another unchanged.

Testing should concentrate on failures that would cause the most harm or disruption.

Maintenance

Could another developer operate the application if its creator left?

At minimum, the business may need access to the current source code, deployment instructions, configuration details and a way to recover its data. The amount of documentation depends on the application's importance.

Deciding what happens next

The outcome of a review should relate to the application's intended use.

A tool might be ready for a limited trial using internal data, provided a few defects are corrected. Another might need work on access controls before customer information can be entered. Some prototypes are easier to rebuild than repair.

The report should identify confirmed faults, unresolved questions and restrictions on use. It should also say what was tested. A review of one version does not automatically cover later changes to the application or its connected services.

The business can then decide whether to run a pilot, commission fixes, arrange further testing or stop the project.

Employee-built software can solve problems that existing products have overlooked. A review gives the company enough information to decide whether the application can become part of its normal operations.