This week, the Google test blog newsletter was about GTAC, the Google Test Automation Conference. I found this session on AI applied to testing really relevant to:

Free Tests Are Better Than Free Bananas: Using Data Mining and Machine Learning To Automate Real-Time Production Monitoring (Celal Ziftci of Google)

The session was about Google’s assertion framework which runs against their production logfiles. The framework runs on real time logs, checking for inconsistencies. Examples of meaningful assertions:

transaction.id > 0, transaction.date != null.

If any assertions fail, a notification is sent to a developer to take some kind of action. Usually a developer would have to design assertions, but now they use a tool to assist.

Daikon invariant detector identifies invariants (rules which are always true for a certain section of code). Although Daikon has been designed to analyse code, they have modified it to work on data like logfiles. Daikon starts with a set of hypotheses about what your input will look like, and as you push data through it, it eliminates those which prove to be false. The rules identified can be used as assertions, thereby automatically generating test cases. These test cases still need a developer to determine value and validity, however.

The other technique they use is Association Rule Learning, which finds relationships between different data items, e.g.

when country is uk, time zone is +0

These, too, are added into the assertion framework to identify issues occurring in production.

In this case, the work that developers used to do in defining test cases is now being done by machines. But human beings are still needed to made a decision on whether the identified rules make sense and add value or not.

The AI system, at times, identifies trivial rules, but is also capable of identifying complex relationships that would be less obvious to humans.