Welcome to the World of Automation Powered by AI
With one API call, you can add the power of AI to your mobile test automation. The team at test.ai has teamed up with Jonathan Lipps, the lead contributor of Appium and founder of Cloud Grey, to add a bit of AI to Appium. The AI finds common elements in mobile apps such as search text boxes, login buttons, etc., so test developers don’t have to worry about all those magic IDs, CSS, or XPaths. Just tell the AI what you want it to find and it will find it for you on the page — even if the element changes color, text, location, or position in the DOM. With AI, tests will be a little quicker to create and will break a little less often. Welcome to the world of automation powered by AI!
How to Get Started
How do you leverage the power of this AI brain in your code?
Simply update your Appium project to the latest revision (see Jonathan Lipp’s Appium Pro article for the details)
Then, find elements using a new custom AI-search strategy such as:
driver.elementByCustom(‘ai:cart’);
// This code asks the AI to find a shopping cart image on the screen for you
I really can’t imagine it getting easier than this to add AI to your automation project. In fact, it is faster and simpler than the traditional methods of finding IDs, CSS or XPaths and using these more complicated search strategies.
AI For the Planet
Open source is key. No team should have to find the XPath or CSS Selector of an element or beg a developer to add a magic ID for them to use in their test code. No team should have to re-invent basic AI element classifiers or re-label 100,000+ images either — what a waste of humanity to duplicate that work. Therefore, the classifier is open source. Many vendors consider this type of IP their magic sauce, but that means that most test automation engineers can’t afford it, or don’t want to integrate it into their own code base. Open source means this tech is awesomely universally accessible to all.
Extensibility is key. This is the ‘hello world’ of bringing AI to element finding. Jonathan made sure this was a pluggable system, so any classifier can be used, or even other element search algorithms can be easily shared and plugged directly into Appium. Test.ai just open-sourced the default/reference implementation and donated to the community in the interest of sharing the power of AI with every test developer on the planet. Our mission at test.ai is to test the world’s apps. What better way is there than to help every test developer with the basics of finding elements inside of their own apps?
Customization is key. The testing community can improve the AI. Anyone can add new training data, alternative training methods, more rigorous relevance testing, or new labels. The AI is the property of the community, and we hope to help bootstrap every test team on the planet with a foundation of AI for their own projects. The test.ai team has shared all the training data on Kaggle, so the world can fork the data, clean it up, add it to their proprietary test frameworks, or compete to improve these classifiers. Crowdsourced, open data for AI testing systems? It is a new world.
Reuse is key. The AI can be forked and/or re-used in other open source and proprietary frameworks. The goal at test.ai is to spread the usage of AI in all aspects of testing, in the interest of faster and smarter test automation, and ultimately better software in the world. The neural networks are based on the open source TensorFlow framework from Google. These models can be run in the cloud, locally, on mobile devices, or in a project not even thought of yet.
Become an AI Test Automation Engineer
Whether you are an AI expert, test automation geek, or just getting started with AI and testing, you can bring AI into your team and your project today. A free, open source, and single API call is all it takes. You can be the hero that brings a bit of AI to your engineering team. You can even contribute to this transformation in testing by helping add new training data or add a similar call to your favorite test framework — be an AI test automation engineer today. Ultimately, it will take our community to bring the power of AI to our entire field.
Seeing is believing and Jonathan Lipps created the first intro video demonstrating this working on both Android and iOS. What about web? It works there too, but it’s far less tested.
This is a hello world of real AI integrating with test automation tools to make our lives just a little easier, and hopefully more fun.
By the way, it is awesome working with Jonathan Lipps — truly today’s top expert in mobile test automation. Thanks to the team of machine learning and integration engineers at test.ai for the willingness and bravery to open source something you have poured so much energy and talent into the past year. And, special thanks to our investors who thought this was a great idea when I brought it up.
“That’s one small step for AI, one giant leap for test automation” :)
— Jason Arbon CEO, test.ai