The method we use for Agile automation is straightforward: The toolsmith watches a tester work and determines how tools can help with what the tester is doing. The toolsmith's attitude is to help the tester on the tester's terms, rather than to push some private agenda.

If you have no toolsmith, then each individual tester should be on the lookout for automation opportunities. Keep in mind, the more the tester knows about tools and programming, the more effective he will be at finding useful tools.

Examples of Our Agile Approach to Test Automation
A tester had previously done a manual spot check on two comma-separated value (CSV) files and found no differences. With assistance from Danny, the tester used a tool to compare the two CSV files. The tool found an entire column of data that was wrong. After about an hour of further research, they found another free tool that did a better job of finding non-matching data.

James helped a test team query an auction system for a full report of auction status at any moment. The tool enabled them to automatically confirm that they had tested the scenarios and conditions they wanted to test, instead of simply hoping they had made no mistakes in executing their test cases. This team had tested for nearly two years without such a report, yet writing the tool took only three hours from start to delivery.

Danny has used the Perl WWW::Mechanize module to write a quick hack that loaded tens of thousands of records into a Web-based database front-end overnight. The next day, he quickly identified a performance problem with the application that was related to running with a large database.

We have both found situations where installation test tools could help the tester see exactly what an installation procedure did to the filesystem and registry.

In all of these examples, we produced real value with only a few hours of effort. We helped testers improve their testing with the help of tools. To us, that is the very definition of test automation: tool supported testing. This extends the scope of test automation far beyond the idea of happy robots that test while you sleep. Why not do automated test design? We've done that. Why not use automated test probes that alert the tester to certain kinds of problems? We've done that. It's all test automation, but it's not your father's test automation.

Deliver Something Useful in Less than Forty Hours, or Else...
Or else you picked the wrong task.

Incremental delivery is a key part of Agile automation. We find that there is a quantum difference between solutions delivered in less than a work week and those that take longer. Longer projects are riskier and absorb resources that might be used for other quicker tasks. Long tasks can be broken up into discrete deliverables of just a few days each—each one delivered and working for a tester before embarking on the next.

If your automation project wasn't going well, wouldn't you rather know about it a week into the project rather than months later? What if you're over-engineering your solution? With an Agile approach, after a few incremental deliveries, the tester might decide that the automation he has is good enough, and the toolsmith will move on to other automation priorities.

With short tasks, it is more feasible for the toolsmith and tester (especially if they are the same person) to self-approve their projects, eliminating a lot of management oversight. Automation tasks that exceed a week in duration are likely to require approval from higher levels, which further complicates and slows the automation process.

If you predict that some task will take more than forty hours, or if you approach forty hours of effort on something you thought wouldn't take so long, it's time to step back. Have you really broken it down into small enough chunks? Is your approach really viable?

No Automation Cathedral
So many "classic" automation efforts we've seen seem to withdraw into the proverbial cathedral or ivory tower. In one case, James witnessed a grand automation strategy that was so remote from its clients that the testers he interviewed didn't even know it existed--despite nine solid months of development work. Danny has seen a test team rebel against a centralized automation group that didn't take the team's needs into account.

What is the opposite of the cathedral? Automation driven by the day-to-day test process, with as many different tools as there are booths in a bazaar; and test automation serving specific immediate needs, rather than future needs. Certainly there is a place for long-range automation projects that establish an infrastructure for automating difficult tests. However, our experience is that the big strategy can blind people to the many small solutions that can have a big impact immediately.

The essential ingredients for an Agile test automation are this: Think broadly about where tools can assist the full range of testing tasks that you perform. Download, create, or buy tool elements to automate a task for the first time or to improve existing automation, and complete each piece in less than a week. Try to establish a toolsmith role in your organization, and don't allow an automation cathedral to slow your progress. There is so much potential for tools to help you. Now get to it!

Authors' Note: Thanks to Neill McCarthy for his early feedback on this article.

Further Reading
Agile Test Automation whitepaper by James Bach
"The Cathedral and the Bazaar" by Eric Raymond
"Manifesto for Agile Software Development"

About the Author
Danny R. Faught is a testing consultant, author, and trainer. Danny is a regular Weekly Columnist on StickyMinds.com and has spoken at several Software Quality Engineering conferences. Danny He is the publisher of Open Testware Reviews and proprietor of Tejas Software Consulting based in Fort Worth, Texas. Visit tejasconsulting.com for more information. You can reach him at faught@tejasconsulting.com. James Bach is the founder of Satisfice, Inc., a test training and consulting company. James is coauthor (with Cem Kaner and Bret Pettichord) of Lessons Learned in Software Testing. He has written many StickyMinds.com columns and spoken at Software Quality Engineering conferences. He can be reached at james@satisfice.com.

Back to Top 
 

	Comment:     by Deepthi D 3/9/2007 Hi Danny & James, It a very informative article !!! Especially the examples used in Agile approach of Test Automation. Thank You ! Regards, Deepthi Author's Response: 3/9/2007     You're welcome! Please let us know how your own automation efforts go. -Danny
	Comment:     by Vishal Verma 3/8/2007 Hello Danny, Well these are for sure nice guidelines for starters(like me) who are just climbing the path of Test Automation and its meaning tool supported testing, I found this article very useful, well Danny i also want to be a toolsmith in my team and i have chosen "Ruby" to be my companion please comment will it ba able to make my life easier.. Author's Response: 3/8/2007     Hi, Vishal. Yes, learning a scripting language can make your life much easier. Just make sure to manage your time, so that in your enthusiasm to learn and use this new skill you don't neglect your responsibilities. Try to get this learning process reflected in your schedule. Also, beware of creating tools that require so much maintenance that you could have done the work more quickly by hand.
	Comment:     by Harvindar S. Devgan 1/17/2005 What are Automated Test Design? Automated Test Probes?
	Comment:     by neill mccarthy 1/13/2005 I was impressed with the article and even more impressed with the comments, I no longer feel alone :) There are more people going through transitions to agility and moving towards pragmatic automation and augmented manual test. Two comments, which I did not think of earlier (sorry): Firstly the Tool Smith approach and agile approaches can be used successfully to improve “traditional automation” with regression packs of the current generation of vendor tools. I accept it is less agile, but better then throwing out that (miss) investment. Secondly that this can also be used by selecting the right tools be an excellent way of...Read On
	Comment:     by Roshan Joseph 1/13/2005 I like this article. I was involved in doing these tiny things that create a big effect. Ability to find oppurtunitnites to apply tools depends on the knowledge that you posses on test tools, existing test processes, project context and customer needs. Abraham lincoln once said "When the only tool you own is a Hammer, every problem begins to resemble a nail". In my opinion, test tools is any utility (How ever small or simple ) that helps me to find defects early,cheaper and faster. A good craftsman is known by the tools that he uses, Testing is a craft and is still evoloving. The value add that a test engineer can bring is...Read On
	Comment:     by Madhavan Jagannathan 1/13/2005 Thanks for the article..This could be THE approach for the management to convince automation 'nay sayers' to automate a few things. The advocates of Manual testers often point out that there is no ROI for automation.To compound the problem is a test architecture which takes a while before we hit upon the right architecture to effectively get a good automation suite. The Agile approach could be a good approach to start automating the different components of the product. "Deliver Something Useful in Less than Forty Hours, or Else..." automatically ensures that there IS ROI. One could then weave this into a complete suite...Read On Author's Response: 1/14/2005     The smaller the project you're seeking approval for, the easier it is to get it approved. This is especially true after you have a track record of successfully delivering small increments.
	Comment:     by Umakanth Srinivasan 1/13/2005 "This extends the scope of test automation far beyond the idea of happy robots that test while you sleep". Impressive sentence increasing the value of testers. Thank You. This Testing domain is a compelling environment where we are forced to find some tools to complete the testing at the earliest. But it is hard to test two tools (AUT & New test tool) at same time. Which I think would be the bottleneck for this approach. Author's Response: 1/13/2005     Yes - the more complex a new tool is, the harder it is for you to gain confidence that it works. One thing to keep in mind is that a toolsmith will keep going back to tools that have worked well before. For example, for the past few days, I've been using James' perlclip tool to generate test data. I've used it several times before, and it's simple, so I was confident in using it right away. With its help, I reported several bugs. In fact, I think there will come a time that people regularly see perlclip's counterstrings like "*3*5*7*9*12*" in bug reports and test databases. On the other hand, I've found that evaluating a load testing tool that I've never used before can be very time-consuming.
	Comment:     by Gary Feldman 1/12/2005 The column is certainly good common sense and consistent with agile approaches - there's nothing to disagree with -- but it still leaves me unsatsified. Agility is more than just incremental delivery, a broad knowledge of available tools, and customer contact. I believe that to do the subject justice, it needs to take test driven design, customer acceptance tests, user stories, etc. into account. There are really two problems: how can a test automation team function agilely, and how does a test automation effort function where the development team is diving into agile methods. The latter raises the question as to whether there...Read On Author's Response: 1/12/2005     Gary, thanks for your comments. Yes, there are several interesting things we could discuss about test automation on an agile project. But I want to make sure that people understand that they can use an agile approach to automation on any project, not just a project that is developing the product in an agile fashion. I'd like to hear about whether anyone has used the Extreme Programming concepts you mentioned on a test automation project. I'm experimenting with test-driven development myself.
	Comment:     by Vijay Rangari 1/12/2005 Excellent. I am glad that someone well known has started this discussion. I have been developing bespoke Test Harness solutions since 1996 and have found that there is as much customisation of 'off-the'shelf' frameworks required as there is work in creating your home-grown solution - except the latter has fewer limitations that cannot be worked around. In an ideal world, Developers would produce Unit Test Harnesses which can then be used by Testers who would have a more meticulous approach to Test Conditions applied (This can usually be integrated into the build process!!). And bespoke System Test tools should be designed by the Testers and...Read On
	Comment:     by Joe Polvino 1/12/2005 Any tester with even basic programming experience should seriously evaluate perl as a language for automation and utilities. Once past the sometimes confusing syntax, most folks will find that perl can do virtually anything. With the added benefits of low acquisition cost (free), portability, massive support base, extensive libraries, and wealths of just-in-time resources, it would be foolish to dismiss perl without giving it a fair shake. Author's Response: 1/12/2005     Joe, I should first point out that I didn't coerce you into posting this. :-) Perl is indeed my favorite language. I've also found significant pockets of advocates for using Python, Ruby, and Tcl for testing, plus a few places where JavaScript might be appropriate. I'm rounding up an analysis of all of these for Open Testware Reviews right now.
	Comment:     by Vinit Awasthi 1/12/2005 A very boosting article to apply agile automation process in Testing.I think that agile automation process may not be more benifical in testing Products because you have to do lot of automation testing not some small set of tests which you can develop in weeks. Through this technique you will be able to know which tool will be more benifical in case of your application (Desktop,Webbased,Client Server)and using which language application is being developed.If i am understanding something wrong then correct me. thanks Author's Response: 1/12/2005     Hi, Vinit. I don't completely understand your comments, but I'd be glad to discuss it if you send me email. Regarding automated test cases, let's consider separately the task of setting up the tools that facilitate writing automated test cases, and writing the test cases themselves. We were thinking of the tools when we wrote this article. Ideally, you don't have to occupy your toolsmith with writing all of the test cases, so you have a larger group of people available to implement the test cases. Anyway, the model still fits - once you have a framework that can support at least a few of the tests you need, then each test case the team develops is a separate deliverable that takes far less than a week to complete. If you spend a week developing test cases and none of them are working yet, then management needs to figure out why. -Danny. James Bach also had this to say: "The whole point of the article is that there are many opportunities for greatly improving testing that don't have anything to do with writing a thousand test cases. For instance, I typically write a program that automatically generates test cases, or that makes a new kind of testing possible. Look for those opportunities and I bet you will discover them."
	Comment:     by Prakash Marar 1/12/2005 Hi Danny/James, Good to see an article like this. I have been working on Opensource based testing tool for some time now and really wonder the resource available for test automation. There are plenty of tools, with no cost or low cost and well fit into agile testing or Xtreme testing. I also wanted to point out the homebrew test automation (Danny had initiated a wonderful roundtable on this here) and how this helps in Agile approach. However, having worked on Agile testing for some time, I also see some negative facts. Not all testing can be automated using agile approach especially there are small iteration time span. By the time QA...Read On Author's Response: 1/12/2005     Thanks for your feedback, Prakash. You bring up a good question about doing agile automation on an agile project. Note that you can do agile automation on any kind of project. But if your project is using short iterations (the shortest I've seen is two weeks), then you do have to consider whether the tools you acquire can be used in the current iteration or the next one. I think that it will often be feasible for an automation effort that takes a week to have an impact on the current iteration, depending on when you start the automation work. But it may be okay if you target it for the next iteration. If doing this automation work means that you can't do any manual testing, though, then you probably don't have sufficient resources to handle any such large automation efforts. Many agile automation tasks only take an hour or two of effort, maybe even just a few minutes, and just about anyone should be able to work that into a project.
	Comment:     by Anuj Malik 1/11/2005 Hi , wonderful article on the test automation stuff, this has cleared a lot of questions i had on what exactly is test automation. Though i have some questions, on the agile testing approach that is mentioned. The project that i have been working involved a business critical application, where only manual testing was done during the development. Now that the project is live, but for future releases of the same application, the management is toying the idea where the automated test scripts could be made for validating that enhancements made to the application do not actually break the original application. How helpful would automation be in...Read On Author's Response: 1/12/2005     Hi, Anuj. You're talking about regression testing. All automated functional testing (tests of individual features) is regression testing. Yes, this can be helpful if you have a well-designed test framework that doesn't become a maintenance burden. Maintenance issues are a very frequent of cause of failure for automated testing. By the way, I think that having automated unit tests (written by the developers) is more important than having automated tests that go through the user interface.
	Comment:     by Eitan Burcat 1/11/2005 I support the general approach, although, it doesn't fit when you automate in order to regression test your product. Incorporating many tools into your regression tests can lead to quite a mess. I'd use this approach, but for tests I know I probably won't regress. Author's Response: 1/11/2005     Eitan, I don't understand what kind of tests you're automating that aren't regression tests - perhaps they're not functional tests? Anyway, you're right that it can get messy when we stack several tools together, and that's often the way we need to do it because it's a very powerful approach. As a previous comment mentioned, though, the commercial tools can also turn into a mess. In both cases, we have to put some effort into making the tools easier to debug and maintain.
	Comment:     by John Daughety 1/11/2005 Having implemented test automation in a few companies now, using the expensive "off-the-shelf" tools, I can say this article is very timely and deserves careful reading. While tools like SilkTest and WinRunner can be helpful, they require a great deal of support in the beginning to set up the architecture and create the low-level functions that an evolved set of test scripts must use. This is nice, but it is only part of the solution. I recently was in a position where the code was so complex it would take months, if not years, to create a suite of automated tests that would perform adequate testing. If the developers had...Read On Author's Response: 1/11/2005     Thanks for your comments, John. Actually, the first GUI test I ever wrote was a Perl script that used the Win32::GuiTest module to drive Notepad. It's worth noting that the commercial GUI test tools are generally more feature-rich than the free ones. So the question is whether you need those extra features or not.
	Comment:     by Andrew Raybould 1/10/2005 I do not know of any successful way to develop software without constructing an automated test harness as you go, and I never cease to be amazed by how many developers and development manegers seem blind to how much it is costing them, in bug-induced delays, rework and all the less direct costs of poor quality, not to have such a facility. Without automation, you cannot do enough testing, and do it thoroughly and accurately enough, to keep up with your progress. I learned very quickly to be ready to test my work before I did it, and that almost always involves the techniques you describe here: scripting (including scripts that write...Read On Author's Response: 1/11/2005     Hi, Andrew. It doesn't always make sense to automate your black box test execution. But very often it's worthwhile to write automated unit tests, as you mention. Yes, the test harness must work well - I don't trust one until I see it correctly report a failing test, for all the ways a test can fail.
	Comment:     by Frits Bos 1/10/2005 Right on! It is amazing how many people equate test automation with the purchase of a massive piece of software when they are not even prepped to understand what it can do for them. Not that there is anything wrong with investing in good products, mind you, the point you made very clearly is that such an approach is both limiting and costly if you are not quite sure what your needs are. I too have collected an impressive array of freeware and shareware products, and still I have had to develop software for test script creation based on fundamental requirements and specifications. Many people believe they get all the tools they need when...Read On Author's Response: 1/10/2005     Hi, Frits. Yes, many test tools are not advertised as such. For example, I recently downloaded a stopwatch program to help me conduct a test that had an element of timing. You're right that you can make a case to management to spend more than 40 hours on a tool project iteration, but first you should think really hard about why you can't produce an incremental result that does something useful in less than a week. In many cases, managers should decline requests to break this rule, because in a large majority of cases, it's a good rule to stick to. It's much easier to manage small iterations and you can figure out more quickly when something isn't going work as you hoped it would.
	Comment:     by Mike Whittaker 1/10/2005 FYI - Windows file comparison tool - "Beyond Compare" - $30. Good support too: I made a couple of suggestions for improvements, which appeared in the next release ! Used it for cross-checking vast Java debug/trace dumps for discrepancies, among many other things. Author's Response: 1/10/2005     Thanks for the recommendation. An interesting comparator I recently discovered is KDiff3, which is open source. A participant on the swtest-discuss mailing list recommended it. I welcome discussions about agile uses of tools on swtest-discuss - see http://lists.topica.com/lists/swtest-discuss/.
	Comment:     by raja ramasamy 1/10/2005 This is an wonderful article, iam stared in search of automation tools for further expand my knowledge in tools. Author's Response: 1/10/2005     Thanks, Raja. I also recommend learning a scripting language, because agile automation often involves writing short homebrew tools. I do best at learning a language when I have a real task I'm trying to automate, so I'm not just writing toy programs.
	Comment:     by ramesh khare 1/10/2005 excelent article. Author's Response: 1/10/2005     Thanks, Ramesh!

Back to Top