On exploratory testing

Cem and James talked about exploratory testing — Cem talked about exploratory testing in terms of categorizing knowledge — by risk, test technique, or approach. Though I wasn’t there, from other comments (and talks with James), I would guess he talked about the use of heuristics and skeptical questioning of the application. He gave the URL for the Satisfice Heuristic Test Strategy Model. This model presents 32 perspectives in 4 areas (though Cem makes it 5 areas, calling risk out separately).

Back to Cem — people (such as Giri Vijayaraghavan) have made taxonomies of various bugs that could exist in a certain type of application (such as shopping cart apps, available at http://www.testingeducation.org). He also talked about the use of “attacks” — patterns (not in formalistic definition of pattern) of trying to find bugs in software. He mentioned Al Jorgenson’s dissertation where he identified a series of tasks that focus on various constraints of the application — input constraints, output constraints, device interaction constraints, and file system constraints (file system as a logical concept, not a device concept). James Whittaker has written a book (How to Break Software) on these attacks as well.

So, a large part of exploring for Cem is having a large batch of ideas of how to test software, and then determining which ideas to use. This is critical, and Cem thinks it’s the most difficult technical area of testing.

James suggested that to learn more about testing, one should read NTSB accident reports for examples of excellent testing and excellent thinking. He also recommended learning more about thinking scientifically. He also said that if you’re going to do exploratory testing really well, you need to have a model in your mind of what is a test — what are the parts of a test and how those parts can break down. He gave examples of coverage and oracles. For coverage, a tester will think of the areas they’ve covered, but they might not think of areas they hadn’t tested and have a wrong impression of what they’ve covered. Oracles are the method by which you recognize that a problem has occurred (generally, an oracle arrives at an answer via an independent method, though there can be other types of oracles as well). But, oracles can have problems too, and no oracle is going to be perfect and enable you to see every problem in the application.

The final element of exploratory testing that James talked about is humility. Even given intelligence, heuristics, and experience, people will make mistakes. Recognizing that and working to get differing viewpoints from others makes exploratory testers more effective.

Day 3 wrap-up

You may have noticed that I’m blogging less and less on each day. Some of this is because the planning and procedural stuff is becoming less emphasized as we work more on getting a story done. While I could say “we wrote code” it doesn’t seem the same to me in terms of what I think of writing about. I think I’m getting a backlog of information to process (never doubt that I’m a reflective learner . I’m going to try to do some of that in this post, so it might wander a bit. I’ve also changed from a write a blog entry during the discussion to making notes. We’ll see if that helps or not – maybe tomorrow I’ll switch back (though tomorrow afternoon we’re taking a break, so there’ll probably be less blogging tomorrow no matter what approach I take).

So, info processing:

• Big changes in the vettest project — The team has asked to be adopted by the RubLog project. Brian and I will work on the testing application offline after this week is over so he doesn’t have to hang his head in shame when he walks down the halls of the vet school. The project was a good one, but it was more suited to a pair of programmers than a team. Plus, this way, everyone gets to pair with the people on the other team, too, so we can learn from them as well. Some of the things that came up in today’s group retrospective
  • we didn’t have a clear definition of the initial config stuff and so it got slapped together willy-nilly
  • We went too long without checking in code. We pretty much spent all day yesterday and part of today without checking in which meant some people were waiting to get interfaces from things other people were working on
  • We weren’t as conscious of our acceptance tests as we might otherwise have done. This led to a less clear picture of where we were going
  • Our individual test files need to be run separately at the moment. We hadn’t gotten an overall tests file made yet, and hadn’t figured out how to make Eclipse/Test::Unit run all the files names test*.rb yet. I think it might be possible to do that, but have yet to work out how exactly to make it work. I’ll probably get a hold of one (or more) of the recently published books on Eclipse sometime this summer and see what I can figure out.
• When both teams got together, we learned that the RubLog group has developed better documentation for the tool and are close to having completed an automatic install.
• James talked about how a vital aspect of testing was diversity. Having different machines is a good thing, as it exposes bugs that might not otherwise have been visible. However, we need to make sure we have ways of illuminating the fixing of problems illustrated by diversity so that management will be aware and not just see that no new features are being developed.
• James also mentioned the sport of Extreme Ironing, which he heard about on a recent trip to the UK. Apparently, the whole point is to be photographed ironing in difficult to get to places (such as dangling from a rope hanging between two sides of a gorge). More info for the curious at http://www.extremeironing.com or http://extremeironing.info. Perhaps the organizers of this sport knew about this study and decided to get creative.
• Tonight, Michael Feathers talked about his patterns for handling legacy code. I looked at the draft of the book he’s writing (at http://groups.yahoo.com/group/welc) and it looks like there’s some interesting stuff there. I’ll be reading it in more detail once I get through the information processing of this week.
• It’s been interesting to see the principles of XP demonstrating value even in just a few days. For paired programming, I’ve had the experience of finding a problem that would have meant rework and having someone else find a problem I had missed. I wasn’t sure that I’d be completely comfortable with pairing (despite being able to see how the benefits ascribed to it might occur), but I haven’t had any of the desire to work alone or thoughts that I would be able to go faster alone that I was a little afraid I might have. Frequent check-ins and test-first development both have shown great use to me over the past few days as well. I guess the whole thing was something that intellectually I understood, but having the chance to actually do some has really driven things home. I tend to do most of my programming alone (being at school, most of my programming is homework or personal projects, and I prefer to do my homework solo even when allowed to work on it in groups), and this week I’ve been thinking about how to change how I develop (amazingly tonight, after I had started thinking about it, I found that Chad Fowler has posted an entry in his blog on the very same thing. I’m going to try to keep his idea of consciously being mindful of the code I write (along with PragDave’s code katas in mind this summer and into the next school year. I’ve had this type of thing happen before (where I’ve had a thought and then read someone else’s blog where they mention the same thing) and find it to be quite interesting when it happens.) There’ll be a bigger emphasis on acceptance tests. You’d think being a tester myself, I’d already have this, but now there’ll be more. I’m thinking that this will probably be in FIT (in the appropriate incarnation for the language I’m using at the time). Test-first development will play a big part (using xUnit), I’ll have source control (even on my single projects) — possibly CVS though I already have SourceSafe installed on my laptop from a previous license I had. I think I’ll keep Apache running on my laptop (or some other computer that’s on more often) and use it to display the results of the tests (and maybe some status reports of automated tasks I have execute). For a lot of people whose blogs I read, this is all old hat, I’m sure and in some ways, it’s all stuff that I was aware of, but like I said, actually experiencing it for the first time is very cool.

  Anyhow, that’s it for tonight. More tomorrow!

  Update: I removed one bullet point from the above. it wasn’t my idea, and I didn’t feel comfortable talking about it, but I needed to write some of it down to be able to move on to other knowledge. My intention was to remove it before clicking Post, but I forgot to do that, so I did now. I also cleaned up some link typos (I thought MT was going to automatically make the URLs links).

More thoughts

Over lunch today, we (the vettest team) started discussing the event so far. I asked Brian what types of things we needed to be doing to meet the goal he stated on the first day of this being a “seminal event” and whether he thought we were on track for that. We didn’t get far in that discussion (Brian made observations that the other two seminal events he’s been at started off feeling like a disaster and only jelled towards the end) and he and I both stated we’re not good at drawing conclusions from an event while it’s happening. This did, however, lead to a discussion of whether there were things that might be more useful to us (us as a team, not just Brian and I). Concerns were expressed that the vettest application is not complex enough (or maybe we’re not far enough along) for the developers to be seeing a lot of new testing types of things.

One of the things that occurred to me while we were talking was a bit of a crisis of identity. I’m generally considered a tester (and that’s what I’ve been paid to do) by everyone who knows me professionally except myself. I’ve always had far more confidence in my ability to develop an application than my ability to test it. In thinking about Brian’s comfort zone stuff from last night, I realized that even though I’m learning things about XP, I don’t feel like I’m being pushed at all outside my comfort zone. In some ways, it feels more like being forced to test this application would be more outside my comfort zone than developing it. I know that the last time I was here at Satisfice, the times when I felt James was pushing me the most was when he had me do some exercises he uses in his testing class.

So am I really a programmer who has happened to spend the last 8 years in a tester’s job? Most of my testing experience was in doing automated testing, and I pretty quickly stopped using the recorder except for getting control recognition strings and treated it like development. I used to have little patience (ok, no patience) for doing manual tesitng, and have been known to avow that I hated it. I also have been known to state emphatically that I was a developer not a tester (referring to developing automation rather than test cases). Part of going back to school was to force myself to learn more about manual testing and get past these feelings. That happened. I still prefer writing code to writing manual tests, but I enjoy the testing part, too, now, and I like the challenge of outthinking the bugs in the code. Part of my mind will always be thinking about automating test execution (and creation, probably), but were I a consultant still, I wouldn’t refuse an assignment because it was intended to be all manual testing.

We tabled the discussion of changing focus so that we could finish the first story. As an update on the vettest team project, we are getting very close to having the web page be created. We have an acceptance testing framework that still has some problems, and the result of the first story is available at http://www.satisfice.com/cgi/acvim.rb