Informal testing must be done before formal testing
我以前认为”Good Testing = Formal Testing + Informal Testing”, 注意是Formal Testing在前，Informal Testing在后，即Formal Testing辅以Informal Testing。一次和James Bach的偶遇让我开始重新思考这个问题。下文是我们的部分讨论内容，尽管当时我并未被James所说服，可是渐渐地，我在事后一次次地深入地思考这个问题时，结合实际项目中的测试过程，结合那个骰子游戏，结合测试的本质，我愈发觉得自己原来的结论站不住脚。现在，我已能深刻理解＂Informal Testing是Formal Testing的基础＂这句话了。我也不再说ET（探索性测试）是ST（脚本化测试）的补充、基于经验的测试是基于规格测试或自动化测试的补充之类的话了。其间道理，还需要每个人自己反反复复地、客观地、深入地思考方可得！
Xiaomei: > If your definition of formal testing is “highly specific, planned,rigorously executed tests, with test results documented and recorded and
showed to officials”, how much formal testing have you been doing in your past testing work? How much formal testing, do you think, would a general tester do in his work?
James:That’s not my definition of formal testing, really. I would define formal testing as ANY testing TO THE DEGREE that it must be PERFORMED IN A SPECIFIC WAY as a criterion for success.
Informal means: it is not expensive, it means we can make mistakes and we can try the unimportant things to gain more understandings. The example of paper airplanes: I practice and practice. I’m not just reading and preparing a long time before I actually practice and create a good paper airplane.
The very formal testing means the test results are documented and recorded and showed to officials.
Xiaomei: > I ask this question because I usually have a different definition of formal testing from yours.
Suppose we have thousands of testers. Most testers’ work procedure is like this:
- First, some testing tasks are determined during early requirement phase;
- Second, parallel with designers’ design phase work, some senior testers
designed test cases for each test feature, based on design specifications
and their experience on this field, and based on discussions with designers,
and based on company’s test design procedures;
- Third, after the software is transfered to test department, some junior
testers will execute these test cases.
We usually call this formal testing process.
James:I would also call that formal testing.
Xiaomei: After one or two runs, the designed test cases were executed and bugs are fixed, some testers may do more informal testing, they may supplement more test cases with no certain rules or techniques, they may execute more tests with no evident test
scripts, they may do exploratory testing, etc. We call this informal testing process. (Ofcourse, in this formal testing process, the test executers will not absolutely follow test scripts, they may do a lot of exploratory testing.)
James:Okay, so it’s not very formal testing, but it’s somewhat formal.
Xiaomei:Based on these definition, from a project test manager’s perspective, he would hope his testers do formal testing first, then he can get an certain confidence into the system under test, then if there is still time left, he would arrange some informal testing, in order to find more interesting bugs.
James:I still say all excellent formal testing must be based on informal
testing. Why do you presume that the formal test ideas promoted by
your test designers are any good? They are designing without the
benefit of the vital learning that comes from trying out their ideas
in real life.
In my experience, it is generally irresponsible to rely on test ideas
as formal testing that were not tested through the process of informal
Xiaomei: Yes, in our model of “Formal testing + Informal testing” (instead of “IT + FT”), people may not do a good formal testing if they don’t have much experience. But with more and more practice, the better and better of testing effectiveness
James:I wouldn’t say “better and better.” Yes, experience and practice can
help, but you’ve given me no indication that your test designers learn
fro their experiences. Where is the feedback loop? This division
between test designers and testers actually shields your test
designers from getting the learning they need.
If you were driving a car blindfolded, being instruction by a sighted
person in the back seat, you would also get “better and better” with
experience, but that wouldn’t justify using such a bad method of
driving. That would never be considered safe driving.
Xiaomei: By the way, I don’t think this pattern (FT + IT) is a best pattern. However, maybe it is a more suitable pattern for us, since we don’t have so many experienced testers and our system is rather complex and we have lots of inexperienced testers in China.
James:A better way is to have you experience testers personally work with
and train your inexperienced testers, rather than try to control them
by writing things down.