Informal testing must be done before formal testing


我以前认为”Good Testing = Formal Testing + Informal Testing”, 注意是Formal Testing在前,Informal Testing在后,即Formal Testing辅以Informal Testing。一次和James Bach的偶遇让我开始重新思考这个问题。下文是我们的部分讨论内容,尽管当时我并未被James所说服,可是渐渐地,我在事后一次次地深入地思考这个问题时,结合实际项目中的测试过程,结合那个骰子游戏,结合测试的本质,我愈发觉得自己原来的结论站不住脚。现在,我已能深刻理解"Informal Testing是Formal Testing的基础"这句话了。我也不再说ET(探索性测试)是ST(脚本化测试)的补充、基于经验的测试是基于规格测试或自动化测试的补充之类的话了。其间道理,还需要每个人自己反反复复地、客观地、深入地思考方可得!

Xiaomei: > If your definition of formal testing is “highly specific, planned,rigorously executed tests, with test results documented and recorded and
showed to officials”, how much formal testing have you been doing in your past testing work? How much formal testing, do you think, would a general tester do in his work?

James:That’s not my definition of formal testing, really. I would define formal testing as ANY testing TO THE DEGREE that it must be PERFORMED IN A SPECIFIC WAY as a criterion for success.
Informal means: it is not expensive, it means we can make mistakes and we can try the unimportant things to gain more understandings. The example of paper airplanes: I practice and practice. I’m not just reading and preparing a long time before I actually practice and create a good paper airplane.
The very formal testing means the test results are documented and recorded and showed to officials.

Xiaomei: > I ask this question because I usually have a different definition of formal testing from yours.
Suppose we have thousands of testers. Most testers’ work procedure is like this:
- First, some testing tasks are determined during early requirement phase;
- Second, parallel with designers’ design phase work, some senior testers
designed test cases for each test feature, based on design specifications
and their experience on this field, and based on discussions with designers,
and based on company’s test design procedures;
- Third, after the software is transfered to test department, some junior
testers will execute these test cases.
We usually call this formal testing process.

James:I would also call that formal testing.

Xiaomei: After one or two runs, the designed test cases were executed and bugs are fixed, some testers may do more informal testing, they may supplement more test cases with no certain rules or techniques, they may execute more tests with no evident test
scripts, they may do exploratory testing, etc. We call this informal testing process. (Ofcourse, in this formal testing process, the test executers will not absolutely follow test scripts, they may do a lot of exploratory testing.)

James:Okay, so it’s not very formal testing, but it’s somewhat formal.

Xiaomei:Based on these definition, from a project test manager’s perspective, he would hope his testers do formal testing first, then he can get an certain confidence into the system under test, then if there is still time left, he would arrange some informal testing, in order to find more interesting bugs.

James:I still say all excellent formal testing must be based on informal
testing. Why do you presume that the formal test ideas promoted by
your test designers are any good? They are designing without the
benefit of the vital learning that comes from trying out their ideas
in real life.
In my experience, it is generally irresponsible to rely on test ideas
as formal testing that were not tested through the process of informal
testing.

Xiaomei: Yes, in our model of “Formal testing + Informal testing” (instead of “IT + FT”), people may not do a good formal testing if they don’t have much experience. But with more and more practice, the better and better of testing effectiveness

James:I wouldn’t say “better and better.” Yes, experience and practice can
help, but you’ve given me no indication that your test designers learn
fro their experiences. Where is the feedback loop? This division
between test designers and testers actually shields your test
designers from getting the learning they need.
If you were driving a car blindfolded, being instruction by a sighted
person in the back seat, you would also get “better and better” with
experience, but that wouldn’t justify using such a bad method of
driving. That would never be considered safe driving.

Xiaomei: By the way, I don’t think this pattern (FT + IT) is a best pattern. However, maybe it is a more suitable pattern for us, since we don’t have so many experienced testers and our system is rather complex and we have lots of inexperienced testers in China.

James:A better way is to have you experience testers personally work with
and train your inexperienced testers, rather than try to control them
by writing things down.

Informal testing must be done before formal testing》有 4 条评论

  1. thomas说:

    James说的太精彩了!也谢谢博主的分享。
    I still say all excellent formal testing must be based on informal testing. Why do you presume that the formal test ideas promoted by your test designers are any good? They are designing without the benefit of the vital learning that comes from trying out their ideas in real life.
    这是卓越和有效的测试,测试就是探寻未知的过程,不断的学习未知,并且要有探寻和学习的兴趣,并能在探寻中获得快乐,这是与只是把测试当成任务有本质的区别的。

    A better way is to have you experience testers personally work with and train your inexperienced testers, rather than try to control them by writing things down.
    我理解,这是传统的管理方式向团队自管理的一个转变的建议。

    回复

  2. 天堂旅人说:

    因为在我当前的公司都是FT+IT的模式,没有实践过IT+FT这样的模式,所以不好评论。当然,James的观点也有道理,通过IT可以对被测对象更为了解,最后的FT也会更全面一些,但问题是,IT需要做到什么样的程度?在hw这样项目进度紧张的情况下,如何去使用IT+FT模式呢?

    回复

    • 邰晓梅说:

      Informal Testing无处不在,不用把Informal Testing想象为一个明确的、独立的测试阶段,你在评审、测试设计、测试执行等任何活动中,都需要充分调动“人”的能动性和潜能,而不是按部就班地套用标准、规程,那么你已经在用Informal Testing了。
      实际上,在越是紧张的项目中,测试越要灵活应对。如果只是一味地遵循标准、流程、模板、工具,测试的有效性和高效性难以保证。当然测试流程、标准、方法、工具等仍然要用,但是我们把这些当做启发式的方法(heuristics),以人为中心,根据不同的测试上下文,灵活决定做哪些测试、采取什么策略、将IT和FT有机结合起来。

      回复

  3. 天堂旅人说:

    有道理,IF和FT本身就不可能孤立的分割吧。

Comment Box is loading comments...