Discussions: formal testing, informal testing, structures


During StarEast2011, I had a chance to discuss with James Bach about formal testing, informal testing and structures. Right after that conversation, I recalled most of what we said and noted it down.

Maybe it is a good idea to share part of these now in my blog, so that more and more people can learn something from it.  Welcome any more discussions about this topic.


Xiaomei

Yesterday in the lightning stroke part, you said that formal testing must be done after informal testing. This idea is totally different from what I thought before. I often told my team that you should do formal testing, and supplementing with informal testing.

James

I see. We’re having a conversation right now, right?

Xiaomei

Yeah.

James

Did you write down what you’re going to say before coming here?

Xiaomei

No. I met you by chance.

James

Did you speak formally? Speaking formally means what you say is rigorously planned in advance.

Xiaomei

Maybe this is different. This is an informal talk, not a formal speech.

James

But why we don’t do it in a formal way? Is it better?

Xiaomei

There are different occasions. Sometimes formal, sometimes informal.

James

What I’m saying is if you want to construct a formal communication, and you have never done any communication before, your formal communication will be very poor. It’s all the informal communication we do that prepares us to do formal communications. Every one who speaks formally is based on that skills, knowledge, experience they gain from speaking informally.

Xiaomei

Yeah.

James

My point is if you want to test something which is complex and important, if you begin with formality without experiments and trying things first, then because I do with experiments and trying things first, I’ll know much more about the product than you will know, my formal testing will be far better than your formal testing.

Xiaomei

Maybe there is a difference of the definition of formal testing between us.

James

That’s possible. What I talk about formal testing is highly, specific planned, rigorously executed tasks.

Xiaomei

And the process is structured.

James

The process of all testing is structured. 

Xiaomei

I mean, the traditional way of testing: first we plan and design our tests, and then we execute them.

James

That’s a particular kind of structure. All testing is structured. Some testing is structured differently from another testing. I’m speaking to you in a structured way, I organize my words in a right sequence, so you can understand what I’m saying. So language is a structure. The fact that I have to write down what I’m saying , the fact that I have more plans before speaking does not mean my speaking is in a structured way. My speaking too has to have a structure. The question is what is the structure and who control the structure. Now if I have to think for an hour what I want to say to you, my structure will seem a little differently. And my structure will use such as a processer to control what I’m saying. That is a structure of a certain kind, different from my previous structure. The problem we have when we talk about testing is that that is not a structure, such as exploratory testing is not structured. That is not correct. They are confused. How can exploratory testing be effective if it is not structured? Actually, it is effective because it is structured. It is structured, however, in a way that controlled by a tester, not controlled by someone else in a different time and place. That’s why how you can drive a car. You drive a car with all practice in it. You are in a car, you see what’s happening. If someone else try to specify how to drive a car to someone who didn’t know how to drive a car, that kind of structure is a poor kind of structure. This is not effective. In a same way, in highly scripted testing, we write down procedures, which is an ineffective structure to find important problems. So when we speak about structures, the question is not whether or not it is a structure, but how is the structure.

My preferred structure is the kind of structure controlled by a tester. It is dynamic. It is created as the tester goes. It builds up from simple to more complicated structue as you go. Now, if I’m going to do really good formal testing, I need to know which variables to test, right?

Xiaomei

Yeah.

James

How do I discover which variables important to test? How do I know? The requirements tell me? They might tell me some important things.

Xiaomei

You should do test analysis. Based on the requirements, you can get the variables.

James

How am I supposed to be able to know? Here is a requirement. They  said this: power on ,power off, power on again. Nowhere in the requirements, nowhere in the design documentation, nowhere in any communication, anywhere that is given to me that they tell me that the actual situation of power is like this. So if I was doing my analysis, I was be able to do my analysis based on this requirement which is wrong, rather than on this which is ture. What I need to do is to gather data so that I was analyzing something that is true and real. And you could say that is a bug. The problem is that they can’t fix that bug. so that they have to change the requirements to accept that bug. So the experience that I got from my informal testing enables me to say that is a bug.

Xiaomei

But before you get that conclusion, you have done formal testing. You read the requirements, doing this kind of testing, and then you find such a problem.

James

I would call that informal testing. First of all, my testing results might not be recorded officially. I don’t have to file my test result to any authority.

Xiaomei

But you can record it.

James

I could, yes. But I didn’t need to. When I say formal testing, what I mean is that my test results are documented and filed to someone else.

Xiaomei

That’s very formal.

James

Yes, indeed. Let’s say we have to prepare for a paper airplane competation.  I’m not going to think and think and think and then make an airplane at one time. I’m going to practice and make lots of paper airplanes before I compete with other people. And I’m going to make paper airplanes in different ways and try different things. The more formal I do it, the more expensive it would be. I don’t want it to be expensive. You can think of your informal testing as practice. You don’t want your practice to be expensive, so that you can do a lot of practice. I’m not ready to compete yet. But I notice that I’m practicing, I’m doing it better, maybe I can observe what other people do, maybe I need to study something, eventually I can do it better. So informal means less expensive, means that I can make mistakes, I can try a lot of things that I don’t know that are unimportant, I’m not sure whether they are important or not. What I’m saying is that there are a lot of things to do, to learn, to prepare before the first moment we see a product. When I want to test a medical device, I have five months before I got to see the real product. So what I’m going to do during these five months? I studied similar medical devices, I went to the company’s website and study the website, I learned about all other technologies that are relevant, I read about regulations and standards, I try to discover what kind of technologies that are going to be used…

Xiaomei

We do the same thngs. We do these kind of work, and we call it the input of our test design. Based on this input and other specifications, we got our output – test cases. And we call this formal testing.

James

Well, what you’re missing here is the most important things. You get the input, you get the output, what is the thing that transits the input to output?

Xiaomei

The test analysis.

James

What is the thing that does test analysis?

Xiaomei

Based on the requirements.

James

What is the process that does test design?

Xiaomei

We have a very formal process.

James

No, you don’t.

Xiaomei

Based on our company’s process.

James

In that process I won’t see any part that tells me I can do a good test design. Your process description does not describe how to do a good test design.

Xiaomei

Then what about test techniques?

James

Test techniques maybe say very tiny bit of how to do a good test design.

Xiaomei

Then what about testing experience?

James

Well, you’re missing that it is a human who can do a good test design.

Xiaomei

Based on their past experience.

James

Not just based on their past experience, but based on their skills. And their skills are not documented. You can not always recognize a human’s skill. So you’re missing here from the input to the output is that the humans, the humans’ skills.

Xiaomei

Everything is done by a human.

James

What I’m saying is that the human must have the right skills, the right learning, the right training, the right knowledge, the right experiences. So what I think of is that not the inputs to get  transited to outputs, but the humans get prepared to test. Ofcourse, the human takes this input in, but we don’t know exactly what it is going on in your head. You can speak a few things out, such as test techniques, that is true. But most of it is the human who is under developed, who is practicing his skills.



Discussions: formal testing, informal testing, structures》有 1 条评论

  1. christina说:

    Very wonderful conversation, thanks for sharing, learn a lot from it!

Comment Box is loading comments...