Is Monkey testing a kind of Exploratory Testing?
When people say "Monkey Testing", do they mean the same thing? I doubt it. Different people may have different mental model of what Monkey Testing means in their own minds. And sometimes, this may introduce confusions when people have discussions about testing. The following conversation between James Bach and me is such a case.
What I learned from this conversation is:
- You need to clarify your view of something or some words clearly enough to avoid misunderstandings.
- Exploration is a search process, which is the flip side of self-management. Exploration testing is a self-managed testing.
- Testing is a cognitive engagement process. And cognition is not just explicit knowledge and skills, it can be meta-cognition. Testing is cognitively sophisticated process, and exploration is not necessarily as sophisticated.
- All testing is exploratory to some degree.
- Testing is evaluation through experimentation.
- Exploration does not necessarily involve thoughtful thinking, but testing does involve that.
- Exploration testing means self-managed testing, a testing approached in an exploratory way.
- Pure scripted testing is not technically testing.
- There is no requirement for responsibility in ET. But we do emphasize responsibility of the tester for his or her testing.
- "Is arbitrarily pounding of keys an exploratory testing?" It can be. Sometimes it is, sometimes not. It depends on the context.
OK. Read the following conversation if you're interested in finding the above key points.
[14-8-12 下午5:37:39] Tai Xiaomei: Is monkey testing a kind of ET? e.g.: a child playing the keyboard randomly without knowing what he is doing.
[14-8-12 下午5:38:20] Tai Xiaomei: Previously I think it is an extreme ET - the freestyle ET. But now I would think it is not an ET process. What would you think?
[14-8-12 下午7:26:59] James Bach: Yes, of course it is exploratory. The better question is whether it is testing.
[14-8-12 下午7:27:17] James Bach: Why would you think it is not ET?
[14-8-12 下午8:43:35] Tai Xiaomei: I think it is testing, but not exploratory. The word 'exploratory', to me, means involvement of cognitive engagement of the tester. As in the definition of ET by Cem Kaner, ET emphasizes personal freedom and responsibility of the tester. The child playing the keyboard doesn't have any responsibility of his "playing" although he has lots of freedom to play. His mind may wonder at some things rather than this playing keyboard activity - i.e. it is hard to identify any structured process in this playing process.
[14-8-13 上午2:05:38] James Bach: If there is no cognitive engagement, then it cannot be testing!
[14-8-13 上午2:06:48] James Bach: All testing is exploratory.
[14-8-13 上午8:22:28] Tai Xiaomei: But sometimes we do monkey testing - the child playing with the keyboard freely without cognitive engagement, suddenly a bug is triggered. This method can be used to verify system's robustness or reliability, to some degree. Don't you think this is testing?
[14-8-13 上午8:29:02] James Bach: It is testing and it is cognitively engaged.
[14-8-13 上午8:29:14] James Bach: and it is exploratory
[14-8-13 上午8:30:34] Tai Xiaomei: I don't think the child is cognitively engaged. "Cognitively engaged process" means you know what you're doing, why you're doing this not that...
[14-8-13 上午8:34:33] James Bach: yes, the child is cognitively engaged.
[14-8-13 上午8:34:52] James Bach: cognition is not just explicit knowledge and skill
[14-8-13 上午8:35:23] James Bach: you are speaking of meta-cognition
[14-8-13 上午8:38:04] Tai Xiaomei: Ok, let's change the child to a monkey. Is this monkey doing testing?
[14-8-13 上午9:22:57] James Bach: no, the monkey is a test tool
[14-8-13 上午9:23:49] James Bach: we draw a line at non-humans
[14-8-13 上午9:24:21] James Bach: testing is a cognitively sophisticated process
[14-8-13 上午9:24:38] James Bach: exploration is not necessarily as sophisticated
[14-8-13 上午9:24:48] James Bach: ants can be exploratory, and are
[14-8-13 上午9:25:15] James Bach: exploratory testing is testing that is approached in an exploratory way
[14-8-13 上午9:26:22] Tai Xiaomei: So you mean the range of exploration is bigger than the range of testing?
[14-8-13 上午9:26:53] James Bach: I mean that testing is evaluation through experimentation
[14-8-13 上午9:27:06] James Bach: and exploration is self-managed search
[14-8-13 上午9:27:19] James Bach: ants do self-managed search
[14-8-13 上午9:27:42] James Bach: but no one would say they perform experiments
[14-8-13 上午9:31:13] Tai Xiaomei: I agree ants do exploration, they know their goals and do their search thoughtfully. I also agree a child does exploration when he is playing his toys. But I think it would be different when a child is simply clicking the keys randomly without knowing what he is doing and without any evident purpose of doing it - This kind of behavior ( often called monkey testing) is not exploratory. And if you call it exploratory testing, it is a meaningless exploration.
[14-8-13 上午9:32:43] Tai Xiaomei: Maybe the line shouldn't be drawn at non-humans, but be drawn at non-animals.
[14-8-13 上午10:09:38] Tai Xiaomei: Haha, finally I opened the Procreate-User-Guide.pdf file today. Never give up trying!
[14-8-13 上午11:47:28] James Bach: I don't think it is reasonable to say that ants know their goals. Ants do not do meta-cognition.
[14-8-13 上午11:47:53] James Bach: I would not say that they do their search thoughtfully. They are little machines.
[14-8-13 上午11:48:19] James Bach: I think you are pretty confused about what exploration is
[14-8-13 上午11:48:31] James Bach: children are definitely exploratory, just as are ants
[14-8-13 上午11:48:53] Tai Xiaomei: I believe ants have their own minds, though simpler than human's.
[14-8-13 上午11:49:12] James Bach: ants do not have meta-cognition
[14-8-13 上午11:49:17] James Bach: they just don't
[14-8-13 上午11:49:21] James Bach: they are just like machines
[14-8-13 上午11:49:37] James Bach: read some books about ants
[14-8-13 上午11:49:44] Tai Xiaomei: Yes, about the word exploration. I think exploration involves thoughtful thinking
[14-8-13 上午11:49:56] James Bach: it doesn't not necessarily involve thought
[14-8-13 上午11:50:07] James Bach: I mean, not necessarily higher thought
[14-8-13 上午11:50:16] James Bach: testing DOES involve that
[14-8-13 上午11:50:27] James Bach: ants are exploratory without being thoughtful
[14-8-13 上午11:50:41] James Bach: once again
[14-8-13 上午11:50:48] James Bach: exploratory testing means
[14-8-13 上午11:50:58] James Bach: self-managed testing
[14-8-13 上午11:51:27] James Bach: whereby the tester makes his own decisions about what to do
[14-8-13 上午11:51:48] James Bach: the tester is not scripted
[14-8-13 上午11:52:04] James Bach: (to some degree)
[14-8-13 上午11:52:08] James Bach: the testing is informal
[14-8-13 上午11:52:14] James Bach: (to some degree)
[14-8-13 上午11:52:40] James Bach: exploration is a search process
[14-8-13 上午11:52:51] James Bach: which is the flip side of self-management
[14-8-13 上午11:53:44] Tai Xiaomei: In Chinese, "exploration" is a rather complimentary word, involving thoughtful thinking; "testing" is a neutral word.
[14-8-13 上午11:53:59] James Bach: I'm speaking English
[14-8-13 上午11:54:46] James Bach: exploration should not be translated into Chinese
[14-8-13 上午11:54:56] James Bach: it's not the WORD that gets translated
[14-8-13 上午11:55:04] James Bach: it's the idea behind the word
[14-8-13 上午11:55:13] Tai Xiaomei: So exploratory testing means self-managed testing. And there is another kind of testing which is not so self-managed, just like Monkey testing.
[14-8-13 上午11:55:28] James Bach: no
[14-8-13 上午11:55:30] Tai Xiaomei: This is what I understand before.
[14-8-13 上午11:55:38] James Bach: "Monkey testing" is also self-managed
[14-8-13 上午11:56:01] Tai Xiaomei: What is the "self-managed" part in it?
[14-8-13 上午11:56:02] James Bach: isn't it?
[14-8-13 上午11:56:16] James Bach: who decides what to do in "monkey testing?"
[14-8-13 上午11:56:23] James Bach: the monkey
[14-8-13 上午11:57:06] Tai Xiaomei: No, the monkey doesn't know what he's doing, what will be done next.
[14-8-13 上午11:57:23] James Bach: no one but the monkey makes that decision
[14-8-13 上午11:57:25] Tai Xiaomei: If a bug is triggered, it is by chance.
[14-8-13 上午11:57:30] James Bach: so what?
[14-8-13 上午11:57:40] James Bach: the monkey is making the decision
[14-8-13 上午11:57:43] James Bach: so the monkey is managing
[14-8-13 上午11:57:47] Tai Xiaomei: It is not a decision. It is a sub-conscious action.
[14-8-13 上午11:58:00] Tai Xiaomei: Or it is a random action.
[14-8-13 上午11:58:01] James Bach: it doesn't matter whether it is conscious or not
[14-8-13 上午11:58:11] James Bach: it doesn't matter if it is random
[14-8-13 上午11:58:28] James Bach: but in the case of a monkey
[14-8-13 上午11:58:32] James Bach: it's NOT TESTING
[14-8-13 上午11:58:39] James Bach: it IS exploratory
[14-8-13 上午11:58:44] James Bach: but it is NOT TESTING
[14-8-13 上午11:59:04] Tai Xiaomei: In the case of a child, it is testing, and an exploratory testing?
[14-8-13 上午11:59:32] James Bach: if the child has the intent to test, then yes, it is testing
[14-8-13 上午11:59:38] James Bach: it is probably not very good testing
[14-8-13 上午11:59:58] James Bach: it is exploratory if the child controls the testing
[14-8-13 下午12:01:01] Tai Xiaomei: I think we have different understandings about the words "control", "conscious", "exploratory", etc. That is the key point.
[14-8-13 下午12:01:40] James Bach: Well, I know what I'm talking about. I can't control what you do.
[14-8-13 下午12:01:57] James Bach: But my way of speaking about this is the result of many years of work
[14-8-13 下午12:02:10] James Bach: I am using my words carefully
[14-8-13 下午12:02:46] James Bach: This is a topic that I was the first in the industry to popularize
[14-8-13 下午12:02:54] James Bach: I created the first exploratory testing class
[14-8-13 下午12:03:02] James Bach: I wrote the first formal process of exploratory testing
[14-8-13 下午12:03:24] James Bach: I've explained this to you
[14-8-13 下午12:03:29] James Bach: I don't know what else to do
[14-8-13 下午12:04:49] Tai Xiaomei: In a word, I think "a child playing with the keyboard randomly" is testing which is generally called monkey testing, but it is not exploratory testing; and you think it is an exploratory testing, with controlled self-management in it. Right?
[14-8-13 下午12:05:03] James Bach: wrong
[14-8-13 下午12:05:31] James Bach: well
[14-8-13 下午12:05:34] James Bach: not wrong, I guess
[14-8-13 下午12:06:11] James Bach: I wouldn't say "controlled" self-management
[14-8-13 下午12:06:21] James Bach: I would just say self-management
[14-8-13 下午12:06:31] James Bach: controlled self-management is a strange phrase
[14-8-13 下午12:07:10] James Bach: If you are going to re-define what exploratory testing means, then I have to distance my work from yours
[14-8-13 下午12:07:23] James Bach: this is an established idea
[14-8-13 下午12:07:45] Tai Xiaomei: OK. But to say self-management is to say controlled by the person himself, right?
[14-8-13 下午12:07:53] James Bach: yes
[14-8-13 下午12:08:13] James Bach: so "controlled self-management" is redundant and therefore ambiguous
[14-8-13 下午12:08:41] James Bach: All testing is exploratory to some degree
[14-8-13 下午12:08:47] James Bach: all testing is scripted to some degree
[14-8-13 下午12:10:22] Tai Xiaomei: In the spectrum of ST and ET, except the pure scripted point, each point have some exploration in it and should satisfy the ET definition. Right?
[14-8-13 下午12:10:49] James Bach: pure scripted testing is not technically testing
[14-8-13 下午12:10:57] Tai Xiaomei: Yes, I agree.
[14-8-13 下午12:11:03] James Bach: but yes
[14-8-13 下午12:11:26] James Bach: I can find exploratory testing (to some degree) in any act of testing
[14-8-13 下午12:12:17] Tai Xiaomei: Then in your opinion, the child playing the keyboard randomly lies in the extreme point of "freestyle testing" in the spectrum?
[14-8-13 下午12:12:33] James Bach: you keep saying randomly
[14-8-13 下午12:12:38] James Bach: children do not play randomly
[14-8-13 下午12:12:48] James Bach: children are incapable of playing randomly
[14-8-13 下午12:13:26] James Bach: a child playing is an example of bad testing
[14-8-13 下午12:13:32] James Bach: but it is testing
[14-8-13 下午12:14:01] James Bach: and it is exploratory, but NOT NECESSARILY extreme
[14-8-13 下午12:14:05] James Bach: it is also scripted
[14-8-13 下午12:14:24] Tai Xiaomei: Yes, in my initial question, what I'm asking is "playing randomly, without knowing what the person is thinking of", is it exploratory? I know in most cases children don't play randomly, they are exploring the world with their toys.
[14-8-13 下午12:14:34] James Bach: just because it's a child does not mean there is no script to it
[14-8-13 下午12:14:57] James Bach: children are incapable of playing randomly
[14-8-13 下午12:15:14] James Bach: the word you are looking for is "arbitrary"
[14-8-13 下午12:15:17] James Bach: not random
[14-8-13 下午12:15:40] Tai Xiaomei: Oh, yes, arbitrary may be more suitable word.
[14-8-13 下午12:15:53] James Bach: arbitrary means "without a known reason"
[14-8-13 下午12:16:09] James Bach: random means "equal probability of any action"
[14-8-13 下午12:16:23] Tai Xiaomei: Oh, sorry, I confused the two words.
[14-8-13 下午12:16:56] James Bach: often in English, people say "random" when they mean arbitrary
[14-8-13 下午12:17:20] Tai Xiaomei: So playing arbitrarily is exploratory testing?
[14-8-13 下午12:18:21] James Bach: is it testing? I think so (although it's poor testing, probably).
[14-8-13 下午12:19:17] Tai Xiaomei: I think it is a testing, but not a exploratory testing, because exploratory testing involves not only the tester's freedom but also the tester's responsibility for his or her testing.
[14-8-13 下午12:19:20] James Bach: Is it controlled more the outside or from the inside? I think from the inside. That makes it exploratory/informal/non-scripted
[14-8-13 下午12:19:38] James Bach: You have misunderstood the definition
[14-8-13 下午12:20:22] James Bach: there is no requirement for responsibility in ET
[14-8-13 下午12:20:36] James Bach: not any particular special sort of responsibility
[14-8-13 下午12:20:51] James Bach: obviously the child is responsible in some sense
[14-8-13 下午12:20:55] James Bach: but not every sense
[14-8-13 下午12:21:20] Tai Xiaomei: Controlled from inside? Every testing is controlled from inside.
[14-8-13 下午12:21:26] James Bach: the definition does not *require* responsibility
[14-8-13 下午12:21:48] James Bach: scripted testing is controlled from the outside
[14-8-13 下午12:21:55] Tai Xiaomei: If the child is unconscious about what he is doing, can the testing be controlled?
[14-8-13 下午12:23:00] James Bach: yes
[14-8-13 下午12:23:11] James Bach: the child is somewhat conscious
[14-8-13 下午12:23:55] Tai Xiaomei: If the scripted testing is done by a human, it is also controlled from the inside.
[14-8-13 下午12:24:04] Tai Xiaomei: But, what do you mean by "the definition does not *require* responsibility"?
[14-8-13 下午12:24:16] James Bach: yes, that's why we say that all testing is at least a little bit exploratory
[14-8-13 下午12:24:44] James Bach: because the tester interprets the script
[14-8-13 下午12:25:04] James Bach: I helped write that definition
[14-8-13 下午12:25:15] James Bach: I ran the event where we wrote that
[14-8-13 下午12:26:12] James Bach: I can tell you that the definition is intended to emphasize responsibility, but not require it
[14-8-13 下午12:26:28] James Bach: and the definition is not the only possible definition of ET
[14-8-13 下午12:26:37] James Bach: it's my favorite, though
[14-8-13 下午12:27:22] Tai Xiaomei: So can I understand that that definition is for a good ET process, not necessarily for every single exploratory testing process.
[14-8-13 下午12:27:34] James Bach: you might say that
[14-8-13 下午12:28:22] James Bach: I have to go
[14-8-13 下午12:28:26] James Bach: it's after midnight here
[14-8-13 下午12:28:32] James Bach: I need to sleep soon
[14-8-13 下午12:28:48] Tai Xiaomei: So the freestyle ET, like the child playing keyboard arbitrarily, might not be in full consistence with this definition.
[14-8-13 下午12:29:04] James Bach: it IS fully consistent
[14-8-13 下午12:29:15] James Bach: the child IS responsible
[14-8-13 下午12:29:27] James Bach: in at least one sense of that word
[14-8-13 下午12:29:36] Tai Xiaomei: OK, let me think this for a while.
[14-8-13 下午12:29:43] Tai Xiaomei: Have a good sleep.
[14-8-13 下午12:29:47] James Bach: one meaning of responsibility is
[14-8-13 下午12:29:58] James Bach: "being the cause of"
[14-8-13 下午12:30:04] James Bach: as in
[14-8-13 下午12:30:19] James Bach: "I am responsible for breaking that glass"
[14-8-13 下午12:31:29] James Bach: emphasizing freedom and responsibility is not limited to the sense of responsibility as exercising due care, but also includes the sense of being the source of the work.
[14-8-13 下午12:32:30] Tai Xiaomei: OK. Then is the child learning, designing, and investigating at the same time during his arbitrary exploration?
[14-8-13 下午12:35:17] Tai Xiaomei: One thing I learned today is that the difference between "emphasize" and "require". People may have different understandings about the definition of ET. So when I teach, I will tell people this is what I'm learned from this definition, you may have your own understandings. And all of us will gradually evolve our understandings about exploratory testing, which itself is a exploration, too.
[14-8-13 下午12:35:59] James Bach: yes, I think the child is doing that
[14-8-13 下午12:36:12] James Bach: there is a theory of play in children
[14-8-13 下午12:36:25] James Bach: that says children are constantly looking for inconsistencies
[14-8-13 下午12:36:41] James Bach: and making of rules
[14-8-13 下午12:36:44] Tai Xiaomei: Sometimes , when I think deeply about some English words, I became confused.
[14-8-13 下午12:36:45] James Bach: and breaking rules
[14-8-13 下午12:38:04] Tai Xiaomei: Oh, just reminding, what I'm talking about is not childen playing their toys, but a child clicking the keys arbitrarily.
[14-8-13 下午12:38:33] James Bach: is the child watching what happens?
[14-8-13 下午12:38:42] James Bach: is the child reacting to what happens?
[14-8-13 下午12:38:53] James Bach: if not, then it is obviously not testing
[14-8-13 下午12:38:55] Tai Xiaomei: Maybe he is angry, so he does this kind of things.
[14-8-13 下午12:39:01] James Bach: then it's not testing
[14-8-13 下午12:39:02] Tai Xiaomei: No, he is not watching
[14-8-13 下午12:39:12] James Bach: come on, Xiaomei
[14-8-13 下午12:39:27] James Bach: then this is OBVIOUSLY not testing
[14-8-13 下午12:39:34] James Bach: it's not even exploring
[14-8-13 下午12:39:46] James Bach: it's just scattered actions
[14-8-13 下午12:39:52] Tai Xiaomei: But some people called this Monkey testing.
[14-8-13 下午12:40:02] James Bach: well those people are wrong to call it that
[14-8-13 下午12:40:23] James Bach: that is not what anyone I know calls monkey testing
[14-8-13 下午12:40:24] Tai Xiaomei: And sometimes, you do find some bugs accidentally.
[14-8-13 下午12:40:41] James Bach: not really
[14-8-13 下午12:40:48] James Bach: not if the child is not looking
[14-8-13 下午12:41:02] James Bach: how could the child possibly find a bug if he is not looking???
[14-8-13 下午12:41:39] Tai Xiaomei: OK. Then we have the same understanding now. I just want to confirm with you.
[14-8-13 下午12:41:39] James Bach: if someone ELSE is looking, then THEY are the ones doing the testing, not the child
[14-8-13 下午12:41:48] James Bach: the child in that case would be a test tool
[14-8-13 下午12:42:15] James Bach: you might as well drop a rock on the keyboard
[14-8-13 下午12:42:29] James Bach: if you drop a rock on the keyboard
[14-8-13 下午12:42:33] James Bach: and find a bug
[14-8-13 下午12:42:42] James Bach: it is not the rock that found the bug
[14-8-13 下午12:42:57] Tai Xiaomei: Then, are TEHY DOING exploratory testing? Or just testing arbitrarily?
[14-8-13 下午12:43:07] James Bach: the child?
[14-8-13 下午12:43:22] Tai Xiaomei: The child and the person who are watching this
[14-8-13 下午12:43:31] James Bach: the child is neither testing nor exploring if he is doing only pounding of keys
[14-8-13 下午12:43:53] James Bach: the person watching is potentially testing
[14-8-13 下午12:44:18] Tai Xiaomei: Or the question would be "Is arbitrarily pounding of keys an exploratory testing?"
[14-8-13 下午12:44:34] James Bach: it can be
[14-8-13 下午12:44:56] James Bach: I assumed you were speaking of arbitrary pounding in the context of looking at the screen and learning
[14-8-13 下午12:45:07] James Bach: but if you just mean arbitrary pounding
[14-8-13 下午12:45:20] James Bach: that's not testing
[14-8-13 下午12:45:24] James Bach: and not exploring
[14-8-13 下午12:45:44] Tai Xiaomei: OK, thanks, James, I understand now.
[14-8-13 下午12:46:32] Tai Xiaomei: I confirmed with you about what I believed before.
[14-8-13 下午12:46:39] Tai Xiaomei: Thanks and good night.