Blog : Learning from peers

Social Software Testing Approaches

Social Software Testing Approaches

Software teams, agile teams, in particular, are well aware of the need for their developers, testers, product owners, and other team members to collaborate. The usual selection of agile ceremonies creates space for people sharing the planning of the work: planning meetings, product backlog refinement meetings, three amigos sessions for coming up with acceptance examples, and daily meetings just to mention a few. For much of the rest of the time, people are hunched over their own keyboards and screens, working on their own tasks – more so in the time of forced remote work. Even if developers work with each other doing pair programming, testers often work on their own.

What if, instead of sitting in meetings to create a shared understanding, we recognized that a lot of the understanding gets created while the work is being done? Accepting this would drive us to seek approaches to do the work together, and move us from co-creation, in a flow of tasks mutually agreed, to continuous collaboration both between multiple testers and between testers and other roles. This is what I call social software testing approaches. I want us to test together, not only talk about testing together!

Social software testing approaches are recipes testers can adopt in order to work together and improve their testing by contributing to the work and simultaneously learning about the work. The testing work done in a social frame could be exploring a new feature, applying a test technique, creating test automation to document core flows or reviewing and testing software as it is being written through bringing your own perspectives to writing it. These approaches include:

  • Traditional Pair Testing
  • Strong-Style Pair Testing
  • Ensemble (mob) Testing
  • Bug Bashes

When doing social activities a physical space is a common go-to, yet these activities work well as virtual over a video call. Embracing the virtual also enables us to expand these from our own colleagues to the community at large, doing a Testing Tour to learn more, working with other software professionals connected through social media and online courses.

From Planning to Doing – Together

Social Software Testing approaches call for a change in how we think about planning and doing the work. These techniques of doing the work instead of planning the work in a social frame encourage getting the best of multiple people into the work as it is being done, instead of optimizing for getting most of the work done separately, risking rework with feedback coming in later.

Work together can be Planning or Doing

Planning together isn’t the same thing as doing together. Planning together optimizes the work with the idea that when we first create a common understanding, we can effectively work individually. Doing together we know best through pair programming, and it optimizes the work with the idea that in knowledge work, a lot of the relevant learning happens when we are doing the work. No plan survives the action without adaptation. No learning can be transferred through a document, even if the document can be a helpful starting point for someone starting to learn an area they haven’t worked with.

For doing together, there is more than one person doing the work, and learning more while doing. The social frame of doing the work makes sense. Even if it is not getting the most out of us, it gets the best out of us into the deliverable we are working in and can help avoid rework. Instead of going back later to review and confirm the correctness of the result, we can co-generate the options and choose something that works in practice (shown through doing) over a theory (written in a plan). We avoid the ping pong, we generate ideas bouncing off one another we would not come to alone, and we sharpen our knowledge and skills learning in the context of doing the work.

The Options for Doing Together

To do testing together, a few things should be in place. We should have something to test, people to do the testing with, and a specific social software testing approach to apply. We could be testing a unit of code or a fully integrated version of a system. For the testing to be in the frame of social, there should be a minimum of two people (pair) up to whatever number (group).

Pair or Group?

In a pair, you’d expect two people engaged in the work. Two is more than one and a definite upgrade from solo work, but in a pair, both people need to be always on. When one needs a break, both take a break from the paired work. Sometimes people don’t connect too well and may need to exert a high level of energy into the collaboration, even if the connection can be significantly improved through focused work on empathy and understanding power dynamics.

A group is more than two. It is a higher level of investment into sharing knowledge, but it also enables someone to step out of the group to take a break, and rejoin later. It often appears that people are on their best behavior in a group, avoiding some of the difficult pair dynamics by design.

The power dynamic in play shows particularly well when you have people with different levels of experience (junior and senior) or from different functional disciplines (tester and developer). In a pair, there can be only one of each. In a group, we can significantly change dynamics with the numbers of representatives. Also, for three amigos to really work together (tester, developer, and business analyst), we are already past a pair needing more perspectives into the collaborative work we are doing.

For a pair, you can apply a promiscuous pattern, meaning rotation of pairs. This enables everyone to work with everyone. Yet a lot of times in a pair programming organization, testers report that they are left out of the pairs and could benefit from the group setting to join as a third in addition to the pair.

Traditional Pair Testing

In traditional pair testing, we give the work to a pair. The pair divides into roles of driver and navigator, and agree on how they switch the roles. The driver sits in front of the keyboard, having control over typing. As the pair works, conversation guides what gets typed, and the navigator can continuously review the work artifact created. While the driver controls the testing on the keyboard, the navigator often keeps notes of what is going on.

Traditional pair testing can work great when there is a high level of trust and a great connection between the two people. A common problem people report from this type of pairing is that all the control is on the driver, and pairing can become watching the other type and explain. To keep the work energetic, switching the keyboard after completing work from one idea, getting both people in the pair to take turns on the keyboard, is a good practice.

Traditional Pairing In Software Testing and Development
Traditional Pairing. “I have an idea, give me the keyboard!”

”On keyboard” is a little more for tactical focus, whereas “off keyboard” is a little more of strategic focus reverse engineered from mutual agreements and observed actions. Thinking out loud is an important part, as very little testing happens on the keyboard. Rather, testing is an activity using the application as an external imagination to find possible problems and the engaged tester plays a central role. To get your own idea worked through in the pair, you typically take the keyboard, and explain as you go.

Strong-Style Pair Testing

In strong-style pair testing, we still give the work to a driver-navigator pair, with the driver’s hands on the keyboard and an agreement of rotating roles in the pair. The rules of communication are different though. By introducing a rule that any idea must go through both people in the pair, ideas must be spoken out loud to be typed in by the other person.

The driver on the keyboard is not only obeying commands but reviewing the work going through them. While trusting that one’s navigator is good, speaking back, confirming ideas, and filling in the blanks in mutual understanding is welcome. When navigating in such a pair, we start from intent (“what should we get done”), and using things happening at the keyboard as feedback, we can then drill down to pointing where to start, conceptual instructions or step by step instructions.

The notes are common, and the connection between the pair is enforced with the communication rule. This style becomes particularly important when working across experience levels or functional disciplines, as it strips away the idea that the more powerful one is on the keyboard by default.

Strong Style Pairing in Software Testing and Development
Strong-Style Pairing. “I have an idea, you take the keyboard!”

With strong-style pair testing, tactical focus is shared, whereas strategic focus is placed with the navigator. This frees the person on the keyboard to pay attention to the details of operating the tools of the trade. Instead of thinking out loud, the driver focuses on listening and translating ideas into action. To get your own idea worked through in the pair, you pass the keyboard to the pair and take the navigator role.

Ensemble Testing

Ensemble testing (also known as mob testing), scales the strong-style pairing to a group. Using the communication rule that whoever is on the keyboard isn’t allowed to be the source of the idea of what to do next, ideas must come from the group of navigators. Often the groups use a pattern of a designated navigator who channels the group, and roles rotate frequently.

Mod or ensemble testing and developing
Ensemble Testing

The distinction in scaling up strong-style over traditional pairing becomes evident when the group grows. Holding the idea of what we are doing with the group keeps people engaged, whereas watching someone else do and explain does not create the same kind of connection for multiple people.

Ensemble testing puts people together on the same task, in the same space, at the same time and on the same computer. If testing tasks require multiple input devices, they are still all controlled from the group of navigators orchestrating the testing using their words.

Bug Bashes

Bug Bashes (or Group Hugs as Lisa Crispin likes to call them) are a tester community go-to mechanism for group sessions with the intent of finding problems with multiple people engaged in testing. In a bug bash, we prepare a session for solo or pair testing to happen in the same space at the same time, but not all on the same computer engaged in one testing activity, but a group of themed testing activities.

A typical use of a bug bash is to collect and share issues in an end to end scenario, or to accept a new feature by ensuring everyone from the team takes their time in trying it out in a coordinated fashion where results from multiple perspectives get to cross-pollinate and change people’s perspectives.

It focuses more on sharing the results of testing and leveraging the energy of the group when ensemble testing focuses more on getting everyone to contribute to the same task at hand.

Making Your Choices

While there are organizations that do pair testing (and pair programming) or ensemble testing (or ensemble programming) full time, most of us may benefit from these social software testing approaches as a spice to create a more versatile test strategy.

Approach Best Used For
Traditional Pair Testing Working together in an equal pair requiring little rule assistance.
Strong-style Pair Testing Working together in an unequal pair.
Ensemble Testing Learning and innovating a shared way of working.
Bug Bashes Seeing problems from many perspectives.


There’s a lot of variation you can introduce to these basic recipes, and we encourage you to do so – whatever you do, make it work for you.

The social software testing approaches are great for:

  • Larger organizations where functional silos continue to exist (Testers get together to test)
  • Agile teams where we need to share testing skills (Teams get together to test)
  • Teaching anything testing related to a group (A training group gets together to test)
  • Introducing testing perspectives to programming activities (Testers joining ensemble programming)

With my work at the office, I use these techniques to break out of both team and functional silos. And I know of no better way to learn more practical tips on testing than to get together a group and observe while working together. People don’t know what they don’t know, but they notice what they did not know in the context of seeing someone else doing it.

Taking Social Software Testing Approaches on a Tour

In addition to learning within your team, you can extend from learning within the team to the community at large: testing tours applying these approaches give you hands on experiences working with others even in other companies.

I learned the practice of these approaches in pairs and groups of testers and developers across the world. Back then, as the only tester amongst 20 developers, I needed the social frame to learn from my peers. Calling out in social media (Twitter and Slack communities), I found people willing to practice with me. Pair testing with people later started a whole movement of Testing Tours where people call out for doing testing practice together, online, and blog about it. I have had the pleasure of being an early stop for multiple people systematically learning from one another. These Testing Tours are about meeting to learn together, and on some of them people may apply testing tours, the test technique, to illustrate taking perspectives on application.

None of this new style of deep, serendipitous learning would be possible without the social software testing approaches. They may not come easy at first, but they are skills you can practice on.


Maaret Pyhäjärvi

Maaret Pyhäjärvi

Maaret Pyhäjärvi is a collaborative software specialist with an emphasis on testing. She works full time as a testing specialist in a product company and helps other organizations on the side as a testing coach/trainer.



Continue Reading

Ru Cindrea, 5 concepts from BBST that will help you create more powerful automated tests

Maaret Pyhäjärvi, Mob Programming Guidebook

Maaret Pyhäjärvi, Five Years of Mob Testing, Hello to Ensemble Testing


5 concepts from BBST that will help you create more powerful automated tests

5 concepts from BBST that will help you create more powerful automated tests

A lot of projects and companies nowadays no longer have dedicated testers. That doesn’t mean they no longer do testing; they simply share the responsibility of testing inside a development team. Testing becomes an activity that everyone in the team does, but there’s also a strong focus on automation and trying to create large regression suites that cover as much as possible from the overall functionality of the application.

I’ve also seen automated scripts created in several contexts where the people creating them were focused on solving the programming challenges, but they seemed to overlook one key element: how to make their tests powerful. There were lots of hours involved, lots of tools and frameworks, lots of lines of code, but there was little understanding of the application and superficial interest in what the tests will find and cover. So the teams put a lot of effort in creating extensive automated test suites but the question that remained was “Do they bring enough value?”

In contexts like these, it’s easy to forget that, even though you no longer have team members that call themselves testers and even though you no longer do exploratory testing but you focus only on automation, you still need to develop your testing skills.

One way to do that, for example, is through our BBST courses, which give you a chance develop your skills by practicing them. Many of the concepts that are taught in the courses address automation related challenges and help you come up with solutions for problems that are specific to automation and to designing good automated suites.

In this article, I want to give you an insight into some of these concepts and give you a glimpse of what you might learn and what skills you would be working on if you decide to take these courses.

1. Partial Oracles

In testing, an oracle is a mechanism that helps us recognize potential problems. It helps us determine if the product we are testing has passed or failed a test.

A survey into the oracle problem in software testing by Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz and Shin Yoo, nicely categorizes oracles in 4 different types:

  1. Specified Oracles. One of the most common ways of deciding if a behaviour is correct or not is by referring to a formal specification. As the survey authors explain, the specified oracles are not referring to textual documentation, but “model-based specification languages, state transition systems, assertions and contracts, and algebraic specifications.”
  2. Derived Oracles. When specified test oracles are not available, distinguishing between correct or incorrect behavior can be done using derived oracles: how do other similar systems work? How did the previous version of this product work? Is there a model that can be generated from the textual documentation?
  3. Implicit Oracles. These rely on general implicit knowledge, like: an application shouldn’t crash, a web page cannot be too slow, etc.
  4. The Human Oracle.  The tester or observer uses their own knowledge and understanding as well as the knowledge of others around them to decide if the behaviour they are observing is correct or not.

While doing exploratory testing, our evaluation of the product comes from our general expectations and product-specific knowledge that we have. We use the human oracle a lot and we also use specified oracles as much as possible.

In automation, however, our ability to write good automated tests depends on our ability to choose and to use good test oracles, since we need to detect programmatically if the product passed or failed the test. Here are some examples of oracles that are addressed in the BBST courses that can be extremely useful when writing automated tests.

Regression oracle

A “regression” is a problem in the product that used to work as expected in a previous version, and thus regression TESTING often involves re-running the same set of tests from one version to the other. This type of testing is often automated since the suite of tests tends to grow larger with each new version of the product. However, regression testing doesn’t necessarily use the regression ORACLE. In most cases, testers use the human oracle alongside specified (e.g. assertions) and other derived oracles (like textual documentation) when they are designing the tests that will be included in the regression suite.

When doing regression testing and using the regression ORACLE, the previous version of the product is the reference. Even though we might not know how the product should behave in a certain case and even though we don’t have a specified oracle to help us, we know it should work exactly the same way as it did before, in a previous version. If we have a set of tests that we know have passed on version x of the product, we know they should also pass on version x + 1, so we can use the results from version x as an oracle version x + 1.

In typical regression testing suites, we have a set of test scenarios where we do something in the program and assert that this or that has happened afterwards, so we typically define in detail what the expected result is. This is often cumbersome (there are many aspects to verify and assert on to make sure that a page is correctly displayed, for example). Instead, using regression oracle can be very simple as it relies on a “straightforward comparison”, as Cem Kaner describes in the BBST Foundations course.

Regression testing, in general, comes with a disadvantage: if the application changes, then the oracle will give false positives and it will identify as problems aspects that have been changed on purpose. To avoid this, the tests need to be continuously maintained. There are, however, situations when this oracle can prove to be a good option, especially if the comparison between results from different versions is trivial. If that’s not the case, you may be spending a lot of time counteracting the disadvantages, so use it wisely.

Self-verifying data

Self-verifying data is test data that embeds the expected output of a test. For example, when testing the checkout feature of a webshop, we could use a small database of testing credit cards, where:

  • The names of the card owner tell us if the payment with that card will be successful or not and what the error will be, if any (e.g.: First names: “Insufficient Funds”, Last name: “Declined”).
  • The first digits of the card number tell us the expected error code: (e.g.: Card number: 431223456906789, expected error code for Declined is “431”)

An automated test would pick a credit card from the list of available ones, try to use it for payment, then check the result against the information contained in the credit card number and first and last name to see if it’s correct or not.

Self-verifying data is great in automation because it helps create better test data and stronger asserts.

State Model Oracle

This is one of the oracles that are more and more popular these days as people start to see its value in testing if your system can be treated as a finite state machine. If your system is more complex than that, you may consider using statecharts instead.

Once you create a model of the SUT’s behaviour, you can use it to test it

How does this work?

You create a diagram or matrix with the states the system can take and all the transitions to those states. Let’s assume we want to test an access control system based on access cards. I will oversimplify the example to keep it short and easy to follow. We have a card reader and a control panel that either keeps the door locked or unlocks it if it can find the card code in the access control list.

The states for the access control systems are:

  • LOCKED – default state, it will not allow the door to be opened
  • UNLOCKED – for this, we need a card that has a code registered in the access control list. This state will transition back into LOCKED after a number of seconds
  • FORCE LOCKED – during the weekend, no card should be able to open the door, even if the card is registered in the access control list
  • FORCE UNLOCKED – in case of fire or other major incidents, the door should be unlocked automatically and not return into LOCKED state unless the emergency has ended

The other component of this model is the transition. It reflects the change from one state to another. One example is regular_unlock, that changes the state of the system from LOCKED to UNLOCKED.

Here’s an example of how you can model this system:

State Model Oracle Example

Now that you have the model, you know what you expect from the system in terms of transitions from one state to another. You can either test the system in an exploratory way, or you can create automated tests. If you use an algorithm that implements Markov chains, you can go through the graph and let it decide based on the assigned probability which transition it will take from one state to another. By adding this randomness, you can test a lot of scenarios/paths that would be hard to follow/script and run otherwise. 

You can have two objectives: 

  1. focus on the graph coverage by going through all the states and transitions.  
  2. test the reliability of the system by running the tests for long time intervals.  

The BBST Foundations course also discusses the idea that all of these oracles are incomplete. This is, perhaps, one of the most important concepts to keep in mind when using oracles in testing in general and in automation in particular:

All of the oracles that we use in testing only pay attention to a part of the result.

The regression oracle tells us how the program used to behave in version x, but knows nothing about all the new features that we have introduced in version x + 1 or about the bugs that we missed in version x.

Self Verifying Data might help us evaluate the output of a function and decide if it’s correct or not, but it tells us nothing about how fast it should return or what parts of the disk it should write the result to.

State Model oracle gives us the reference about how the states of the system and how it should transition from one state to another, but doesn’t give us any information about the performance of the system.

In automation, the trick is to use not just one, but a combination of these programmable partial oracles. One oracle can look at how the product used to function in a previous version, while another will check that it does it in a reasonable amount of time and with a reasonable amount of resources. Another oracle might look at all the calculations the product does and check that they are mathematically correct, while another one might check the behavior of our product in comparison to that of a competing product.  Besides these 3 oracles, there’s a list of about 10 more that are presented in the BBST Foundations class and that are all useful in automation.

2. Risk-based testing

My experience with projects and teams that use agile methodologies is that some teams design tests based exclusively on the acceptance criteria and using a narrow understanding. These are abstract, high-level requirements and one phrase can include a multitude of expected and unexpected behaviors. How do you get from an acceptance criteria to a set of powerful tests? One technique to do this is to start with the risks. Think about these questions:

  • In what ways can the program fail?
  • What could go wrong?

Create a list with these risks and then design tests based on it. This will give you a lot of testing ideas that you probably wouldn’t have come up with if you only considered the acceptance criteria. You could take this idea further than what’s presented in the BBST Test Design course and prioritise the tests and add them to automation testing suites that are organized based on risks. You can run them selectively, based on the risk(s) you want to cover in a particular version of the application.

3. “The most powerful test/value”

I attended a talk once where a developer was discussing ways to improve your unit tests. He showed a unit test for a function that multiplied a number by itself. The unit test gave the function the input 5 and checked that the output was 25. To improve this test, he showed how to use a library that would allow him to test all the numbers from 5 to 1000000. With very little effort, he explained, he already had 1 million tests instead of one.

One of the testers in the room asked: “But why would you test with those 1 million values? They don’t give us any new information, they are just as likely to fail or pass as 5 was.” The developer replied: “Because we can!” and, in that case, he was right: running the test with 1 million values didn’t seem to take any longer than running it with one value. But then, in real life, how often do we have to test a function that takes a number and multiplies it by itself?

In my experience, the automated tests that we have to write on today’s projects are far more complex than the one in the example and take a lot longer to execute. I’ve recently worked on a project where I had to write a BDD scenario describing the first time use for an online shop. The scenario was to visit the shop, choose a product, choose a size, add it to the cart, go to checkout and create an account to pay for the purchase. While designing these scenarios, I had to decide which product to add to cart. The 5th? Is that enough? I couldn’t test all of them, because there were thousands, and the tests would have taken way too long to complete.

The concept of the “power of a test” from domain testing helps with choosing these values. A test is more powerful than another one if it is more likely to discover a bug than the other one. Out of all the possible values for a variable, some of them are just as likely to find problems (they are equivalent) while others are more interesting.

The BBST Test Design and Domain Testing courses teach us that equivalence classes go beyond just the usual partitioning of a domain. We can have risk-based equivalence classes, and these are excellent when trying to prioritize what test data to cover in automated scripts.

4. Code coverage vs test coverage vs other types of coverage

Many programmers and testers work with the concept of code coverage, mostly measured from running unit and integration automated tests. I’ve worked with teams that have hard to attain code coverage goals that they had to meet, and I’ve heard many testers assume that 100% code coverage means that “everything is tested”.

However, the BBST classes work on understanding that 100% structural code coverage does not mean complete testing. Even though every line and branch in the code is covered by one test, it doesn’t mean that we have tested, for example:

  • Every possible input value for each variable 
  • Every possible combination of inputs
  • Every possible sequence through the program
  • Every possible interrupt, timeout or race condition
  • Every possible configuration
  • Every possible interference

Also, one common misconception about coverage in general that the BBST courses address is the idea that it’s focused on the code only. Coverage, however, means the extent of testing of a given type that has been completed, compared to the total amount of possible tests of that specific type. We can talk about configuration testing and think of coverage from this point of view: how many configurations have we covered? Or we can think about software requirements, design tests for each of them and measure how much of them we have covered after running some of the tests.

Achieving complete coverage would be equivalent to achieving complete testing but, in practice, that is never achievable. It would mean running 100% of all the possible tests of all the possible types, and that is impossible.

5. Measurement dysfunction

“We have 10000 automated tests”, “Let’s measure the testers’ productivity by the number of bugs they find per sprint” “We can write 10 automated tests per day”. Are any of these statements familiar to you?

People like numbers and they will try to find correlations between them that could ease the effort to draw conclusions and make decisions. But to differentiate between valid measurements, where the correlation between the variables is real and the ones that are a set of coincidences, you may need to understand some key concepts. You need to understand what type of measurements are out there, what are the valid models that make the measurements valid.

BBST Foundations class will help you get a solid understanding of the measurement dysfunctions that are more common than you’d expect in the software world. Once you go through the material and you discuss with your peers and instructors, you will probably have a good understanding of what measurements you can rely on for good decisions and which ones you should take with a grain of salt. You wouldn’ want to have all your team get a false sense of safety and be surprised when users start to report unexpected behaviour.

It’s been almost 10 years since Alberto Savoia’s keynote at GTAC 2011, where he declared that “Test is dead!” – and what he meant by that was that traditional testing was transforming. He was worried that, along with their testers, companies were also getting rid of testing altogether. He saw that as a threat then, and it’s still a threat today: even if we don’t have testers, we still need testing. Savoia said “Test is dead”, and continued with “Don’t take that literally, take it seriously”. Especially if you are in automation, developing your testing skills is the key to avoiding that threat.

While this article lists some ideas and concepts that can help you towards developing these skills, the only way to really improve is by practicing them. The BBST courses offer this chance to practice them in-depth in a safe environment and to get valuable feedback from experienced instructors. Join us for the next one! 


Ru Cindrea

Ru Cindrea is a BBST instructor, senior test consultant, and managing partner at Altom Consulting. With over 15 years of experience, she believes all testing is exploratory. She is currently working with the Finnish testing community on building a strong group of testing professionals.



Continue Reading

Maaret Pyhäjärvi, Social Software Testing Approaches

Alexandra Casapu, How well do you advocate for your bugs?

Oana Casapu, Live webinar with Ru Cindrea: Lessons Learned from the Worst Bug I Ever Found


How well do you advocate for your bugs?

How well do you advocate for your bugs?

If you do testing, and recognize how cognitively rich the activities involved in testing are, you probably also recognize the importance of testing skills.

On all the projects I’ve contributed to, good testing, deep testing, involved skills. Asking a random person from the street to test on the project would probably not have led to spectacular results (unless, of course, they happen to be an exquisite tester with awesome testing skills!). Developing those skills requires a lot of work.

On each project, stakeholders expected from me to be able to provide relevant information. This included sometimes:

  • the status of the product,
  • progress of the testing effort,
  • feedback on the development process,
  • and risks worth considering.

This information can be conveyed in different forms, and one of the prevalent ones is creating bug reports. In many of my testing experiences, bug reports have been a main way of communicating relevant information on the project.

Replicate, Isolate, Maximize

I quickly understood that “bug advocacy” did not mean insisting to get EVERY ONE of the bugs I found fixed. It also did not mean coming up with unreasonable arguments for fixing a bug because I wanted to prove something or look good in the face of a developer. I associated the idea of bug advocacy with presenting relevant information in a bug report to help others decide if they want to fix the bug or not.

As I accumulated more experience in doing testing, I realized more and more that I needed to learn new, specific skills to do this work well. For some reason though, bug advocacy skills were not at the center of my learning focus at the beginning. I observed how others around me reported issues, and tried to mimic that. I chose examples from people I thought were doing this well, and tried to follow along. From this process I gathered ideas like including the browser and OS version in my bug report when it was relevant, writing a clear summary, organizing the relevant information in steps to reproduce, include the result I’m getting and the oracle I’m using to decide that a behavior may be a problem. I slowly built a template for that.

In terms of bug investigation, I learned to try to reproduce the issue more than once, that it was important to understand what triggers the problem, and what impacts it could have. I didn’t have a template for that though, so I followed my gut instinct and decided to apply my analytical thinking to guide me through. That was the level of detail I got into analyzing how I do bug investigation.

And then the idea of “bug advocacy” started to take shape in my head. I took the Bug Advocacy course from AST in 2012 and realized there was more to that than I had initially thought. There was a wide array of skills related to doing effective Bug Advocacy. I got familiarized with a framework for investigating bugs and dealing with excuses to not fixing them.

That was when I consciously started considering Isolation, Maximization, Generalization, and Externalization of bugs. So I started practicing on my projects using a structure. Through feedback I understood that a template for bug reports does not always work, that it’s important to find ways to effectively tailor the reports to the context.

I practiced with a follow-up testing framework that would help me understand if the bug was more serious than I initially thought. I practiced with what would be relevant to vary and to consider in order to manage to reproduce the hard-to-reproduce bugs, and to uncorner my corner cases. I started grasping just how powerful bug reports are as a tool for communication. And how being more skillful at this makes a big difference. It influenced how I was perceived by the programmers, how much the project managers trusted my suggestions and appreciated testing efforts.

Generalize, Externalize, Neutral tone

As with other arrays of skills I use in testing, I need to clarify my understanding of the concepts and to practice with bug advocacy skills in order to improve them. And as I go deeper, I find there’s more and more interesting skills to work on. Why, then, have I initially overlooked this area of skills in testing?

I think it’s because I thought I’m doing bug advocacy as good as any other fellow tester. A multitude of skills involved were not visible to me. They were not explicit, but incorporated in the reports, and in the testers’ credibility. I hope they are visible to testers around me now, and I strive to make them so.

As part of that, I got involved in actually teaching the Bug Advocacy course. As part of Altom’s collaboration with Cem, I’m lucky enough to be able to participate in teaching the course alongside him. And it has been enlightening so far. One central feature which I missed when I did the course as a student was the interactive grading session halfway through the course. Since this course focuses on developing bug advocacy skills, it contains practical assignments. Students get to actually contribute to the bug reports of an open source project. After their first contribution to such a report, they have an interactive grading session in which they get feedback on what the did well, what to improve, and how. Afterwards, they get the chance to work on a different bug report and incorporate the feedback. I think this loop of doing a task, getting good feedback, then trying again, is essential to developing skills. And I’ve had the chance to see the students’ evolution in this context. So that’s one of the elements I’m looking for in my own learning process: after advocating for bugs, getting useful feedback, then getting the chance to tweak the process. A challenging aspect is that contexts change all the time, as I work with different teams, and encounter different types of issues. Which is also fun, because there’s always something new to learn in the process.

I hope to internalize the bug advocacy frameworks I’ve learned about more and more as I consciously practice with them and think critically about them.

So, what about you? What’s your way of improving your bug advocacy skills?

PS: You may have noticed that the bugs drawings represent the RIMGEN bug advocacy framework. Thanks to my colleague Levi for creating them! You can download the printable RIMGEN postcards for free.



Register now for the next BBST Bug Advocacy course!

Why BBST® Foundations is Not Only For Novice Testers

The first course in the Black Box Software Testing series is called BBST® Foundations, but don’t let the title fool you: this course is not only for beginner testers. How would you, no longer a novice to software testing, benefit from taking this course? – To help you find an answer to this question, we have compiled a list – tailored for the experienced tester – of valuable takeaways from the course.


Reviewing and deepening your knowledge of the fundamentals of software testing.

BBST® Foundations is an opportunity to go over the essentials and, most importantly, a space facilitating a deeper reflection on these. How often do you stop and reflect on your testing notions and activities? What is the mission of testing? How would you define your role as a tester? What is a bug? – All these are basic, “simple” questions participants in the course are encouraged to ponder, with the ensuing discussions being wonderfully complex and enlightening.  

If we were to take as reference a scientific model to describe this transition to another level of knowledge, we could invoke Anderson and Krathwohl’s revision of Bloom’s classic cognitive taxonomy. The revised taxonomy emphasizes how learning occurs over several – both knowledge and cognitive process – levels. The model includes four knowledge dimensions – factual, conceptual, procedural, metacognitive – and six cognitive processes: remember, understand, apply, analyze, evaluate, create. Anderson and Krathwohl’s model could be represented like this:


Anderson and Krathwohl's revision to Bloom’s classic cognitive taxonomy


Bringing that to our context, a non-novice tester might have a factual and conceptual knowledge but still have yet to achieve procedural and metacognitive. They might remember and understand, but still have issues with apply and evaluate, or have not reached the create yet. This course can come in helpful in this process of transitioning to a superior cognitive layer. For instance, you know what heuristics are but you do not apply them in your work, or you do not come up with heuristics of your own (or you do but are not aware of that).


Preparing for the next course/s in the Black Box Software Testing series.

The BBST® series consists of three courses: Foundations, Bug Advocacy, and Test Design. Each course in the series is designed as a prerequisite for the next. Starting with Foundations, students get a grasp of the structure of the courses, and become familiar with the online platform that hosts them, Canvas. This helps with future organization and time management.


Canvas - BBST Foundations online learning platform


Interacting with your peers and learning from their experience.

BBST® Foundations is a course which places emphasis on practical work and on interaction. Taking part in it is a great way of reminding yourself that testing is a social activity. One of our students has pointed out in the feedback form that BBST® Foundations is an “online course more social than any regular courses I’ve ever attended”.

Students in the course are encouraged to communicate, to connect, to debate, to express their points of view, to share stories and ideas. Topics are discussed with input from both the other students and the instructors. There is so much to be learned from these daily exchanges. For instance, not once has it happened that students have found solutions to testing problems they were facing by participating in the discussions. A testing technique that you are not familiar with described in a post, or a way of organizing the test team that you have not thought of – participants often find inspiration to deal with their daily challenges in experiences described by the others. There is a great deal of learning that comes from this sharing of personal stories and experiments, with their victories and their mistakes.


Working on your review skills and on giving productive feedback.

The course offers plenty of opportunities to exercise these valuable skills: there are assignments that involve reviewing the work submitted by your peer students. Correspondingly, your work will also be reviewed, by your peers and by your instructors. This exposure to different ways of giving and receiving feedback promotes understanding and an increased awareness of the various mechanisms at play, of the dos and the don’ts. We are sure you will find these very helpful when next reviewing the performance of a colleague at work.   


BBST students discussing


Developing your argumentation skills and discovering new ideas for structuring your arguments.

Besides being engaged in reviewing activities, all students are encouraged to participate in forum discussions over the various items in the course. Helpful information is gained from these exchanges. This is good practice for the discussions that will follow in the workplace, helping cement argumentation skills and providing new ideas for you to use and expand upon.

This is one of the most valuable takeaways of the course: you do not simply get to revisit concepts and exercise your skills, you will actually use what you learn in your work as a tester. You will find, for instance, you have more solid guidelines for evaluating the heuristics you use in a particular context, or for searching suitable oracles in different situations.


Taking a valuable course, highly recommended by specialists in the software testing field.

One of the content owners of the course is Cem Kaner, a noted figure in the field of software testing. Professor of Computer Sciences and Cybersecurity at the Florida Institute of Technology, Cem Kaner is also the author of some of the most insightful writings on software testing. He is senior author of the classic Testing Computer Software (with Jack Falk and Hung Quoc Nguyen), and also of Bad Software (with David Pels), and Lessons Learned in Software Testing (with James Bach and Bret Pettichord), the first self-described context-driven book.

Cem Kaner is the primary creator of the BBST® courses and the series has received much acclaim, being considered a first-rate option in terms of meaningful courses for software testers.


Software testing is about exploring, thinking critically, communicating – and a healthy dose of curiosity. There is a lot for the curious tester, whatever their level of experience, to explore in BBST® Foundations: from questions such as “what is testing?” and “how do we measure testing?”, to a diverse assortment of practical assignments, to a multitude of – more or less divergent – points of view on the various aspects of testing. Sometimes there is no need for an additional reason: a tester might simply indulge their curiosity, follow the spirit of exploration and take the class 🙂