My goal in this blog post is to de-mystify what performance testing is and why it matters. I’m going to keep this informal, this is a high level introduction only. For those of you who work as performance testers (or know a lot about it) you’ve probably heard this all before.
What is performance?
Performance is a quality of a software system and it has just three components: response time, capacity, and stability:
At a high level:
- Response time is how long it takes the system to respond after I click a button. Response time is intertwined with user experience.
- Capacity is what volume of business activity and user concurrency can our system cope with. We call this activity “load”.
- Stability is whether our system can continue to perform and behave itself over prolonged periods of time.
Every performance issue you encounter will fit into one or more of the above categories.
Why does performance matter?
None of what I do as a performance tester matters unless I’m giving the businesses I work with confidence that their software solutions will perform. What I’m actually doing is protecting their reputation and revenue – performance is all about business risk:
- If a retail website holds a promotion and expected a huge volume of activity on a particular day, then capacity matters to them. They need to be ready for this extra load.
- If a company has an internal system critical to the day to day operation of the business, what would be the cost if it went down for an hour? A day? A week? If the cost is high then stability matters.
- On any system which is customer facing response time matters because the user experience is tied to the reputation of your brand.
What’s performance testing?
Like any other kind of testing, performance testing is about testing for and identifying performance defects before they go into the real world and impact our business and our customers.
The main differentiation with other kinds of testing is that tools are generally required, and the most common kind of tool we use is called a load testing tool.
Load testing tools, at a very basic level, simulate a lot of users doing ‘stuff’ (applying load) to a software solution. While this load is being applied we observe the behaviour of the solution in order to understand its performance.
How do load testing tools work?
Let’s use the example of a coffee shop. What if I wanted to load test this establishment? How would I go about that? Let’s say I just have one test case; buying a standard coffee (the example on the left):
So how would I go about load testing this? Let’s say I hire some actors, maybe 50 of them. I get them to meet me at a park around the corner, and give them all a piece of paper which is their script. On the script are stage directions (e.g. walk into the cafe, go to the counter, etc) and dialogue (e.g. “can I have a standard coffee please?”).
But that’s not very realistic because not everyone orders a standard coffee in the real world. So I modify some of the scripts so some of the actors will ask for a cup of tea, or an orange juice, or a hot chocolate and a muffin. Because this matters – it might take the person behind the counter more or less time to handle certain orders (i.e. we’re adding in variable test data).
So now what do I do? I could get all the actors to run at the coffee shop and try and buy a coffee at once. That might be entertaining but it doesn’t tell us anything useful because it doesn’t reflect the real world. So let’s say I’ve been studying this place and I know for a fact the busiest period they had in the last year had 30 customers come in over an hour. I use that to inform how often I send each actor into the coffee shop.
We now have a crude way of applying “load” to the coffee shop. I’m sitting inside reading the paper. I’m looking at the person behind the counter – are they sweating? Do they look stressed out? How do they respond to all this activity? I’m monitoring the server (pun intended).
The good news is that if you understand this slightly absurd situation, then load testing a software system is almost exactly the same (as shown by the insurance quoting example in the image above). The only difference is that instead of simulating a verbal conversation between myself and the person behind the counter, we simulate the network conversation between a client and a server. We also don’t use real actors – our load testing tool creates “virtual users” (there are many other terms for this) and also controls the rate at which we introduce them to the system under test.
A typical performance testing engagement
This is just a basic framework and every situation is unique:
- Gather information about the solution architecture, the infrastructure, what business purpose this software serves, what different kinds of users there are, and what external and internal integrations exist.
- Assess the risk to help us prioritise our testing effort (if any is required!). A big part of this assessment is to build a workload model which helps us understand the expected load on our solution (both its volume and nature).
- Build a test strategy which details how we will go about performance testing, as well as making non-testing recommendations. I think about performance holisticaly – if we can manage performance risks in other ways, let’s explore those too (i.e. let’s not just focus on load testing).
- Developing a load test suite usually takes significant effort, but we don’t get any value from it until we run our first test.
- We then have cycles of test execution and analysis. I feed the results back after every test to the customer, discuss the results, and respond accordingly. There needs to be a feedback loop here – you never know what will be uncovered during performance testing.
- We may need to investigate performance issues that were identified during testing. This requires deep technical knowledge and great communication skills.
- Finally, we generally deliver a formal test exit report. This should not contain anything the customer doesn’t already know about – it’s just a summary of what was found during the engagement.
This process is changing to meet the ever-changing way we develop and implement software. This is only a basic framework to open up a discussion.
So that’s a very high level look at what it’s all about. I’d be happy to answer any questions you have – I realise I’ve only scratched the surface.
The content of this blog was inspired by a talk I gave at WeTest Auckland. This is the content from the first half of my talk. In my next post I’ll cover the second half of my talk around what it takes to have a career in performance testing and analysis.