pnp.GIF

How To: Evaluate Systems to Improve Performance Testing Effectiveness

Carlos Farre, Scott Barber

Applies to

  • Performance Testing

Summary

While System Evaluation is an ongoing process throughout the performance testing effort, it offers greater value if it is conducted early on. The intent is to collect information about the project as a whole, the functions of the system, the expected user activities, the system architecture, and any other details that are helpful in guiding performance testing to achieve the specific needs of the project. This information provides a foundation for collecting the performance goals and requirements, characterizing the workload, creating performance test strategies and plans, and assessing project and system risks.

Contents

  • Objectives
  • Overview
  • Steps
  • Step 1. Capture System Functions
  • Step 2. Capture User Activities
  • Step 3. Capture Logical and Physical Architectures
  • Additional Resources

Objectives

  • Learn techniques to effectively and efficiently capture the system's functions.
  • Learn techniques to effectively and efficiently capture expected user activities.
  • Learn techniques to effectively and efficiently capture the system's logical and physical architecture.

Overview

Evaluation of the system under test is critical to a successful performance testing effort. The measurements gathered during later aspects are only as accurate as the models that are developed and validated in this aspect. The evaluation provides a foundation for determining acceptable performance; specifying performance requirements of the software, system, or component(s); and identifying any risks to the effort before testing even begins.

Evaluating the system includes, but isn’t limited to, the following steps:
  • Identify the user facing functionality of the system.
  • Identify non-user-initiated (batch) processes and functions.
  • Determine expected user activity.
  • Develop a reasonable understanding of potential user activity beyond what is expected.
  • Develop an exact model of both the test and production architecture.
  • Develop a reasonable model of actual user environments.
  • Identify any other process/systems using the architecture.

Steps

  • Step 1. Capture System Functions
  • Step 2. Capture User Activities
  • Step 3. Capture Logical and Physical Architectures

Step 1. Capture System Functions

This step identifies the system’s core functions that will help build the performance acceptance criteria. Subsequently, workload models can be assessed to validate both the acceptance criteria and the collection of system functions.
For performance testing, it is essential to know the core functions of the system under test, which makes it possible to initially determine performance acceptance criteria as well as the user community models used to assess the application’s success in meeting the acceptance criteria.
To ensure that all of the system functions are captured, start by meeting with stakeholders to determine the overall purpose of the system or application. Before you can determine how best to test a system, you must first completely understand the intent of the system. It is often the case that the project documents don’t explicitly express all of the functionality implied by the stakeholders’ vision. This is why it is a good idea to start with the stakeholders before moving on to evaluate documentation.
Valuable resources for determining system functionality include:
  • Interviews with stakeholders
  • Contracts
  • Information about how similar applications are used
  • Client expectations
  • Your own experiences with how similar applications are used
  • Design documents
  • State transition diagrams
  • Requirements and use cases
  • Marketing material
  • Project plans

Considerations

  • Meet with stakeholders to capture the overall purpose of the system.
  • Keep in mind that contracts and documentation may deviate from the stakeholders’ views of the system. System functions may be either user initiated or scheduled (batch) processes.
  • Interviews, documents, and plans frequently include high-level functions that include a lot of implied functionality. For example, “provide a secure log-in method” implies session tracking, retrieve lost password, create a new user, user identification, user roles and permissions, etc.

Step 2. Capture User Activities

In this step, you will identify the key user activities, since it is not practical to simulate all user tasks. It is impractical and virtually impossible to simulate every possible user task or activity in a performance test. However, before you can decide which activities are most important to simulate during performance testing you need to determine what the possible user activities are.

One of the most effective places to start is to evaluate the competition’s Web site. Whether it is explicitly stated or not, at some point during the project it is likely to become very obvious that the goal is to allow users to perform all of the activities available from the competitor. Knowing up front what these activities are will keep you from being surprised when they show up in the application – whether or not they appear in any of the documentation.
Valuable resources for determining system functionality include:
  • Information about how similar applications are used
  • Client expectations
  • Your own experiences with how similar applications are used
  • Requirements and use cases
  • Interviews with stakeholders
  • Marketing material
  • Help and user documentation
Once you collect a list of what you believe are all of the activities a user can perform, circulate the list to the team along with the question: “What else can a user of any type possibly do with this application that isn’t on this list?”

Considerations

  • Evaluate the competitor’s Web site, as it will allow a certain degree of anticipation for project evolution, since it is likely that keeping up with the competition will eventually become a project goal.
  • Remember to take all categories of users into account when soliciting possible user activities. Customers, administrators, vendors, and call center representatives are likely to use and have access to very different aspects of the application that may not be easily found in the documentation.
  • Spend extra time soliciting exception and error case activities, which are often implied or buried in documentation.
  • If you find activities missing that seem important to you or that appear in competing applications, consult with the relevant members of the team as soon as possible. These may indicate unintentional oversights.

Step 3. Capture Logical and Physical Architectures

This step identifies the relationship between the application and the structure of the hardware and software. This information is critical when you are designing performance tests to address specific areas of concern, and when you are trying to track down a performance bottleneck.

Of the three aspects of system evaluation, this is frequently the one performed least well. It can lead to adverse affects on performance testing later in the project and add time to the tuning process. To capture the logical and physical architectures, the performance tester generally meets with technical stakeholders, architects, and administrators to determine both the physical and programmatic architecture of the system or application for both the production and test environments. This is critical because designing an effective test strategy requires the performance tester to be aware of which components or tiers of the system communicate with one another and how they do so. It is also valuable to understand the basic structure of the code and contributing external software.

Because the term “architecture” is used in so many different ways by different teams, the following sections have been included for clarity

Logical Architecture

Logical architecture, as it is used in this How To, refers to the structure, interaction, and abstraction of software and/or code. That code may include everything from objects, functions, and classes, or even entire applications such as IIS. You will have to learn the code-level architecture from your team, as every team accomplishes this differently. However, there is a common aspect of logical architectures that is worth exploring further – the concept of logical architectural tiers.

The most basic architecture for Web-based applications is known as the three-tier architecture, where those tiers often correspond to physical machines with roles defined as follows:
  • Client tier (the user’s machine) — presents requested data.
  • Presentation tier (the Web server) — handles all business logic and serves data to client.
  • Data storage tier (the database server) — maintains data used by the system, typically in a relational database.
Evaluate System1.GIF
As architectures became more complex, more tiers were added and those tiers began to be comprised of clusters of machines that served the same role, or of single machines serving as the host for multiple logical tiers.
Evaluate System2.GIF

Specifically, this complexity implies the following:
  • It is reasonable to think of a logical tier as a grouping of related back-end functions.
  • Any tier that is depicted in a logical diagram may span more than one physical machine, share one or more machines with one or more other tiers, or exclusively tied to a dedicated machine.
  • Arrows connecting logical tiers represent a flow of data, not network cables or other physical connections.

To confuse the matter even further, virtually no one uses terms like “file storage tier.” The “file storage tier” has become colloquially named “the file server,” whether or not that tier resides on a dedicated server or not. The case is the same for the presentation tier (Web server), application or business logic tier (application server or app server in shorthand), data storage tier (database server), etc.

Put simply, the key to understanding a logical architecture is the following: In a logical architecture, each tier contains a unique set of functionality that is logically separated from the other tiers. However, even if a tier is commonly referred to as “server,” it is not safe to assume that every tier lives on its own dedicated machine

Physical Architecture

It should be clear that the physical architecture of the environment – i.e., the actual hardware that runs the software – is at least as important as the logical architecture.

Many teams refer to the actual hardware as the “environment” or the “network architecture,” but neither term actually encompasses everything a performance tester should be interested in. What concerns you is generally represented in diagrams where actual, physical computers are shown and labeled with the roles they play along with the other actual, physical computers that they talk to.
Evaluate System3.GIF

System Architecture

  • Authentication and application tiers can be served by two servers.
  • The mapping will allow information to better design performance tests.
  • Performance tests can be targeted at application tier directly for example.

Putting these two pieces of the puzzle together adds most value for the performance testing effort. Having this information at your fingertips, along with the more detailed code architecture of what functions or activities are handled on which tiers is what allows you to design tests that can determine and isolate bottlenecks
Evaluate System4.GIF

Considerations

  • Some teams view testing as a “black-box” activity. It reminds and educates the team of the necessity to design performance tests well and maintain knowledge of the entire system – from the load balancing scheme to the thread-sharing model of code objects. Doing so allows the performance tester to identify high risk areas early.
  • To test a Web farm it is necessary to use IP switching techniques to correctly simulate production because of server affinity for IP addresses.
  • Application or Web servers generally have dual NICs to offload balance of traffic going to /from other servers/clients and SQL. It is important to capture those characteristics. The performance tester will not be as effective if he or she is not accepted by the development team as a technical resource. The process of determining the system architecture is one way the performance tester can establish him or herself as a technical resource with the developers and architects

Additional Resources

<<TBD>>

Last edited Mar 17, 2007 at 1:12 AM by prashantbansode, version 3

Comments

QAThoughtLeader Aug 9, 2012 at 7:49 AM 
Thank you for sharing this post. I agree with you that Evaluation of the system under test is critical to a successful performance testing effort.