pnp.gif

How To: Capacity Plan For Web Applications Using Transaction Cost Analysis (TCA)

J. D. Meier, Prashant Bansode, Scott Barber, Mark Tomlinson

Applies To

  • Performance Testing

Summary

This How To shows you how to perform capacity planning for Web applications by using Transaction Cost Analysis (TCA). Transaction Cost Analysis measures the cost of a user operation on the available server resource.

Contents

  • Objectives
  • Overview
  • Summary of Steps
  • Step 1 – Compile a User Profile
  • Step 2 – Execute Discrete Tests
  • Step 3 – Calculate Cost of Each Operation.
  • Step 4 – Calculate the Cost of an Average User profile.
  • Step 5 – Calculate Site Capacity.
  • Step 6 – Verify Site Capacity.
  • Resources

Objectives

  • Learn how to perform capacity planning.
  • Learn how to use Transaction Cost Analysis for capacity planning.

Overview

When testing projects are limited by time, budget, or resource constraints, it is very common to perform mathematical estimates in an attempt to predict application performance. Transaction Cost Analysis (TCA) is a process for conducting a controlled estimate of application performance, based on key measurements of a single transaction through the application. Although TCA does not serve as a replacement for the complete and thorough performance testing of the entire system, it can be very helpful in determining any obvious risks or serious underestimates related to the hardware required to support the system. This How To will help you understand the basic TCA process and how to perform calculations for extrapolating an estimate of future system capacity.
As you might imagine, with any mathematical calculation, it is critical to ensure very exact inputs to the calculations if you want to have reliable outputs and results from the TCA model.

Summary of Steps

  • Step 1. Compile a User Profile
  • Step 2. Execute Discrete Tests
  • Step 3. Calculate the Cost of Each Operation
  • Step 4. Calculate the Cost of an Average User Profile
  • Step 5. Calculate Site Capacity
  • Step 6. Verify Site Capacity

Step 1. Compile a User Profile

An individual user profile is also commonly referred to as a workload characterization (make link reference here), in which information about transaction volumes and application usage patterns is arranged in a matrix or table format. You can also compile individual user profiles based on analysis of existing production traffic data, or a transactional workload taken from your load test plan. One possible resource for identifying production usage is the Microsoft® Internet Information Services (IIS) log files.

The Usage Profile table should contain the following key information:
  • A list of user profiles – the individual pages or components that are expected to be the major consumers of system resources
  • The average duration of a user session – the length of time a user should take to execute the business process, including any “think time” or delays due to user behavior
  • The total number of operations performed during the session – the number of unique transactions a single user executed during the session
  • The frequency with which users perform each operation during the session – the number of times a single user executed any unique transaction during the session

The following is an example of how to compile a user profile from an analysis of IIS log files:
  • Calculate the number of user requests for each page and the respective percentages. The number of user requests for each page can be extracted from the log files. Divide the number of requests for each page by the total number of requests to get the percentage. The following table illustrates a sample profile.

User Requests per Page
ID URI # of Requests Percentage
1 /MyApp/login.aspx 18,234 35%
2 /MyApp/home.aspx 10,756 20%
3 /MyApp/logout.aspx 9,993 19%
4 /MyApp/SellStock.aspx 4,200 8%
5 /MyApp/BuyStock.aspx 9,423 18%
Total NA 52,606 100%

  • Calculate the logical operations and number of requests required to complete the operation. A user operation can be thought of as a single complete logical operation that can consist of more than one request. For example, the login operation might require three pages and two requests. The total number of operations performed in a given time period can be calculated by using the following formula:

Number of operations = # of requests / # of requests per operation

The Requests per operation column in the following table shows how many times the page was requested for a single operation.

User Requests per Operation
ID URI # of Requests Requests / Operations *# of Operations
1 /MyApp/login.aspx 18,234 2 9,117
2 /MyApp/logout.aspx 9,993 1 9,993
3 /MyApp/SellStock.aspx 4,200 2 2,100
4 /MyApp/BuyStock.aspx 9,423 3 3,141
Total NA 41,850 8 24,351

  • Identify the average user profile, session length, and number of operations per session. You can analyze the IIS log files to calculate the average user session length and the number of operations an average user performs during the session. The session length for the sample application was calculated as 10 minutes from the IIS logs, and the average user profile for the sample application is shown in the following table.
Average User Profile
Operations # of Operations Executed during an average session
Login 1
SellStock 3
BuyStock 2
Logout 1


Of course, if the application has never run in a production environment, you would not have any IIS logs to process. In this case, another way to create the user profile is to use the documents created from performance test planning. For more information about identifying user profiles, see "Workload Modeling" <<Add Url>>

Step 2. Execute Discrete Tests

In this step, you will create a very simple test script that will execute discrete tests for each user operation identified in Step 1. You will configure a load test to place enough stress on your system to reach maximum throughput. For example, you might need to run separate tests for the Login, BuyStock, and SellStock operations, increasing the load on the system to the point where the CPU resource is at 100-percent utilization. The test script only fires the requests for a dedicated user operation, such as a single .aspx page.

The procedure for executing the tests consists of the following tasks:
  • Set up the environment with the minimum number of servers possible. Make sure that the architecture of your test setup mirrors your production environment as closely as possible.
  • Create a test script that loads only the operation in consideration without firing any redundant requests.
  • Define the point at which your system reaches maximum throughput for the user profile. You can identify this point by monitoring the ASP.NET Applications\ Requests/Sec counter for an ASP.NET application when increasing the load on the system. Identify the point at which Requests/Sec reaches a maximum value.
  • Identify the limiting resource against which the cost needs to be calculated for a given operation. List the performance counters you need to monitor to identify the costs. For example, if you need to identify the cost of CPU as a resource for any operation, you need to monitor the counters listed in the following table.

Performance Counters Used to Identify Cost
Object Counter Instance
Processor % Processor Time _Total
ASP.NET Applications Requests/Sec Your virtual directory


Note - It is important to have a throughput measure that is reliable and relates specifically to the requests being emulated by the load testing tool. For ASP.NET, you can use Requests/Sec to calculate the processor cost per request.
  • Run load tests for a duration that stabilizes the throughput of the application. The duration can be somewhere between 15 to 30 minutes. Stabilizing the throughput helps create a valid, equal distribution of the resources over a range of requests. It is important to use the same time range for test execution, for each user operation/test you will execute.

Output:

The output from executing this series of steps for each scenario would be a report like the following:
  • Number of CPUs = 2
  • CPU speed = 1.3 GHz

Following table shows a sample report for the results of the load tests.
Load Test Results comparing the Processor % utilization to the Requests/Sec

User operation Process\% Processor Time ASP.NET Applications\Requests/Sec
Login 90% 441
SellStock 78% 241
BuyStock 83% 329
Logout 87% 510

Step 3. Calculate the Cost of Each Operation

Measure the cost of each operation in terms of the limiting resource identified in Step 2. Measuring the operation cost involves calculating the cost per request and then calculating the cost per operation. Use the following formulas for these tasks:
  • Cost per request. You can calculate the cost in terms of processor cycles required for processing a request by using the following formula:
Cost (Mcycles/request) = ((number of processors x processor speed) x processor use) / number of requests per second 


For example, using the values identified for the performance counters in Step 2, where processor speed is 1.3 GHz or 1300 Mcycles/sec, processor usage is 90 percent, and Requests/Sec is 441, you can calculate the page cost as:
((2 x 1,300 Mcycles/sec) x 0.90) / (441 Requests/Sec) = 5.30 Mcycles/request 
  • Cost per operation. You can calculate the cost for each operation by using the following formula:
Cost per operation = (number of Mcycles/request) x number of pages for an operation 

The cost of the Login operation is:
5.30 x 3 = 15.9 Mcycles 

If you cannot separate out independent functions in your application and need one independent function as a prerequisite to another, you should try to run the common function individually and then subtract the cost from all of the dependent functions. For example, to perform the BuyStock operation, you need to perform the login operation, calculate the cost of login separately, and then subtract the cost of login from the cost of the BuyStock operation.

Therefore the cost of a single BuyStock operation can be calculated as follows:
Single cost of BuyStock operation = Total cost of BuyStock – Cost of Login operation 

The cost of a single BuyStock operation is:
39.36 – 15.92 = 23.44 Mcycles 

Following table shows the cost of each user operation in a sample application using the following scenario.
  • CPU Speed = 1300 MHz
  • Number of CPUs = 2
  • Overall CPU Mcycles = 2,600

Cost per Operation for Login, SellStock, BuyStock, and Logout Operations
User Operations CPU % utilization Total net CPU Mcycles ASP.NET Requests / Sec # of Requests Operation Costs (Mcycles) # pages without login Single Operation cost
Login 90% 2,340.00 441 3 15.92 3 15.92
SellStock 78% 2,028.00 241 5 42.07 2 26.16
BuyStock 83% 2,158.00 329 6 39.36 3 23.44
Logout 87% 2,262.00 510 5 22.18 2 6.26


The operation cost needs to be measured separately for each tier of an application.

Step 4. Calculate the Cost of an Average User Profile

The behavior of actual users can cause random crests and troughs in resource utilization. However, over time these variations even out statistically to average behavior. The user profile you compiled in Step 1 reflects average user behavior. To estimate capacity, you need to assume an average user and then calculate the cost in terms of the limiting resource identified in Step 2.

As shown in following table, during a ten-minute session, an average user needs 147.52 Mcycles of CPU on the server. The cost per second can be calculated as follows:
Average cost of profile in Mcycles/sec = Total cost for a profile / session length in seconds

Thus, the average cost for the profile shown in Table 7 is:
147.52/600 = 0.245 Mcycles/sec

This value can help you calculate the maximum number of simultaneous users your site can support.

Cost of an Average User Profile
Average User Profile Number of operations executed during an average session Cost per operation (Mcycles) Total cost per operation (Mcycles)
Login 1 15.92 15.92
SellStock 3 26.16 78.47
BuyStock 2 23.44 46.87
Logout 1 6.26 6.26
Total 147.52

Step 5. Calculate Site Capacity

Calculating site capacity involves knowing how many users your application can support on specific hardware and what your site's future resource requirements are. To calculate these values, use the following formulas:
  • Simultaneous users with a given profile that your application can currently support. After you determine the cost of the average user profile, you can calculate how many simultaneous users with a given profile your application can support given a certain CPU configuration. The formula is as follows:
Maximum number of simultaneous users with a given profile = (number of CPUs) x
 (CPU speed in Mcycles/sec) x (maximum CPU utilization) / (cost of user profile in Mcycles/sec) 

Therefore, the maximum number of simultaneous users with a given profile that the sample application can support is:
(2 x 1300 x 0.75)/0.245 = 7,959 users 
  • Future resource estimates for your site. Calculate the scalability requirements for the finite resources that need to be scaled up as the number of users visiting the site increases. Prepare a chart that gives you the resource estimates as the number of users increases.
Based on the formulas used earlier, you can calculate the number of CPUs required for a given number of users as follows:
Number of CPUs = (Number of users) x (Total cost of user profile in Mcycles/sec) /
 (CPU speed in MHz) x (Maximum CPU utilization) 

If you want to plan for 10,000 users for the sample application and have a threshold limit of 75 percent defined for the processor, the number of CPUs required is:
10000 x 0.245 / (1.3 x 1000) x 0.75 = 2.51 processors 

Your resource estimates should also factor in the impact of possible code changes or functionality additions in future versions of the application. These versions may require more resources than estimated for the current version.

Step 6. Verify Site Capacity

It is very important to verify that TCA accurately predicts your application capacity and future requirements. Because TCA analysis estimates are based on extrapolations of very specific performance measurements, there is a good possibility that some discrepancies might result. Most test projects use TCA as only an initial estimate of performance, but then continue the load testing effort to execute load tests to verify that the TCA model accurately predicts your application capacity and future requirements. In short, TCA is not recommended as a replacement for load testing, and you should still execute a thorough and robust performance test if at all possible.

Verify the calculated application capacity by running load tests with the same characteristics you used to calculate TCA. The verification script is simply a collection of all TCA measurement scripts, aggregated and run as a single script.

The actual values and the estimated values should vary by an acceptable margin of error (+/- 15%). The acceptable margin of error may vary depending on the size of the setup and the budget constraints. You do not need to run load tests each time you perform TCA.

Additional Resources

Last edited Mar 14, 2007 at 11:04 PM by prashantbansode, version 9

Comments

No comments yet.