pnp.gif

How To: Load Test Web Applications Using Visual Studio Team System

J. D. Meier, Prashant Bansode, Carlos Farre, Scott Barber

Applies to

  • Visual Studio Team Suite
  • Visual Studio Team Edition for Software Testers
  • Performance Testing

Summary

This How To topic will show you how to create a load test using Visual Studio Team System to simulate workload with a mix of user scenarios and validate if the load test is simulating the workload realistically. The load test then will produce performance results, to validate if performance goals are being met with regards of the requirements and performance metrics.

Contents

  • Objectives
  • Overview
  • Scenario
  • Workload
  • Performance Goals
  • Objectives
  • Summary of Steps
  • Step 1 – Create a Test Project
  • Step 2 – Create a Web Test
  • Step 3 – Record Your Web Test
  • Step 4 – Configure Your Web Test
  • Step 5 – Replay Your Web Test
  • Step 6 – Add Your Load Test
  • Step 7 – Specify Your Scenario Settings
  • Step 8 – Specify Your Load Pattern Settings
  • Step 9 – Add Your Web Test to Your Load Test
  • Step 10 – Specify Your Browser Mix
  • Step 11 – Add Your Network Mix
  • Step 12 – Specify Your Computers and Counter Settings
  • Step 13 – Specify Your Run Settings
  • Step 14 – Set The Threshold Values For The Metrics.
  • Step 15 – Run Your Load Test
  • Step 16 – Analyze the results

Objectives

  • Learn how to do load testing to simulate scenario with Visual Studio 2005.
  • Learn how to do data analysis to validate the performance results against performance goals of the scenario.

Overview

Realistic load testing is the process of simulating workload with user scenarios, ensuring realism of test design, so it can provide accurate performance data that yield results that will reproduce performance characteristics of production or projected business volume.

Scenario

A Web based application has the following characteristics extracted from production in IIS logs or with marketing projections:
  • Hourly total number of users: 10000
  • Profile distribution of the user scenarios

User Scenarios % of Work distribution
Browse 50
Search 30
Place Order 20

  • Average user session duration: 8 minutes. The user session is defined as the complete navigation path to correspond to a user scenario. For example the user session for Browse scenario can be understood:
    • User opens home page
    • User logs in
    • User opens the product list page
    • User opens the product details
    • User logs off
  • 14% of the users are new users or clients with caching expired in their browsers.

Workload

Given the characteristics of the scenario, the following are the number of concurrent users needed to simulate the scenario.

Total Hourly Number of Users (One user is counted as the user performing a complete scenario one of the below, Browse, Search, Place order 10000
Session Average time - This is the average time among scenarios a user completes them. 7.5 Minutes
Number of Sessions Per Hour - This is the number of scenarios completed in one hour per concurrent user 60/7.5=8
Number of concurrent Users 10000/8=1250

Performance Goals

The performance goals mandated by business are listed in table below:

Objectives Performance goals
# Users Processed Hourly - This is the total number of users scenarios completed ( user sessions ) 10000 total completed scenarios, 50% Browse. 50000 Browse scenarios, 30% Search. 30000 Search scenarios, 20% Place Orders: 20000 Placed Orders scenarios
Orders Transaction Order transaction should complete in less than 8 seconds
Failed Transactions No failed transactions will occurs
Processor Processor time should be less than 75%

Before You Begin

  • Make sure your database used for data binding is created to simulate realistic load tests.
  • Make sure your database used for data binding simulates production data.
  • Ensure database used for data binding resides in a different SQL server than the server being tested.

Summary of Steps

  • Step 1 – Create a Test Project
  • Step 2 – Create a Web Test
  • Step 3 – Record Your Web Test
  • Step 4 – Configure Your Web Test
  • Step 5 – Replay Your Web Test
  • Step 6 – Add Your Load Test
  • Step 7 – Specify Your Scenario Settings
  • Step 8 – Specify Your Load Pattern Settings
  • Step 9 – Add Your Web Test to Your Load Test
  • Step 10 – Specify Your Browser Mix
  • Step 11 – Add Your Network Mix
  • Step 12 – Specify Your Computers and Counter Settings
  • Step 13 – Specify Your Run Settings
  • Step 14 – Set The Threshold Values For The Metrics.
  • Step 15 – Run Your Load Test
  • Step 16 – Analyze the results

Step 1 Create a Test Project

Create a test project that will have the 3 scenarios (Orders, Browse, Search) simulating the workload for the scenario above. Name your project; by default it will have an empty unit test. We will add Web Test and Load Test to this project in later steps.

Creating a Test Project

  • This will open the New Project dialog box.
  • In the New Project dialog box, in the left hand pane, expand the Visual C# node, and select Test.
  • In the right hand pane select the option Test Project, name your test project TestProject1, and then click OK.

Step 2 Create a Web Test

In this step, you create 3 Web tests for 3 user scenarios (Browse, Search and Orders). The Web tests will be used in your Load Test. The Web test will simulate the workload characterization for the scenario.

Creating a Web Test

  1. In Solution Explorer, right-click TestProject1, select Add, and then select Web Test.

Step 3 Record Your Web Test

  1. The Web Test Recorder opens inside a new instance of Internet Explorer.

For this How to we are creating Web test by recording HTTP requests using the Web Test Recorder in a browser session, but you can also build Web tests manually using the Web Test Editor.

For this how to we will be recording a Web Test for all the scenarios identified in scenarios section:
  • Browse
  • Search
  • Orders

Recording Your Web Test

  1. In the Address bar, replace about:blank with the URL address for the three scenarios ( Browse, Search and Orders) individually to create the 3 separate Web tests.
  2. When you press Enter, the page is processed and rendered in the browser window. The actions are recorded for subsequent playback when you run the test.
  3. Go through various steps to create your scenarios.

Step 4 Configure Your Web Test

Once your Web tests are created, configure them for the following items:
  • Data binding for data parameters.
  • Think times.
  • Create Transactions on the Web request(s).

Data binding for data parameters

Set data binding in post or query string parameters to be replaced during runtime. For more information on data binding refer to How to data binding in Visual Studio Team System <<Add url>>

Think times

Setting the correct think times is important to replicate the user behavior. The average user session lasts 7.5 minutes, including the time to process the transactions in the server.
Make sure think time is correct adjusting the value to meet the user’s behavior as identified in scenario section. After the Web test is recorder right click on the Web request and enter the think times. Repeat this step for all Web requests of the Web tests (Orders, Browse, and Search). Leave all remaining values of the properties of the Web request as default.

LoadTestVS1.GIF

Create the Transactions on the Web requests

Create the transaction in the Web request or group of Web requests as identified in the performance goals section. In below example it needs to be created transactions around Orders processing with three Web requests: add to cart, shopping cart and Check out.
Right click on the Web request you want to start the transaction, and click insert transaction. In this case the start transaction will be Add to car page.

LoadTestVS2.GIF

Choose the first page and last page as the transaction range. In the example below three web requests are included in the transaction: AddToCart.aspx, ShoppingCart.aspx and Checkout.aspx. You could include one or several Web requests in one transaction.

LoadTestVS3.GIF

After the transaction is created, the tree view of the script will show the transaction around the Web request(s).

LoadTestVS3a.GIF

Step 5 Replay Your Web Test

In this step you replay the web test to ensure that the web tests are all passing.
  1. In the main window of Web test click the Run Test button in the left hand corner. Or you can right click the Web test and choose the option Run Web Test.
  2. The Web test will be run for a single user, and result displayed in the main window as shown in the figure.
  3. Analyze and make sure there are no errors in any of the requests.
LoadTestVS4.GIF

More Information

The purpose of replaying your Web test is not merely to ascertain if the Web Test passes or fails. Here are the things you might want to check before adding the web test to a Load Test.
  • Check if any of the top request or dependent request has failed, the failure reasons could be URLs called through JavaScript, ActiveX controls, Applets. These problems arise because the recorder can only record requests coming from the Web browser. If you experience any one or more of these problems, you can manually add the necessary requests to the recorded Web test. For more information, see "About JavaScript and ActiveX Controls in Web Tests" at http://msdn2.microsoft.com/en-us/library/ms404678(VS.80).aspx Fix the failure before you add the web test to Load test. This is important because debugging from load test can be painful process.
  • Additionally you can verify if the replay of the Web tests works for data binding parameters. Click on edit run settings and specify the number of iterations. Then click on click here to run again. It will run the number of iterations replacing the parameters.

Step 6 Add Your Load Test

In this step, you add the Load Test to the TestProject1 project. Subsequently we will add the Web Test to the Load Test, which will simulate the load while executing the Web test. You will then set various properties of your Load Test in subsequent steps.

Adding Your Load Test

  1. In Solution Explorer, right-click the TestProject1 project node. Select Add, and then click Load Test.
  2. The New Load Test Wizard starts.
  3. The Welcome page of the New Load Test Wizard is the first page.
  4. Click Next.

Step 7 Specify Your Scenario Settings

Specify name of your scenario. We are performing Load Test for Orders, Browse and Search.
  1. Set the Time Profile Think to Use normal distribution centered on recorded think times. This will use the recorded times from Step 3. Think times represent the time that a user would ponder a Web page before going on to the next page.

LoadTestVS5.GIF

More Information

The choice of Think Time Profiles depends upon intend of usage.
  • By choosing to use recorded think times, Think times are used exactly as they were recorded in the Web test. Because a load test simulates multiple users, using the same think time could create an unnatural load pattern of synchronized virtual users.
  • By choosing normal distribution, think times are used, but varied on a normal curve. Provides a more realistic simulation of virtual users by slightly varying the think time between requests. Normal distribution will simulate more realistically the usage of the Web application where users will differ in think time fluctuation.

For more information on Think Time, please see "About Think Times" at http://msdn2.microsoft.com/en-us/library/ms184790(VS.80).aspx

Step 8 Specify Your Load Pattern Settings

In this step, you will set the load pattern, with the number of concurrent users as identified in the workload .section. The number of concurrent users should simulate the business volume for the scenario (total number of users – sessions in a given time; one hour for our scenario). They do not represent concurrent users, since they will be randomly executing the 3 user scenarios (Browse, Search, Place Orders), with the variance of session time duration.

Using a Constant Load Pattern

  1. Select the Constant Load radio button. Select 1250 users
LoadTestVS6a.GIF

Step 9 Add Your Web Test to Your Load Test

Add all the web tests identified in scenario section with profile workload distribution,(Place Orders 20%, Browse 30%, Search 50%).

Adding Your Web Test to Your Load Test

  1. In the Add tests to a load test scenario and edit test mix screen, Click the Add button.
  2. In the Add Tests dialog box that opens, in the Available tests list box, select the Web test created in previous steps and then click the right arrow to move it to the Selected tests list box.
  3. In the Add Tests dialog box, click OK
LoadTestVS8.GIF

Step 10 Specify Your Browser Mix

In this step, you add the browser mix by specifying types of browsers being used and distribution ratio for each type of browser.

Specifying Your Browser Mix

  1. From the drop-down list of browser types, select the IE6 browser to add to the mix.
  2. As our web site is internet facing web site we would be adding Netscape 6.0 as well, Click Add button from the drop down select Netscape 6.0.
  3. Adjust the distribution of IE user and Netscape as 88% IE and 12% Netscape (based on industry trends).
  4. Then click Next.
LoadTestVS9.GIF

Step 11 Add Your Network Mix

In this step, you add the network mix by specifying types of network, on which the application is accessed and distribution ratio for each type of network.

Specifying Your Network Mix

  1. As our Web Site is internet facing and users from various location are expected to use the Web site, it will be mix of DSL and dial-up connection.
  2. From the Network Type drop down, select the Cable / DSL 384k connection type.
  3. Click Add button and select the Dial-up 56K.
  4. Let the distribution be 70% and 30% each.
  5. Then click Next.
LoadTestVS10.GIF

Step 12 Specify Your Computers and Counter Settings

In this step, you will specify the performance counters to be used for capturing load test results, in order to validate your tests against the objectives identified in scenario section
  • Click Add computer, enter the computer name or IP address. Click on the check box on the left side of the computer. The default counters will be displayed. You may continue to add other computers that need to be monitored.
LoadTestVS11.GIF
  • Select the check boxes, Application, ASP.Net and SQL. In this case SQL and IIS are running on the same machine. Select the counters applicable to different servers, marking the check boxes on the counters under the machine chosen.
LoadTestVS12.GIF

Step 13 Specify Your Run Settings

In this step, specify set of properties which determine how the Load test runs. The run settings determine the length of the test, warm-up duration, maximum number of error details reported, sampling rate, validation level etc.

Specifying Your Settings

  1. On the Run Settings page choose your initial settings.
  2. Set the Warm-up duration as 5 minutes.
  3. Set Run duration 1 hour.
  4. Set Sampling rate as 5 seconds
  5. In the Description edit box specify the description of the Run Settings.
  6. Accept the other default settings.
  7. Click Finish. Your Load test is opened in the Load Test Editor.
LoadTestVS13.GIF

Step 14 Set The Threshold Values For The Metrics.

In this step you will set the threshold values for the metrics identified in the Performance Goals section. During the runtime of a load test the threshold violations will be flagged in the UI for test pass criteria analysis.

Failed Orders Transactions

Click on LoadTest under counter sets then right click on Failed Tests and select add threshold value.
LoadTestVS14.GIF
In the Threshold value window set alert if over to true and warning and critical threshold values to 1. True setting will flag a failure if value goes higher than critical values.
LoadTestVS15.GIF

Processor Time

Do the same procedure for Counter Sets->application->processor->%Processor time.
Set the warning threshold to 70% and critical value to 75%.
LoadTestVS16.GIF

Step 15 Run Your Load Test

In this step you run the Load test to see, how the Web application responds to the web test, under the load simulation.

Running Your Load Test

  1. In the Load Test Editor right click on the Load Test and select Run Test option.
  2. Click View menu in main window then select Full Screen, this will maximize the viewable area.
  3. Once the Load test is complete, it displays a message, "Load Test 'LoadTest1' is complete. The test data currently displayed only represents a portion of the available results. Would you like to view the detailed results from the load test result store?"
  4. Click Yes on the message.

LoadTestVS23.GIF

Step 16 Analyze Load Test Results

  • Analyze if the number of tests completed (Place Orders, Browse, and Search) and failed tests are meeting the performance goals as identified in the Performance Goals section. Click on tables then tests
The number of browse completed scenarios is missing 37 to complete 5000 and the number or search completed scenarios is missing 7 to complete 3000. Also the number of failed tests is one. One Placed Order failed to complete.

LoadTestVS24.GIF
  • Check if there are threshold violations as identified in the Performance Goals section. Click on threshold violations. It will displayed all the violations to the thresholds specify in step 14.

LoadTestVS28.GIF

Considerations

  • Rerun your load test adjusting new think times to check if the session duration is correctly modeled.
  • When doing workload modeling consider load tests for normal and peak loads.
  • Treat workload characterization as a moving target. Adjust new settings for think times and number of users as to model the new total number of users for normal and peak loads

Additional Resources

Last edited Mar 15, 2007 at 9:22 PM by prashantbansode, version 18

Comments

No comments yet.