pnp.gif

How To: Control the Number of Test Iterations During a Load Test in Microsoft Visual Studio Team System

J. D. Meier, Prashant Bansode, Carlos Farre, Scott Barber

Applies to

  • Microsoft® Visual Studio® Team System
  • Performance Testing

Summary

This How To explains how to create and use a Load Test plug-in that will control the number of test iterations for a particular Web test or for all Web tests in a test mix of a load test. It also demonstrates how to terminate the test after the required number of test iterations is reached.

Contents

  • Objectives.
  • Overview.
  • Summary of Steps.
  • Step 1 – Create a Load Test Containing One or More Web Tests in a Test Mix
  • Step 2 – Create a Load Test Plug-In Class to Monitor the Number of Tests to be Executed
  • Step 3 – Add the New Load Test Plug-In Class to the Test Project
  • Step 4 – Add the Load Test Plug-In to the Load Test
  • Step 5 – Add Context Parameters
  • Step 6 – Run the Load Test

Objectives

  • Learn how to control the execution of a load test based on the number of test iterations in a Web test.
  • Learn how to write a Load Test plug-in to control the number of tests in a Web test to be executed during a load test.

Overview

Frequently while designing or conducting load tests, it becomes clear that the only way to achieve the desired test duration is to increase the number of test iterations (e.g., the number of times that a specific scenario represented by a user session) rather than increasing the duration of individual user sessions. For example, this is desirable in the following scenarios:
  • Load tests that are designed to examine the effects of a certain transaction count of a particular Web test, after which another load test starts, characterizing a scenario. For example, a test mix of Browse, Search, and Place Order could be run to replicate a scenario that caused performance challenges in a previous release, such as placing an order after 20,000 orders were entered in the database but are still pending shipping status.
  • Certain workload characterizations require testers to determine the number of users needed to achieve a particular transaction count of a specific Web test over given time period, to reproduce anticipated load patterns. For example, during a peak load a Web site may be expected to process 20,000 orders in a two-hour period. To simulate this scenario, the tester likely need to increase the number of times a transaction is repeated during the given two-hour period to achieve the correct number of transactions without additional virtual users or overlapping sessions.
  • The purpose of a load test that includes several Web tests is to simulate stress on a Web application in order to validate the functionality of a transaction under stress conditions—for example, a load test containing many Web tests in a test mix, one of them placing orders. The load test will run for extreme stress conditions until 20,000 orders are completed. On completion of the test, the tester will check to see if there are 20,000 entries in the orders table.

This How To explains how to control the number of test iterations when using VSTS.

Scenario

This How To considers a typical e-commerce application with Web tests and their workload distribution shown in the following Load Distribution table.

Browse: Open Page->Login->Search->Browse->Logout.
Search: Open Page->Login->Search->Logout
Place Order: Open Page->Login->Search->Browse->AddToCart->Order->Logout

Load Distribution
User Scenarios % of users Users
Browse 50 250
Search 30 150
Place Order 20 100
Total 100 500


Typically, each execution of the above case can be understood as the user session, or the navigation path through the Web site. In the Visual Studio Team System (VSTS), the user session in Web tests can be also be viewed as test iteration. When the Web test is executed from a load test, after going through all pages, it executes the session path again during the test, iterating in a loop cycle.

Summary of Steps

  • Step 1. Create a load test containing one or more Web tests in a test mix.
  • Step 2. Create a Load Test plug-in class to monitor the number of tests to be executed
  • Step 3. Add the new Load Test plug-in class to the test project
  • Step 4. Add the Load Test plug-in to the load test
  • Step 5. Add context parameters
  • Step 6. Run the load test

Step 1. Create a Load Test Containing One or More Web Tests in a Test Mix

First, create a test project that will contain your load test and Web tests. Then, create individual Web tests that reflect key business user scenarios. Finally, create a load test to which you will add your Web tests. While creating the load test, you can set many run-time properties to generate desired load simulation; for example, you can specify the load pattern, browser, and network types and add the performance counters to be monitored.
For more information on creating load tests, refer to “How To: Create a Load Test Using VS.NET 2005” at << to link to how to >>

Step 2. Create a Load Test Plug-In Class to Monitor the Number of Tests to Be Executed.

In this step, you create a plug-in class for your load test project. The Load Test plug-in is an extensibility point that allows code to be executed when test iteration is complete. This Load Test plug-in will allow controlling of load test duration based on the number of test iterations of a particular Web test, or based on the test iterations of all Web tests.
This plug-in can help you gauge the number of users needed to simulate realistic load patterns for workload characterization of a Web site. For example, a load test must simulate placing 20,000 orders over a two-hour period when the toad test is running a mix of Web tests (Browse, Search, and Place Order)—a particularly challenging scenario to simulate for load tests containing many Web tests. Another case scenario is the running of a load test based on transaction count, after which another load test must be executed characterizing a business scenario. The process of writing a Load Test plug-in can be summarized as follows:
  • Create a plug-in class.
  • Implement the Initialize method.
  • Implement the callback function.

Create a plug-in Class

Perform the following steps to create a Load Test plug-in class:
  • Add a new class to your test. To do this, right-click your project, click Add, and then click Class. Name your class MaxTestIterationsLoadTestPlugin
  • Add the LoadTesting namespace to your project. To do this, add a reference to Microsoft.VisualStudio.QualityTools.LoadTestFramework by right-clicking your test project and then clicking Add Reference. Then add the following using statement to your class:

using Microsoft.VisualStudio.TestTools.LoadTesting;
  • Inherit your class from ILoadTestPlugin as follows:

public class MaxTestIterationsLoadTestPlugin: ILoadTestPlugin
  • Add the following member variables to your class:

    private LoadTest mLoadTest;
    private int mMaxTestIterations;
    private int mTestsFinished;
    private bool mAbortTest;
    private string mTestName;


Your class should now look like this:

 using System;
 using System.Collections.Generic;
 using System.IO;
 using System.Text;
 using Microsoft.VisualStudio.TestTools.LoadTesting;
 
 namespace LoadIterationPlugin
 {
 
    public class MaxTestIterationsLoadTestPlugin : ILoadTestPlugin
    {
        #region ILoadTestPlugin Members
        private LoadTest mLoadTest;
        private int mMaxTestIterations;
        private int mTestsFinished;
        private bool mAbortTest;
        private string mTestName;
 
       }
 }


More Information
The purpose of each member variable is as follows:
  • mLoadTest. This variable is initialized with the LoadTest object. The LoadTest object holds the context of a load test’s settings, available run-time variables, and the event handlers to be executed during different stages of load test execution.
The context variables used from the LoadTest object and the event handler used to control the number of test iterations are described below:
  • mMaxTestIterations. This variable specifies the number of test iterations to be executed. It is read from Context Parameter MaxTestIterations of the load test, mLoadTest.Context["MaxTestIterations"] . If the plug-in is used and the context parameter is not defined, the test will run as specified in the time settings of the load test.
  • mAbortTest. This variable acts as a flag to specify whether the test is to be aborted when the condition is met. It is read from Context Parameter AbortWhenMaxTestsCompleted of the load test, mLoadTest.Context.ContainsKey("AbortWhenMaxTestsCompleted") . If the parameter is not defined, the test will stop the load but will continue to execute as specified in the time settings of the load test.
  • mTestName. This variable specifies the test name for which the test iteration is to be counted. This name needs to match the name of the Web test (case-sensitive). For example, a load test may have Browse, Search, and Orders Web tests but should execute only until 1,000 Orders are placed. It does not count when other tests are executed. It is read from Context Parameter TestName mLoadTest.Context["TestName"]. If the parameter is not defined, the test will count all Web test iterations, not the one specified only.
  • mLoadTestTestFinished. This variable represents the callback function hooked to the TestFinished handler of the load test. Every time a test is finished, the callback function mLoadTestTestFinished is executed.

Implement the Initialize method.

In this example, you implement the Initialize method of the ILoadTestPlugin interface, which takes the LoadTest object as a parameter in the Initialize method; assigns the local variables mMaxTestIterations, mAbortTest, and mTestName; and subscribes the event handler mLoadTestTestFinished to the TestFinished event. To do this, add the following code to your class:

 public void Initialize(LoadTest mloadTest)
 {
   //load Test object
   this.mLoadTest = myloadTest;            
   if (this.mLoadTest.Context.ContainsKey("MaxTestIterations"))
   {
      try
      {
         //number of iterations to be executed
	   mMaxTestIterations = 
                int.Parse((string)mLoadTest.Context["MaxTestIterations"]);
      }
      catch (FormatException)
      {
	   throw new ApplicationException(
                "MaxTestIterations not in integer format");
      }
   }
                
   //The function to be executed after every test iteration
   this.mLoadTest.TestFinished += new 
                EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);
 
   if (mLoadTest.Context.ContainsKey("AbortWhenMaxTestsCompleted"))
   {
      try
      {
         // Bool value to specify if test is to be 
         // aborted when condition is met
         mAbortTest =
                bool.Parse((string)mLoadTest.Context[
"AbortWhenMaxTestsCompleted"]);
     }
     catch (FormatException)
     {
        throw new ApplicationException(
                "AbortWhenMaxTestsCompleted not in bool format");
     }
   }
  
   if (this.mLoadTest.Context.ContainsKey("TestName"))
   {
      try
      {
         //The name of the test where the test iterations are to be counted
         mTestName = (string)this.mLoadTest.Context["TestName"];
      }
      catch (FormatException)
      {
         throw new ApplicationException(
                        "AbortWhenMaxTestsCompleted not in string format");
      }
   }
 }


More Information
Before the test starts, the Initialize method is called once; the mLoadTest variable is assigned with LoadTest passed as a parameter; and the local variables mMaxTestIterations, mAbortTest, mTestName are assigned from the context parameters read from the LoadTest object. In addition, the callback function is subscribed to the TestFinished event handler. This will execute when iteration is completed.

Implement the callback function

You will implement the callback function as follows:

 void mLoadTestTestFinished(object sender, TestFinishedEventArgs e)
 {
   if (mMaxTestIterations > 0)
   {
      if (String.IsNullOrEmpty(mTestName) ||
               (String.Equals(mTestName,e.TestName)))
      {
         if (++mTestsFinished >= mMaxTestIterations)
         {
            SetLoadForAllScenariosToZero();
            if (mAbortTest)
            {
               mLoadTest.Abort();
            }
         }
      }
   }
 }


More Information
Every time a test is completed, the TestFinished event is raised, which will be handled by the callback function. Because the mloadTestTestFinished function checks whether mMaxTestIterations is greater than zero, it has been defined as a context parameter as an integer. If this context parameter is not defined, the test will continue as specified in the run-time settings.

If mMaxTestIterations is greater then zero, it checks whether mTestName is null or the same as the test being finished; in this case the condition will be tested for every test, including all of the tests executed for the load test. If mTestName is null or is the same as the test being finished, the counter controlling the tests being executed is incremented using the mTestsFinished local counter. If mTestName is null, the load test will count all iterations for all Web tests in the test mix; if not, it will do so only for that specific Web test.
If the counter mTestsFinished crosses the threshold set by mMaxTestIterations, it calls SetLoadForAllScenariosToZero, which sets the current load to zero for every scenario retrieved from the LoadTestScenarios collection of LoadTest. After setting the load to zero, if mAbortTest was defined, it will abort the test; otherwise the test continues with the load set to zero until the test completes as defined in the run-time settings.
Create the code to set the load for the scenarios as follows:

 void SetLoadForAllScenariosToZero()
 {
   foreach (LoadTestScenario scenario in mLoadTest.Scenarios)
   {
      scenario.CurrentLoad = 0;
   }
 }


More Information
LoadTestScenario is the individual scenario designed in the load test. A load test can have one or more scenarios, each of which can have one or more tests to be executed. The LoadTestScenario includes the CurrentLoad that can be read or set for the current load running on a scenario for each individual agent that is executing the test. If the test is executing locally without agents, CurrentLoad expresses the current load running on the scenario on the local machine.

It is important to note that the Iteration Control plug-in will hold variables pertaining to the local agent executing the load test. If mMaxTestIterations is set to 30, and if there are three agents executing the test, a total of 90 iterations will be executed. Also, when a scenario is set to zero current load as in scenario.CurrentLoad =0 , the load is set for the particular agent executing the load test plug in.

Example Load Testing Plug-In Class

Here is the complete code sample for the plug-in class, which you can simply copy paste.

 using System;
 using System.Collections.Generic;
 using System.IO;
 using System.Text;
 using Microsoft.VisualStudio.TestTools.LoadTesting;
 
 namespace LoadIterationPlugin
 {
    public class MaxTestIterationsLoadTestPlugin : ILoadTestPlugin
    {
        #region ILoadTestPlugin Members
        private LoadTest mLoadTest;
        private int mMaxTestIterations;
        private int mTestsFinished;
        private bool mAbortTest;
        private string mTestName;
 
        public void Initialize(LoadTest loadTest)
        {
            mLoadTest = loadTest;
            if (mLoadTest.Context.ContainsKey("MaxTestIterations"))
            {
                try
                {
                   mMaxTestIterations =
                   int.Parse((string)mLoadTest.Context["MaxTestIterations"]);
                }
                catch (FormatException)
                {
                    throw new ApplicationException("MaxTestIterations 
                                                    not in integer format");
                }
 
                mLoadTest.TestFinished += new 
                EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);

                if(mLoadTest.Context.ContainsKey
                  ("AbortWhenMaxTestsCompleted"))
                {
                  try
                  {
                  mAbortTest =                                               
                             bool.Parse((string)mLoadTest.Context
                             ["AbortWhenMaxTestsCompleted"]);
                    }
                    catch (FormatException)
                    {
                        throw new ApplicationException
                        ("AbortWhenMaxTestsCompleted not in bool format");
                    }
                }
                if (mLoadTest.Context.ContainsKey("TestName"))
                {
                    try
                    {
                        mTestName = (string)mLoadTest.Context["TestName"];
                    }
                    catch (FormatException)
                    {
                        throw new ApplicationException
                                  ("AbortWhenMaxTestsCompleted not in bool
                                    format");
                    }
                }
            }
        }
 
        void SetLoadForAllScenariosToZero()
        {
            foreach (LoadTestScenario scenario in mLoadTest.Scenarios)
            {
                scenario.CurrentLoad = 0;
            }
        }
 
        void mLoadTestTestFinished(object sender, TestFinishedEventArgs e)
        {
            if (mMaxTestIterations > 0)
            {
              if (String.IsNullOrEmpty(mTestName) ||
                 (String.Equals(mTestName,e.TestName)))
                {
                    if (++mTestsFinished >= mMaxTestIterations)
                    {
                        SetLoadForAllScenariosToZero();
                        if (mAbortTest)
                        {
                            mLoadTest.Abort();
                        }
                    }
                }
            }
        }
        #endregion
    }
  }

Step 3. Add the New Load Test Plug-In Class to the Test Project

After the new Load Test plug-in class has been developed, add the class project containing the plug-in into the test project. It is required for configuring the load test properties for the Load Test plug-in.
  1. Right-click the test solution.
  2. Select Add, and then click Existing Project….
  3. Browse to the project containing the Load Test Plug in.
  4. Select the Load Test Plug in. It will be added to the test project.

Step 4. Add the Load Test Plug-In to the Load Test.

In this step, you add the Load Test plug-in to the load test so it can be executed. For a load test to execute the Load Test plug-in, the load test needs to be configured after the Load Test plug-in has been added to the test project. The Load Test plug-in needs to be added in the properties of the root node of the load test, as detailed in the following steps:
  • After the load test has been created, click its root node.
  • In the Properties pane of the load test, click the ellipsis (…).
LoadPlug-In 1.GIF
  • The Set Load Test Plug-in dialog box is displayed, showing available Load Test plug-ins. Select MaxTestInterationsLoadTestPlugin.
LoadPlug-In 2.GIF
  • Click Ok.

Step 5 Add Context Parameters.

Context parameters are data object variables that you pass to the Load Test plug-in. These parameters can be assigned to local variables either during the initialize phase before the load test executes, or during the execution of the callback function. Context parameters are used to allow the passing of variables during execution of a test, to make it possible to change load test settings such as current load, based on conditions such as the Web test name and the number of test executions.
You add the context parameters as follows:
  • In the load test, right-click Run Settings.
  • Click the Add Context Parameter option.
LoadPlug-In 3.GIF
  • In the Properties pane, locate the name default parameter1 with the value empty. Create the flowing three parameters, one at a time:
    • MaxTestIterations. Write the number of test iterations. If this parameter is not defined, the test will execute normally as specified in the run-time settings.
LoadPlug-In 4.GIF
  • AbortWhenMaxTestsCompleted. Set the value to true to abort the test. If this parameter is not defined, the test will not abort.
LoadPlug-In 5.GIF
  • TestName. Set the Web Test Name to count the iterations only for that test. It needs to match the Web Test Name as in the test mix of the scenario. If the test contains three Web tests (Browse, Search, and Orders) and a parameter is created with the value Orders, only the Orders test iteration count is considered for the condition. If this parameter is not defined, the condition counter will be incremented for all Web test iterations.
LoadPlug-In 6.GIF

Step 6 – Run the Load Test

In this step, you run the actual load test. The load test is created with one or more Web tests in the mix setting, with the Load Test plug-in hooked up as shown in previous steps.
The following is an example of load test run with three Web tests (Browse, Orders, and Search Products). The load test will run until 2,000 orders are placed. The load test was configured to run with 20 users and for two hours.
LoadPlug-In 7.GIF

Running Your Load Test

  1. In the Load Test Editor, right-click the load test and then select the Run Test option.
  2. The test runs until the condition is met and an abort message is displayed at the end. Clicking tables and selecting tests from the drop-down list will give the number of tests for each Web test in the test mix, and the total number of tests will be displayed at the bottom of the left pane. The following illustration depicts a result set totaling 2,000 orders executed:
Total tests: 10134
Browse tests: 5128
Orders tests: 2000
SearchProducts tests: 3006

LoadPlug-In 8.GIF

Resources

Last edited Mar 15, 2007 at 10:12 PM by prashantbansode, version 9

Comments

No comments yet.