This project is read-only.

Explained: Load Agent Deployment Scenarios in Visual Studio Team System

- J.D. Meier, Carlos Farre, Prashant Bansode

Applies To

  • Visual Studio Team System

Contents

  • Overview
  • Scenario 1 – Single workstation scenarios
  • Scenario 2 – Local Controller, Local Results Database, and Remote Agent
  • Scenario 3 – Remote Controller, Remote Results Database and Remote Agent(s)
  • Scenario 4 – Remote Controller, Remote Results Database and Multiple Remote Agents
  • V-Lan Infrastructure Examples
  • How Many Machines and Agents Do You Need
  • Connection Pool Mode versus Connection Per User
  • SQL Sizing Considerations
  • Multi-processor Considerations
  • Security Considerations
  • Additional Resources

Overview

You can do performance and load testing with Visual Studio Team System solution, comprising of different products. The test and load agent solution complements the Visual Studio Team System and includes controller and agent, allowing for scalable test configuration, generating extra test load distributed across several machines.

The agent is used to simulate multiple clients from a single workstation. You can use multiple agents to increase the number of clients you simulate. The controller controls the test execution of the agents and collects performance counter metrics. You can share the controller among different Visual Studio workstations in two different scenarios:
  • The Visual Studio workstations will share the same agents, and in that case load test executions coming from different workstation machines will not happen simultaneously; load tests will be executed on first come basis, with remaining tests queued up by the controller.
  • The Visual Studio workstations will use their own exclusively assigned agents, and in that case load test executions coming from different workstation machines will happen simultaneously. .

For doing performance and load testing following two main options
  • Using Single Workstation
  • Using Separate Load Test Controller and Agent.

Using Single Workstation (Scaled Down Load Test Framework)

In the simplest scenario, you can run load tests on your local workstation, without a separate controller and agent. In this case, VSTestHost.exe will act as the agent and controller. This scenario is a scaled down load test framework:
  • Case 1 – you are a developer doing preliminary testing for a component or related set of components and you want to test performance/concurrency.
  • Case 2 – you want to load test an application and you have a number of virtual users that you can simulate from your workstation.
  • Case 3 – you don’t need to share the performance testing infrastructure. Each developer or tester is running tests from their single workstation.

Using Separate Load Test Controller and Agent

If you need to simulate more users beyond what a single workstation can handle or if you need to create a shared environment for performance testing, then it will be required to implement scalable test configuration with controller and agent(s). The Visual Studio workstation acts both as a test development platform and local load generator; the Visual studio load agent acts as a distributed load testing platform, with controller hosted by QTController.exe and agent hosted QTAgentService.exe acting as a dispatcher and process QTAgent.exe acting as the load generator engine.

Scenario 1 – Single Workstation Scenario

Scenario1.GIF

Single Workstation Scenario

Key Characteristics

  • It’s a scaled down version of the load testing platform.
  • Visual Studio acts as development and load testing platform.
  • The test is hosted by local Visual Studio VSTestHost.exe process.

When to use

  • Use this option for testing scenarios with limited scope like performance testing components or assemblies on their local workstation.
  • Use this option, for load test with limited number of virtual users.
  • Use this option, for cases when you cannot handle the high number of virtual user but you need to do some preliminary performance testing that allows finding bugs earlier in the development cycle, before handing the code to the test team.

Key Points

  • The client is used to develop tests, select tests to run, and view test results. The client is also used to execute the load and collect the test results.
  • With this configuration the execution of tests is affected by any locally running processes, which may skew performance testing results.
  • If the application being tested is in an isolated private VLAN, it forces Visual Studio to be installed in machine in the same VLAN that can access the application.
  • For data collection it is required that machine running Visual Studio to have access to machines being monitored for performance data.
  • It is not possible to disconnect from the machine running Visual Studio.
  • Visual Studio machine cannot be closed without interrupting the test.
  • If the test crashes VsTestHost.exe there is no restart functionality.
  • Using Visual Studio to generate load, depending on the number of virtual users, the think times and the transaction response times may cause processor bottleneck and skew the performance testing results.
  • Using Visual Studio and results database in the same machine may cause memory pressure or disk IO activity, depending on the amount of data collected and sampling rate. This may skew the performance testing results. Consider increasing memory or using better disk system.
  • Using Visual Studio load test an application using Connection per user mode may cause memory pressure, depending on the number of virtual users. In this case each virtual user will be represented by a running thread. Consider increasing memory to handle more users. The connection pool model conserves the resources on the load test agent by sharing connections to the web server among multiple virtual web test users, with less memory footprint. However if the user load is larger than the connection pool size, then web tests executing on behalf of different virtual users may need to wait before issuing a request when another web test is using the connection. Consider tracking the counter load test performance counter “Avg. Connection Wait Time”. This number should ideally be a less than the average response time for a page. If it is not, then the connection pool size is probably too small.
  • It does not allow sharing the performance testing lab infrastructure.

More Info

  • To increase capacity for handling more number of Virtual Users if you have reached processor, memory or Disk IO limits consider using a distributed test deployment configuration with controller and agent(s).
  • To be able to use shared lab infrastructure consider using a distributed test deployment configuration with controller and agent(s).
  • Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

Scenario 2 – Local Controller, Local Results Database, and Remote Agent


Scenario2.GIF

Local Controller, Local Results Database, and Remote Agent Scenario

Key Characteristics

  • The Visual Studio controller service and the results database run on the same machine.
  • It is not possible to disconnect from the machine running the tests, because controller service and visual studio run in the same machine.
  • Visual Studio can be closed without interrupting the test, because controller service runs in a separate process and will continue to collect performance metrics, and controlling the test execution on the agents.
  • The agent runs in a separate machine.
  • It does allow for sharing the performance testing lab infrastructure. Other Visual Studio machines can connect to the controller and execute load tests on the agents. If Visual Studio workstations share the same agents among themselves, the load test executions coming from different workstation machines will not happen simultaneously; load tests will be executed on first come basis, with remaining tests queued up by the controller. If Visual Studio workstations will use their own separated exclusively assigned agents, the load test executions coming from different workstation machines will happen simultaneously.

When to use

  • Use this option to share machine with Visual Studio and the controller.
  • Use this option when memory and disk are not a limiting factor when hosting the Visual Studio for development, the controller and results database in the same machine.
  • Use this option when moving the controller to different VLANs is not a limiting factor, because Visual Studio is attached to the same machine than the controller.

Key Points

  • It enables you to share development machine with the controller.

More Info

  • Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

Scenario 3 – Remote Controller, Remote Results Database and Remote Agent(s)

Scenario3.GIF

Remote Controller, Remote Results Database and Remote Agent(s)

Key Characteristics

  • The Visual Studio is in a separate machine.
  • The controller, agent and the results database run on the same machine.
  • Optionally other agents can run a separate machine.
  • It is possible to disconnect from the machine running Visual Studio, because Visual Studio and controller service run on separate computers.
  • Visual Studio machine can be closed without interrupting the test, because controller service runs in a separate process and will continue to collect performance metrics, and controlling the test execution on the agents.
  • It does allow for sharing the performance testing lab infrastructure. Other Visual Studio machines can connect to the controller and execute load tests on the agents. If Visual Studio workstations share the same agents among themselves, the load test executions coming from different workstation machines will not happen simultaneously; load tests will be executed on first come basis, with remaining tests queued up by the controller. If Visual Studio workstations will use their own separated exclusively assigned agents, the load test executions coming from different workstation machines will happen simultaneously.

When to use

  • Use this option to allow mobility for machine running Visual Studio.
  • Use this option for reduce memory and disk IO consumption on machine running Visual Studio for development.
  • Use this option to share machine with controller and agent.

Key Points

  • The controller runs on separate machine and is used to manage the agents and collect the test results.
  • One agent runs on the same machine as the controller.
  • Installing an agent on the same computer as the controller can interfere with results collection. If you choose to do this, use the Agent Weighting property to reduce the amount of load generated by the agent installed on the same computer as the controller. It is recommended that you install the agents on their own computer. Anything that is processing at the same time the agent is running affects the accuracy of the test. Agent weighting reduces the impact, but inaccuracies are still introduced. The inaccuracies will be determined by the amount of resources the agent is using, processor and memory, and the sampling rate of data collection during load test
  • It is possible to use Visual Studio for development in a mobile workstation or laptop because Visual Studio and controller are in separate machines.

More Info

  • Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

Scenario 4 – Remote Controller, Remote Results Database and Multiple Remote Agents

Scenario4.GIF

Remote Controller, Remote Results Database and Multiple Remote Agents Scenario

Key Characteristics

  • The Visual Studio is in a separate machine.
  • The controller/results database and agent(s) run on the separate machines.
  • It is possible to disconnect from the machine running Visual Studio because Visual Studio and controller service run on separate computers.
  • Visual Studio machine can be closed without interrupting the test, because controller service runs in a separate process and will continue to collect performance metrics, and controlling the test execution on the agents.
  • It does allow for sharing the performance testing lab infrastructure. Other Visual Studio machines can connect to the controller and execute load tests on the agents. If Visual Studio workstations share the same agents among themselves, the load test executions coming from different workstation machines will not happen simultaneously; load tests will be executed on first come basis, with remaining tests queued up by the controller. If Visual Studio workstations will use their own separated exclusively assigned agents, the load test executions coming from different workstation machines will happen simultaneously.

When to use

  • Use this option to allow mobility for machine running Visual Studio 2005.
  • Use this option for reduce memory and disk IO consumption on machine running Visual Studio for development.
  • Use this option to reduce resource consumption on agent, that could skew performance testing results.
  • Performance data collection on same machine as the controller may cause memory and disk IO consumption. Consider adding memory and/or improving disk subsystem.

Key Points

  • The controller runs on separate machine and is used to manage the agents and collect the test results.
  • Agents run on the separate machines.
  • Using Controller and results database in the same machine may cause memory pressure or disk IO activity, depending on the amount of data collected and sampling rate. Consider increasing memory or using better disk system.

More Info

  • Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

V-Lan Infrastructure Examples

Visual Studio Team System, controller and agents can be deployed in virtual lans in two ways:
  1. The lab infrastructure is not shared. Each performance testing project will have its own agents and controller assigned to only one Visual Studio workstation.
  2. The lab infrastructure is shared. Several performance testing projects will share the agents and controller and they will be assigned to different Visual Studio workstations, or each Visual Studio workstation will share the controller, but will be assigned unique agents.

Lab Infrastructure Not Shared

Following figure is representation of the lab performance infrastructure that is not shared.

Lab1.GIF

Lab Infrastructure Not Shared

Lab Infrastructure Shared

Following figure is representation of the lab infrastructure that is shared.
  • Visual Studio workstations will share the controller and same agents
  • Visual Studio workstation will share the controller, but will use unique assigned agents.

Lab2.GIF

Lab Infrastructure Shared

How Many Machines and Agents Do You Need

To determine the number of workstations and agents that you need, consider the following:
  • Licensing Standpoint – Licenses are per processor. For example, if you run the controller on a dual-processor workstation, then you need two licenses. If you have three agents, each on a single processor workstation, then you will need three licenses (one for each processor).
  • Technological Standpoint – From a technological standpoint, the number of virtual users that you can simulate is limited by workstation resources, mainly processor, memory, disk and network.

Processor and Memory Limits

The amount of load that a specific load agent can generate varies widely from test to test. Most tests are bound by the processor. Processor use is directly proportional to requests per second (RPS). For other load tests, memory is the limiting factor. The RPS you can expect to drive from a load agent depends on the following factors:
  • User load
  • Think time
  • Authentication scheme
  • Size of requests and responses
  • Response time
  • Level of response validation
  • Connection mode
  • Data binding payloads
  • Test Type under load (either Web Test or Unit Test)

Resources Used By the Controller, Agents and SQL Server

When a load test is created, it defaults to critical performance counters and thresholds of all the agents in your test configuration. Monitor those performance counters on the agents during load test execution. Increase the number of agents if threshold violations are reported. The table below reports the summary of resources used by controller, agents and SQL Server for collecting data results.

Agent Controller Application Tier Controller Data Tier Controller Application tier/Data tier
CPU Depending on the test, the CPU is frequently the limiting factor Not heavily used Not heavily used Not heavily used
Disk Not heavily used Not heavily used 10 GB space required for 24 hours of test data 10 GB space required for 24 hours of test data
Memory Depending on the test, memory will be the limiting factor Not heavily used Heavily used by SQL Server Heavily used by SQL Server


Connection Pool Mode versus Connection Per User

Connection mode of execution is a load test setting, configurable during load test design. Agents can be bound by memory on tests that use the Connection Per User connection mode. Two connection modes can be configured in the load test run settings:
  • ConnectionPerUser
  • ConnectionPool

ConnectionPerUser

In Connection Per User mode, each user has a connection that consists of two actual connections open to the server. The ConnectionPerUser model most closely simulates the behavior of a real browser: each virtual user running a web test uses one or two connections to the web server that are dedicated to that virtual user. The first connection is established when the first request in the web test is issued. A second connection may be used when a page contains more than one dependent request; these requests may be issued in parallel using the two connections. These connections are re-used for subsequent requests within the web test, and are closed when the web test completes. The drawback of the ConnectionPerUser model is that the number of connections held open on the agent machine may be high (as high as 2 times the user load), and the resources required to support this high connection count may limit the user load that can be driven from a single load test agent.

ConnectionPool

In Connection Pool mode the connections are pooled but each user still uses two connections when active. In this mode all virtual users are multiplexed over the connection pool. The ConnectionPool model conserves the resources on the load test agent by sharing connections to the web server among multiple virtual web test users. When the Connection Model is ConnectionPool, the Connection Pool Size specifies the maximum number of connections to make between the load test agent and the web server. If the user load is larger than the connection pool size, then web tests executing on behalf of different virtual users will share a connection. This may mean that one web test may need to wait before issuing a request when another web test is using the connection, the average time that a web test waits before submitting a request is tracked by the load test performance counter “Avg. Connection Wait Time”. This number should ideally be less than the average response time for a page. If it is not, then the connection pool size is probably too small.

SQL Sizing Considerations

When deploying SQL database to act as a load test repository, there are considerations regarding the sizing, directly related with the number of samples for the performance testing metrics.
  • By default, SQL Express is installed on the controller and is used by the controller as the default SQL store for load test results. The SQL Express database is license-limited to store 10 GB of data, which is around 24 hours of load test data for a typical load test. The space that is required for load test data varies greatly depending on the test.
  • During a load test, samples are collected for each counter instance on each computer. Therefore, the amount of space that is required in the database depends on the following factors: the number of counters collected, the number of computers involved in the test, and the number of samples taken, as controlled by the sample rate.
  • If appropriate, consider using a separate database to store the load test data. The database can be stored on either the controller computer or on a different computer. To change the data store, submit the SQL commands contained in the .sql file to the SQL server instance you want to use for the load test results store. These are two ways to do this. One way is to use the command sqlcmd from a command prompt and specify the options needed to connect to the desired database. Use the –i option to specify the path to loadtestresultsrepository.sql. Another way is to use one of the GUI interfaces to SQL, such as query analyzer, and open the .sql file and submit the connects

Multi-processor Considerations

If you are generating load from a multi-processor machine, consider enabling server GC. By default the process will use workstation GC.

To enable your application to use server GC, you need to modify either the VSTestHost.exe.config or the QTAgent.exe.config. If you are not using a Controller and Agent setup, then you need to modify the VSTesthost.exe.config. If you are using a controller and agent, then modify the QTAgent.exe.config for each agent machine.

Open the correct file. The locations are
  • VSTestHost.exe.config - C:\Program Files\Microsoft Visual Studio 8\Common7\IDE
  • QTAgent.exe.config - C:\Program Files\Microsoft Visual Studio 2005 Team Test Load Agent\LoadTest

To enable gcserver you need to add the following line in the runtime section:

<gcServer enabled="true" />

Below is the sample config file:
<?xml version ="1.0"?>
<configuration>
    <runtime>
        <gcServer enabled="true" />
        <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
            <probing privatePath="PrivateAssemblies;PublicAssemblies"/>
        </assemblyBinding>
    </runtime>
</configuration>

Security Considerations

If the computers in a rig are running in Work Group mode, that is, they are not in a domain, or they are running in different non-trusted domains, create local computer accounts on the controller. The accounts should have a matching password for each user who will access the controller, including the Agent service users.

Below is an example of machines: Controller1, Agent1 and Agent2:
Computer Name User Account Password
Controller1 ControllerService ControllerServicePassword
Agent1 ControllerService ControllerServicePassword
Agent2 ControllerService ControllerServicePassword
Controller1 AgentService AgentServicePassword
Agent1 AgentService AgentServicePassword
Agent2 AgentService AgentServicePassword

Additional Resources

Last edited Sep 12, 2007 at 11:14 PM by prashantbansode, version 2

Comments

No comments yet.