How To: Step Through Creating a Load Test in Visual Studio Team System
J.D. Meier, Prashant Bansode, Carlos Farre, Larry Brader, Scott Barber
- Microsoft® Visual Studio® 2005
This How To topic shows you how to create a load test using Microsoft Visual Studio Team System. To create a load test in Visual Studio Team System, you first create a Test Project. You then create individual Web tests that simulate user scenarios. Finally, you create a load test, to which you will add your Web tests. While creating your load test, you can set many runtime properties to generate the desired load simulation; for example, you can specify the load pattern, browser, and network types and add the performance counters to be monitored.
- Before You Begin
- Summary of Steps
- Step 1 – Create a Test Project
- Step 2 – Create a Web Test
- Step 3 – Record Your Web Test
- Step 4 – Replay Your Web Test
- Step 5 – Add Your Load Test
- Step 6 – Specify Your Scenario Settings
- Step 7 – Specify Your Load Pattern Settings
- Step 8 – Add Your Web Test to Your Load Test
- Step 9 – Specify Your Browser Mix
- Step 10 – Add Your Network Mix
- Step 11 – Specify Your Computers and Counter Settings
- Step 12 – Specify Your Run Settings
- Step 13 – Run Your Load Test
- Learn the end-to-end steps for creating a load test using Visual Studio 2005.
- Learn how to configure the various load test runtime properties.
You use load testing to verify application behavior under normal and peak load conditions. The load-testing process allows you to verify that your application can meet your desired performance objectives. Load testing enables you to measure response times, throughput rates, and resource utilization levels, as well as identify your application's breaking point.
This How To walks you through a simplified example of creating a load test using Visual Studio 2005. The purpose of this How To is to familiarize you with the end-to-end steps involved in creating a load test using Visual Studio 2005. Once you are familiar with these steps, you can focus on specific aspects of load testing. For example, you can focus on modeling the user experience to match your specific context and produce better test results.
Before You Begin
You will need an application to run your load test against. In this example scenario, you will use the Small Business Starter Kit. You can download and install the starter kit from the following location:
: The small business starter kit will install a template that can be access in the following steps:
- Open a new instance of Visual Studio Team System, go to File menu, and select New.
- Select Web Site…, In the New Web Site dialog box select the Small Business Starter Kit below my Templates.
- In the Location specify File System (you can choose HTTP as well, but it adds dependency to IIS). Make sure the location path is set correctly.
- Click OK button.
- This will open the Web Site solution in your Visual Studio instance.
- You can hit Ctrl+F5 and the web site will be running under ASP.NET Development Server the icon will be displayed in the system tray
- Your test site is loaded and running and the address is similar to following http://localhost:1380/SmallBusiness/
For this How To, you will use the Small Business Starter Kit. Assuming that the Web site in the scenario is a Small Business Web site, it will be an Internet-facing Web site.
From an application point of view, the critical performance area is the ability of site users to access the Items page and then navigate to a specific item. Therefore you need to load-test a scenario in which a user browses a catalog on the Web site.
Summary of Steps
- Step 1. Create a Test Project
- Step 2. Add a Web Test
- Step 3. Record Your Web Test
- Step 4. Replay Your Web Test
- Step 5. Add Your Load Test
- Step 6. Specify Your Scenario Settings
- Step 7. Specify Your Load Pattern Settings
- Step 8. Add Your Web Test to Your Load Test
- Step 9. Specify Your Browser Mix
- Step 10. Add Your Network Mix
- Step 11. Specify Your Computers and Counter Settings
- Step 12. Specify Your Run Settings
- Step 13. Run Your Load Test
Step 1. Create a Test Project
In the initial step, you will create a Test Project named TestProject1. By default, this Test Project will include an empty unit test. You will add a Web test and a load test to this Test Project in later steps. Note
– The language you choose for your Test Project affects the language used when coded Web tests are generated.
Creating a Test Project
- Open a Microsoft Visual Studio 2005 instance.
- Click the File menu, click New, and then click Project.
- In the New Project dialog box that opens, in the left hand pane, expand the Visual C# node and then select Test.
- In the right pane, select the Test Project option, name your test project TestProject1, and then click OK.
Step 2. Add a Web Test
In this step, you will create a Web test to be used as part of your load test. The Web test simulates how an end user will interact with the Web application.
Creating a Web Test
- In Solution Explorer, right-click TestProject1, select Add, and then select Web Test.
The Web Test Recorder
opens inside a new instance of Internet Explorer.
Step 3. Record Your Web Test
For this How To, you will create a Web test by recording Hypertext Transfer Protocol (HTTP) requests using the Web Test Recorder in a browser session. Note that you can also build Web tests manually using the Web Test Editor
Also for this How To, you will record a Web test for a scenario in which the user browses a catalog, using the Small Business Starter Kit as described in the Scenario section above.
Recording Your Web Test
- In the Address bar, replace about:blank with the URL address of the Small Business Starter Kit web site for which you want to run the Load Test(for example, http://localhost:1380/SmallBusiness/) and press ENTER.
- When you press '''Enter''', the page is processed and rendered in the browser window. The actions are recorded for subsequent playback when you run the test.
- Go through various steps for accessing item detail as follows.
- Click the Items link available on the default page.
- Click the Amet hyperlink (First category level).
- Click the Sed eget Magna hyperlink (Second category level).
- Click the Proin Vitae hyperlink taking user to the item details page. .
- Click Stop on the Web Test Recorder to stop recording and return to the project. You now have a Web test that contains a list of recorded actions in the Web Test Editor as a list of URLs.
Coded Web Tests
For scenarios where you need to add looping and branching constructs, or dynamically change the number of requests in the test, or dynamically generate the set of URLs that the test requests, you may need to create coded Web tests. A coded Web test is a Microsoft .NET class that generates a sequence of WebTestRequests. A coded Web test is typically created by converting an existing, recorded Web test into a coded Web test.Note
: Once a recorded Web test is converted to a coded Web test and is edited, you cannot convert it back to a declarative test.
For more information see "How to: Create a Coded Web Test" at http://msdn2.microsoft.com/en-us/library/ms182550(VS.80).aspx
Step 4. Replay Your Web Test
It is important to replay your Web test to make sure it works. Always fix any failure that might occur before you add the Web test to the load test, because it is simpler to debug errors at this point rather than debugging from a load test, which can be a difficult process.
Replaying Your Web Test
- In the main window of Web test click the Run Test button in the left hand corner. Alternatively, you can right click the Web test and choose the option Run Web Test.
- The Web test will be run for a single user, and result displayed in the main window as shown in the figure.
- Analyze and make sure there are no errors in any of the requests.
The purpose of replaying your Web test is not merely to ascertain if the Web Test passes or fails. You might want to check the following before adding the Web test to a load test:
- Verify that the response time for each request meets the requirements for a single user; for example, 5 seconds or less for intranet applications and 8 seconds or less for Internet-based applications. Add your load test step only if the requirements are met If they are not, investigate the reason for the delayed response time and fix the issue as appropriate.
- Verify that the file sizes of top requests and dependent requests meet the stated requirements. If they do, go ahead with adding your load test step. If not, try to fix the underlying issue.
Step 5. Add Your Load Test
In this step, you add the Load Test to the TestProject1 project. In a later step you will add the Web test to the load test in order to simulate the load while executing the Web test. You will set various properties of your load test in subsequent steps.
Adding Your Load Test
- In Solution Explorer, right-click the TestProject1 project node, select Add, and then click Load Test. The New Load Test Wizard starts
- The Welcome page of the New Load Test Wizard (the the first page), Click Next.
Step 6. Specify your scenario settings
In this step, you specify the details for the user scenario, for which we are developing the Load Test.
Specifying Your Scenario Settings
- Specify name of your scenario. We are performing Load Test for browsing catalog scenario, so the scenario name could be "Browsing Catalog."
- Set the Think Time Profile Think to Use normal distribution centered on recorded think times. Think times represent the time that a user would examine a Web page before going on to the next page. For this exercise, the think time will be the time the user spends reading the catalog, choosing a category, and narrowing down his or her search to a specific item. This scenario will use the recorded times from step 3, Record Your Web Test.
The choice of Think Time Profiles depends upon their intended usage.
- When you choose to use recorded think times, the think times are used exactly as they were recorded in the Web test. Because a load test simulates multiple users, using the same think time could create an unnatural load pattern of synchronized virtual users.
- When you choose normal distribution, think times are used but vary based on a normal curve. This approach provides a more realistic simulation of virtual users by slightly varying the think time between requests.
- If you choose not to use think times, they are ignored. This approach does not provide realistic user interaction with the application and should be used only when you are doing performance tuning in a situation where you are concerned about throughput rather then actual user experience.
For more information on think times, please see "About Think Times" at http://msdn2.microsoft.com/en-us/library/ms184790(VS.80).aspx
Step 7. Specify your load pattern settings
In this step, you will set the load pattern—in this case, the Step Load pattern is used. This pattern increases the user load until you reach the Maximum user count.
Using a Step Load
- Select the Step Load radio button.
- Set Start user count to the value that you estimate your minimum user count will be. For this scenario, set the number to 10.
- Step duration is the time interval, in seconds, between opening the minimum number of user connections and opening additional user connections. For this scenario, set the number to 10.
- Step user count is the number of additional user connections opened during each step. For this scenario, set the number to 10.
- Maximum user count is the maximum number of users for which you are load testing. For this scenario, set the number to 200.
– The total test duration specified in Step 11 should be more than the total step time to reach the maximum user count. If not the, load test will stop after the elapsed time and will not reach the Maximum user count
You can use the Step Load pattern to increase the load until the server reaches a point where performance diminishes significantly. As the load increases, the server will be able to keep up until it runs out of resources. Using the Step Load pattern is a good way to determine the number of users at which diminished performance occurs. With the Step Load pattern, you also have to monitor agent resources closely to make sure that the agents can generate the desired load.
When load testing using the Step Load pattern, you need to allow the system being tested to adjust to the new load step before increasing the load in the next step. Hence the step duration is critical and should be adjusted optimally.
Information regarding the loading pattern and specific inputs can be obtained from the Service Level Agreement (SLA) for your application or the performance objectives set.
Step 8. Add Your Web Test to Your Load Test
In this step, you will add a Web test to your load test. You will also set the work distribution for each Web test.
Adding Your Web Test to Your Load Test
- In the Add tests to a load test scenario and edit test mix screen, click Add… button.
- In the Add Tests dialog box that opens, in the Available tests list box, select the Web test created in previous steps and then click the right arrow to move it to the Selected tests list box
- In the Add Tests dialog box, click OK.
- In the Add one or more tests to the mix and specify a distribution dialog box, you can use the sliders to adjust the test distribution. In the current scenario, the distribution will be 100% because only one Web test is being run.
- Click Next.
The load test will use the Web test to simulate the actual user interaction with the Web application and would make concurrent HTTP requests to simulate a real-time environment.
Ideally, your application will have more than one workflow, in which case you might consider creating a separate Web test for each workflow. You would add all of these Web tests to your load test. Depending on the work distribution for each workflow, you can adjust the test distribution in terms of percentage by using the sliders. The load test would then run the various Web tests according to the ratio of test distribution.
Step 9. Specify your browser mix
In this step, you will specify your browser mix by determining the types of browsers being used and the distribution ratio for each type of browser.
Specifying Your Browser Mix
- In the Browser Type drop-down list, select the Internet Explorer 6.0 browser to add to the mix.
- Because the example Web site is an Internet-facing site, you should add Netscape 6.0 to the mix as well. Click the Add button and then select Netscape 6.0 in the drop-down list.
- Adjust the distribution of Internet Explorer users and Netscape users to be 88-percent Internet Explorer and 12-percent Netscape (based on industry trends).
- Click Next.
The browser mix is a combination of two factors: browser types and their distribution. The types of browsers being supported can be determined from either the requirement specifications or the SLA.
For a new Web application, browser distribution is based on specified requirements or industry trends and is set in terms of percentage.
For existing Web applications, browser distribution is based on information extracted from Microsoft Internet Information Services (IIS) logs.
Step 10. Add your network mix
In this step, you will add the network mix by specifying the types of network on which the application is accessed and the distribution ratio for each type of network.
Specifying Your Network Mix
- Because the example Web site is Internet-facing and users in various locations are expected to access it, your network mix will be a combination of digital subscriber lines (DSL) and dial-up connections.
- From the Network Type drop down list, select the Cable / DSL 1.5 Mbps connection type.
- Click Add button and then select the Dial-up 56K.
- Set the distribution at 50% for each type of connection.
- Click Next.
A network mix, is a combination of two factors: the network types and their distribution. The types of network being supported can be determined from either the requirement specifications or the SLA. Depending on specific requirements or industry trends, the network mix distribution can be determined and set in terms of percentage.
Step 11. Specify your computers and counter sets to monitor
In this step, you will specify the performance counters to be used for capturing load test results, in order to gauge the performance of the application.
Specifying Your Computers and Counter Sets
Because the Web site in this scenario is deployed on the local machine, you will use the default counter set.
The Performance counters are organized according to the technologies that are useful to monitor during a load test. Counter sets are part of the load test and apply to all of its scenarios. The counter sets include Load Test, IIS, ASP.NET, and SQL.
When you create a load test by using the Load Test Wizard, you add an initial set of counters that offer you a set of important, predefined counter sets for your load test. These counter sets are useful when you are analyzing a server running IIS, ASP.NET, or Microsoft SQL Server™.
If the Web Application and database is hosted on remote machine, click the Add Computer button and enter the names of the computers. Once you have added the computers, you will see nodes that you can select below the new entry; for example, ADO.NET, IIS, SQL and others. Select the check boxes next to the nodes you want to select. The new counters appear in the Preview selections pane. Note
– You need to have user permissions to run performance monitors for remote servers.
Step 12. Specify Your Run Settings
In this step, you will specify a set of properties that determine how the load test runs. The run settings determine the length of the test, warm-up duration, maximum number of error details reported, sampling rate, validation level, and so on.
By default, the Warm-up duration is 0, the Run duration is 10 minutes, the Sampling rate is 5 seconds, the Maximum error details number is 100, and the Validation level is Low.
Specifying Your Settings
- On the Run Settings page choose your initial settings.
- Specify the '''Warm-up duration''' in hh:mm:ss (hours/minutes/seconds) format. This is the period between the beginning of the test and the time when recording of the data samples begins.
- Specify Run duration in hh:mm:ss format. This is the actual length of the test.
- Specify Sampling rate in seconds. This is the interval at which to capture performance counter values
- In the Description edit box, specify the description of the Run Settings.
- Specify the Maximum error details.This is the maximum number of request and response details of failed requests that are stored. This number is important because detailed error results can consume a large amount of database storage. If you do not want to record error details, use a value of 0.
- Specify the Validation level. This defines the highest level of validation rule that will run in a load test. Validation rules are associated with Web test requests. Each validation rule has an associated validation level: high, medium, or low. This load test run setting will specify which validation rules will run while the Web test is run in the load test. For example, if this run setting is set to Medium, all validation rules marked as medium or low will be run.
- Click Finish. Your Load test is opened in the Load Test Editor.
The default settings cause the load test to run for duration of 10 minutes while recording the sample data every 5 seconds from the start of the test. If there are any failed requests, maximum 100 failed requests and their response details are stored.
At the default Warm-up duration of 0, test data will be captured from the start of the test. This might be useful when there is no startup overhead for the application, or when the startup overhead might be experienced by end users as well, because it helps to capture realistic test data. In contrast, for ASP.NET applications, there is always a first-time just-in-time compiler (JIT) compilation overhead associated with the application, which users might never experience. In such scenarios it makes sense to add appropriate Warm-up time, so that you do not capture such ambiguous data during your testing.
Validation rules help verify that a Web application is working correctly by validating the existence of text, tags, or attributes on the page returned by a Web request. The default setting of Low executes the fewest rules and can be used for heavy load test and stress runs. Note
– The Run duration specified should be more than the time required to reach the Maximum user count specified in the Load Pattern (step 4), to ensure that the test does not stop before the reaching the maximum user count.
Step 13. Run Your Load Test
In this step, you will run the Load test to observe how the Web application responds to the Web test under the load simulation.
Running Your Load Test
- In the Load Test Editor, right click on the Load Test and select '''Run Test''' option.
- In the main window, on the View menu, click Full Screen to maximize the viewable area.
- Once the Load test is complete, the following message appears: "Load Test 'LoadTest1' is complete. The test data currently displayed only represents a portion of the available results. Would you like to view the detailed results from the load test result store?" Click Yes
The threshold violation, errors, and warnings are displayed using different colors and icons. Counters violating thresholds can be dragged onto the graph to generate the violation graph. This can be done while the test is running.
What to Do Next
You can analyze the data from your load test runs to locate bottlenecks, identify errors, and measure improvements in your application as follows:
- The Summary pane at the left of the results window indicates the total number of tests (user sessions executed) along with the failure count. Check here to see if any of the requests has failed.
- If any errors occurred, an errors hyperlink appears on the load test status bar and specifies the number of errors that occurred. To display the errors table, click the hyperlink.
For more information, see "Analyzing Load Test Errors" at http://msdn2.microsoft.com/en-us/library/ms404697(VS.80).aspx
- If any violations occurred, a threshold violations hyperlink appears on the load test status bar. Click the hyperlink to display the threshold violations table that specifies the number of violations that occurred.
For more information, see "Analyzing Threshold Rule Violations" at http://msdn2.microsoft.com/en-us/library/ms404694(VS.80).aspx