pnp.GIF

How To: Account for User Abandonment

Scott Barber,

Applies To

  • Performance Test Design
  • Workload Modeling for Performance Testing
  • Performance Testing

Summary

When you adopt concurrent, simultaneous, or overlapping users as a performance testing input parameter but fail to account for user abandonment, you run the risk of creating loads that are highly unrealistic and improbable. It is important to simulate user abandonment as realistically as possible. If you don’t, you may be creating a type of load that will never occur in a real-world business environment, while creating bottlenecks that might never happen with actual users. At the same time, you would be ignoring one of the most important load testing results: the number of users that might abandon your Web site due to poor performance. In other words, your test might be useless. This How To addresses modeling and implementing user abandonment in your performance tests.

Contents

  • Objectives
  • Overview
  • Summary of Steps
  • Step 1. Determine if You Need to Account for User Abandonment
  • Step 2. Determine Abandonment Thresholds
  • Step 3. Interpreting Abandonment Statistics
  • Consequences of Not Accounting for User Abandonment
  • Consequences of Improperly Accounting for User Abandonment
  • Resources

Objectives

  • Learn how to determine if your performance test needs to account for user abandonment.
  • Learn how to determine user abandonment parameters.
  • Learn how to report abandonment statistics.

Overview

It is well documented that people who use the Internet often abandon the Web sites they visit, and that they do so for a variety of reasons. One major reason is that users get tired of waiting for a Web page to load and exit the site before completing the task they had in mind. When designing your performance tests, it is important to ask whether the tests account for this type of behavior. If you are using the default settings on your load-generation tool, the tool probably won’t allow for user abandonment. In fact, if you don’t manually write a custom abandonment routine for your load-generation tool, the tool either won’t account for abandonment at all or will account for it in a way that isn’t useful for testing purposes.

One of the advantages of most Web sites is that if the load gets too big for the system/application to handle, the site slows down, causing visitors to abandon it, thus decreasing the load until the system speeds back up to acceptable rates. Conversely, imagine the side effects to your business if a site slowed down and stayed slow until someone fixed the server. Fortunately, abandonment can offset any negative impacts in most cases. Assuming that the site performs well enough under a “reasonable” load, performance is generally self-regulating, but at the cost of losing some customers/users. One reason to correctly account for user abandonment when designing your test is to determine how many users will likely abandon a slow-running site. Another reason is to determine the actual volume of transactions your application can maintain before you start losing users. Yet another reason to account for user abandonment is to avoid simulating, and subsequently resolving, bottlenecks that might not even be possible in a real-world scenario.

Steps

  • Step 1. Determine if You Need to Account for User Abandonment
  • Step 2. Determine Abandonment Thresholds
  • Step 3. Interpreting Abandonment Statistics

Step 1. Determine if You Need to Account for User Abandonment

In the vast majority of cases, accounting for user abandonment is critical to conducting a performance test that provides accurate results as the response time starts to increase. However, there are some cases where not accounting for abandonment is an accurate representation of a real-world situation. One such case would be a Web-based application that has been exclusively created for an audience that has no choice but to wait for it to load; for example, a company that has a Web-based time-management system and a policy that all employees must enter the hours they worked in the past week between noon and 5 P.M. every Friday (with a few trivial exceptions). Because roughly 3,500 people accessed the time-management system during a five-hour period in addition to the normal traffic of users performing other tasks on the Web site, the system got very slow. Under other circumstances, users might abandon the site and try later or go somewhere else, but in this case, the users didn’t have a choice, so accounting for abandonment as part of the performance testing would make the tests less representative of a real-life situation. In this case it was more valuable to determine how long the employees would have to wait under different usage scenarios.

To determine whether accounting for user abandonment is appropriate for your application, consider the following questions:
  • Can your users accomplish the same task via another Web site or an alternate method?
  • Can your users accomplish the task via your site at another time?
  • Are your users willing to accept the consequences of giving up on accomplishing the task?

If you can answer yes to any one of these questions, you should account for user abandonment when conducting performance testing.

Step 2. Determine Abandonment Thresholds

Presuming that you have determined that your site is subject to user abandonment, you now need to determine, for each page or class of page, how long users will wait for a page before abandoning it. To do this, you will need to establish the following three parameters for each type of page on the site:
  • Minimum abandonment time
  • Maximum abandonment time
  • Abandonment distribution

These parameters are determined by either parsing Web server logs, researching sites with functionality similar to the one you are testing, or conducting your own informal research by observing representative users. It is important to note that abandonment parameters that are off by a few seconds one way or another are not going to have a significant impact on your results. These parameters need to be reasonable estimations, not precise measurements. It is also important to note that you can obtain valuable information by executing the same test with different parameters to determine ranges for abandonment, with the understanding that user expectations and loyalty are bound to change over the life of your application.

Minimum Abandonment Time

The minimum abandonment time is your estimate of the amount of time that you anticipate all users will wait for a page to load. If you do not have Web server logs from which to extract this information, you can estimate the time as falling somewhere between the performance goal for a given page and 1.5 times the goal for that page, depending on how much motivation a user has to complete his or her task, right now and specifically on your Web site. Obviously, the less motivated a user is, the shorter the minimum abandonment time should be.

Maximum Abandonment Time

No matter how patient a user may be, Web sites sometimes simply fail to respond due to circumstances such as the following:
  • Unanticipated application or infrastructure performance bottlenecks
  • Temporary (or permanent) Internet connectivity interruptions on either end
  • Browser “page cannot be displayed” time-out errors
  • Secure session time-out (requiring users to either abandon or start over)

While some of these situations don’t technically count as user abandonment (because the Web site abandoned the user and not the other way around), they do provide the upper boundary for how long users could potentially wait before ending their Web session.

The maximum abandonment time parameter is the most scientific of the parameters. The maximum abandonment time is simply the time after which either your browser stops waiting for a response (frequently, 120 seconds); the secure session expires and the user has to reestablish the session, typically by logging in again (most session managers have a default setting of 20 minutes); or some other number determined by parsing Web server logs. If you don’t already know these numbers, you can simply ask the architect/developer responsible for the presentation tier (or Web server) to provide you with the information.

Abandonment Distribution

The reasons that a user may abandon a Web site are extremely variable, not only between different users but also between visits by the same user. A user’s tolerance for waiting might change dramatically from session to session. While this makes it difficult to predict when an individual might abandon a session, it also means that abandonment is likely to follow standard distribution models such as normal or linear distributions when taken over a large enough sample. Your next task is to determine, for each page type, what distribution model best represents the likely abandonment behavior of users.

In the vast majority of cases, either a normal (bell curve) or a uniform (linear) distribution will be sufficiently accurate. If you have ever taken a statistics or psychology course, you know that almost everything that real human beings do (over a large enough sample) can be represented by a bell curve. You may also recall that the key to an accurate bell curve is the standard deviation. Virtually all relevant research indicates two things about standard deviations when it comes to Web usage: (1) they are exceptionally large (statistically) in comparison to the range of values, and (2) they are almost impossible for non-mathematicians to calculate accurately. What this means is that in most cases, if you were to graph abandonment rates for a production site, you would very likely end up with a very flat bell curve that, in effect, approaches a linear distribution. Statistics aside, if you do not have a strong reason to do otherwise, choose either a normal or a uniform distribution based on your best judgment. If you really don’t know which distribution is best in your specific situation, use a linear distribution because it is almost always close enough and can always be modified later as you obtain more information.
Abandonment Linear Dist.GIF
Abandonment Normal Dist.GIF
Programming or configuring your load generation tool to handle user abandonment with these three parameters is unique to every tool and out of scope for this How To

Step 3. Interpreting Abandonment Statistics

As with any metric or statistic, abandonment statistics can be grossly misinterpreted. Remember the following when reviewing your abandonment data:
  • Check your abandonment rate before you evaluate your response times. If your abandonment rate for a particular page is less than about 5 percent, look for and handle outliers. If your abandonment rate for a particular page is more than about 5 percent, you probably have a problem worth researching further on that page. {see How To – Handle Outliers for more information}
  • Check your abandonment rate before drawing conclusions about the load. Remember that every user who abandons a session is one less user applying load. Your response time statistics may look good, but if you have 25-percent abandonment, your load may be 25-percent lighter than you were expecting.
  • If your abandonment rate is more than about 20 percent, consider disabling the abandonment function and re-executing the test to help gain information about what is causing the problem.

Consequences of Not Accounting for User Abandonment

If you do not account for abandonment at all when you should, the performance script will wait indefinitely to receive the page or object it requested. When it eventually receives that object, it will move on to the next object as though nothing ever happened. If the object is never received, the script never ends. This adds no value to the performance-testing effort, unless there is a need to demonstrate to some stakeholder that, for example, “Under the specified conditions, the average page-load time was roughly 2.5 hours.” Unfortunately, points like this sometimes can only be made convincingly by generating meaningless numbers, which does not constitute a performance test and does not bring you any closer to delivering a quality, well-performing application.

Consequences of Improperly Accounting for User Abandonment

Improperly accounting for user abandonment is what most load-generation tools do by default. This is because most tools assume that all users abandon at a predetermined time (frequently 240 seconds) yet still continue on, requesting the following page as though nothing happened. Of course, with most tools, you can change settings to improve the situation by changing the time limit or having the virtual user actually exit, but that still is not context-specific by page. Regardless of your tool’s default settings, or how difficult or easy it may be to change those defaults, improper accounting for abandonment can bring results that are even more misleading than if abandonment were not accounted for at all. Consider the following examples and their side-effects.
  • At 240 seconds, stop trying to get this object, log a message, and move on to the next object. (Common Load Generation Tool default)
  • This scenario may cause unexpected errors in situations where subsequent objects need the former objects to have loaded successfully, because the tool will now be “forcing” the application to serve pages that a real user couldn’t reach.
  • This scenario will also skew page and object load times, because you don’t actually know how long the object would have taken to load, yet the 240 seconds figure is calculated as if the download were successful.
  • Subsequent errors caused by the skipped object not downloading completely may mask the actual cause of the errors, making it appear as if the script is flawed.
  • The additional load applied after the time-out (that a real user would not be applying) may skew your results.
  • Only log when people would have abandoned for analysis but don’t actually exit the virtual user.

While this may be useful during early testing, it is a very inaccurate representation of the actual abandonment rate for a variety of reasons, including:
  • Once a user encounters one page that is slow enough to abandon, the user will usually encounter more if they haven’t already exited the site, resulting in statistics showing an artificially high abandonment rate.
  • Allowing a user who would have abandoned the site to continue keeps the current load on the system rather than reducing the total load for others, thereby causing other users to experience response times in the abandonment range, once again resulting in statistics showing an artificially high abandonment rate.
  • If the abandonment level response time was actually due to an error, subsequent page requests may also produce errors, making the actual problem (one slow page) much more difficult to detect.

Please note that the above examples represent cases where actual abandonment rates were grossly misrepresented. Inaccurately modeling the abandonment range by a few seconds is not going to cause problems on this scale. Your abandonment model needs to be reasonable, not perfect.

Resources

Last edited Mar 16, 2007 at 9:37 PM by prashantbansode, version 4

Comments

No comments yet.