• Jobs
  • About
  • Performance testing using Selenium January 26, 2011

    I consider myself a test automation guy but I do have some experience doing performance testing. With this background of mine I have been frequently asked if we could do some performance testing with selenium. In fact it has come up so many times that I thought it would be interesting to share them here and possibly get some of you to share your experiences as well.

    By the way, even though I mention Selenium here, this applies to any functional UI testing tool.

    Performance testing goals using Selenium

    Load testing

    1. First off most people realize generating several concurrent virtual users requires a lot of hardware resources when using a real browser.
    2. Secondly, the load on the System Under Test cannot be random. It needs to be very precise with respect to when the concurrent sessions must start and how much think time should exist between transactions on each of the threads. This is very hard, if not impossible, to control.

    Browser rendering times

    So, let us say the goal of the tests using selenium is not to do load testing but simply measuring browser rendering times of a single thread. I have actually tried this and the problems here are:

    1. The variance in the page response times in Selenium has been a lot more than a performance testing tool under the same circumstances.
    2. If you are trying to measure the total rendering time of pages that have a lot of AJAX etc, you may have to add several wait for conditions to ensure all the processing is complete. This might be tricky to do and also with new releases, the maintenance cost of such scripts can be high.

    Performance testing measurements using Selenium

    Incorrectly written code that does the measurement

    Another set of problems I have seen is that the code that does the measurement is written in a manner that either incorrectly detects the end of the operation, or the sample rate of the measurements is large. Look at this simple example:

    // some action
    timer.start();
    while (!browser.isElementPresent(locator)) {
        try {
            Thread.sleep(100);
        } catch (InterruptedException e) { /* Do nothing */ }
    }
    timer.stop();
    

    In this case, isElementPresent method can itself take a long time to evaluate, which adds to the total time of the, say ajax, operation to complete. Also, the sample rate of 100 milliseconds sample rate could be accounting for a larger portion of the total percentage of the operation.

    Data Validation

    Assuming you collected page response times after running your Selenium tests, how do you validate the data is accurate? With performance testing tools, you usually average out several execution results and validate if the data from that run is usable by applying Little’s Law against it. to that you need to first measure

    • Transactions per second
    • Hits per second
    • Response time
    • Number of virtual users being simulated.

    and check if it applies the law:
    Actual Number of Users in the System = (Response time + Think time) * Transactions per second

    You will find that achieving this with Selenium will be very hard.

    Bottleneck detection

    Another factor many people forget is that all the data you get is useful to someone who wants to improve the performance only if you can pinpoint where the bottleneck is. For those of us who are test automation guys and have little to no performance background, let me first explain bottlenecks a little.

    Simply speaking a bottleneck is that place in your entire system where performance is being limited to. There may be several inefficiencies in the system but there is always only one bottleneck in the system at a particular load. So, even if you improve performance elsewhere the users of the system will not see an improvement in the response time of that operation. For example, if there are multiple user sessions competing for a single resource (mutex), or even in a single session case, multiple components on the page competing for a single resource, then no matter how much you tune the database, the performance wont improve.

    With that in mind, the next question is how do we find these bottlenecks. There are two categories of bottlenecks – Software bottlenecks and Hardware bottlenecks. RAM, CPU and IO (includes network, disks etc) fall under the hardware bottleneck category and App, Database and OS are the software bottlenecks. If you have the correct monitors for each of these, you should have the data needed to detect the bottleneck. If you use a performance testing tool, most likely you will/ can purchase, the monitors that work together. This makes aligning all the data of the monitors and analyzing it a lot easier. If you wanted to setup your own system using functional testing tools, this can get really complicated.

    Everything looks like a nail

    In several discussion/ interviews I have come across people who have tried it too and it all turned out to fail for the same reasons. On reflection, it almost always turns out that the reason for using a functional testing tool for performance testing or performance testing tool for test automation is because there was a high amount of investment in a tool and so it seemed to make sense to use it for other purposes. A classic case of “To a man with a hammer, everything looks like a nail”.

    The solution is ofcourse to add more tools and the appropriate tools in your tool belt. The most widely used tool for performance testing is still Load Runner from HP. This is a pretty expensive software though. If you are just interested in measuring browser performance, I like a cheaper tool called NeoLoad by Neotys. It is a simple to use GUI based tool that provides most of the control you need to create the load scenarios you need.

    I hope this helps some of you, and for those who have a different take, I would love to hear your views on this subject as well.

    Posted by Rahul Poonekar in : Selenium

    3 responses to “Performance testing using Selenium”

    1. PT&AT says:

      Dude.. I am a Perf test engineer , mostly worked on loadrunner. Though i do not vouch for loadrunner the fact is , there are NO perf test tools that simulate the application perf testing as loadrunner does.

      Currently im working for a product company…. there seems to be this buzz about open source (Which i whole hearted`ly agree )!!!!!!!!
      Jus like selenium… Jmeter and so on…. but does it help!!!!!!!
      Well not completely…. Not anywhere close to what Mercury(HP) Loadrunner is… It has defined How perf testing should be carried out.

      Good Article though!!!!! Something i was looking out for.:)

      Cheers.

    2. AIR says:

      Hi guys.. how about the performance testing tool WAPT. Is it comparable with Load Runner? I heared, it is cheaper compare to LR.

      Thanks

    3. Miloni says:

      Can u plz give me URL for downloading Loadrunner.

    Leave a Reply

    Your email address will not be published. Required fields are marked *