• Jobs
  • About
  • Domain Specific Languages (DSL) vs. General Purpose Languages for Functional Test Automation March 15, 2010

    What are these and how do they relate to functional test automation?

    General Purpose Language

    All procedural languages like perl, python, ruby etc and Object-Oriented Languages, like Java, C++, .NET etc fall in this category fall under the category of General Purpose languages.

    Procedural Languages

    Between these two categories, people often choose procedural languages to create scripts for achieving test automation. Scripting languages are easier to learn and do not require compilation (which saves time). Even between scripting languages, languages that have a large follower base, with lots of libraries to choose from to solve various automation problems, tend to be a better choice.

    Object-Oriented Languages

    Usually Object Oriented languages are chosen for test automation when the application under test is developed using an Object Oriented language and the developers of that application have a greater influence in the architecture of the solution. The most common object oriented languages like Java and .NET have lots of tools and libraries options that can be used for automation.

    So which one is better of the two?

    At the end of the day, the success of the solution depends on which tools are best suited for building the solution and how well the test framework is supported so that it meets the needs of its consumers. In the past few years, most good tool choices for test automation support both procedural as well as object-oriented languages. Assuming that’s the case with your solution, in the long run if the group of people who end up supporting the solution feel comfortable with a particular language, then that should be chosen for building the solution. However, if the group feel equally comfortable or are equally divided, then the benefits of the procedural language usually ends up being a better choice.

    That being said, before you start diving into building scripts, consider the pros of writing tests as specifications as well.

    Domain Specific Language (DSL)

    DSL is a specification language that is dedicated to a particular problem domain, a particular problem representation technique, and/or a particular solution technique. The DSL is a software solution that focuses the domain experts to spend their time more effectively describing or solving a problem as opposed to battling with a programming language. Even though this concept has been around for several years, but its implementation in the test automation space, in my opinion, is still in its early stages.

    Imagine documenting your test case steps (Specifications) pretty much as you would do in your test case management system but being able to “run” these tests in an automation fashion without any coding! This is the promise that this solution can provide.

    There are some tools that allows creating tests as Specifications. FitNesse and Cucumber are two such examples.

    Test Specifications vs Scripts

    Lets now compare Tests as Specifications to traditional automation scripts approach.

    • Learning curve:
      • A programming language comes natural to developers. We have a tendency to write code to solve almost all problems. Writing tests steps that look like sentences requires a mind shift and can be a real challenge for some.
        QA on the other hand are generally more open to this concept.
      • If you are considering learning curve of existing employees, all languages need some learning curve. For creating specifications, you do not incur the cost of learning the programming language.
        If you are hiring to do the job though, you can however always hire people with the programming skills necessary. In comparison, its not always possible to hire domain experts, or atleast the pool of people to choose from is smaller.
      • Some argue that the consumers of the solution have to learn how to describe the solution to their domain problems using DSL. This is true, however, that’s no different than the effort it takes to familiarize yourself with the reusable modules when we use a programming language.
    • Speed of Development: When we compare speed of development between specifications and programming language options in the real world, we are really comparing speed of development of specifications by domain experts vs speed of development of functionality by programmers using a programming language. Even though often the programmers program functionality faster than domain experts describe specifications, the functionality developed by developers are meant to solve domain problems. If you take the time it takes for the developers to understand the domain problems and solve it correctly, you will realize that the speed of progress i.e. building solutions for domain problems is faster when domain experts write specifications.
      Within a QA organization, the number of people who know how the domain works are almost always more than the number of QA engineers who know the programming language. This ends up with a faster and wider adoption of automation, which ensues with more tests being automated sooner.
    • Documentation: DSL are self documenting. This makes
      • the intent of the test very clear,
      • it easy to review the tests,
      • makes it easy for new employees to understand how the system works
    • Maintenance cost:
      • If the test steps do not change, but the way you interact with the application changes, you can just modify the DSL implementation. This does not affect the end users who use the DSL to write the tests.
      • If you build your own DSL without IDEs that will support it, if the test steps change, refactoring become harder to do.
      • In a programming language approach, its very easy and common to create several methods that are very similar. This makes scripts and the reusable modules hard to maintain in the long term.

    So which one is better?

    Most companies start with developing tests using a programming language and it works great in the beginning. However after about a year they fail (I would say about 90% of them fail in the first attempt) mostly because of the lack of maintainability and/ or lack of adoption. When the end users of a Specifications based solution focus on what they know best, solving domain problems, in this case QA engineers focusing on writing automated tests that look like test cases – as opposed to battling with the programming language, the success rate is much higher. So, it should be pretty obvious that specifications is the way to go.

    However, the biggest hurdles with Specification Languages are

    • designing a good specifications based solution that can describe all the tests.
    • grasping the specification concept! As I mentioned devs tend to have more difficulty in making this paradigm shift.
    • Also, even if we build a great specifications based solution with tools like FitNesse, requests often are made to provide conditional branches and looping, which if provided, would make the tests less like specifications more like scripts.

    If you could get over this initial hump though and stay the course, then you should be successful!

    Feedback please

    I have expressed what I have learnt from my experience with both approaches and after discussing these approaches with few others. However, I would love to hear from all of you and learn from your successes and failures with these approaches.

    Posted by Rahul Poonekar in : Concepts

    One response to “Domain Specific Languages (DSL) vs. General Purpose Languages for Functional Test Automation”

    1. Tim says:

      With regard to choice of language, all of the ‘procedural’ languages you cite have object support. The main difference for me is between strongly and weakly typed languages: weakly typed languages are generally easier to get started with, but suffer in the longer term because the lack of discipline encourages poor programming practice.

      Automation is software, and should be treated like software: larger test systems need their framework aspects tested, code should be inspected for refactoring opportunities, etc

    Leave a Reply

    Your email address will not be published. Required fields are marked *