• Jobs
  • About
  • Error handling in FitNesse August 24, 2010

    If you follow my blog, or if you have worked in a company where I architected the test automation solution, you already know how big of a believer I am in creating human readable tests. Hiding the implementation details enables the test experts who want to focus on designing tests rather than having them struggle with code. But it’s not only about creating human readable tests, but after executing the tests, the results and the errors should be human readable too.

    FitNesse is one framework that allows you to achieve these goals. Since its a framework and not a tool to test an interface like Soap UI or HtmlUnit can, to achieve capabilities to test browsers, web services etc, you have to write code to integrate tools that can test these interfaces with FitNesse. While doing so, if you do not design your extensions right, you could loose some of the goals FitNesse tries to achieve in its architecture. Designing your Specification Language using FitNesse covers several aspects of how to design your fixture, but, one topic that needs a separate post is how to handle and display exceptions.

    Friendly Exceptions in FitNesse

    When you learn about java exceptions, one of the common examples used to demonstrate it is a method for dividing two numbers. Let us assume you have such a method in your fixture:

        public double divideBy(double x, double y) {
            return x/y;
        }
    

    If you see a positive test case for this method, it would look like this after execution:

    Now let us see how it behaves if you tried to pass in a string:

    What happens here is that, FitNesse recognizes that the parameter passed is not a double and throws a type of ParseException. The fitnesse traversal logic detects this exception and instead of throwing a stacktrace, it shows a meaningful error – Invalid number. Also, it shows it next to the parameter so you know exactly which parameter is invalid.

    Unfortunately, all other types of exceptions generated by the code you write will show up as stacktraces. For instance, if you have a method that looks like this:

        public void doSomething() {
            throw new RuntimeException("Nothing to do...");
        }
    

    will cause an exception that appears like this:

    This is useful for debugging the framework code, but it is not very useful for a tester who does not care about the code but rather wants to know why the test step didn’t work as expected. What I believe is the best solution, is a way to allow test framework developers to get to the stacktrace in the logs but show testers an easily readable error that gives them the information they need to reproduce, or better yet, file, the bug. Even if the tester and the framework developer is the same person performing two roles, there are several advantages to separate a pointer within the framework of where the error occurred with the functional failure information. The stacktrace is just noise, and say, the tester wants to share the test script with the developer so he can reproduce the error, even he would appreciate an error that tells him what’s functionality is broken without all the stacktrace that is irrelevant to him.

    Designing Exceptions in FitNesse test framework

    Fortunately FitNesse provides ways to plug in your own implementation of the parsing logic. In this case we can simply extend the existing traversal logic and modify just the way it displays exceptions in the wiki and put the stacktrace information in the logs.

    First create the exceptions that you will throw in your framework

    Base exception class:

    package net.qaautomation.exceptions;
    
    /**
     * Base exception for all errors that happen during test case execution.
     *
     * @author rpoonekar
     */
    public abstract class TestCaseExecutionException extends RuntimeException {
        protected boolean fatal = false;
    
        /**
         * Constructs a new runtime exception with the specified detail message.
         * The cause is not initialized, and may subsequently be initialized by
         * a call to <code>Throwable.initCause(java.lang.Throwable)</code>.
         *
         * @param message The detail message. The detail message is saved for later
         * retrieval by the <code>Throwable.getMessage()</code> method.
         */
        public TestCaseExecutionException(String message) {
            this(message, false);
        }
    
        /**
         * Constructs a new runtime exception with the specified detail message.
         * The cause is not initialized, and may subsequently be initialized by
         * a call to <code>Throwable.initCause(java.lang.Throwable)</code>.
         *
         * @param message The detail message. The detail message is saved for later
         * retrieval by the <code>Throwable.getMessage()</code> method.
         * @param fatal Signifies if the test case needs to be terminated or not.
         */
        public TestCaseExecutionException(String message, boolean fatal) {
            super(message);
            this.fatal = fatal;
        }
    
        /**
         * Constructs a new runtime exception with the specified detail message and cause.
         * Note that the detail message associated with cause is not automatically incorporated
         * in this runtime exception's detail message.
         *
         * @param message the detail message (which is saved for later retrieval by the
         * <code>Throwable.getMessage()</code> method).
         * @param cause The cause (which is saved for later retrieval by the
         * <code>Throwable.getCause()</code> method). (A <code>null</code> value is permitted,
         * and indicates that the cause is nonexistent or unknown.)
         */
        public TestCaseExecutionException(String message, Throwable cause) {
            this(message, cause, false);
        }
    
        /**
         * Constructs a new runtime exception with the specified detail message and cause.
         * Note that the detail message associated with cause is not automatically incorporated
         * in this runtime exception's detail message.
         *
         * @param message the detail message (which is saved for later retrieval by the
         * <code>Throwable.getMessage()</code> method).
         * @param cause The cause (which is saved for later retrieval by the
         * <code>Throwable.getCause()</code> method). (A <code>null</code> value is permitted,
         * and indicates that the cause is nonexistent or unknown.)
         * @param fatal Signifies if the test case needs to be terminated or not.
         */
        public TestCaseExecutionException(String message, Throwable cause, boolean fatal) {
            super(message, cause);
            this.fatal = fatal;
        }
    
        public void setAsFatal(boolean fatal) {
            this.fatal = fatal;
        }
    
        public boolean isFatal() {
            return fatal;
        }
    }
    

    Implementation of an exception class

    package net.qaautomation.exceptions;
    
    /**
     * Meant to capture System Exceptions like when you cannot connect to a url or database.
     *
     * @author rpoonekar
     * @since Jul 20, 2007
     */
    public class SystemException extends TestCaseExecutionException {
        /**
         * Constructs a new runtime exception with the specified detail message.
         * The cause is not initialized, and may subsequently be initialized by
         * a call to <code>Throwable.initCause(java.lang.Throwable)</code>.
         *
         * @param message The detail message. The detail message is saved for later
         *                retrieval by the <code>Throwable.getMessage()</code> method.
         */
        public SystemException(String message) {
            super(message);
        }
    
        /**
         * Constructs a new runtime exception with the specified detail message.
         * The cause is not initialized, and may subsequently be initialized by
         * a call to <code>Throwable.initCause(java.lang.Throwable)</code>.
         *
         * @param message The detail message. The detail message is saved for later
         * retrieval by the <code>Throwable.getMessage()</code> method.
         * @param fatal Signifies if the test case needs to be terminated or not.
         */
        public SystemException(String message, boolean fatal) {
            super(message, fatal);
        }
    
        /**
         * Constructs a new runtime exception with the specified detail message and cause.
         * Note that the detail message associated with cause is not automatically incorporated
         * in this runtime exception's detail message.
         *
         * @param message the detail message (which is saved for later retrieval by the
         *                <code>Throwable.getMessage()</code> method).
         * @param cause   The cause (which is saved for later retrieval by the
         *                <code>Throwable.getCause()</code> method). (A <code>null</code> value is permitted,
         *                and indicates that the cause is nonexistent or unknown.)
         */
        public SystemException(String message, Throwable cause) {
            super(message, cause);
        }
    
        /**
         * Constructs a new runtime exception with the specified detail message and cause.
         * Note that the detail message associated with cause is not automatically incorporated
         * in this runtime exception's detail message.
         *
         * @param message the detail message (which is saved for later retrieval by the
         * <code>Throwable.getMessage()</code> method).
         * @param cause The cause (which is saved for later retrieval by the
         * <code>Throwable.getCause()</code> method). (A <code>null</code> value is permitted,
         * and indicates that the cause is nonexistent or unknown.)
         * @param fatal Signifies if the test case needs to be terminated or not.
         */
        public SystemException(String message, Throwable cause, boolean fatal) {
            super(message, cause, fatal);
        }
    }
    

    Create the traversal logic

    package net.qaautomation.fitnesse;
    
    import com.netsuite.testautomation.exceptions.TestCaseExecutionException;
    import fit.Fixture;
    import fitlibrary.exception.AbandonException;
    import fitlibrary.exception.FitLibraryException;
    import fitlibrary.exception.IgnoredException;
    import fitlibrary.exception.method.AmbiguousActionException;
    import fitlibrary.exception.method.AmbiguousNameException;
    import fitlibrary.exception.method.MissingMethodException;
    import fitlibrary.global.PlugBoard;
    import fitlibrary.suite.InFlowPageRunner;
    import fitlibrary.table.Cell;
    import fitlibrary.table.Row;
    import fitlibrary.table.Tables;
    import fitlibrary.traverse.workflow.DoCaller;
    import fitlibrary.traverse.workflow.DoTraverse;
    import fitlibrary.traverse.workflow.caller.*;
    import fitlibrary.utility.TableListener;
    import fitlibrary.utility.TestResults;
    
    import java.util.ArrayList;
    import java.util.List;
    
    /**
     * Enhancing default DoTraverse
     *
     * @author Rahul
     * @since Aug 23, 2010
     */
    public class DoMoreTraverse extends DoTraverse {
        public DoMoreTraverse(Object sut) {
            super(sut);
        }
    
        public Object interpretRow(Row row, TestResults testResults, Fixture fixtureByName) {
        	if (testResults.isAbandoned()) {
    			row.ignore(testResults);
    			return null;
    		}
        	final Cell cell = row.cell(0);
        	if (cell.hasEmbeddedTable()) {
        		setExpectedResult(null);
        		interpretInnerTables(cell.innerTables(),testResults);
        		return null;
        	}
        	setExpectedResult(Boolean.TRUE);
        	try {
        		DoCaller[] actions = {
        				new DefinedActionCaller(row, this),
        				new MultiDefinedActionCaller(row, this),
        				new SpecialCaller(row,switchSetUp()),
        				new PostFixSpecialCaller(row,switchSetUp()),
        				new FixtureCaller(fixtureByName),
        				new ActionCaller(row,switchSetUp()) };
    			checkForAmbiguity(actions);
                for (DoCaller action : actions)
                    if (action.isValid()) {
                        Object result = action.run(row, testResults);
                        if (testResults.isAbandoned() && !testResults.problems())
                            row.ignore(testResults);
                        return result;
                    }
    			methodsAreMissing(actions,row.text(0, this));
        	} catch (IgnoredException ex) {
        		//
        	} catch (AbandonException e) {
        		row.ignore(testResults);
        	} catch (Exception ex) {
                Throwable t = ex.getCause();
                if (t instanceof TestCaseExecutionException) {
                    handleFriendlyErrors(row, testResults, (TestCaseExecutionException) t);
                } else {
        		    row.error(testResults, ex);
                }
        	}
        	return null;
        }
    
        private void handleFriendlyErrors(Row row, TestResults testResults, TestCaseExecutionException exception) {
            exception.printStackTrace(System.err);
    
            StringBuffer sbMessage = new StringBuffer();
            sbMessage.append("[");
            String exceptionClassName = exception.getClass().getName();
            sbMessage.append(exceptionClassName.substring(exceptionClassName.lastIndexOf('.') + 1));
            sbMessage.append("] ").append(exception.getMessage());
    
            Throwable nestedException = exception.getCause();
            while(nestedException != null) {
                exceptionClassName = nestedException.getClass().getName();
                sbMessage.append("<blockquote>[").
                        append(exceptionClassName.substring(exceptionClassName.lastIndexOf('.') + 1)).
                        append("] ").
                        append(nestedException.getMessage()).
                        append("</blockquote>");
                nestedException = nestedException.getCause();
            }
    
            String htmlErrorMessage = Fixture.label(sbMessage.toString());
    
            if (exception.isFatal()) {
                htmlErrorMessage = "<div style=\"color: red\">" + htmlErrorMessage + "</div>";
                try {
                    TestResults.setAbandoned();
                    tearDown();
                    abandonStorytest();
                } catch (Exception e) {
                    System.err.println("Could not terminate test execution");
                    e.printStackTrace(System.err);
                }
            }
    
            row.cell(0).error(testResults, htmlErrorMessage);
        }
    
        private void interpretInnerTables(Tables tables, TestResults testResults) {
    		new InFlowPageRunner(this,testResults).run(tables,0,new TableListener(testResults),true);
    	}
    
    	private static void checkForAmbiguity(DoCaller[] actions) {
    		final String AND = " AND ";
    		String message = "";
    		List<String> valid = new ArrayList<String>();
            for (DoCaller action : actions) {
                if (action.isValid()) {
                    String ambiguityErrorMessage = action.ambiguityErrorMessage();
                    valid.add(ambiguityErrorMessage);
                    message += AND + ambiguityErrorMessage;
                }
            }
    		if (valid.size() > 1)
    			throw new AmbiguousActionException(message.substring(AND.length()));
    	}
    
    	private void methodsAreMissing(DoCaller[] actions, String possibleFixtureName) {
    		// It would be better to pass all these exceptions on in a wrapper exception.
    		// Then they can be sorted and organised into <hr> lines in the cell.
    		final String OR = " OR: ";
    		String missingMethods = "";
    		String missingAt = "";
    		String ambiguousMethods = "";
            for (DoCaller action : actions)
                if (action.isProblem()) {
                    @SuppressWarnings({"ThrowableResultOfMethodCallIgnored"})
                    Exception exception = action.problem();
                    if (exception instanceof MissingMethodException) {
                        MissingMethodException missingMethodException = (MissingMethodException) exception;
                        missingMethods += OR + missingMethodException.getMethodSignature();
                        missingAt = missingMethodException.getClasses();
                    } else if (exception instanceof AmbiguousNameException) {
                        AmbiguousNameException ambiguousNameException = (AmbiguousNameException) exception;
                        ambiguousMethods += OR + ambiguousNameException.getMessage();
                    } else if (exception instanceof ClassNotFoundException) {
                        ClassNotFoundException cnf = (ClassNotFoundException) exception;
                        if (cnf.getCause() != null) {
                            System.out.println("methodsAreMissing(): CNFE: " + exception.getMessage() + ": " + cnf.getCause().getMessage());
                        } else
                            System.out.println("methodsAreMissing(): CNFE: " + exception.getMessage());
                        missingMethods += OR + exception.getMessage();
                    } else
                        missingMethods += OR + exception.getMessage();
                }
    		String message = "";
    		if (possibleFixtureName.contains("."))
    			message += "Missing class or ";
    		if (!"".equals(missingMethods))
    			message += "Missing methods: "+missingMethods.substring(OR.length());
    		if (!"".equals(ambiguousMethods))
    			message += " "+ambiguousMethods.substring(OR.length());
    		if (!"".equals(missingAt))
    			message += " in "+missingAt;
    		throw new FitLibraryException(message.trim());
    	}
    }
    

    Enable the traversal class in your Fixture

    First instantiate the traversal class:
    private DoTraverse traverse = new DoMoreTraverse(this);

    Set the traversal object in your DoFixture constructor
    setTraverse(traverse);

    Throw your standard exceptions

        public void doSomething() {
            throw new SystemException("Nothing to do...");
        }
    

    Let us now look at how this affects the display of the previous test step after execution:

    You can see that the wiki does not have the stacktrace anymore. It just contains the error message. Now all you have to do is make sure any exception you trow, has a meaningful message. The stacktrace of this exception is now in the logs that you can view by clicking the Tests Executed OK link.

    Posted by Rahul Poonekar in : FitNesse

    Leave a Reply

    Your email address will not be published. Required fields are marked *