What is an automation framework?
In technical terms, an automation framework is a set of rules or guidelines which include standardized code, data structures, algorithms, folders, reports, configuration files, databases etc.
Why do we need an automation framework?
An automation framework is very helpful for every team because it provides team consistency, it is scalable and modular, it could be reused for other purposes, you could easily generate reports based on test results, but the most important thing is the reduced cost and it is easy to maintenance.
Why do we need a custom automation framework?
Most of the automation teams are using an existing framework (eg. Selenium, RobotFramework etc), but there are also teams which build their own framework. A custom automation framework is used by the whole team, not only by automation team. A custom framework has a desktop/web interface easy to use by anyone, technical or non-technical. It gives you the power to build your own tools for generating reports, sending custom emails or any other tool which could help your team. Other benefits of having a custom automation framework are the possibility to implement tests using different programming languages.
Let’s talk about the architecture
In this article I want to talk about a custom automation framework created using the following programming languages, frameworks, servers, and software project management:
- Java – Spring Framework (Back-end)
- Maven (Project management)
- Bootstrap (Front-end)
- MySQL (Database server)
- Tomcat (Application server)
MVC – Model–View–Controller – is a software architectural pattern used for implementing user interfaces on computers. It has three interconnected parts – the model, the view and the controller. An easy way to explain how it works: a user is sending a request to the controller, the model manipulate the data and the view displays the manipulated data to the user. MVC software architecture is very powerful at a parallel development and at code reuse.
Java – Spring Framework (Spring MVC)
Why Java? Because Java is the most popular programming language and it is also very powerful. Another argument for this choice is that I worked with Java and I know it and it’s capacities. It is also open source and cross-platform, so you can deploy the application on any operating system.
Why Spring Framework? Spring MVC is the most popular Java framework and it provides everything you need to build a web application from scratch.
If you need to create a web application using Java, Spring MVC is the best choice for you.
Maven is a software project management tool and it is one of the most popular tools for it’s purpose. Combined with Spring MVC it gives you everything you need: build, install, test, package the project and also it has a powerful dependencies system. Theoretically, you are able to import every external Java library using Maven dependencies.
Twitter Bootstrap is a front-end framework. You can choose any other framework for this, but I chose this because it has a lot of fancy icons, buttons, and other elements. You can write your own, but it’s easy to choose a free framework.
Choosing a database server could be difficult. You should think in advance if there will be other tools on your framework which will require database storage and what type of storage it needs. For a relational database, the best choice is MySQL. It is a free tool, and it is also open source, so you are able to make changes on it. For our purpose, it is the best, because we won’t have billions of rows, so it could handle our reports very well.
Apache Tomcat is the most used application server for JSPs. I don’t think we need to talk about it, because everyone should hear about tomcat and how customizable it is.
Integrate tests into the application
You should follow a standard in all your automated test cases:
- naming convention – every test case should have a unique name
- logging – after a test is run, the assertion and a message should be logged into the database
There are also 2 mandatory parameters on each test suite:
- test run id – this is generated when you trigger a suite
Now, the main question is: how can we integrate our tests into the application and view the results in a live way(on demand)?
First – you have to declare an alias for each framework/programming language used by your tests. Eg. for JMeter tests you have an alias “JMeter” which points to the JMeter path. Why? Because we will use the command line to trigger the tests. Theoretically, every programming language or automation framework could be run from a command line on any operating system (Windows, MacOS or Linux).
Second – you have to create the command for each suite. You can create this in 2 ways:
- store the command into the database after you create the suite. In this way, it appears dynamic into the web application
- create the command on the running phase based on the test script extension
Eg. for running a JMeter script: JMeter -n -t scriptPath/scriptName.jmx -JpropertyName=propertyValue
Third – you have to execute the command. This is the easiest step:
Process process = Runtime.getRuntime().exec(command);
Following these 3 steps, you are able to integrate automated test cases written in every programming language/automation framework. I integrated test cases written in Java, Python, JMeter, RobotFramework, Shell, Perl, and Ruby into our custom automation framework and we didn’t have any issues so far.
The runner page should contain all suites with the ability to choose what suites you want to run. You could also have some custom parameters on this page if your tests require additional parameters.
After you press the “Start” button the following should happen “in the background”: validate the request and parameters, and if everything is all right you have to create a new entry into the database for this new test run. Extract the auto-generated ID and the active session username and pass them as parameters to the suite. Execute the command in a separate thread and return the response to the user. If the response is ok, the API which returns the logs is called from an interval(every 2 seconds should be ok) and the logs are displayed until the suite is finished and you will receive an email with the results.
Again…why do we need a custom automation framework?
After talking about the architecture and implementation there are a couple of questions. One of them is again “why do we need a custom automation framework?” Now I can give you another answer. We need a custom automation framework because it has the following features:
- intuitive interface – an interface is very useful for everyone. Having an intuitive interface helps you to understand everything about the application. It could be used by anyone working on the project even if they don’t have information about the tests, suites or other technical information.
- platform independent – taking a look at the software architecture you are seeing all are open source and free and are cross-platform. This is giving the ability to deploy and run the application on any operating system.
- plug-and-play – the application is developed in such a way so that any tests can be ran from it with minimal implementation.
- portal for multiple tools – give you the power to implement any other tool inside it. You can add external tools, internal tools or custom made tools to the app.
- exposed API – every feature has it’s own API, so you can integrate the features into any other app (like Jenkins)
- Jenkins? Jenkins is a powerful tool for CI. It is also used by automation teams to run their tests. If your project has a lot of test environments and a lot of test suites it is very difficult to maintain Jenkins, because you have to create a separate Jenkins job for each suite on each environment or combination of suites on environments. Think about how many combinations are. Another con for Jenkins is it’s reporting system. You are not able to generate reports based on the combination of suites from data x to date y. Also, you have a history of the last x builds(x is configurable). Jenkins is a tool which requires a couple of technical skills to work with it. If you want to share your automated test cases with everyone, you should take into consideration not everyone knows how to create a Jenkins job or how to run it.
- jUnit/testNG? Because for these 2 you have to write all your tests in Java. If some of the tests are not written in Java it’s a little bit more complicated to integrate them. And there are also the cons from Jenkins: technical skills are required and you are not able to generate reports based on the combination of test suites and timeline.
- Test runner – this is the “main()” function of the application. From this page, you configure a test suite and run it.
- Live view – live view or on-demand view is the feature which displays you every log with proper and understandable messages
- Database stored logs – every log from every test cases ran is stored into the database. This gives you the ability to generate graphs or reports based on the test case, test suite, environment, and timeline.
- Test runs comparison – you are able to compare the results between 2 test runs
- Graphs – you are able to generate graphs based on the test suite(s), environment and timeline. The graphs should be generated by number of tests passed and failed or by percentages of tests passed and failed
- Schedules (Cron jobs) – you could create a cron job for a particular test run configuration
- Email results – after a test suite/test configuration is finished, you will receive a detailed report on your email address
Having a custom automation framework is very helpful for you and your team. Everyone working on the project is able to run the suites and receive the reports. You could integrate the tool with the CI machine to run a particular suite after a new build is generated.
I hope everything is clear and you will proceed by creating your own custom automation framework using these easy steps listed above.