You are starting to work on a new project and you should design the architecture for an automation framework for REST API testing. But…you are new in this “world” and you don’t know how to start. Which is the best framework and programming language to use?
In this article I’m presenting some frameworks from my experience and then I’m explaining their pros and cons.
You should take into consideration that all these frameworks listed below should work with Softvision’s custom automation framework . So there were a couple of challenges to integrate each of them with our custom solution. But these challenges are the beauty of building and implementing a new software architecture.
When I started to work on my current project the automation framework used for API testing was jMeter. Every tester/dev should know or have heard about jMeter.
jMeter is a very powerful tool for load/performance/stress/capacity testing. But it is also very powerful for API testing, because it is very customizable. Choosing jMeter as the principal automation framework is a good choice if you have a small project. In the case you work on a big project, with a lot of endpoints, a lot of testing environments and also a CI tool to run the scripts, all of these could cause some issues.
Why should I use jMeter? This question has a lot of answers. One of the best answers is that jMeter is pretty easy to use and has a GUI. It also has a mature community where you could find all your answers. This is the first pro for jMeter.
Another pro for jMeter is the “modularization.” You could create jMeter test fragments and reuse them in other suites (jMeter Test plans) using “Include module” controller. Using this style of creating your test framework you could parameterize every test case and you could run which test case you want, not all the tests at a time. But having a lot of modules could cause some issues.
A big pro for jMeter is it is open source. You could create plugins for it. You could write native Java code into a lot of it’s samplers. Having the possibility to write Java code gives you many opportunities to better handling exceptions or assertions.
Now let’s talk about the possible issues. We have five jMeter scripts (suites) with 100 test cases (min. 100 HTTP requests) with at least two assertions(response code and response body). We have to run these scripts on five different test environments. Running the scripts one by one is not causing any issues. The issues appears when you are running all of them in parallel. jMeter is using a lot of memory and CPU if our APIs return a lot of data and it has to process it. You will see that the scripts will run very, very slow. This is the first con for jMeter.
I said having many jMeter modules could cause issues. If you import 100 jMeter test fragments into a script it will load the script very, very slow => a lot of CPU and memory is consumed by jMeter and this will cause a slowness of test running.
- PROs: Easy to use, GUI, run native Java code, modular
- CONs: Having a lot of test could cause many issues if your test environment doesn’t have the necessary resources to handle a lot of CPU and memory usage
On our project we are using jMeter for some REST API backend tests. But when the number of jMeter scripts had reached into the hundreds we started to face those possible issues listed above.
What was our next move? Researching for a new automation framework.
jUnit’s embedded http request
Every Java software engineer heard about jUnit. jUnit is a testing framework for Java programming language and it is very important in the development of test driven development.
jUnit is very powerful, but it has pros and cons on using it as a principal automation framework.It has HTTP requests embedded in it’s framework. This is a powerful feature of jUnit, but it has it’s limitations. The url should be constant(final), so you are not able to parameterize it. This is a big CON, and it was enough for us to restart the research.
- PROs: Easy to use
- CONs: Http request url should be final
From Java to Python. This is a long, long way… But everything has its starting point. This was ours.
RobotFramework is an automation framework which is actually a wrapper for selenium 2 library. It is a Keyword driven development automation framework implemented in Python. You could write Python or Java functions and import them into the project and use them as keywords. So, instead of calling a function getSomething(param) you write “getSomething param” – this is the keyword.
Being implemented using Python and Selenium2Library it is actually a powerful UI automation tool, not a backend API automation tool. This could be a CON for our purpose. We have to decide to start implementing new keywords to handle API testing or choose another framework. We had to choose another one, but we didn’t stop using it. We are using it on automating UI tests.
If you are starting an automation project and you are new in this type of testing this is the easiest framework to use. This is a big pro for RobotFramework.
- PROs: Easy to use, powerful tool for UI automation, written in Python and Selenium, modular
- CONs: Not designed for REST API testing
Back to Java again. RestAssured is a very powerful automation framework written in Java. It is designed to automate HTTP Requests – so REST APIs. For us this was a big pro. And we started to use it.
If your main programming language is Java, then RestAssured is the best automation framework for REST APIs.
The syntax is like BDD – given() -> when() -> then():
- given() – add parameters, headers, cookies, body
- when() – request method and request url
- then() – assertions, extractors
- PROs: Dedicated REST API automation framework, supports every http request method, cookies, headers, params, path extractors for xml & json, built-in assertions, modular
- CONs: So far we didn’t find any con (as we are in the early stages of implementation)
Only a few words about this framework. Why? Because I’ll let you discover the beauty of it.
This conclusion is based only from our point of view. Some people will choose RobotFramework to automate API, other people will choose to use the HTTP Request library embedded into the programming language etc. Each software engineer has his own preferences in choosing the programming language, the framework and any other resources they need.
So, from our perspective the best automation framework for REST APIs is RestAssured. It is simple to use, modular and you can manipulate the request in any way you want.
If you are planning to start a new automation project for testing REST APIs and you don’t know how to start you should start to read about RestAssured and a testing framework, like jUnit or TestNG. RestAssured and jUnit are the perfect match for your purpose at the beginning of your journey.