Comparison of Static Analysis Tools – A DAR Report!

Its been quite some time since I last wrote a blog post, and here I am back at it again! I’m going to forgive myself for that, because after all, its my first time in Europe and things had to settle down! Loving it at trivago!

We are off to an adventure here to select a tool that would best suit the need for a static code analysis tool to be used for validating the code quality of the project that I am currently working on. At the end of it all, we need a tool that compliments Kotlin and ReactJS with Typescript. I have gone with the Weighted Sum Model to rank the tools according to the selected criteria in order to arrive at a result (Power of proof matters :)). So let’s get started!!

Points Considered:

Here’s a list of requirements that I considered as the influence factors in the selection of this static analysis tool.

  • Ability to analyse the code quality of Kotlin, React JS and its Typescript files.
  • Ability to add rules on demand.
  • Active support.
  • Support for open source tools and linters.
  • Integration with coverage reporting tools.
  • Slack integration.
  • Jira integration.
  • Visual representation.
  • Pricing/Free Open source

So here’s the list of contenders going at it and I have chosen to go with the “most known” static code analysis tools out there in the market.

For Kotlin, detekt and ktlint have been evaluated along with the possibilities of providing a better visual experience on the results.

Sonarqube:

A powerful static analysis tool that is also open source. Supports over 26 languages and is capable of not just finding bugs in the code, but also coding rules, test coverage, duplications, API documentation, complexity and much more visually represented through a single dashboard. It also provides moment in time snapshot of the code quality as well as the trends of lagging or leading. The default profiles for specific languages could be customised to add new rules and to remove rules that may not suit the flow of a project. There’s also a comprehensive set of metrics that could be used in tracking the quality of the code. It can also be integrated into Slack, Jacoco and Jira, providing a complete coverage on the development life cycle. Provides active support through its GitHub page and community forum.

Criteria Weight Points Score
Ability to analyse the code quality of Kotlin, React JS and its Typescript files 30 30 30
Ability to add rules on demand 15 15 15
Support for open source tools and linters 10 10 10
Visual representation 10 8 8
Pricing/Free Open source 9 9 9
Active support 8 7 7
Integration with coverage reporting tools 6 6 6
Slack integration 6 6 6
Jira integration 6 6 6
Total 97

https://www.sonarqube.org/index/clean-code.png

97 is a score to beat! let’s see what the others have in store for us!

Code Climate:

A popular code quality tool that identifies potential code errors, security vulnerabilities and provides quantitative and qualitative metrics tracking progress over code based trends. Provides an effective user interface that makes the provided figures and charts easier to understand. Test coverage and maintainability are graded from A to F based on various measures and percentages. There have been instances logged where lots of false positives are shown when the analysis is run for the 1st time, making it not-so-accurate. The metrics applicable are also related to complexity and duplicated code. It can also report code coverage in a comprehensive form of per-file.  It also provides integration with Slack and Jira, though issues have been reported on its compatibility with Jacoco. Support is limited as it is provided as requests through their website.

Criteria Weight Score Comments
Ability to analyse the code quality of Kotlin, React JS and its Typescript files 30 30
Ability to add rules on demand 15 10 Checks, plugins and patterns can be added or removed through a yml or json config file.
Support for open source tools and linters 10 10 css, scss, sass etc. available.
Visual representation 10 8
Pricing/Free Open source 9 0 Paid/Enterprise versions available.
Active support 8 3 Support available through a request form in the website.
Integration with coverage reporting tools 6 4 Coverage tools of popular languages are available. Issues raised over functionality with Jacoco.
Slack integration 6 6 6
Jira integration 6 6 6
Total 77

https://frozencloud.files.wordpress.com/2018/07/af548-1b1c6s5dxaroqrmrwmej4va.png?w=1040

77 isn’t bad. May be the next one will do better!

Codebeat:

A code quality tool that grades the projects similar to Code Climate, but using a 4.0 scale system instead of A-F. Provides a comprehensive user interface that is smooth to use. Tool happens to be more accurate than Code Climate as it is even capable of differentiating similar code to identical code. It also contains a section where it lists down the top 5 issues that affects the code quality in a project. Main drawback for the tool comes from its inability to allow adding of more plugins or rules, leaving the users to stick to the default provided ones. Comes in free for public repositories and paid version for private repositories (Cheaper than Code Climate). Supports integration with Slack and Jira though there it does not accept the output files from Jacoco. Support is available through an online forum though it doesn’t seem to be active.

Criteria Weight Score Comments
Ability to analyse the code quality of Kotlin, React JS and its Typescript files 30 30
Ability to add rules on demand 15 0 Works with default rules.
Support for open source tools and linters 10 0 Doesn’t provide any linters for CSS, SCSS etc.
Visual representation 10 8
Pricing/Free Open source 9 3 Free only for public repositories.
Active support 8 4 Online forum available but not actively responded.
Integration with coverage reporting tools 6 3 Doesn’t support integration with Jacoco
Slack integration 6 6
Jira integration 6 6
Total 60

https://frozencloud.files.wordpress.com/2018/07/6a3ac-1uwwfpuvpejsispcpd2phsg.png?w=1040

Expected much better from this one, still 60 is a good score!

Codacy:

The last tool to be considered, having a great user interface that is clean and easy on the eyes. Contains a popular user base with companies such as Adobe and Paypal. Provides more metrics compared to Code Client and Codebeat, in the form of code complexity, compatibility, error-prone, security etc. Allows defining goals per file or category and helps providing steps to tackle the issues and meet the goals. Capability to measure code quality for Javascript seems to be lesser compared to other tools. Similar to Codebeat, this also provides a free version for public repositories while different paid plans are provided for private repositories. Integration with Slack, Jacoco and Jira are available. Support is available through requests submitted through the website.

Criteria Weight Score Comments
Ability to analyse the code quality of Kotlin, React JS and its Typescript files 30 30
Ability to add rules on demand 15 10 Allows adding custom extensions and patterns.
Support for open source tools and linters 10 10 Community linters are available
Visual representation 10 10
Pricing/Free Open source 9 3 Free only for public repositories.
Active support 8 3 Requests submitted online.
Integration with coverage reporting tools 6 6 Doesn’t support integration with Jacoco
Slack integration 6 6
Jira integration 6 6
Total 81

https://frozencloud.files.wordpress.com/2018/07/eff4f-1ibalqiz-z3bv5ldtzla7dq.png?w=1040

81 was a close call! but we know who stood out!

Based on the rankings resulting from the weighted sum model, Sonarqube seems to be the best option having configurable quality profiles, risk based views, comprehensive reports, custom rules, community support and a downloadable free version that we can setup ourselves.

Static analysis plan for Kotlin:

ktlint and detekt are two of the most popular static analysis tools for Kotlin.

ktlint:

  • Provides no configuration options, as it enforces the official code style from kotlinlang.org and Android Kotlin Style Guide.
  • Contains a built in formatter.
  • Provides command line interface reports or in XML format.
  • More of a CheckStyle for Kotlin, and we still need to have Lint for all other checks.

detekt:

  • Highly configurable with custom rules.
  • Contains checks for more code smells compared to ktlint.
  • Analysis can be run both at build.gradle file level and command line interface level.
  • Allows to add more extensions.

detekt contains the edge over ktlint having the capability to find more code smells, is updated quite often and contains lots of configuration options. detekt could also be integrated to Sonarqube with https://github.com/arturbosch/sonar-kotlin. This allows the configurations to be done through sonarqube user interface and provides the reports in an easily readble format through the sonarqube dashboard itself.

In conclusion, Sonarqube is what I’m going ahead, with detekt, for a comprehensive visual representation of the issues and for providing more configuration options (What’s more, both happen to be free! 😀 ).

Aaand so that brings us to a close! Oh on another note, life in Europe has been quite a change, especially for someone who couldn’t even make tea back at home! But then again, here I am making biriyanis for Muslim friends ^_^

 

 

Advertisements

Getting Started with Selenium Grid

What and Why!

It’s been a while since I blogged anything and here I am about to put my Selenium Grid experience to the vault. If your problem is to either save execution time on a large test suite or is to run several test cases in parallel on different platforms, then Selenium Grid is the answer. Selenium Grid allows you to test any combination of operating system running any browser, in parallel of course.

The Architecture

Selenium Grid is purely a network of hubs and their relevant nodes. A hub would work as the central point establishing control over all the connected nodes. This is where all the scripting would be done. A node on the other hand is any PC on which your test cases would get executed. These nodes could be a Windows machine with a Chrome, a Linux machine with a Firefox, Mac with a Safari or even an Android device. Now, having Android doesn’t mean that we could execute test cases of a mobile application! This rather means that we can execute our web application on a mobile browser. You can find a descriptive image of the architecture at toolsqa.com

Throughout this tutorial we’ll be using Selenium Grid  2.0 which has the remote control bundled with the Selenium Server jar file itself. Therefore we’ll be needing only the server jar file to host the grid. Using 2.0 also means that we need not to have an Apache Ant installation as required by 1.0.

First of all let us download and setup the Chrome driver for Google Chrome and Gecko driver for Firefox. So in our example we will be launching our tests on both Google Chrome and Firefox, in parallel! Now that you’ve got a basic idea, let us now get into the setup process!

Setting Up the Grid

Setting up the Web Drivers

In order to launch browsers to load our web application, we first need to setup the relevant web drivers. These web drivers are separate executables that allow us to control the relevant browsers. Since we’ll be running our test suite in this tutorial on both Chrome and Firefox, we’ll be needing the Chrome Driver executable and Gecko Driver executable.

Let us first setup the Chrome driver by following the below mentioned commands. In the below steps we’ll be downloading the Chrome driver, specifying executable permission to the downloaded executable and moving it to the usr/local/share directory. We’ll then be creating symbolic links to the usr/local/bin and usr/bin directories.

Now its time to setup the Gecko driver. Steps here are exactly similar to the ones followed for the Chrome Driver.

Getting the Grid up and running!

Okay now that we got the client drivers covered, let us now get the Selenium server hub up and running. As mentioned earlier, this is where all your scripting would be done at.

  • Download the latest version of Selenium Server.
  • Open a terminal and navigate to the downloaded location.
  • Execute the command shown below to run the server.

java -jar selenium-server-standalone-3.5.3.jar -role hub

Up on successful execution, your terminal would look something like this.

Capture.PNG

Execution of the above command would start a Jetty server on port 4444. You could pass the parameter “-port” to specify a custom port number of your choice. Now open a browser and visit the page http://localhost:4444/grid/console to verify that server is running without any issues. Your page should look something like this.

Screenshot from 2017-09-23 19-43-16.png

Next up is to start the node! Again, this would be the computer on which the browser would be launched and your tests would get executed. Follow the below mentioned steps to register an instance of the node on the hub that we set up in the above steps.

  • Download the latest version of Selenium driver in the node machines as well.
  • Open a terminal and execute the command given below to run the server on the node machine. Note that your role has now changed to “node”.

java -jar selenium-server-standalone-3.5.3.jar -role node -hub http://<IP of the Hub machine>:4444/grid/register

URL of the hub that is used in the above command would be the one displayed in the terminal for hub that we opened in the earlier step. In that terminal, find the “Nodes should register to” output as shown below.

You could also specify the Chrome driver location with the Selenium server command itself, if you haven’t created a Symbolic link to it. This is a popular option among Windows users as well. So your command to run the node would now look something like this.

java -jar -Dwebdriver.Chrome.driver=<location of the Chrome driver executable> selenium-server-standalone-3.5.3.jar -role node -hub http://<IP of the Hub machine>:4444/grid/register

Up on successful completion of registering the node at the hub your hub terminal would now show the entry shown below. Note that both the hub and node IP addresses are the same for me as I am running both of them in the same machine.

Capture.PNG

Visiting the URL http://localhost:4444/grid/console in the node machine would now show you a page similar to what has been shown below.

Screenshot from 2017-09-23 19-36-46

The above page shows you the information on how many nodes have got registered (Only 1 node in my case) and how many instances of different browsers could be opened at one go.

Well guess what, that’s all we had to do to setup our Selenium Grid!

Our Grid in action!

Let us now run a couple of tests in parallel and see our grid in action. For this purpose I am going to be using TestNG with Eclipse to write the test suite. Ignore the below steps if you already have an Eclipse with TestNG plugin installed.

Setting up Eclipse with TestNG

  • Download the latest version of Eclipse.
  • Open Eclipse and select a directory to be used as the work space for our project (If you are not familiar with the concept of work space, read through this).
  • Find TestNG plugin in the Eclipse Market place and drag and drop the “Install” button into your Eclipse.
  • Eclipse will now launch a new window with the TestNG plugins listed. Installation steps from here on will be pretty much self explanatory. If you’re having any doubts, read through this tutorial.
  • Once the plugin has been installed, restart your Eclipse software.

Creating the test suite using TestNG

Now that we’ve got the development environment setup, let us now create a test suite that contains two test cases. Let’s write a test case to launch Chrome and navigate to Yahoo page. We’ll validate the title of the page to pass the test. Our next test case is going to be launching Google on a Firefox browser and verifying the title in a similar fashion. First let’s start by creating the project.

  • Go to “File” -> “New” and select “Java Project”.
  • Specify a project name and click “Finish” with other options set to their default values.
  • Expand the project and you will find the “src” directory. This is will act as the source directory where we will be creating the TestNG classes.
  • Right click on the “src” directory and then select “New” -> “Other”.
  • Find and expand “TestNG” directory, select “TestNG class” and click “Next”.
  • Click on “Browse” next to the “Source Folder” and select the “src” directory. As mentioned earlier, this folder will contain our TestNG classes.
  • Specify a package name or you can leave it blank.
  • Specify “TestBase” as the “Class name” and click “Finish”.

Bringing in Maven

Now before we get to the implementation, let us convert the created project to a Maven project. Maven is a build tool which we are going to use to manage our dependencies. By having the dependencies listed through Maven, we won’t be needed to download the required libraries manually and attach to the build path. If you’re not familiar with Maven, here’s a good tutorial.

In order to convert our project to Maven, we need to install the Maven plugin for Eclipse from the Eclipse Market Place. The installation steps would be similar to how we installed TestNG plugin. Once installed, right click on the Java project we created, go to “Configure” and select “Convert to Maven project”. Now you will see that a pom.xml file will be created. Paste the content below to your pom.xml file. All we are doing here is, adding selenium and testng libraries to our project so that we could use their methods.

<dependencies> 
 <dependency> 
   <groupid>org.seleniumhq.selenium</groupid> 
   <artifactid>selenium-java</artifactid> 
   <version>3.5.3</version> 
 </dependency>
<dependency> 
 <groupid>org.testng</groupid> 
 <artifactid>testng</artifactid> 
 <version>6.11</version> 
 <scope>test</scope> 
 </dependency>
</dependencies>

Now to the Test Suite!

Now let’s resume our implementation. TestBase.java class is where we are going to establish connection to the Hub and specify the capabilities for the platform and browser. These have been parameterized to make it configurable from outside the code and we will be retrieving the values from the TestNG xml (We’ll be creating this XML in while and this will be used to specify the test methods that we’d like to run and the order in which they should get executed.).


public class TestBase
{
//Declare ThreadLocal Driver for ThreadSafe Tests
protected ThreadLocal<RemoteWebDriver> driver = null;
//Do the test setup
@BeforeMethod
@Parameters(value={"browser","platform"})
public void setupTest (String browser, String platform) throws MalformedURLException {
//Assign driver to a ThreadLocal
driver = new ThreadLocal<>();
//Set DesiredCapabilities
DesiredCapabilities capabilities = new DesiredCapabilities();
//Set Platform
capabilities.setCapability("platform", platform);
//Set BrowserName
capabilities.setCapability("browserName", browser);
//This is where we are invoking the browser driver
driver.set(new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capabilities));
}

public WebDriver getDriver() {
return driver.get();
}
@AfterClass
public void tearDown() throws Exception {
getDriver().quit();
}
}

Now let us create the first test to be executed. Follow the steps we followed above to create another TestNG class and call it “FirstTest”. In this class we are going to write a test to launch the Yahoo home page and verify its title. Content of this class would be as follows.


public class FirstTest extends TestBase {

@Test
public void firstTest() throws Exception {
  System.out.println("First Test Started!");
  getDriver().navigate().to("http://www.yahoo.com");
  System.out.println("First Test's Page title is: " + getDriver().getTitle());
  Assert.assertEquals("Yahoo", getDriver().getTitle());
  System.out.println("First Test Ended!");
 }
}

Now to the second test case to launch Google.com through a browser and verify its title.


public class SecondTest extends TestBase{

@Test
 public void secondTest() throws Exception {
  System.out.println("Second Test Started!");
  getDriver().navigate().to("http://www.google.com");
  System.out.println("Second Test's Page title is: " + getDriver().getTitle());
  Assert.assertEquals("Google", getDriver().getTitle());
  System.out.println("Second Test Ended!");
 }
}

Now let’s configure the TestNG xml file to contain the following configuration. Here we will be specifying the test methods to be executed and they’ll get executed in the order they’ve been specified. We will also be passing values to the parameters browser and platform that we had set up in the TestBase class. This allows us to specify on which platform and browser we need to run the relevant test cases.


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="5" name="Suite" parallel="tests" verbose="1">
 <test name="FirstTest">
  <parameter name="browser" value="chrome"/>
  <parameter name="platform" value="LINUX"/>
  <classes>
    <class name="FirstTest"/>
  </classes>
 </test> <!-- First Test -->
 <test name="SecondTest">
  <parameter name="browser" value="firefox"/>
  <parameter name="platform" value="LINUX"/>
  <classes>
    <class name="SecondTest"/>
  </classes>
 </test> <!-- Second Test -->
</suite> <!-- Suite -->

That’s all for the implementation. Pretty straight forward isn’t it? Now let’s run the test suite and see it in action! Right click on your TestNG.xml and select the “Run Test Suite” option. You will now see that two browsers would be opened. Chrome will be opening Yahoo.com and Firefox will be opening Google.com, both at the same time! That’s the whole point of the Grid!

What If!

In case you get an empty browser loaded or if your browser doesn’t load at all, verify that you have set up the Chrome driver and Gecko driver properly. Ensure that you are using the latest versions of the drivers as well the browsers. If not, please update your browsers to the latest versions to conform to the driver executable versions. Also if you are running the hub and node on different machines and if you run into any issues when establishing communication between them, make sure that there are no firewall issues. If not, relevant in bound and out bound rules have to be created to allow communication between the two machines.

Well that’s all folks for now. Happy gridding alright! ^_^