TestNG Report Example

When you create and run a TestNG project, it will generate a lot of Html or XML report files. The report files include a huge amount of useful information that can help you to check the test case status, debug test case steps and show test case execution results to your manager. So it is very important to know all the key reports that TestNG generated for you.

1. What You Will Learn?

1. Create a TestNG project with one main suite, the main suite will contain two child suites.
2. Create some pass and fail tests in the above suite. Some have data providers and some are grouped.
3. Generate TestNG report files.
4. Introduce each Html and XML report generated.

2. Create TestNG Suite Project.

You can read TestNG Eclipse Plugin, TestNG Tutorial to learn how to crate TestNG project. Below we only show you how to add main and sub suite xml in it.

1. Right-click the project name, click ” New —> Others…“, choose “TestNG class” in the popup dialog, click the Next button to open the New TestNG class wizard dialog.
2. Browse the Source folder to your project src folder ( in my example is /TestNGExampleProject/src). Input the Package name ( com.dev2qa.example.testng.report ) and Class name( TestCaseClass1 ). In the XML suite file input box, input the file path relative to the src folder such as ./com/dev2qa/example/testng/report/main-suite.xml.
3. After clicking the Finish button, you can see main-suite.xml exists under the report folder in the eclipse project left panel.
4. Create other test case classes and suite1.xml and suite2.xml as below.

   C:\WORKSPACE\WORK\DEV2QA.COM-EXAMPLE-CODE\TESTNGEXAMPLEPROJECT
   ├───src
   │   └───com
   │       └───dev2qa
   │           └───example
   │               └───testng
   │                   └───report
   │                       │   main-suite.xml
   │                       │   suite1.xml
   │                       │   suite2.xml
   │                       │   TestCaseClass1.java
   │                       │   TestCaseClass2.java
   │                       │   TestCaseClass3.java
   │                       │   TestCaseClass4.java
   │                       │   TestCaseClass5.java
   │                       │   TestCaseClass6.java

3. Code Example Files Content.

1. main-suite.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <suite name="MainSuite" parallel="false">
     <suite-files>
     	<suite-file path="./suite1.xml" />
     	<suite-file path="./suite2.xml" />
     </suite-files>
     <test name="MainSuiteTest">
       <classes>
         <class name="com.dev2qa.example.testng.report.TestCaseClass1"/>
         <class name="com.dev2qa.example.testng.report.TestCaseClass2"/>
       </classes>
     </test> <!-- MainSuiteTest -->
   </suite> <!-- MainSuite -->

2. TestCaseClass1.java

   public class TestCaseClass1 {
     
   	@Test /* This method will pass. */
   	public void tc1m1() {
   	  
   	}
   	
   	@Test /* This method will fail. */
   	public void tc1m2() {
   	  String str = null;
   	  System.out.println(str.length());
   	}
   	
   	@Test(enabled=false) /* This method will ignore. */
   	public void tc1m3() {
   
   	}
   }

3. TestCaseClass2.java

   public class TestCaseClass2 {
   
   	  /* tcc2m1 and tcc2m2 are all in group1. */
   	  @Test(groups="group1")
   	  public void tcc2m1() {
   		  Assert.assertTrue(false);
   	  }
   	  
   	  @Test(groups="group1")
   	  public void tcc2m2() {
   		  Assert.assertTrue(true);
   	  }
   	  
   	  /* tcc2m3 in group2*/
   	  @Test(groups="group2")
   	  public void tcc2m3() {
   		  Assert.assertTrue(true);
   	  }
   	  
   	  @Test
   	  public void tcc2m4() {
   		  Assert.assertTrue(true);
   	  }
   
   	  /* tcc2m5 has a data provider. */
   	  @Test(dataProvider="exampleDP")
   	  public void tcc2m5(String param) {
   	  }
   	  
   	  @DataProvider
   	  private Object[][] exampleDP() {
   		  return new Object[][]{{"Java"}, {"Selenium"}};
   	  }
   }

4. suite1.xml

   <suite name="Suite1" parallel="false">
     <test name="Suite1Test1">
       <classes>
         <class name="com.dev2qa.example.testng.report.TestCaseClass1"/>
         <class name="com.dev2qa.example.testng.report.TestCaseClass3"/>
       </classes>
     </test>
     <test name="Suite1Test2">
       <classes>
         <class name="com.dev2qa.example.testng.report.TestCaseClass4"/>
       </classes>
     </test>
   </suite>

5. TestCaseClass3.java

   public class TestCaseClass3 {
     @Test(groups="group3")
     public void tcc3m1() {
   	  /* Reporter will log the message in TestNG Reporter output. */
   	  Reporter.log("This method is TestCaseClass3.tcc3m1");
     }
     
     @Test(groups="group3")
     public void tcc3m2() {
   	  
     }
   }

6. TestCaseClass4.java

   public class TestCaseClass4 {
   
   	  @Test
   	  public void tcc4m1() {
   		  
   	  }
   	  
   	  @Test
   	  public void tcc4m2() {
   		  Reporter.log("This method is TestCaseClass4.tcc4m2");
   	  }
   }

7. suite2.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <suite name="Suite2" parallel="false">
     <test name="Suite2Test1">
       <classes>
         <class name="com.dev2qa.example.testng.report.TestCaseClass5"/>
         <class name="com.dev2qa.example.testng.report.TestCaseClass6"/>
       </classes>
     </test>
   </suite>

8. TestCaseClass5.java

   public class TestCaseClass5 {
     @Test
     public void tcc5m1() {
   	  Reporter.log("This method is TestCaseClass5.tcc5m1");
     }
   }

9. TestCaseClass6.java

   public class TestCaseClass6 {
   	  @Test
   	  public void tcc6m1() {
   		  Reporter.log("This method is TestCaseClass6.tcc6m1");
   	  }
   }
   

4. Generate TestNG Reports.

1. Right-click main-suite.xml file, Click ” Run As —> TestNG Suite ” in popup menu.
2. When execution complete, you can see the execution output in the eclipse console.
3. Right-click the project name, click ” Refresh ” in the popup menu. Then you can see the test-output folder. There are many HTML and XML report files there.

5. Report Files Introduction.

5.1 test-output / index.html。

1. This is the HTML report enter page. Open it in a web browser, you can see the Main Suite, Suite1, Suite2 in the left panel and related links. Click the link, the right panel will show the test suite detail information.
2. In the left panel, each suite has an Info and Results section.
3. Info — Information about the suite. Such as main-suite.xml, test status, group status link.
4. Results — Test result link, pass methods, fail methods, etc.
5. Below is the Info section primary links introduction.
6. main-suite.xml — The content of suite XML file.
7. Reporter output — Click this link to show Reporter.log() logged messages in the right panel. Reporter.log() method can be used in any test method to log custom messages.
8. Ignored methods — List all ignored test methods such as the method with @Test(enabled=false) annotation.
9. Chronological view – List test methods by their execution time in chronological order.
10. Detail Info Panel: Whichever link you click in the left panel. The right panel will show the detailed information related to it. For example, If you click the Failed methods name in the left panel, the detail panel will show the detailed error information about the failure.

5.2 test-output / emailable-report.html.

1. This report file is generally used to send to other team members by email. It includes all test suite and test method status information. It has been divided into three sections.
2. The test suite summary list all the test executed in this suite and sub suite. The first Test column in the HTML file is the TestNG test suite name, you can click the name link to see the detailed test suite information.
3. When you click any test name in the above test suite list, it will direct you to the detailed execution status section. This section will list all the test method name and their execution status.
4. When you click any test method name in the above list, it will show the detailed execution information include error messages if exist.

5.3 Other Important Report Files Under test-output Folder.

1. old / index.html: This is the old-style TestNG reports.
2. testng-failed.xml: This file include only failed test class and methods. You can easily rerun these failed methods again by right click this XML file and then click ” Run As —> TestNG Suite “.
3. junitreports folder: Junit format report XML files for all test classes.
4. testng-results.xml: XML format file for the results report.
5. Suite1, Sutie2 folder: Include individual HTML and XML reports for each test case.