TestNG Reporting Customization

We have introduced how to create TestNG default report Html and XML files in the article TestNG Report Example. But you may find it is not enough for you. Because the original reports generated by TestNG have some inconvenience, it is so basic. TestNG provide an IReporter interface that you can implement to create a test context listener. Then you can customize the report in its generateReport() method. This example will show you how to customize the TestNG report step by step.

1. Default TestNG Generated Reports.

We will customize the default report generated in the article TestNG Report Example. Below are the testng.xml and test java files used in that article.

  1. main-suite.xml
  2. suite1.xml
  3. suite2.xml
  4. TestCaseClass1.java
  5. TestCaseClass2.java
  6. TestCaseClass3.java
  7. TestCaseClass4.java
  8. TestCaseClass5.java
  9. TestCaseClass6.java

default-testng-emailable-report

The default TestNG emailable report contains the test suites summary area and suite test class summary area, it has the below disadvantages.

  1. The Time column is not user-friendly.
  2. Do not provide test start time and end time.

We will customize it to the below report. It has below improvements.

  1. Add Browser, Start Time, and End Time columns for test suite summary report.
  2. Add Start Time, Parameter, Reporter Message, and Exception Message columns for test method summary report.
  3. Convert Execution Time value to a more human-readable format.

2. Customize TestNG Report Steps.

  1. customize-emailable-report-template.html : This is the template Html for customized reports. It contains Html layout and three customize TestNG report element place holder.
      <body>
        <table>
          <tr><center><b>$TestNG_Custom_Report_Title$</b></center></tr>
          <thead>
            <tr>
              <th>Test Name</th>
              <th># Total Method</th>
              <th># Passed</th>
              <th># Skipped</th>
              <th># Failed</th>
              <th>Browser</th>
              <th>Start Time</th>
              <th>End Time</th>
              <th>Execute Time (mm:hh:ss:ms)</th>
              <th>Included Groups</th>
              <th>Excluded Groups</th>
            </tr>
           </thead> 
           $Test_Case_Summary$
        </table>
        <table id="summary">
          <thead>
            <tr>
              <th>Class</th>
              <th>Method</th>
              <th>Start Time</th>
              <th>Execution Time (mm:hh:ss:ms)</th>
              <th>Parameter</th>
              <th>Reporter Message</th>
              <th>Exception Message</th>
            </tr>
          </thead> 
          $Test_Case_Detail$
        </table>
      </body>
  2. main-suite.xml : Add test listener in this TestNG suite XML file. We will introduce the test listener later.
    <listeners>
    	<listener class-name="com.dev2qa.example.testng.report.custom.CustomTestNGReporter" />
      </listeners>
  3. CustomTestNGReporter.java : This is the java class that implements the IReporter interface. It just needs to override the IReporter interface generateReport() method. You can see below code comments for more explanation. Besides you add the TestNG listener class in XML settings files, you can also add it in the java class use the below source code.
    // Create an instance of the TestNG listener.
    private CustomTestNGReporter listener = new CustomTestNGReporter();
    
    // Create an instance of the TestNG.
    TestNG testng = new TestNG();
    
    // Add the TestNG listener to the TestNG object.
    testng.addListener(listener);

    Package : com.dev2qa.example.testng.report.custom

    public class CustomTestNGReporter implements IReporter {
    	
    	//This is the customize emailabel report template file path.
    	private static final String emailableReportTemplateFile = "C:/WorkSpace/dev2qa.com/TestNGExampleProject/src/com/dev2qa/example/testng/report/custom/customize-emailable-report-template.html";
    	
    	@Override
    	public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
    		
    		try
    		{
    			// Get content data in TestNG report template file.
    			String customReportTemplateStr = this.readEmailabelReportTemplate();
    			
    			// Create custom report title.
    			String customReportTitle = this.getCustomReportTitle("Custom TestNG Report");
    			
    			// Create test suite summary data.
    			String customSuiteSummary = this.getTestSuiteSummary(suites);
    			
    			// Create test methods summary data.
    			String customTestMethodSummary = this.getTestMehodSummary(suites);
    			
    			// Replace report title place holder with custom title.
    			customReportTemplateStr = customReportTemplateStr.replaceAll("\\$TestNG_Custom_Report_Title\\$", customReportTitle);
    			
    			// Replace test suite place holder with custom test suite summary.
    			customReportTemplateStr = customReportTemplateStr.replaceAll("\\$Test_Case_Summary\\$", customSuiteSummary);
    			
    			// Replace test methods place holder with custom test method summary.
    			customReportTemplateStr = customReportTemplateStr.replaceAll("\\$Test_Case_Detail\\$", customTestMethodSummary);
    			
    			// Write replaced test report content to custom-emailable-report.html.
    			File targetFile = new File(outputDirectory + "/custom-emailable-report.html");
    			FileWriter fw = new FileWriter(targetFile);
    			fw.write(customReportTemplateStr);
    			fw.flush();
    			fw.close();
    			
    		}catch(Exception ex)
    		{
    			ex.printStackTrace();
    		}
    	}
    	
    	/* Read template content. */
    	private String readEmailabelReportTemplate()
    	{
    		StringBuffer retBuf = new StringBuffer();
    		
    		try {
    		
    			File file = new File(this.emailableReportTemplateFile);
    			FileReader fr = new FileReader(file);
    			BufferedReader br = new BufferedReader(fr);
    			
    			String line = br.readLine();
    			while(line!=null)
    			{
    				retBuf.append(line);
    				line = br.readLine();
    			}
    			
    		} catch (FileNotFoundException ex) {
    			ex.printStackTrace();
    		}finally
    		{
    			return retBuf.toString();
    		}
    	}
    	
    	/* Build custom report title. */
    	private String getCustomReportTitle(String title)
    	{
    		StringBuffer retBuf = new StringBuffer();
    		retBuf.append(title + " " + this.getDateInStringFormat(new Date()));
    		return retBuf.toString();
    	}
    	
    	/* Build test suite summary data. */
    	private String getTestSuiteSummary(List<ISuite> suites)
    	{
    		StringBuffer retBuf = new StringBuffer();
    		
    		try
    		{
    			int totalTestCount = 0;
    			int totalTestPassed = 0;
    			int totalTestFailed = 0;
    			int totalTestSkipped = 0;
    			
    			for(ISuite tempSuite: suites)
    			{
    				retBuf.append("<tr><td colspan=11><center><b>" + tempSuite.getName() + "</b></center></td></tr>");
    				
    				Map<String, ISuiteResult> testResults = tempSuite.getResults();
    				
    				for (ISuiteResult result : testResults.values()) {
    					
    					retBuf.append("<tr>");
    					
    					ITestContext testObj = result.getTestContext();
    					
    					totalTestPassed = testObj.getPassedTests().getAllMethods().size();
    					totalTestSkipped = testObj.getSkippedTests().getAllMethods().size();
    					totalTestFailed = testObj.getFailedTests().getAllMethods().size();
    					
    					totalTestCount = totalTestPassed + totalTestSkipped + totalTestFailed;
    					
    					/* Test name. */
    					retBuf.append("<td>");
    					retBuf.append(testObj.getName());
    					retBuf.append("</td>");
    					
    					/* Total method count. */
    					retBuf.append("<td>");
    					retBuf.append(totalTestCount);
    					retBuf.append("</td&gt;");
    					
    					/* Passed method count. */
    					retBuf.append("<td bgcolor=green>");
    					retBuf.append(totalTestPassed);
    					retBuf.append("</td>");
    					
    					/* Skipped method count. */
    					retBuf.append("<td bgcolor=yellow>");
    					retBuf.append(totalTestSkipped);
    					retBuf.append("</td>");
    					
    					/* Failed method count. */
    					retBuf.append("<td bgcolor=red>");
    					retBuf.append(totalTestFailed);
    					retBuf.append("</td>");
    					
    					/* Get browser type. */
    					String browserType = tempSuite.getParameter("browserType");
    					if(browserType==null || browserType.trim().length()==0)
    					{
    						browserType = "Chrome";
    					}
    					
    					/* Append browser type. */
    					retBuf.append("<td>");
    					retBuf.append(browserType);
    					retBuf.append("</td>");
    					
    					/* Start Date*/
    					Date startDate = testObj.getStartDate();
    					retBuf.append("<td>");
    					retBuf.append(this.getDateInStringFormat(startDate));
    					retBuf.append("</td>");
    					
    					/* End Date*/
    					Date endDate = testObj.getEndDate();
    					retBuf.append("<td>");
    					retBuf.append(this.getDateInStringFormat(endDate));
    					retBuf.append("</td>");
    					
    					/* Execute Time */
    					long deltaTime = endDate.getTime() - startDate.getTime();
    					String deltaTimeStr = this.convertDeltaTimeToString(deltaTime);
    					retBuf.append("<td>");
    					retBuf.append(deltaTimeStr);
    					retBuf.append("</td>");
    					
    					/* Include groups. */
    					retBuf.append("<td>");
    					retBuf.append(this.stringArrayToString(testObj.getIncludedGroups()));
    					retBuf.append("</td>");
    					
    					/* Exclude groups. */
    					retBuf.append("<td>");
    					retBuf.append(this.stringArrayToString(testObj.getExcludedGroups()));
    					retBuf.append("</td>");
    					
    					retBuf.append("</tr>");
    				}
    			}
    		}catch(Exception ex)
    		{
    			ex.printStackTrace();
    		}finally
    		{
    			return retBuf.toString();
    		}
    	}
    
    	/* Get date string format value. */
    	private String getDateInStringFormat(Date date)
    	{
    		StringBuffer retBuf = new StringBuffer();
    		if(date==null)
    		{
    			date = new Date();
    		}
    		DateFormat df = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
    		retBuf.append(df.format(date));
    		return retBuf.toString();
    	}
    	
    	/* Convert long type deltaTime to format hh:mm:ss:mi. */
    	private String convertDeltaTimeToString(long deltaTime)
    	{
    		StringBuffer retBuf = new StringBuffer();
    		
    		long milli = deltaTime;
    		
    		long seconds = deltaTime / 1000;
    		
    		long minutes = seconds / 60;
    		
    		long hours = minutes / 60;
    		
    		retBuf.append(hours + ":" + minutes + ":" + seconds + ":" + milli);
    		
    		return retBuf.toString();
    	}
    	
    	/* Get test method summary info. */
    	private String getTestMehodSummary(List<ISuite> suites)
    	{
    		StringBuffer retBuf = new StringBuffer();
    		
    		try
    		{
    			for(ISuite tempSuite: suites)
    			{
    				retBuf.append("<tr><td colspan=7><center><b>" + tempSuite.getName() + "</b></center></td></tr>");
    				
    				Map<String, ISuiteResult> testResults = tempSuite.getResults();
    				
    				for (ISuiteResult result : testResults.values()) {
    					
    					ITestContext testObj = result.getTestContext();
    
    					String testName = testObj.getName();
    					
    					/* Get failed test method related data. */
    					IResultMap testFailedResult = testObj.getFailedTests();
    					String failedTestMethodInfo = this.getTestMethodReport(testName, testFailedResult, false, false);
    					retBuf.append(failedTestMethodInfo);
    					
    					/* Get skipped test method related data. */
    					IResultMap testSkippedResult = testObj.getSkippedTests();
    					String skippedTestMethodInfo = this.getTestMethodReport(testName, testSkippedResult, false, true);
    					retBuf.append(skippedTestMethodInfo);
    					
    					/* Get passed test method related data. */
    					IResultMap testPassedResult = testObj.getPassedTests();
    					String passedTestMethodInfo = this.getTestMethodReport(testName, testPassedResult, true, false);
    					retBuf.append(passedTestMethodInfo);
    				}
    			}
    		}catch(Exception ex)
    		{
    			ex.printStackTrace();
    		}finally
    		{
    			return retBuf.toString();
    		}
    	}
    	
    	/* Get failed, passed or skipped test methods report. */
    	private String getTestMethodReport(String testName, IResultMap testResultMap, boolean passedReault, boolean skippedResult)
    	{
    		StringBuffer retStrBuf = new StringBuffer();
    		
    		String resultTitle = testName;
    		
    		String color = "green";
    		
    		if(skippedResult)
    		{
    			resultTitle += " - Skipped ";
    			color = "yellow";
    		}else
    		{
    			if(!passedReault)
    			{
    				resultTitle += " - Failed ";
    				color = "red";
    			}else
    			{
    				resultTitle += " - Passed ";
    				color = "green";
    			}
    		}
    		
    		retStrBuf.append("<tr bgcolor=" + color + "><td colspan=7><center><b>" + resultTitle + "</b></center></td></tr>");
    			
    		Set<ITestResult> testResultSet = testResultMap.getAllResults();
    			
    		for(ITestResult testResult : testResultSet)
    		{
    			String testClassName = "";
    			String testMethodName = "";
    			String startDateStr = "";
    			String executeTimeStr = "";
    			String paramStr = "";
    			String reporterMessage = "";
    			String exceptionMessage = "";
    			
    			//Get testClassName
    			testClassName = testResult.getTestClass().getName();
    				
    			//Get testMethodName
    			testMethodName = testResult.getMethod().getMethodName();
    				
    			//Get startDateStr
    			long startTimeMillis = testResult.getStartMillis();
    			startDateStr = this.getDateInStringFormat(new Date(startTimeMillis));
    				
    			//Get Execute time.
    			long deltaMillis = testResult.getEndMillis() - testResult.getStartMillis();
    			executeTimeStr = this.convertDeltaTimeToString(deltaMillis);
    				
    			//Get parameter list.
    			Object paramObjArr[] = testResult.getParameters();
    			for(Object paramObj : paramObjArr)
    			{
    				paramStr += (String)paramObj;
    				paramStr += " ";
    			}
    				
    			//Get reporter message list.
    			List<String> repoterMessageList = Reporter.getOutput(testResult);
    			for(String tmpMsg : repoterMessageList)				
    			{
    				reporterMessage += tmpMsg;
    				reporterMessage += " ";
    			}
    				
    			//Get exception message.
    			Throwable exception = testResult.getThrowable();
    			if(exception!=null)
    			{
    				StringWriter sw = new StringWriter();
    				PrintWriter pw = new PrintWriter(sw);
    				exception.printStackTrace(pw);
    				
    				exceptionMessage = sw.toString();
    			}
    			
    			retStrBuf.append("<tr bgcolor=" + color + ">");
    			
    			/* Add test class name. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(testClassName);
    			retStrBuf.append("</td>");
    			
    			/* Add test method name. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(testMethodName);
    			retStrBuf.append("</td>");
    			
    			/* Add start time. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(startDateStr);
    			retStrBuf.append("</td>");
    			
    			/* Add execution time. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(executeTimeStr);
    			retStrBuf.append("</td>");
    			
    			/* Add parameter. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(paramStr);
    			retStrBuf.append("</td>");
    			
    			/* Add reporter message. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(reporterMessage);
    			retStrBuf.append("</td>");
    			
    			/* Add exception message. */
    			retStrBuf.append("<td>");
    			retStrBuf.append(exceptionMessage);
    			retStrBuf.append("</td>");
    			
    			retStrBuf.append("</tr>");
    
    		}
    		
    		return retStrBuf.toString();
    	}
    	
    	/* Convert a string array elements to a string. */
    	private String stringArrayToString(String strArr[])
    	{
    		StringBuffer retStrBuf = new StringBuffer();
    		if(strArr!=null)
    		{
    			for(String str : strArr)
    			{
    				retStrBuf.append(str);
    				retStrBuf.append(" ");
    			}
    		}
    		return retStrBuf.toString();
    	}
    
    }
  4. Right-click main-suite.xml, click the Run As —> TestNG Suite menu item in the popup menu list to run it.
  5. After execution, you can see the custom-emailable-report.html in the left project panel under the test-output folder. Open it in your favorite web browser, you can see the customized TestNG reports.

3. Question & Answer.

3.1 How to include test method execution result in the TestNG customize report.

  1. I am looking for a method to include the test method executed result in the TestNG Email-able report. In my test case, the test method will return a UUID, an error code number, and an error message if the test method throws errors. And I want to include the test method returned UUID, error code number, and error message in the customized TestNG report with separate columns. I think this need to use the TestNG listener to implement it, but I do not know how to implement it. Please give me some help, thanks a lot.

3.2 How to customize the TestNG report summary section.

  1. The TestNG emailable report is very helpful, and I also learned from this article that I can customize the TestNG emailable report with a Listener class. But what I need is to add a more detailed information table in the report summary section to show the more detailed test method execution result in that area. Is there any method to implement this?
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

26 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
shubham
shubham

Did you find the solution. if ys then please share

Cruze
Cruze

Hi ,

When I run the XML suite using maven build pom.xml file with TestNG plugin and when there are test failures and maven build gets failed . The customized report is not getting generated. Please let me know if i should add any parameters in pom.xml file. Thanks in advance.

Sapan Sharma
Sapan Sharma

Hi,
When I am getting custom emailable report on my target folder but nothing is written into it. My code is same as above mentioned. Please help me .

Sisira
Sisira

Thanks for sharing,
This implemenation working fine with my project except failure details not passing,
please support on following enhancements?
1. How to get report for multiple browser executions
2. How to get screen captures on failures + why failure details not passing

nesli
nesli

i am getting lot of errors, it doesn’t work 🙁

swati
swati

Hi,

I am getting the same error. Can anyone help me with this

S P
S P

Need help….

How is the suite-parameter named “browserType” set? I don’t see it being set anywhere in JAVA or XML files

S Patel
S Patel

Thanks for the nice example.

I have one question, though. How is the suite’s parameter “browserType” passed to this listener. I checked the java code, and the TestNG XML files – and did not find a mention of the parameter.

nishadhi
nishadhi

When I implement the custom-emaiable-report.html now in my test output directory I am getting both the default emaialble-report.html and custom-emailable-report.html. But I want to have only the custom report in the output directory. Is there a possible way to programmatically disable the creation of default emailable-report.html

sreedevi
sreedevi

i am getting java.lang.IllegalArgumentException: Illegal group reference at below lines :

// Replace report title place holder with custom title.
customReportTemplateStr = customReportTemplateStr.replaceAll(“\\$TestNG_Custom_Report_Title\\$”, customReportTitle);

// Replace test suite place holder with custom test suite summary.
customReportTemplateStr = customReportTemplateStr.replaceAll(“\\$Test_Case_Summary\\$”, customSuiteSummary);

// Replace test methods place holder with custom test method summary.
customReportTemplateStr = customReportTemplateStr.replaceAll(“\\$Test_Case_Detail\\$”, customTestMethodSummary);

please help.

Ajay
Ajay
Reply to  sreedevi

Did you find any solution for this? I’m also getting same error

Arno
Arno
Reply to  sreedevi

If you change “customReportTitle” to “Matcher.quoteReplacement(customReportTitle)” it will work. Same thing with the other two lines.

For me it was also failing here, because my exception messages included $ character. By using Matcher.quoteReplacement these $ characters will not be treated as special characters.

Balazs Gyore
Balazs Gyore
Reply to  sreedevi

I did change line 55. From replaceAll. to replace, also change the regexp and it’s working now.

line 55 looks like that now:

customReportTemplateStr = customReportTemplateStr.replace(“$Test_Case_Detail$”, customTestMethodSummary);

venkat
venkat
Reply to  Balazs Gyore

Yes, super bro now working fine for exceptions also.

Rakesh kumar Kandanuru
Rakesh kumar Kandanuru

Tests and results in the report are not published in executed order. In TestNG XML i have mentioned preserve-order=”true”, still the same.
Is there a way i can publish test results(test methods names) in the order of execution.

vik
vik

Hi – is there a way to get the log messages in different rows instead of all getting congested in the same cell one after the other ?

and all the headers like execution time etc. are not very clearly visible. Can’t we add borders to those headers ?

Murali V
Murali V

Hi,

Thank you so much for code.It works fine for all test cases passed.

Some of the points identified while working on above script:

+ I am unable to get the customized template report when test case failed.
+ If any exception raises at script level, which doesn’t show in customized report.
+ It doesn’t show total count of methods in the particular project/product

Could you please help us in above points.

Thanks in Advance.

Best Regards,
Murali V.

viki
viki
Reply to  Murali V

I am also not getting failed test cases, its showing everything passed,

Murali V
Murali V

@Senthil, You can import statements below:

I am getting Customized report successfully for every run.Hope it helps for you.

import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Set;

import org.apache.batik.bridge.UserAgent;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.BrowserType;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.xml.XmlSuite;

Senthilkumar S Selvaraj
Senthilkumar S Selvaraj

please share the import statements

i am getting lot of errors

Senthilkumar S Selvaraj
Senthilkumar S Selvaraj

import java.awt.List;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Map;
import java.util.Set;

import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.Reporter;

what imports are missing here?

Senthilkumar S Selvaraj
Senthilkumar S Selvaraj

finally i fixed all the errors

running the testng suite , not generating the customer report

Please help

Ami
Ami

unable to automatically create custom emailable report html file , can you help ? I have followed every steps right

happyzhaosong
happyzhaosong
Admin
Reply to  Ami

What error have you encountered? Can you show the error messages?

shubham
shubham
Reply to  Ami

Did you get solution for this? because i am getting same problem

Murali V
Murali V

Nice Article…Very Helpful..:)

26
0
Would love your thoughts, please comment.x
()
x