This article describes how to run performance tests using either our .NET or Java Report Engines. Windward's engines are fast, very fast. The number one thing that we hear from our clients moving to Windward from competing solutions is the performance gain they receive. We realize that running performance tests under ideal conditions are not always reflected in real environments. We routinely update the performance PDF and this article with the latest versions of our Report Engine benchmarks. However, we also provide this performance test as a part of our Report Engine software package in both demo and production versions. We encourage you to test the Windward Report Engine in your own environment to see how it performs before you purchase.
The steps below will outline how we test our product internally and how you can run these tests for yourself in your own environment. As always, we are curious to know your results and are always interested to see how our product performs in different environments. If you would like assistance running these tests and are willing to share your results then please feel free to contact us at firstname.lastname@example.org and our Marketing team will reach out to you.
Download the Full Report here: Windward Engine Performance.pdf
- The Numbers - the latest performance analysis report for the current version of the Windward Engine
- The Measurement System - Here we will explain how we run our tests including hardware specifications, template size, output size and data source details.
- How do I run the Tests?
The below tests all use the files in WindwardPerformance.zip
- The OData Server contains the Northwind database. The database server is located in the cloud.
- The JSON file is a 1.27M file which contains a dump of the SQL Server Northwind database. The file is on the local machine.
- The SQL Server contains the Northwind SQL database. The database server is located on the test machine.
- The XML file is a 1.35M file which contains a dump of the SQL Server Northwind database. The file is on the local machine.
- The template is 2 pages long and the report it generates is 17 pages long and is run 50 times.
For this latest version of the Windward Report Engine, we tested the performance by creating a report 50 times and averaging the result. Performance results (number of pages created per second [PPS]) depended on which engine is used, what sort of data source is used, and the number of simultaneous threads called by the Windward application.
- Processor: Intel® Xeon® E5520 @ 2.27GHz 8 virtual processors
- Memory: 32.0 GB
- Operating system: Windows Server 2016 Standard 64-bit
- The Windward AutoTag template is a Microsoft® Word document two pages long. It contains text, images, headers, footers, multiple loops, and equations.
- The generated report is a 17-page Word document (DOCX). The number of pages generated per second is not affected by the output format. Both DOCX and PDF reports were generated in the same amount of time. The report is run 50 times for each test.
For each test of the Java Report Engine, we ran with four data sources (JSON, OData, SQL, and XML) and four different thread counts (2, 4, 6 and 8 threads). The exact results are listed in the table below.
*Running on an Intel® Xeon® E5520 @ 2.27GHz with 32 GB RAM on Windows Server 2016 Standard (64-bit)
These Report Engine performance tests assume the worst case performance in several aspects, so your performance could easily be faster.
In this test:
- We open the template each time the Engine generates a report. When you run the same template in bulk through the Windward Engine, it scans the template one time and caches it. This results in a speed increase as the time to read the template are removed from each successive run.
- The data sources are loaded again each time, which may not be necessary in your use case.
- Both DOCX and PDF reports were generated in the same amount of time.
We ran these tests on a 4 core system. Typically we recommend that you do not exceed 2 times the number of total cores on your target system. You will notice in most of the tests above that performance starts to plateau after 8 threads (2 x 4 cores = 8 threads). This is a result of a phenomena called processor thrashing which is a state where the processor performs less productive work and more work swapping between all simultaneously running processes or threads. This yields diminishing returns over time as the processor is spending more time trying to manage all simultaneous processes than actually processing them.
The -performance:1234 option is available in the sample RunReport application shipped with both the .NET and Java engines. We strongly recommend that you run this program using your sample sets of templates and data on the server you will use.
One of the key factors to the performance of generating reports is based on the hardware being used. The greater the processing power and the larger memory your system has will translate into improved performance of generating reports. That being said, here are some important stats to know in regards to generating reports.
There are a couple of generalizations we feel safe making about Windward Reports:
- Its performance time is linear. It will take 10x longer to run 100,000 reports than it takes to run 10,000 reports.
- For every CPU core, you should generally have a maximum 2 threads generating reports.
- Its memory usage is constant. Running 1,000 reports uses the same amount of memory as running 100,000 reports.
- DOCX, XLSX, PPTX, PDF, HTML, and TXT reports all take about the same time.
- File I/O is a big part of the time required.
- Logic commands are the most expensive computationally. IF/ELSE and SWITCH/CASE statements, when used excessively, will slow down your report generation (this is only noticeable if you have more than 150 tags of this type in a single template)
Navigate to the directory where you installed the Java Engine
- Windows: C:\Program Files (x86)\Windward Studios\Windward Java Engine\
- Unix: path where you extracted the Windward JAR files
Make sure all Windward JAR files are present in your CLASSPATH
- Windows: this is done automatically, no further action needed
- Unix: you will have to alter your CLASSPATH to include all JAR files in the Windward /lib directory
Change to the "test" directory
- Windows: C:\Program Files (x86)\Windward Studios\Windward Java Engine\test
- Unix: /Your_Windward_Path/test
Run the command below
When executing the RunReport program from the command line of the Windward Engine with no parameters you will see the following output below will all options that can be used.