Benchmarks are an incredibly powerful industry standard way to ensure your nodes are robust. This guide gives a walkthrough of the best practices on how to set up benchmark reporting to get the most out of this feature.


Benchmarks provide a way to validate that important configuration settings on your nodes are secure and robust. This guide provides a high level, step-by-step guide to getting the most out of benchmarks, and how to systematically build up your benchmarking capabilities with UpGuard Core.

Exploring and Planning

Before creating node groups or assigning benchmarks, it is a good idea to explore which benchmarks you might like to apply and to which nodes. To see which benchmarks are available, navigate to Control > Policies and click on the Public filter on the left panel.

Some points to consider when planning:

  • Choose a single benchmark that can be applied to a small group of your nodes to begin with. For example, you could choose to try parts of the CIS Red Hat 7 benchmark on all of your RHEL 7 nodes.
  • If the benchmarks come in a Level 1, Level 2, etc, start with the Level 1 checks.
  • Click into a benchmark to look at the individual checks, see if there is a section that you’d like to focus on first. For example, the RHEL 6 Level 1 benchmark has some good user access control checks in section 5.

Creating Dynamic Node Groups

The next step is to create some dynamic node groups based on the types of benchmarks you want to assign. Dynamic node groups allow you to group nodes based on common properties, and here we are going to create groups based on operating system and version. The benefit of dynamic node groups here is that as your inventory of nodes evolve over time, nodes that match these properties will always be automatically assigned into the relevant node groups.

For more information on how to create dynamic node groups, please visit our guide on Node Groups. In this guide we will be using complex dynamic queries. For a more complete background on these types of queries, please visit our guide on Complex Dynamic Group Queries.

As an example, I’d like to create a group called “Windows 2012 Servers”. I could create a new node group with the following dynamic node group query:

inventory:osfamily WITH value:Windows EXACT AND inventory:operatingsystem WITH value:2012

Assigning Benchmarks to Node Groups and Scheduling

Next, you should choose a benchmark to assign to one of your node groups. This will allow you to both customize the benchmark to meet your requirements, but also allow you to see how a benchmark can be executed across some of your nodes. For more information on how to assign and schedule benchmark reports, please visit our guide on How to schedule benchmark reports.

To begin with, you may not want to enable a scheduled report to run. That is, when you are configuring which parts of a benchmark to initially run, you can run a benchmark across a node or node group in an ad-hoc fashion. Later, once you are happy with a particular benchmark configuration, you should schedule your benchmarks to be run, for example, weekly over your node groups.

Once a benchmark is assigned to a node group, you can also customize which checks you want to run to begin with, and modify the parameters of any value based check. For more information on customizing benchmarks, please visit our guide on Customizing Benchmarks. A recommended practice is to disable all checks in a benchmark, then re-enable a particular section that you can focus on first. For example, the user access and password policy sections of many OS level benchmarks are the most popular, so enabling those sections first is a great place to start.


When a benchmark is executed against a set of nodes in a node group it generates a Benchmark Report. Report results can be accessed by navigating to Reports > Benchmark Reports. This page lists all benchmarks that have been executed against nodes and their pass, fail and error rates.


Clicking into a report will show a breakdown view of that benchmark against all nodes over a selected period of time. The left panel allows you to filer down to specific node groups, nodes or environments and adjust the time span to view results for.


Check results fall into 4 categories:

  • Pass: the check ran and the node passed this check,
  • Fail: the check ran and the node failed this check,
  • Error: the check failed to execute at all, and
  • Ignored: the check isn’t applicable in the context of the node.

Taking Action

If you have customized your benchmark down to just a few checks, then the best place to start is to filter the benchmark report down to just errors and failures. You can then either visually action each failing or erroring item from the UI, or export the filtered results to PDF or CSV.

Erroring checks are the most important type of result to correct first as they are preventing the actual benchmark check from giving a pass or fail result. After that, failing checks should be worked through, check by check rather than node by node so that you can focus on the context of a particular check, then apply it across the board.

Once all of your results are either passing or not applicable (ignored) then loop back to the benchmark, enable another section and see how your nodes perform under that section of checks. Remember that there might be some sections of a benchmark that you never enable as they are not applicable to your business’s requirements.

I’ve Inherited Benchmarks, How Do I Get Started?

This guide has focused on getting you up to speed with benchmarks and node groups from scratch. However, sometimes you may have inherited the resposibility of managing the benchmark reporting from a colleague. This section outlines a systematic approach to coming up to speed with the current deployment and helps you identify where the current process can be improved.

Which benchmarks are in use?

The best place to start is the Benchmark Reports page, which can be accessed via Reports > Benchmark Reports. Aside from showing you how your current benchmarks are performing across the board, this list gives you a view of which benchmarks are actually currently in use.

If you then click into one of the benchmark reports to bring up the details view, you can see if the benchmark is still actively running on nodes, or it has run in the past and has since been disabled. To do this, locate the Options panel on the left side and change the Time Span option to Past Week. If you see some results then you know that the benchmark is currently running across nodes on a schedule. If you don’t see any results, try Past 2 Weeks for example. If you can’t see any results appearing for a reasonable amount of time, then this benchmark might have been active in the past, but has been disabled along the way.

Which nodes have these benchmarks assigned to them?

For both active and stagnant benchmarks, you can see which nodes groups have this benchmark attached by selecting the Filter By option on the left panel as Node Group and then viewing the list of node groups that are included in the adjoining drop down. It might be worth checking if the nodes in these node groups are still actively scanning, if the node group actually has any nodes in it anymore, or if the benchmark isn’t applicable to these nodes.

There are too many failures?

One of the most daunting things new and existing benchmark report users experience is a report with tens or hundreds of failures. The first step here is to navigate to the benchmark itself via Control > Policies, locate the benchmark and find whole sections you want to disable for the time being. For example, user access and password controls are one of the most popular sections, so disabling all other sections is a good start. For more information on disabling sections, please see our guide on Customizing Benchmarks.

After limiting the scope of the benchmark to a smaller set of checks, you should run the benchmark again to generate a fresh report. This will allow you to view just the latest results. If it helps in grouping, sorting and digesting the result data for action, filter down to failures and errors and then export to CSV.

How do I view scheduled benchmark reports?

Once benchmarks are configured in a desirable state then should be scheduled to run on a regular basis. Most users run benchmark reports weekly, but you can configure to be more frequent, if required. To view existing scheduled benchmark reports, navigate to Control > Job Schedule. This should list all types of scheduled jobs. You can use the Type heading to filter for Benchmark type scheduled jobs. You can temporarily disable a scheduled benchmark report by using the drop down arrow on the right side of each scheduled job and clicking Disable.

If you would like to configure your own scheduled benchmark reports, please visit our guide on How to schedule benchmark reports.