UpGuard Core can be used to validate deployments as part of your existing CI/CD process. This guide outlines the technical specifics of how to set up Core to monitor your deployments via the API.

Overview

UpGuard Core can be used as validation tool to confirm that a deployment went successfully and surface changes that occurred between the pre-deployment and post-deployment state of your servers. This guide provides a practical guide to integrating UpGuard into your build process, using the API to:

  • Initiate a node scan pre-deployment to snapshot the state of your server
  • Create or update a policy based on the build artifacts from your CI/CD tool that you are about to deploy
  • Initiate a node scan post-deployment that will provide both a reference point and allow you to validate the state of your server post-deployment
  • How to set up alerting when a policy failure occurs
  • How to pull policy results in PDF format back into your CI/CD tool as a deployment validation artifact

Here we are going to use the use case of confirming that files deployed to a server exist after deployment, and that their checksums, as they appear in the build tool, are correct. You can naturally extend the techniques discussed here to other configuration types modified during a deployment such as environment variables and or registry keys.

Prerequisites

This guide assumes that the servers that you deploy to are already being actively scanned by UpGuard or at least have their connection and credential information configured correctly to be able to scan.

Create a Post-Deployment Policy

We will assume that you are able to gather the filenames and MD5 checksums of your deployed files from your CI/CD tool. We will use a CSV file for this example, but you may be able to gather this data programatically.

The following curl command creates a fresh blank policy.

$ curl -X POST -H "Content-Type: application/json" --user <api_key:sec_key> https://app.upguard.com/api/v2/policies.json -d '{"policy":{"name":"Deployment Policy 123"}}'
{
  "id": 123,
  "name": "Deployment Policy 123",
  "description": "",
  "created_by": 40,
  "updated_by": 40,
  ...
}
$

This will give you back the ID of an empty policy ready to populate with policy checks. Next we are going to iterate over the file paths and content checksums in a given CSV file and create two checks for each file: a presence check to make sure the file was actually deployed to the target machine; and a content checksum check to ensure that the deployed file matches the build checksum from our CI/CD tool.

The following ruby script reads through a given CSV file and adds the two checks to the known policy ID from the previous step. Here we are making use of the Policies > Add File Check endpoint.

require 'httparty'

api_key = "1234"
sec_key = "5678"
policy_id = 123

content = File.read("files.txt")
content.split(/\r?\n/).each do |line|
  tokens = line.split(",")
  next if tokens.count != 2

  file_path = tokens[0]
  checksum  = tokens[1]

  # add a presence check
  response = HTTParty.post("https://my-app.upguard.com/api/v2/policies/#{policy_id}/add_file_check.json?section=Files&file_path=#{file_path}&attr=present&check=present&expected=true",
                           :headers => {
			     "Authorization" => "Token token=\"#{api_key}#{sec_key}\""
			   })
  if response.code.to_s == "204"
    puts "Accidentally tried to double add the same check"
  elsif response.code.to_s == "201"
    puts "Check added"
  else
    raise response.body
  end

  # add the checksum check
  response = HTTParty.post("https://my-app.upguard.com/api/v2/policies/#{policy_id}/add_file_check.json?section=Files&file_path=#{file_path}&attr=checksum&check=equals&expected=#{checksum}",
                           :headers => {
			     "Authorization" => "Token token=\"#{api_key}#{sec_key}\""
			   })
  if response.code.to_s == "204"
    puts "Accidentally tried to double add the same check"
  elsif response.code.to_s == "201"
    puts "Check added"
  else
    raise response.body
  end
  
end

You can then attach this policy to a known node group using the following bash/curl script.

#!/bin/bash

NODE_GROUP_ID=123
POLICY_ID=456

curl -X POST --user <api_key>:<sec_key> https://my-app.upguard.com/api/v2/node_groups/$NODE_GROUP_ID/add_policy_version.json?policy_id=$POLICY_ID&version=latest

Initiate a Pre and Post Deployment Node Scan

Preparation

Initiating a scan is a simple process of:

  • (optionally) looking up the node’s ID based on the name, hostname or an external ID of the node, or from membership in a node group.
  • using the node ID to initiate a node scan, which returns a Job ID
  • polling for the completion status of the node scan job

The pre and post deployment scans should be identical in their execution. That is, your deployment should be structured with these intermediate steps:

  • scan your device and wait for the scan to complete
  • update the associated policy based on the about-to-be deployed assets
  • deploy
  • scan your device and wait for the scan to complete

Ensuring you have the node ID

In most cases, you will be deploying code or assets to fixed servers or devices, so you could encode the IDs of the nodes into your build process. However, you may have a dynamic environment and prefer to generically look up the ID of a node based on its name or hostname (as may be the case with auto scaling group cloud nodes), or based on membership in a particular node group.

To look up a node’s ID based on the name, hostname or an external ID, you can make use of the Node > Look Up endpoint. Below is a sample curl command that can retrieve the node ID for a node in my account called prod01.

$ curl --user <api_key:sec_key> https://app.upguard.com/api/v2/nodes/lookup.json?name=prod01
{"node_id":123}
$

To lookup a set of node IDs based on a known node group you can use the Node Group > Nodes endpoint. Below is a curl command example that lists nodes (and their IDs) currently in my EC2 Instances node group with ID 123.

$ curl --user <api_key:sec_key> https://app.upguard.com/api/v2/node_groups/123/nodes.json
[
  {"id":43,"name":"node-1", ...},
  {"id":44,"name":"node-2", ...}
]
$

Perform a node scan

Once you have the ID of the node you want to scan, you can initiate a scan of that node using the Nodes > Start Scan endpoint. Below is a curl example of how you would initiate a node scan for the node with ID 123. Here you’ll notice we’re making use of the optional label field to help tag this node scan with a little more context.

$ curl --user <api_key:sec_key> https://app.upguard.com/api/v2/nodes/123/start_scan.json?label=pre-deployment
{"job_id":1337}
$

The API endpoint above will return the ID of the node scan job, which you can poll to pause your build process until the scan is complete. To track the status of a job, you can poll the Jobs > Show endpoint, looking particularly at the returned status field.

At the time of writing, a job status of 2, 3 or -1 shows that the job has finished (one way or another). Please consult the Jobs > Show page for the most up to date description and list of values for the returned status property. Below is a curl example of how to extract out the job status as a single value that could be placed into a loop with a 1 or 10 second sleep, for example:

$ curl --user <api_key:sec_key> https://app.upguard.com/api/v2/jobs/123.json | jq '.["status"]'
2
$

Once a node scan job has finished executing, you can continue with the next step in your build process.

Policy Failure Alerting

If a node has policies attached to it (via membership in a node group), then the policies are automatically executed against a node’s new scan data when a scan completes. This is another reason why it’s important to make sure that you update your associated policy after the pre-deployment node scan and before the post-deployment node scan.

To be alerted when this policy detects a failure, you should set up an Event View with the following query:

type=Policy Ran AND variables.success=false AND variables.policy=<The Name of My Policy>

You can then associated an appropriate Action to this event happening. For more information on creating Event Actions please visit our guide on Event Actions.

Pull Policy Results via the API

To access the latest policy results for a particular policy and node, you can make use of the Policies > Latest Results endpoint. If you have confirmed programatically that the scan job has finished, you should be able to place the following code as the next step in the build process. Below is an example of how to pull the latest policy results for a given node and policy.

$ export NODE_ID=123
$ export POLICY_ID=456
$ curl --user <api_key:sec_key> https://app.upguard.com/api/v2/policies/$POLICY_ID/latest_results.json?include_policy_results=true&node_id=$NODE_ID
{
   ... the JSON payload containing policy result information ...
}
$

By default, this endpoint returns policy result data in JSON format, but you can also specify to download the data in PDF format. Below is an example of how to perform the same action as the above example, but with PDF output. (Note the path extension changes from .json to .pdf).

$ export NODE_ID=123
$ export POLICY_ID=456
$ curl --user <api_key:sec_key> -o report.pdf https://app.upguard.com/api/v2/policies/$POLICY_ID/latest_results.pdf?include_policy_results=true&node_id=$NODE_ID
$ ls report.pdf
report.pdf
$

What Next?

In addition to validating deployed assets are correctly installed, it is recommended that you also construct and assign a policy that validates the more static parts of your environment, such as installed packages, users and groups.

Tags: api