UpGuard Core can be used to alert on many events such as detected changes, policy failures or user access events. This guide covers how to set up event logging into splunk from scratch.

Overview

Monitoring your infrastructure for changes, policy failures and other system events can be a daunting task. Trying to find the balance between coverage and “alert fatigue” can be difficult to find. Additionally, different tools are more applicable to different types and levels of alerting volume and specificity.

This guide walks through how to post UpGuard events from the Control > Events feed into your splunk instance. We will be using Splunk’s HEC (HTTP Event Collector) data source.

Please note that posting events from UpGuard into splunk only requires a HTTP POST event type. For more information on a more complete Splunk Integration that enables pulling data back into UpGuard from Splunk, please visit our guide on the Splunk Integration.

Setting up Splunk to Accept Events

First we need to configure Splunk to accept HTTP based events. Navigate to Settings > Data > Data Inputs.

w600shadow

From the Local inputs section, select HTTP Event Collector.

w400shadow

Here you will see a list of tokens enabled for potentially other feeds into Splunk. It’s a good idea to create a new token for the UpGuard Integration so that you have per-service access control in place. To create a new token, click the New token button.

w300shadow

Name the token something descriptive and optionally leave a short description outlining the use case of this HTTP data source.

w600shadow

On the input settings screen, you can select Automatic for the data type to begin with and configure later once we have a complete integration set up on both the Splunk and UpGuard sides.

After reviewing the tokens settings you should be presented with a new token. You can also view the value for this token on the Data Inputs > HTTP Event Collector page.

w500shadow

If this is the first time you are creating a HEC token, then you may need to enable the token in the Global Settings screen. To access Global Settings, click the Global Settings button next to the New Token button and then set All Tokens to Enabled and then click Save.

w500shadow

Make sure you have access to the HEC token generated here for the UpGuard configuration steps below.

Adding REST Integration to UpGuard

To add a new integration to post events into Splunk, navigate to Control > Integrations and click Add Integration.

w500shadow

Select the REST Endpoint and then fill out the fields that identify your Splunk instance. Below the we have used the example value myserver.splunk.com - please replace this with either your hosted or on-prem/local Splunk hostname.

Since the HEC integration into splunk uses the following raw HTTP Header format

Authorization: Splunk 1234...

you only need to specify the value of the Authorization header (as shown below).

w400shadow

Depending on your Splunk instance, you may like to switch between http and https for the URL protocol and if on-prem and using a self-signed certificate you may like to Bypass Certificate Validation. Depending on your Splunk version, it may also be advisable to add the following header to the Custom Headers section:

Content-Type: application/json

Clicking Done will attempt to send the text message to your Splunk instance and if the test event was posted successfully, the integration should now appear in your list of integrations.

w600shadow

You now have a basic UpGuard to Splunk event integration set up that can be applied to a range of different Event Action pairs.

Posting Events to Splunk

Now that you’ve configured UpGuard to be able to post events to Splunk, you will need to create one or more Event Actions to post real events into your Splunk feed. Here we are going to focus more on the Action side of the Event Action feature pair. For more information on filtering events and creating custom event views, please visit our guide on Events.

Once you have a defined Event View of the type of events you want to post into Splunk, navigate to the Actions tab of the event view and select Add Action. Then select Send a message to REST Endpoint.

Give the action a descriptive name, for example, the type of event that this action is posting for and where it is posting to. Here we have assigned a PCI compliance policy to our PCI Environment node group and have created a custom Event View for when this policy fails on any node. For the REST Endpoint Integration, select the integration you created in the previous section.

For the Body section, you may need to play around with the exact structure of the JSON payload to fit well with the expected structure and fields required by the Splunk HEC endpoint. In the example below, we’re posting an event object under the "event" key in the top level content. For example, we’re logging the name of the node using the `` substitution against the "name" field.

w500shadow

Clicking Done will create the new action and you should see the action listed in this Event View’s Action tab list. If your Event View has seen a particular type of event before, then you can test this particular action on the most recent instance of event filtered into this Event View. Click the ... to the right of the action and then click Test most recent.

w600shadow

What Next?

For more information on Event types and Event views, please visit our guide on Events.

For more information on other types of Actions, please visit our guide on Event Actions to get started.

UpGuard also has a more complete Splunk Integration where data from Splunk can be accessed from and pulled into UpGuard. For more information, please visit our guide on the Splunk Integration.

Tags: events splunk