Splunk Add-Ons

It was never clear what my responsibilities were to be with Fiserv. I was asked if I could create a Splunk add-on to provide the missing ‘monitoring maintenance’ functionality. Not having any Splunk or Python experience, I was hoping for on-premise expertise. I soon learned the knowledge necessary to fulfill this request did not exist. I learned enough Python and Splunk to design a solution. Unknown at the time was the version of Python used at the core of Splunk. It was 15 years old. Needless to say, in the technology sector that was worthy of a Jurassic title. Of course it was a major obstacle as several helpful libraries were unavailable. Suffice it to say, I was able to craft a unique solution.

In parallel I recommended they improve upon the email alert that Splunk provides.

PROBLEMS:

• Splunk’s email philosophy is to provide a platform to deliver emails with a variety of options.
  This, though flexible, severely limited the HTML capabilities; such as not allowing embedded links, not color coding alerts, and providing limited information.
• Microsoft Outlook is severely handicapped as it is using an HTML rendering engine over 20 years old.
  Furthermore, nearly all formatting must be done ‘in-line’ and all indenting done by embedding tables
  within tables.
• Splunk, for its own part, is using a Python version (ver. 2.7) which is 10 years old.
• Splunk’s email data delivery is two dimensional tables. This leads to allot of page real estate
  consumed by redundant data.
• Python reorders data when it converted to a dictionary.
• Current Fiserv email alerts have different line items for each metric’s reporting period. Since,
  the only difference is the timestamp (which is consistently 5 min prior), the smart solution is to
  combine line items separating the two metrics as ‘current’ and ‘previous’

SOLUTION: To provide a more intuitive operational flow, a fuller breadth HTML needed to be realized.
It then had to be delivered to the Microsoft Outlook client. A decision was made to create
a ‘templating’ solution; allowing a choice of ‘templates’ in the application UI framework. The
templates are HTML files. It’s important to note there are TWO templates:

1. Page template: This is the overall HTML page complete with as much formatting as possible in the
   CSS section. This allows for ‘page’ level reformatting, styling, or omissions.
   This has a string (ie ‘data‘) embedded in it which is replaced by the final formatted data.
2. Line template: The actual data received from splunk is ‘line’ by line( HTML: ).
   Each column ( HTML: ) is acted upon depending on it’s name and relationship to the pivot key.
   There are two distinctly different column formats which are addressed in different fashions:
   • Splunk – this has threshold arguments within the data. For this data format we needed to design
     KV store ‘lookup’ methodology into the SPL for the thresholds. This requires creating KV store
     items for all KPIs. Within the SPL, the threshold arguments are looked up and placed within the data
     just as they’re stored in the KV store (ie arg1 and arg2). This order is esential to determine
     if the threshlds are ‘descending’, ‘ascending’, or ‘equal’. If arg2 is greater than arg1, they
     are ‘ascending’; meaning that if a KPI exceeds arg2 it is ‘critical’. A quick example would be
     ‘percent disk consumption’. If it exceeds arg2, it is ‘critical’. However, if arg2 is less than
     arg1, then it is considered ‘descending’. Let’s take the same disk scenario except reverse the
     KPI to ‘percent disk remaining’. Since the arguments are in descending fashion, the alert will
     trigger if the ‘percent disk remaining’ is less than arg2. If they are equal the check is ignored.
   • ITSI – this has the current state of the KPI information. Since all the treshold analysis is done
     within the SPL the only thing left for the htmlEmail application to do was to assign ‘colors’ to
     best represent the ‘state’.
   A tiered data structure allows the removal of redundant data to be placed in a tiered format.
   The user is required to select a ‘pivot’ field in the configuration UI. This is the field by
   which the resulting data pivots. In other words, all columns to the left of it are removed from
   the table and placed in a tier. All columns (including the pivot field) remain in the table.