Machine Learning Alerts
Introduction
Machine Learning alerts are dynamic alerts which adjust based on historic data. These alerts are not visible in the “Rules” page. These type of alerts are available for filebeat-*, office365-*, windows-* and auditbeat-*.
These types of alerts can be identified by the tags field within the alerts, typically recognized by the ml keyword[s].
Fundamentals of Machine Learning Rules
Machine Learning Alerts fundamentally work on deviations from a previously known baseline, with the severity adjusted based on persistence of the data, until a baseline is established.
Two useful fields are expected and current. These fields show us what the ML algorithm expected the approximate baseline to be vs the current actual result.
As we can see in the below example, we can immediately see why the alert triggered. Apart from the below, the ML algorithm also compares previously unseen items.
Example
Since these alerts are dynamic, they cannot be tuned using static logic format as we see in Rule Management document.
Example Scenario
A Volume Anomaly alert from a local IP address
10.0.23.1scanning multiple ports with theComputerNamefieldPRTG-Server. On most environments, this is expected activity, given that services such as PRTG is within their remit to do such activities.
Optimization and Tuning
To tune these alerts, we need to tell the Machine Learning algorithm to ignore specific datasets. We can achieve this by adding an ignore tag.
We will use JMESPath and JSON. In the below demonstration, we will be using NiFi, http://[YOUR SIEM IP]:9900/nifi/
Syslog
Step 1 - Choosing your Log Source
When you first access NiFi, you will see some of the pre-configured process groups currently available in your implementation, the below process groups are being pushed to Sentio SIEM. We call it the backbone of the SIEM.
At this point, determine the log source of the alert you wish to filter out. These can be determined by the name of the alert such Win_Success_Login_Volume (Windows)
So we will go to the “Windows Process Group”
The above may vary from environment to environment
Step 2 - Creating your Flow
In this example we’re going to use the above example. We are going to tag any every firewall traffic with the ComputerName PRTG-Server.
JMESPath Query:
ComputerName=='PRTG-Server'
Then you are going to create a new processor called “
JMESPathMatcher”Add a “Property” in the
JMESPathMatcherGive it a friendly name such as
ml ignore(Can be anything)Add your JMESPath query
Step 3 - Adding the ml_ignore tag
After Step 2, add a
JSONPatchto your pipelineGo into
JSONPatchand add the following property to theJSON Patch
[ { "op": "add", "path": "/cybersift_ml", "value": "ignore" } ]Step 4 - Connecting Everything Together
Now, it’s time to connect our pipeline. Switch off all the processors before you start working on your pipeline
Redirect the “Success” pipe from
PutElasticCSto our newly createdJMESPathMatcher. And select the property “Success” as it was before.Then, connect the
JMESPathMatcherprocessor to theJSONPatchwith the relationshipYour friendly name(Property).Connect the
JMESPathMatcherto thePutElasticCSwith the relationshipsFailedandUNMATCHEDConnect your
JSONPatchprocessor to thePutElasticCSprocessor with the relationshipsFailureandSuccessSwitch everything back on
Diagram of Pipeline
graph TD
ListenSyslog["ListenSyslog"]
SyslogParser["SyslogParser"]
JmesPathMatcher["JmesPathMatcher"]
JsonPatch["JsonPatch"]
PutElasticCS_Left["PutElasticCS"]
PutElasticCS_Right["PutElasticCS"]
ListenSyslog --> |"invalid, success"| SyslogParser
SyslogParser --> |"Success"| JmesPathMatcher
SyslogParser --> |"Alerts"| PutElasticCS_Right
JmesPathMatcher --> |"ml ignore"| JsonPatch
JmesPathMatcher --> |"FAILED, UNMATCHED"| PutElasticCS_Left
JsonPatch --> |"failure, success"| PutElasticCS_LeftStep 5 - Check to See if it worked!
If you did the above correctly, you should be able to see a new field cybersift_ml with the value ignore.
Now the Machine Learning algorithm will ignore any logs with your predefined parameters.
tags:
Filtering out for Machine Learning alerts
Volume Anomalies exclusions
ml_ignore