# Machine Learning Alerts

## Introduction

Machine Learning alerts are dynamic alerts which adjust based on historic data. These alerts are not visible in the “Rules” page. These type of alerts are available for `filebeat-*`, `office365-*`, `windows-*` and `auditbeat-*`. 


These types of alerts can be identified by the `tags` field within the alerts, typically recognized by the `ml` keyword[s]. 


## Fundamentals of Machine Learning Rules

Machine Learning Alerts fundamentally work on deviations from a previously known baseline, with the severity adjusted based on persistence of the data, until a baseline is established.


Two useful fields are `expected` and `current`. These fields show us what the ML algorithm *expected* the approximate baseline to be vs the *current* actual result. 


As we can see in the below example, we can immediately see why the alert triggered. Apart from the below, the ML algorithm also compares previously unseen items.

 ![](https://outline.cybersift.io/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzL2MwNWNhZDYzLTA0ZmEtNGFmZC04NjMwLTI3MjNjZDYwZWZkNS9lOTc1Yzg3MS1hOTBhLTRjODktOWYyZS02YTcwOWY1MGQ0NzUvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTUxNDcyMDAsImV4cCI6MTc5NTIzMzYwMH0.x7hXeM4QLqvP18NzVzF2Qb205f3MOaFkigotinsoGc8 " =742x354")

### Example 

Since these alerts are dynamic, they cannot be tuned using static logic format as we see in [Rule Management](/doc/810f7403-5102-4831-a7fd-d55c3553c4ae) document. 

> ***Example Scenario ***
>
> *A Volume Anomaly alert from a local IP address* `*10.0.23.1*` *scanning multiple ports with the* `*ComputerName*` *field* `*PRTG-Server*`*. On most environments, this is expected activity, given that services such as PRTG is within their remit to do such activities.*

## Optimization and Tuning

To tune these alerts, we need to tell the Machine Learning algorithm to *ignore* specific datasets. We can achieve this by adding an ==ignore== tag. 

We will use JMESPath and JSON. In the below demonstration, we will be using NiFi, `http://[YOUR SIEM IP]:9900/nifi/`


+++++++
## Syslog

### Step 1 - Choosing your Log Source

When you first access NiFi, you will see some of the pre-configured process groups currently available in your implementation, the below process groups are being pushed to Sentio SIEM. We call it the backbone of the SIEM.


At this point, determine the log source of the alert you wish to filter out. These can be determined by the name of the alert such `Win_Success_Login_Volume` (Windows)


So we will go to the “Windows Process Group”

 ![The above may vary from environment to environment ](https://outline.cybersift.io/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzL2MwNWNhZDYzLTA0ZmEtNGFmZC04NjMwLTI3MjNjZDYwZWZkNS9lYzkwZTZkMS03OTgyLTRlMTYtYmZiZi00ZmU0NzUyMDRmYjEvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTUxNDcyMDAsImV4cCI6MTc5NTIzMzYwMH0.LJ7EEBVr0op1F9Bc6XCNquZ8fI7DTnpUYlwFiv1AaBE " =670x320")

### Step 2 - Creating your Flow 

In this example we’re going to use the above example. We are going to tag any every firewall traffic with the `*ComputerName PRTG-Server*`*.* 

> JMESPath Query: 
>
> `ComputerName=='PRTG-Server'`


1. Then you are going to create a new processor called “`JMESPathMatcher`” 
2. Add a “Property” in the `JMESPathMatcher`
3. Give it a friendly name such as `ml ignore` (Can be anything)
4. Add your JMESPath query

 ![](https://outline.cybersift.io/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzL2MwNWNhZDYzLTA0ZmEtNGFmZC04NjMwLTI3MjNjZDYwZWZkNS9kNDkyY2ZjZS0yMTY5LTQ4YjctOTRlNy1iNTczNTY4ODRmZjYvYnJhdmVfVkRrdlgzNDY0US5naWYiLCJ0eXBlIjoiYXR0YWNobWVudCIsImlhdCI6MTc5NTE0NzIwMCwiZXhwIjoxNzk1MjMzNjAwfQ.Jr3ssjQSKiujTbPZ_xyuPDKkgvB054a2ZAmrAwsFY6E " =697x333")


### Step 3 - Adding the `ml_ignore` tag


1. After Step 2, add a `JSONPatch` to your pipeline
2. Go into `JSONPatch` and add the following property to the `JSON Patch` 

```json
[ { "op": "add", "path": "/cybersift_ml", "value": "ignore" } ]
```


### Step 4 - Connecting Everything Together


1. Now, it’s time to connect our pipeline. Switch off all the processors before you start working on your pipeline
2. Redirect the “Success” pipe from `PutElasticCS` to our newly created `JMESPathMatcher`. And select the property “Success” as it was before.
3. Then, connect the `JMESPathMatcher` processor to the `JSONPatch` with the relationship `Your friendly name` (Property).
4. Connect the `JMESPathMatcher` to the `PutElasticCS` with the relationships `Failed` and `UNMATCHED`
5. Connect your `JSONPatch` processor to the `PutElasticCS` processor with the relationships `Failure` and `Success`
6. Switch everything back on

 ![](https://outline.cybersift.io/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzL2MwNWNhZDYzLTA0ZmEtNGFmZC04NjMwLTI3MjNjZDYwZWZkNS8yN2NkMmJmYS02ZDQyLTQwNTMtYWIwMi1mNDc0NWU5YTZlN2QvYnJhdmVfajEzZUVQb1BpQS5naWYiLCJ0eXBlIjoiYXR0YWNobWVudCIsImlhdCI6MTc5NTE0NzIwMCwiZXhwIjoxNzk1MjMzNjAwfQ.kI4MaZHaTpIUPmCLTwzPY5YyjHS9ENHHqrH7yikEUk4 " =722x345")


+++++
Diagram of Pipeline

```mermaid
graph TD
    ListenSyslog["ListenSyslog"]
    SyslogParser["SyslogParser"]
    JmesPathMatcher["JmesPathMatcher"]
    JsonPatch["JsonPatch"]
    PutElasticCS_Left["PutElasticCS"]
    PutElasticCS_Right["PutElasticCS"]

    ListenSyslog --> |"invalid, success"| SyslogParser
    SyslogParser --> |"Success"| JmesPathMatcher
    SyslogParser --> |"Alerts"| PutElasticCS_Right
    JmesPathMatcher --> |"ml ignore"| JsonPatch
    JmesPathMatcher --> |"FAILED, UNMATCHED"| PutElasticCS_Left
    JsonPatch --> |"failure, success"| PutElasticCS_Left
```

+++++

### Step 5 - Check to See if it worked!

If you did the above correctly, you should be able to see a new field `cybersift_ml` with the value `ignore`.

 ![](https://outline.cybersift.io/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzL2MwNWNhZDYzLTA0ZmEtNGFmZC04NjMwLTI3MjNjZDYwZWZkNS8wMjMyN2U3My0zNWNjLTQxNmEtYTA0MC04ZGI0Nzg1YWE0M2MvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTUxNDcyMDAsImV4cCI6MTc5NTIzMzYwMH0.-9heuVCgGSPIKagVLpeMygQRNqoAh1Y7gqH2NM2XDmg " =648x293")

Now the Machine Learning algorithm will ignore any logs with your predefined parameters. 

+++++++

---


:::info
tags:

* Filtering out for Machine Learning alerts
* Volume Anomalies exclusions
* `ml_ignore`

:::

---

**Documents**

- [Logs and Architecture](https://outline.cybersift.io/s/sentio-siem/doc/logs-and-architecture-acvUiDgvI5)
- [Sentio](https://outline.cybersift.io/s/sentio-siem/doc/sentio-KuYAbF5grr)
- [ISO27001:2022](https://outline.cybersift.io/s/sentio-siem/doc/iso270012022-7S5IzvF6a0)
- [FAQ](https://outline.cybersift.io/s/sentio-siem/doc/faq-vVyRNyNb60)