Amazon Web Services (AWS)

Support Notice

AWS Cloudtrail logs are ingested via S3 buckets. While it is possible to ingest logs after setting up only an S3 bucket, this is highly inefficient and can lead to performance issues. The reason is that the S3 API does not list "last modified", forcing us to download the entire list of files stored in a bucket. Instead, S3 will "publish" an "object created" event to an SQS queue. CyberSift filebeat is then setup to read these events from that queue and download the event instead of "polling" the S3 Bucket ARN and listing all files.
[AWS Documentation]

Requirements

Step

Instruction

Notes

1

Log in to the AWS Management Console

Navigate to the AWS Management Console and sign in to your AWS account.

2

Open the SQS Service

Open the SQS Service: Click on “Services” in the top left corner and then select “SQS” under the “Application Integration” section. Alternatively, you can type “SQS” in the search bar and select it.

3

Create a New Queue

In the Amazon SQS dashboard, click the “Create Queue” button.

Configure the SQS Queue

You’ll need to provide some information to configure the queue

Field

Instruction

Queue Name

Enter a name for your queue. The name must be unique within your AWS account and follow the naming rules (letters, numbers, hyphens, and underscores allowed)

Standard Queue

Make sure the “Standard Queue” option is selected. This is the default queue type.

Other Settings

You can configure additional settings like visibility timeout, message retention period, and other advanced settings as needed. These settings determine how messages are handled in the queue.


Create the Queue

Once you’ve configured the queue settings, click the “Create Queue” button at the bottom of the page.


Configure S3 bucket

Amazon S3 provides a feature called “S3 events” that allows you to trigger actions or notifications based on events that occur within your S3 buckets. The SQS Queue must be in the same region as the AWS S3 bucket. These events can include object creation, deletion, or modification, and they can be used to automate various tasks.

For the purposes of the CyberSift SIEM, only “object creation” events are of interest to us

  1. Create Bucket: Create a S3 bucket if it already does not exist.

  2. Set Permissions: Replace the access policy that's attached to the queue. In it, provide your Amazon SQS ARN, source bucket name:

    {
        "Version": "2012-10-17",
        "Id": "example-ID",  
        "Statement": [
            {
                "Sid": "example-statement-ID",
                "Effect": "Allow",
                "Principal": {
                    "Service": "s3.amazonaws.com"
                },
                "Action": [
                    "SQS:SendMessage"
                ],
                "Resource": "SQS-queue-ARN", // <---- change this to your newly created SQS ARN
                "Condition": {
                    "ArnLike": {
                        "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1" // <-- change this to your S3 ARN
                    }
                }
            },
            {
                "Sid": "allow-receive-message",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::account-id:user/example_user" // <---- change this to your newly created SQS ARN
                },
                "Action": "SQS:ReceiveMessage",
                "Resource": "SQS-queue-ARN" // <---- change this to your newly created SQS ARN
            }
        ]
    }
  3. Define Event Notifications: Navigate to the bucket properties and configure event notifications. Select “Object Creation”.

  4. Select the Destination: Choose the destination for these events. In our case, we want to send events to an SQS queue.



  5. Test the Configuration: To verify your setup, you can upload an object to the S3 bucket and check if it triggers a message in the SQS queue.


NiFi Provisioning

Cybersift Support Required

At this point in time, Cybersift will do the NiFi provisioning for you.

Please provide the below details to support@cybersift.io


  • The SQS ARN in URL format:

    https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}

  • Credentials for the AWS user:

    Please provide both the below details using a service such as https://onetimesecret.com/en/

    • Access Key ID

    • Secret Access Key


tags:

  • Amazon Web Services (AWS) Logs Setup

  • NiFi AWS Setup

  • S3 Bucket

  • Cloudtrail