AWS CloudTrail
Amazon Web Services (AWS)
Support Notice
AWS Cloudtrail logs are ingested via S3 buckets. While it is possible to ingest logs after setting up only an S3 bucket, this is highly inefficient and can lead to performance issues. The reason is that the S3 API does not list "last modified", forcing us to download the entire list of files stored in a bucket. Instead, S3 will "publish" an "object created" event to an SQS queue. CyberSift filebeat is then setup to read these events from that queue and download the event instead of "polling" the S3 Bucket ARN and listing all files.
[AWS Documentation]
Requirements
Step | Instruction | Notes |
|---|---|---|
1 | Log in to the AWS Management Console | Navigate to the AWS Management Console and sign in to your AWS account. |
2 | Open the SQS Service | Open the SQS Service: Click on "Services" in the top left corner and then select "SQS" under the "Application Integration" section. Alternatively, you can type "SQS" in the search bar and select it. |
3 | Create a New Queue | In the Amazon SQS dashboard, click the "Create Queue" button. |
Configure the SQS Queue
You'll need to provide some information to configure the queue
Field | Instruction |
|---|---|
Queue Name | Enter a name for your queue. The name must be unique within your AWS account and follow the naming rules (letters, numbers, hyphens, and underscores allowed) |
Standard Queue | Make sure the "Standard Queue" option is selected. This is the default queue type. |
Other Settings | You can configure additional settings like visibility timeout, message retention period, and other advanced settings as needed. These settings determine how messages are handled in the queue. |
Create the Queue | Once you've configured the queue settings, click the "Create Queue" button at the bottom of the page. |
Configure S3 bucket
Amazon S3 provides a feature called "S3 events" that allows you to trigger actions or notifications based on events that occur within your S3 buckets. The SQS Queue must be in the same region as the AWS S3 bucket. These events can include object creation, deletion, or modification, and they can be used to automate various tasks.
For the purposes of the CyberSift SIEM, only "object creation" events are of interest to us
Create Bucket: Create a S3 bucket if it already does not exist.
Set Permissions: Replace the access policy that's attached to the queue. In it, provide your Amazon SQS ARN, source bucket name:
{ "Version": "2012-10-17", "Id": "example-ID", "Statement": [ { "Sid": "example-statement-ID", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": [ "SQS:SendMessage" ], "Resource": "SQS-queue-ARN", // <---- change this to your newly created SQS ARN "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1" // <-- change this to your S3 ARN } } }, { "Sid": "allow-receive-message", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::account-id:user/example_user" // <---- change this to your newly created SQS ARN }, "Action": "SQS:ReceiveMessage", "Resource": "SQS-queue-ARN" // <---- change this to your newly created SQS ARN } ] }Define Event Notifications: Navigate to the bucket properties and configure event notifications. Select "Object Creation".
Select the Destination: Choose the destination for these events. In our case, we want to send events to an SQS queue.
Test the Configuration: To verify your setup, you can upload an object to the S3 bucket and check if it triggers a message in the SQS queue.
NiFi Provisioning
Cybersift Support Required
At this point in time, Cybersift will do the NiFi provisioning for you.
Please provide the below details to support@cybersift.io
The SQS ARN in URL format:
https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}Credentials for the AWS user:
Please provide both the below details using a service such as https://onetimesecret.com/en/
Access Key ID
Secret Access Key
tags:
Amazon Web Services (AWS) Logs Setup
NiFi AWS Setup
S3 Bucket
Cloudtrail