Architecture and Design
Cybersift Architecture
The below diagram is a simplified, high level view of the architecture behind the CyberSift SIEM.
Internet Access Requirements
Cybersift Public Access Requirements:
CyberSift requires access to the following IPs and ports:
51.91.75.82
Port 80
Port 443
13.81.122.195
Ports 80
Port 443
Port 7443
Access to the following URLs is also necessary (note that no static IP is supported due to limitations):
Just like any unix system, CyberSift also requires access to yum and python “pypi” repositories for regular OS updates
Common Architecture
TCO: Total Cost of Ownership
Typically architectures start with smaller deployments in which one server is an “all-in-one” which hosts the entire SIEM system including the NiFi data processing pipeline, the ML server, and the OpenSearch backend. Once log ingestion saturates the underlying hardware, usually one of the following architectures is considered for scaling up.
Servers split by role
In this architecture setup the heaviest roles are split onto two separate servers. The heaviest resource consumers are typically
NiFi data processing pipeline
OpenSearch backend
By splitting these two roles onto separate machines, they are no longer contending with each other and can ingest a far larger amount of data each. For most deployments, with appropriately spec’d hardware, this architecture tends to be the best performance to cost ratio. Cold data can be stored on a seperate, storage optimised server (typically an NFS NAS)
There is no data redundancy or high availability in this setup. The focus here is best performance for minimum TCO (Total Cost of Ownership)
Redundant Architecture with High Availability
In the following architectures we illustrate introducing High Availability and Data Redundancy
Completely Redundant Setup
In the below architecture a completely redundant setup is shown. Note the following:
The use of an external load balancer to detect if one of the NiFi servers goes down
Multiple NiFi Servers
Multiple OpenSearch server (or clusters) allowing for redundant hot data
Multiple storage servers allowing for redundant cold data
Hot and Cold Redundant Data
In this scenario the TCO is lowered by:
keeping hot and cold data redundancy but removing the redundant NiFi servers, and so the external LB becomes unnecessary
failover to a new NiFi server is manual and usually takes a couple hours if appropriate backups are taken
Only Hot Redundant Data
In this architecture TCO is further reduced by removing cold storage data redundancy, and leaving only hot storage redundancy
Cold data is simple ZIP’d text files which can be offloaded to any cloud storage service like AWS S3 or Cloudflare R2
Scaling Out
The below architecture illustrates the “scaling out” concept where different log sources are routed via config to different NiFi servers, and each of these servers feeds into a common (or independent) OpenSearch cluster. This allows maximum flexibility as once the capacity of the current setup is reached, simply adding another NiFi server and more OpenSearch Data nodes allows to provision more capacity
Again be wary of your data redundancy and HA requirements
tags:
High level architecture
High availability
Cybersift Design and Architecture