Architecture and Design

Cybersift Architecture

The below diagram is a simplified, high level view of the architecture behind the CyberSift SIEM.



Internet Access Requirements

Cybersift Public Access Requirements:

  • CyberSift requires access to the following IPs and ports:

    • 51.91.75.82

      • Port 80

      • Port 443

    • 13.81.122.195

      • Ports 80

      • Port 443

      • Port 7443

Just like any unix system, CyberSift also requires access to yum and python “pypi” repositories for regular OS updates

Common Architecture

TCO: Total Cost of Ownership

Typically architectures start with smaller deployments in which one server is an “all-in-one” which hosts the entire SIEM system including the NiFi data processing pipeline, the ML server, and the OpenSearch backend. Once log ingestion saturates the underlying hardware, usually one of the following architectures is considered for scaling up.

Servers split by role

In this architecture setup the heaviest roles are split onto two separate servers. The heaviest resource consumers are typically

  • NiFi data processing pipeline

  • OpenSearch backend

By splitting these two roles onto separate machines, they are no longer contending with each other and can ingest a far larger amount of data each. For most deployments, with appropriately spec’d hardware, this architecture tends to be the best performance to cost ratio. Cold data can be stored on a seperate, storage optimised server (typically an NFS NAS)

There is no data redundancy or high availability in this setup. The focus here is best performance for minimum TCO (Total Cost of Ownership)



Redundant Architecture with High Availability

In the following architectures we illustrate introducing High Availability and Data Redundancy

Completely Redundant Setup

In the below architecture a completely redundant setup is shown. Note the following:

  • The use of an external load balancer to detect if one of the NiFi servers goes down

  • Multiple NiFi Servers

  • Multiple OpenSearch server (or clusters) allowing for redundant hot data

  • Multiple storage servers allowing for redundant cold data


Hot and Cold Redundant Data

In this scenario the TCO is lowered by:

  • keeping hot and cold data redundancy but removing the redundant NiFi servers, and so the external LB becomes unnecessary

  • failover to a new NiFi server is manual and usually takes a couple hours if appropriate backups are taken


Only Hot Redundant Data

In this architecture TCO is further reduced by removing cold storage data redundancy, and leaving only hot storage redundancy

Cold data is simple ZIP’d text files which can be offloaded to any cloud storage service like AWS S3 or Cloudflare R2


Scaling Out

The below architecture illustrates the “scaling out” concept where different log sources are routed via config to different NiFi servers, and each of these servers feeds into a common (or independent) OpenSearch cluster. This allows maximum flexibility as once the capacity of the current setup is reached, simply adding another NiFi server and more OpenSearch Data nodes allows to provision more capacity

Again be wary of your data redundancy and HA requirements




tags:

  • High level architecture

  • High availability

  • Cybersift Design and Architecture