Eran Alshech |
Data processing and storage pose many challenges for complex organisations migrating to the cloud. Security teams struggle to maintain effective cybersecurity processes while staying within operational budgets.
However, when CyberProof’s analyst team searched for a way to help clients optimise data processing and storage costs where zero trust was a security requirement, we uncovered the significant potential of Microsoft’s Azure Data Explorer (ADX).
A fully managed platform-as-a-service solution for big data analytics, ADX parses, tags and processes data at incredible speeds. It provides real-time analysis on large volumes of data streaming from applications, websites, internet of things devices, and more. We decided to move the long-term retention of data logs, and the processing of custom collected logs, to ADX. This enables us to manage massive quantities of data for clients at a significantly lower cost.
Today, all our clients who work with Microsoft Azure Sentinel – a cloud-native security information event management solution – use the CyberProof Log Collector (CLC). The CLC was specifically designed to enhance the functionality of Azure Sentinel by enabling logs from custom sources to be collected and ingested for security analytics. We use this solution together with ADX to facilitate data ingestion and processing.
When CyberProof first started using ADX for an enterprise client, we conducted a proof of concept where we measured data samples every 12 hours for 48 hours, recording more than one billion events within this period.
In the first step, we focused on formatting, cleansing and tagging the data and combining fields, which reduced data storage requirements by 46 per cent. This is because formatting was enabled within ADX via a mapping rule on ingestion, ensuring only relevant data was retained.
In the second stage, we transferred relevant data from ADX to Sentinel and mapped out the log sources. We only took events that were relevant to managed detection and response requirements. The combination of both aspects of our work led to a 60 per cent decrease in the volume of stored data.
By integrating ADX with CLC to filter and process data, CyberProof helps large enterprise clients to use Azure Sentinel more efficiently and significantly cut costs, while delivering faster cyberthreat detection and response services to reduce the risk of cyberattacks.
Eran Alshech is chief technology officer at CyberProof
This article was originally published in the Autumn 2022 issue of Technology Record. To get future issues delivered directly to your inbox, sign up for a free subscription.