Editing article

Title

Summary

Content

<h3>Securing Anthos Workloads With Chronicle Backstory — A comprehensive approach</h3>Implementation process, threat detection strategies, and remediation workflows to get started.<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*RpN2FuNx2P5E8wai.jpg" /></figure>How cool would it be to have a single glass-pane to secure Anthos as well as GKE clusters on the go? Helping implement automation to be granular security findings before it escalates?The world of multiple clouds and hybrid systems poses distinct security challenges. The Anthos platform from Google Cloud makes it easier to deploy applications in a variety of environments, but securing these workloads calls for a thorough strategy. With the recent integration of Chronicle Backstory, a threat detection and investigation tool, with Anthos, we can now leverage extended detection and response (XDR) capabilities throughout this intricate environment. This blog delves deeply into the technical aspects of using Chronicle Backstory to secure Anthos workloads, including data ingestion, threat hunting queries, and utilizing the integrations that are already built in.<h3>Implementing Backstory with Anthos</h3>Chronicle Backstory uses pre-existing data sources to identify potential threats. Our main goal in integrating it with Anthos will be to consume information from two main sources:Cloud Audit Logs: Your GCP projects’ administrative activity, including that of Anthos clusters, is recorded in these logs. GKE Logs: The Kubernetes Engine (GKE) logs offer valuable information about the activities of containers and possible security incidents that occur in your Anthos workloads.<h4>Configuring cloud audit logging for Backstory</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e6I_qtdeaCqpVhuqPvmYXg.gif" /><figcaption><a href="https://images.app.goo.gl/dM7fV7uT2ZPAD6ZD9">https://images.app.goo.gl/dM7fV7uT2ZPAD6ZD9</a></figcaption></figure>We start with enabling cloud audit logs for our GCP project as starters, run the command below to add an IAM binding to the Chronicle ingestor service account.<pre>gcloud projects add-iam-policy-binding PROJECT_ID \ --member=&quot;serviceAccount:chronicle-ingestor@backstory.iam.gserviceaccount.com:user&quot; \ --role=&quot;roles/logging.logWriter&quot;</pre>Now that we have a log ingestor in place we can go ahead and create a logging sink to backstory. With this setup, Anthos cluster activity from Cloud Audit Logs can be ingested by Backstory.<pre>gcloud logging sinks create backstory-sink \ --log-filter=&quot;resource.type=cluster&quot; \ --destination=&quot;projects/PROJECT_ID/sinks/backstory-sink&quot; \ --destination-type=backstory</pre>Enabling GKE logging for backstory includes two steps:<ul><li>Enable Stackdriver Kubernetes Engine Monitoring for your Anthos cluster.</li><li>Create a sink within Stackdriver Monitoring to export logs to Backstory:</li></ul><pre># Configure the Stackdriver Logging Agent apiVersion: logging.k8s.io/v2 kind: LoggingDeployment metadata: name: backstory-agent spec: sinkRefs: - name: &quot;backstory-sink&quot; namespace: &quot;logging&quot; # Replace with your Backstory ingestion endpoint outputDataset: &quot;projects/your-project-id/datasets/anthos-logs&quot; # Filters to select relevant container logs selectors: - expression: &quot;resource.type=k8s_container&quot; --- # Define the Stackdriver Logging Sink to route logs to Backstory apiVersion: logging.k8s.io/v2 kind: LoggingSink metadata: name: backstory-sink spec: # Replace with your Backstory ingestion credentials secretRef: name: backstory-credentials destination: # Configure secure HTTPS destination for Backstory destination: &quot;https://your-backstory-endpoint.google.com/v2/ingest&quot; # Define the log format for Backstory ingestion outputFormat: &quot;json&quot;</pre>With this setup, Backstory receives GKE logs from your Anthos workloads that show container activity.<h3>Threat Detection With Backstory Queries</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*mkKqN1LfvAgBx4jO.jpg" /></figure>Using Chronicle Query Language, Backstory is highly proficient in threat detection (CQL). Here are a few instances:<h4>Detecting Suspicious APIs</h4>This query finds instances of unauthorized users within Anthos clusters making API calls to the Secrets API.<pre>SELECT resource.labels.cluster_name, timestamp, protoPayload.methodName, protoPayload.request.principalEmail FROM audit_log WHERE protoPayload.methodName LIKE &#39;%/v1/secrets%&#39; AND NOT protoPayload.request.principalEmail LIKE &#39;%admin@yourdomain.com&#39; ORDER BY timestamp DESC;</pre><h4>Unusual Container Activity Detection</h4>This query finds containers that crash frequently in Anthos workloads, which may be a sign of suspicious activity.<pre>SELECT resource.labels.cluster_name, container.name, timestamp, jsonPayload.reason FROM container WHERE jsonPayload.reason LIKE &#39;%CrashLoopBackOff%&#39; ORDER BY timestamp DESC;</pre><pre># Find container executions with unusual resource usage SELECT process.name, container.name, resource.usage.cpu.usage_in_cores, resource.usage.memory.usage_in_bytes FROM logs WHERE resource.type = &quot;k8s_container&quot; AND resource.usage.cpu.usage_in_cores &gt; (AVG(resource.usage.cpu.usage_in_cores) + 3 * STDDEV(resource.usage.cpu.usage_in_cores)) OR resource.usage.memory.usage_in_bytes &gt; (AVG(resource.usage.memory.usage_in_bytes) + 3 * STDDEV(resource.usage.memory.usage_in_bytes)) ORDER BY resource.usage.cpu.usage_in_cores DESC, resource.usage.memory.usage_in_bytes DESC</pre><h4>Suspicious Login Attempts</h4>This query looks for login attempts made during a specified time period from odd locations. Additional filtering options include user accounts or unsuccessful login attempts.<pre>SELECT user_email, source_ip, timestamp FROM events WHERE event_type = &#39;login.attempt&#39; AND timestamp &gt;= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1d) AND geo_country(source_ip) NOT IN (&#39;US&#39;, &#39;GB&#39;) -- Replace with trusted countries</pre><h4>Potential Lateral Movements</h4>This query looks for user activity that may indicate lateral movement across clusters involving multiple GCP resources. Events can be narrowed down by particular resource kinds or activities that take place within a constrained time frame.<pre>SELECT user_email, resource_type, resource_name, timestamp FROM events WHERE event_type IN (&#39;resource.create&#39;, &#39;resource.access&#39;) GROUP BY user_email, resource_type, resource_name HAVING COUNT(*) &gt; 5 -- Adjust threshold based on expected activity</pre><h4>Unusual File Access</h4>This query looks for file access events coming from source IP addresses or unexpected user accounts. Additional filters can be applied based on particular file types or attempts to access data after business hours.<pre>SELECT user_email, source_ip, file_path, timestamp FROM events WHERE event_type = &#39;file.access&#39; AND (user_email NOT IN (&#39;admin@example.com&#39;, &#39;service_account@project.com&#39;) -- Trusted accounts OR geo_country(source_ip) NOT IN (&#39;US&#39;)) -- Trusted location </pre><h3>Remediation?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x9a8vmb4z1_JhlwL" /></figure>For automated remediation, Backstory integrates with a number of tools. Here are some examples using Cloud Functions (because that’s what I found closest at hand).Isolating infected workloads on the cluster requires a cloud function to be triggered via Backstory findings after which we can add the function:<pre>def isolate_workload(data, context): # Extract cluster name and pod details from Backstory alert. cluster_name = data[&#39;resource&#39;][&#39;labels&#39;][&#39;cluster_name&#39;] pod_name = data[&#39;container&#39;][&#39;name&#39;] # Use Kubernetes API to cordon the infected node. from kubernetes import client, config config.load_kube_config() v1 = client.AppsV1Api() v1.patch_namespaced_daemon_set( &quot;kube-system&quot;, &quot;kube-dns&quot;, body={&quot;spec&quot;: {&quot;template&quot;: {&quot;spec&quot;: {&quot;taints&quot;: [{&quot;effect&quot;: &quot;NoSchedule&quot;, &quot;key&quot;: &quot;infected&quot;}]}}}} )</pre>With the addition of a taint to stop additional pod scheduling, this Cloud Function automatically isolates the compromised node.Further, you can implement something like:<pre>def remediate_backstory_finding(data, context): &quot;&quot;&quot;Cloud Function triggered by Backstory detection.&quot;&quot;&quot; # Parse the Pub/Sub message data pubsub_message = json.loads(data) backstory_finding = json.loads(pubsub_message[&quot;data&quot;]) # Extract relevant details from the detection finding_name = backstory_finding[&quot;findingName&quot;] threat_type = backstory_finding[&quot;externalSystems&quot;][0][&quot;threatType&quot;] # Implement logic for remediation based on threat type if threat_type == &quot;MALWARE&quot;: # Example: Isolate the affected workload print(f&quot;Isolating workload associated with finding: {finding_name}&quot;) # Replace with your specific isolation workflow (e.g., API call to Anthos) elif threat_type == &quot;PORT_SCAN&quot;: # Example: Block suspicious IP addresses print(f&quot;Blocking suspicious IP addresses from finding: {finding_name}&quot;) # Replace with your specific IP blocking workflow (e.g., firewall rule update) else: print(f&quot;Unrecognized threat type: {threat_type} for finding: {finding_name}&quot;) # Implement logic for handling unknown threats or sending notifications </pre>A Pub/Sub message with the Backstory detection details in JSON format initiates the function. After parsing the message data, the threat type and finding name are extracted. The function carries out particular remediation actions based on the type of threat. Including examples of workload isolation for malware and IP blocking for port scans in this case.<h3>Conclude</h3>These are but a few simple instances. Depending on your unique Anthos environment, security posture, and the threats you want to find, you’ll need to modify the queries. As the integration develops, it’s also advised to refer to the official Backstory documentation for the most recent syntax and functionalities.<h3>Get in touch??</h3><a href="https://linktr.ee/imranfosec">imranfosec | Instagram | Linktree</a><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcf4a9a3a78b" width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/securing-anthos-workload-with-chronicle-backstory-a-comprehensive-approchg-fcf4a9a3a78b">Securing Anthos Workload With Chronicle Backstory — A comprehensive approchg</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author

Link

Published date

Image url

Feed url

Guid

Hidden blurb

--- !ruby/object:Feedjira::Parser::RSSEntry
title: Securing Anthos Workload With Chronicle Backstory — A comprehensive approchg
url: https://medium.com/google-cloud/securing-anthos-workload-with-chronicle-backstory-a-comprehensive-approchg-fcf4a9a3a78b?source=rss----e52cf94d98af---4
author: Imran Roshan
categories:
- ai
- cloud-computing
- gcp-security-operations
- google-cloud-platform
- cybersecurity
published: 2024-05-13 02:17:57.000000000 Z
entry_id: !ruby/object:Feedjira::Parser::GloballyUniqueIdentifier
 is_perma_link: 'false'
 guid: https://medium.com/p/fcf4a9a3a78b
carlessian_info:
 news_filer_version: 2
 newspaper: Google Cloud - Medium
 macro_region: Blogs
rss_fields:
- title
- url
- author
- categories
- published
- entry_id
- content
content: '<h3>Securing Anthos Workloads With Chronicle Backstory — A comprehensive
 approach</h3>Implementation process, threat detection strategies, and remediation
 workflows to get started.<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*RpN2FuNx2P5E8wai.jpg"
 /></figure>How cool would it be to have a single glass-pane to secure Anthos
 as well as GKE clusters on the go? Helping implement automation to be granular security
 findings before it escalates?The world of multiple clouds and hybrid systems
 poses distinct security challenges. The Anthos platform from Google Cloud makes
 it easier to deploy applications in a variety of environments, but securing these
 workloads calls for a thorough strategy. With the recent integration of Chronicle
 Backstory, a threat detection and investigation tool, with Anthos, we can now leverage
 extended detection and response (XDR) capabilities throughout this intricate environment.
 This blog delves deeply into the technical aspects of using Chronicle Backstory
 to secure Anthos workloads, including data ingestion, threat hunting queries, and
 utilizing the integrations that are already built in.<h3>Implementing Backstory
 with Anthos</h3>Chronicle Backstory uses pre-existing data sources to identify
 potential threats. Our main goal in integrating it with Anthos will be to consume
 information from two main sources:Cloud Audit Logs: Your
 GCP projects’ administrative activity, including that of Anthos clusters, is recorded
 in these logs. GKE Logs: The Kubernetes Engine (GKE) logs offer
 valuable information about the activities of containers and possible security incidents
 that occur in your Anthos workloads.<h4>Configuring cloud audit logging for
 Backstory</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e6I_qtdeaCqpVhuqPvmYXg.gif"
 /><figcaption><a href="https://images.app.goo.gl/dM7fV7uT2ZPAD6ZD9">https://images.app.goo.gl/dM7fV7uT2ZPAD6ZD9</a></figcaption></figure>We
 start with enabling cloud audit logs for our GCP project as starters, run the command
 below to add an IAM binding to the Chronicle ingestor service account.<pre>gcloud
 projects add-iam-policy-binding PROJECT_ID \ --member=&quot;serviceAccount:chronicle-ingestor@backstory.iam.gserviceaccount.com:user&quot;
 \ --role=&quot;roles/logging.logWriter&quot;</pre>Now that we have a log
 ingestor in place we can go ahead and create a logging sink to backstory. With this
 setup, Anthos cluster activity from Cloud Audit Logs can be ingested by Backstory.<pre>gcloud
 logging sinks create backstory-sink \ --log-filter=&quot;resource.type=cluster&quot;
 \ --destination=&quot;projects/PROJECT_ID/sinks/backstory-sink&quot; \ --destination-type=backstory</pre>Enabling
 GKE logging for backstory includes two steps:<ul><li>Enable Stackdriver Kubernetes
 Engine Monitoring for your Anthos cluster.</li><li>Create a sink within Stackdriver
 Monitoring to export logs to Backstory:</li></ul><pre># Configure the Stackdriver
 Logging Agent apiVersion: logging.k8s.io/v2 kind: LoggingDeployment metadata: name:
 backstory-agent spec: sinkRefs: - name: &quot;backstory-sink&quot; namespace:
 &quot;logging&quot; # Replace with your Backstory ingestion endpoint outputDataset:
 &quot;projects/your-project-id/datasets/anthos-logs&quot; # Filters to select
 relevant container logs selectors: - expression: &quot;resource.type=k8s_container&quot; --- #
 Define the Stackdriver Logging Sink to route logs to Backstory apiVersion: logging.k8s.io/v2 kind:
 LoggingSink metadata: name: backstory-sink spec: # Replace with
 your Backstory ingestion credentials secretRef: name: backstory-credentials destination:
 # Configure secure HTTPS destination for Backstory destination:
 &quot;https://your-backstory-endpoint.google.com/v2/ingest&quot; # Define
 the log format for Backstory ingestion outputFormat: &quot;json&quot;</pre>With
 this setup, Backstory receives GKE logs from your Anthos workloads that show container
 activity.<h3>Threat Detection With Backstory Queries</h3><figure><img alt=""
 src="https://cdn-images-1.medium.com/max/600/0*mkKqN1LfvAgBx4jO.jpg" /></figure>Using
 Chronicle Query Language, Backstory is highly proficient in threat detection (CQL).
 Here are a few instances:<h4>Detecting Suspicious APIs</h4>This query finds
 instances of unauthorized users within Anthos clusters making API calls to the Secrets API.<pre>SELECT
 resource.labels.cluster_name, timestamp, protoPayload.methodName,
 protoPayload.request.principalEmail FROM audit_log WHERE protoPayload.methodName
 LIKE &#39;%/v1/secrets%&#39; AND NOT protoPayload.request.principalEmail LIKE
 &#39;%admin@yourdomain.com&#39; ORDER BY timestamp DESC;</pre><h4>Unusual Container
 Activity Detection</h4>This query finds containers that crash frequently in Anthos
 workloads, which may be a sign of suspicious activity.<pre>SELECT resource.labels.cluster_name,
 container.name, timestamp, jsonPayload.reason FROM
 container WHERE jsonPayload.reason LIKE &#39;%CrashLoopBackOff%&#39; ORDER
 BY timestamp DESC;</pre><pre># Find container executions with unusual resource usage SELECT
 process.name, container.name, resource.usage.cpu.usage_in_cores, resource.usage.memory.usage_in_bytes FROM
 logs WHERE resource.type = &quot;k8s_container&quot; AND resource.usage.cpu.usage_in_cores
 &gt; (AVG(resource.usage.cpu.usage_in_cores) + 3 * STDDEV(resource.usage.cpu.usage_in_cores)) OR
 resource.usage.memory.usage_in_bytes &gt; (AVG(resource.usage.memory.usage_in_bytes)
 + 3 * STDDEV(resource.usage.memory.usage_in_bytes)) ORDER BY resource.usage.cpu.usage_in_cores
 DESC, resource.usage.memory.usage_in_bytes DESC</pre><h4>Suspicious Login Attempts</h4>This
 query looks for login attempts made during a specified time period from odd locations.
 Additional filtering options include user accounts or unsuccessful login attempts.<pre>SELECT
 user_email, source_ip, timestamp FROM events WHERE event_type = &#39;login.attempt&#39;
 AND timestamp &gt;= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1d) AND geo_country(source_ip)
 NOT IN (&#39;US&#39;, &#39;GB&#39;) -- Replace with trusted countries</pre><h4>Potential
 Lateral Movements</h4>This query looks for user activity that may indicate lateral
 movement across clusters involving multiple GCP resources. Events can be narrowed
 down by particular resource kinds or activities that take place within a constrained
 time frame.<pre>SELECT user_email, resource_type, resource_name, timestamp FROM
 events WHERE event_type IN (&#39;resource.create&#39;, &#39;resource.access&#39;) GROUP
 BY user_email, resource_type, resource_name HAVING COUNT(*) &gt; 5 -- Adjust
 threshold based on expected activity</pre><h4>Unusual File Access</h4>This query
 looks for file access events coming from source IP addresses or unexpected user
 accounts. Additional filters can be applied based on particular file types or attempts
 to access data after business hours.<pre>SELECT user_email, source_ip, file_path,
 timestamp FROM events WHERE event_type = &#39;file.access&#39; AND (user_email
 NOT IN (&#39;admin@example.com&#39;, &#39;service_account@project.com&#39;) --
 Trusted accounts OR geo_country(source_ip) NOT IN (&#39;US&#39;)) -- Trusted
 location </pre><h3>Remediation?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x9a8vmb4z1_JhlwL"
 /></figure>For automated remediation, Backstory integrates with a number of tools.
 Here are some examples using Cloud Functions (because that’s what I found closest
 at hand).Isolating infected workloads on the cluster requires a cloud function
 to be triggered via Backstory findings after which we can add the function:<pre>def
 isolate_workload(data, context): # Extract cluster name and pod details from
 Backstory alert. cluster_name = data[&#39;resource&#39;][&#39;labels&#39;][&#39;cluster_name&#39;] pod_name
 = data[&#39;container&#39;][&#39;name&#39;] # Use Kubernetes API to cordon
 the infected node. from kubernetes import client, config config.load_kube_config() v1
 = client.AppsV1Api() v1.patch_namespaced_daemon_set( &quot;kube-system&quot;,
 &quot;kube-dns&quot;, body={&quot;spec&quot;: {&quot;template&quot;: {&quot;spec&quot;:
 {&quot;taints&quot;: [{&quot;effect&quot;: &quot;NoSchedule&quot;, &quot;key&quot;:
 &quot;infected&quot;}]}}}} )</pre>With the addition of a taint to stop additional
 pod scheduling, this Cloud Function automatically isolates the compromised node.Further,
 you can implement something like:<pre>def remediate_backstory_finding(data,
 context): &quot;&quot;&quot;Cloud Function triggered by Backstory detection.&quot;&quot;&quot; #
 Parse the Pub/Sub message data pubsub_message = json.loads(data) backstory_finding
 = json.loads(pubsub_message[&quot;data&quot;]) # Extract relevant details
 from the detection finding_name = backstory_finding[&quot;findingName&quot;] threat_type
 = backstory_finding[&quot;externalSystems&quot;][0][&quot;threatType&quot;] #
 Implement logic for remediation based on threat type if threat_type == &quot;MALWARE&quot;: #
 Example: Isolate the affected workload print(f&quot;Isolating workload associated
 with finding: {finding_name}&quot;) # Replace with your specific isolation
 workflow (e.g., API call to Anthos) elif threat_type == &quot;PORT_SCAN&quot;: #
 Example: Block suspicious IP addresses print(f&quot;Blocking suspicious IP
 addresses from finding: {finding_name}&quot;) # Replace with your specific
 IP blocking workflow (e.g., firewall rule update) else: print(f&quot;Unrecognized
 threat type: {threat_type} for finding: {finding_name}&quot;) # Implement
 logic for handling unknown threats or sending notifications </pre>A Pub/Sub
 message with the Backstory detection details in JSON format initiates the function.
 After parsing the message data, the threat type and finding name are extracted. The
 function carries out particular remediation actions based on the type of threat.
 Including examples of workload isolation for malware and IP blocking for port scans
 in this case.<h3>Conclude</h3>These are but a few simple instances. Depending
 on your unique Anthos environment, security posture, and the threats you want to
 find, you’ll need to modify the queries. As the integration develops, it’s also
 advised to refer to the official Backstory documentation for the most recent syntax
 and functionalities.<h3>Get in touch??</h3><a href="https://linktr.ee/imranfosec">imranfosec
 | Instagram | Linktree</a><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcf4a9a3a78b"
 width="1" height="1" alt=""><hr><a href="https://medium.com/google-cloud/securing-anthos-workload-with-chronicle-backstory-a-comprehensive-approchg-fcf4a9a3a78b">Securing
 Anthos Workload With Chronicle Backstory — A comprehensive approchg</a> was originally
 published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a>
 on Medium, where people are continuing the conversation by highlighting and responding
 to this story.'

Language

Active

Ricc internal notes

Ricc source

Show this article Back to articles