An incident response on a customer account in 2025 went sideways for one reason: the CloudTrail logs were enabled, but the S3 bucket they wrote to was in the same account as the compromised IAM credentials. The attacker had s3:DeleteObject on the bucket via the over-broad role they had stolen, deleted the trail files for the four-hour window they were active, and the forensic team had nothing but the partial events still in the CloudTrail Event History (which keeps 90 days of management events in the console but only that). The baseline below assumes that the account it logs from can be compromised, so the logs live somewhere else with the right keys and the right locks. GuardDuty is the second piece — it produces findings continuously, most of which are noise, and the question is which ones to wake someone for.

What’s happening

CloudTrail records API calls. Every RunInstances, every CreateUser, every GetObject (if data events are enabled) gets a JSON entry. The default trail captures management events for the account it lives in, writes to an S3 bucket, and that is the minimum baseline. The gaps in the minimum baseline:

Multi-region — the default trail is per-region. An API call in us-west-2 is invisible from a us-east-1 trail unless the trail is configured as multi-region. Attackers regularly pivot to obscure regions specifically because monitoring is weaker there.
Cross-account — the S3 bucket lives in the source account. If the source account is compromised, the trail is deletable. The fix is a separate security/logging account that owns the bucket; the workload account’s trail writes to it via cross-account bucket policy.
KMS encryption + Object Lock — server-side encryption is the default but with the bucket-owner-managed key. With a customer-managed KMS CMK in the logging account and Object Lock in compliance mode, the trail files become immutable for the retention period. Even root in the workload account cannot rewrite or delete them.
Organizations trail vs per-account — Organizations CloudTrail captures every member account in one trail, in the management account or a delegated administrator. The advantage is consistency (every member account is auditable from one place); the disadvantage is centralizing risk (one trail to compromise). For most AWS Organizations, the right answer is both — an Organizations trail in the logging account plus per-account trails for workloads that need their own retention.

GuardDuty is the threat-detection layer. It consumes VPC Flow Logs, CloudTrail, DNS query logs (and Kubernetes audit logs, if enabled), runs them against AWS’s threat-intel database and behavioral models, and produces findings. The categories that actually warrant action:

UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS — instance role credentials being used from an IP that is not an EC2 instance. This is the SSRF-to-exfiltration playbook in flight.
CredentialAccess:IAMUser/AnomalousBehavior — IAM principal performing unusual actions for their pattern (e.g. listing S3 buckets they have never touched).
Backdoor:EC2/C&CActivity.B!DNS — DNS query from an EC2 instance to a known command-and-control domain.
PenTest:IAMUser/KaliLinux — API calls from a Kali Linux user agent. Not necessarily malicious (security teams use it), but warrants verification.
Recon:IAMUser/MaliciousIPCaller — API call from an IP on the threat-intel block list.
CryptoCurrency:EC2/BitcoinTool.B!DNS — EC2 instance resolving cryptocurrency mining pool domains. Almost always a compromised instance running an attacker’s miner.

The noise floor is Recon:EC2/PortProbeUnprotectedPort and Recon:EC2/Portscan — the internet is constantly probing every public IP. These findings are real but not actionable in isolation; they only matter when correlated with a successful exploit.

How to verify

The check on CloudTrail:

aws cloudtrail describe-trails \
  --query 'trailList[].[Name,IsMultiRegionTrail,IsOrganizationTrail,IncludeGlobalServiceEvents,KmsKeyId,LogFileValidationEnabled,S3BucketName]' \
  --output table

Every column should be populated. IsMultiRegionTrail: true, KmsKeyId non-empty, LogFileValidationEnabled: true, S3BucketName pointing at a bucket in a different account.

Then check the bucket configuration:

aws s3api get-object-lock-configuration --bucket security-trail-prod
aws s3api get-bucket-policy --bucket security-trail-prod

Object Lock in COMPLIANCE mode with a retention period (commonly 365 days for SOC 2, 7 years for HIPAA). The bucket policy denies s3:DeleteObject and s3:PutBucketPolicy from anyone except a tightly-scoped administrative role in the logging account.

For GuardDuty:

aws guardduty list-detectors
aws guardduty get-detector --detector-id <id>
aws guardduty list-findings --detector-id <id> \
  --finding-criteria '{"Criterion":{"severity":{"Gte":7}}}' \
  --max-results 50

The severity-7-and-above filter cuts the noise. The cleanup target: there should be zero open findings with severity ≥ 7 in a healthy account. Severity 5-7 findings need triage but are routine.

The fix

The Organizations setup (run from the management account, delegating administration to a logging account):

Create the KMS CMK in the logging account with a key policy that allows CloudTrail to encrypt and the security team to decrypt. The policy explicitly denies any other principal — even root in the workload accounts.
```
aws kms create-key \
  --description "CloudTrail encryption" \
  --policy file://trail-key-policy.json \
  --region us-east-1
```

Create the S3 bucket in the logging account with Object Lock enabled at creation time (this is a one-shot — Object Lock cannot be added to an existing bucket without contacting AWS Support):

aws s3api create-bucket \
  --bucket security-trail-prod \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

aws s3api put-object-lock-configuration \
  --bucket security-trail-prod \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 365
      }
    }
  }'

Apply a bucket policy that grants CloudTrail in every Organizations member account s3:PutObject to its prefix, and denies deletion:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailWrite",
      "Effect": "Allow",
      "Principal": {"Service": "cloudtrail.amazonaws.com"},
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::security-trail-prod/AWSLogs/o-abc123/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceOrgID": "o-abc123",
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Sid": "DenyDelete",
      "Effect": "Deny",
      "Principal": "*",
      "Action": ["s3:DeleteObject", "s3:DeleteObjectVersion"],
      "Resource": "arn:aws:s3:::security-trail-prod/*"
    }
  ]
}

Create the Organizations trail from the management account, pointing at the logging account bucket and the KMS CMK:

aws cloudtrail create-trail \
  --name org-trail-prod \
  --s3-bucket-name security-trail-prod \
  --kms-key-id arn:aws:kms:us-east-1:111111111111:key/abc-... \
  --is-multi-region-trail \
  --is-organization-trail \
  --include-global-service-events \
  --enable-log-file-validation

aws cloudtrail start-logging --name org-trail-prod

Enable GuardDuty for the organization with auto-enable on new accounts:

aws organizations register-delegated-administrator \
  --account-id 222222222222 \
  --service-principal guardduty.amazonaws.com

# then from the logging account:
aws guardduty enable-organization-admin-account --admin-account-id 222222222222
aws guardduty update-organization-configuration \
  --detector-id <id> \
  --auto-enable-organization-members ALL

Wire the high-severity findings to EventBridge for auto-response. The rule below routes severity-≥-7 findings to an SNS topic that pages the on-call:

GuardDutyHighSeverityRule:
  Type: AWS::Events::Rule
  Properties:
    Name: gd-high-severity
    EventPattern:
      source: ["aws.guardduty"]
      detail-type: ["GuardDuty Finding"]
      detail:
        severity: [{"numeric": [">=", 7]}]
    Targets:
      - Arn: !Ref SecurityPageSnsTopic
        Id: gd-page

For the specific finding type UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS, add a Lambda target that automatically isolates the instance — revoke the IAM role, snapshot the EBS volumes for forensics, and stop the instance:

GuardDutyExfiltrationRule:
  Type: AWS::Events::Rule
  Properties:
    Name: gd-credential-exfil
    EventPattern:
      source: ["aws.guardduty"]
      detail-type: ["GuardDuty Finding"]
      detail:
        type: ["UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS"]
    Targets:
      - Arn: !GetAtt IsolateInstanceLambda.Arn
        Id: isolate

The Lambda’s logic is small (50 lines of Python) and the failure mode is contained — isolating a legitimate instance during a false positive is reversible.

Three rules across every CloudTrail/GuardDuty deployment we run:

Trail goes to a different account. Same-account trail is forensically useless against an attacker who pivots into the workload role. The cross-account bucket plus Object Lock plus KMS CMK is the baseline.

GuardDuty severity ≥ 7 is paging-grade. The 5-7 band is routine triage; below 5 is noise. The Recon family below severity 5 is interesting only as background pattern, not as paging.

Auto-response on a narrow set of findings only. Credential exfiltration is the canonical auto-isolate trigger. Broader auto-response (auto-disable IAM users, auto-revoke roles) accumulates false-positive lockouts and gets disabled within a quarter. Restraint is what keeps the rule in service.

The full deployment on a fresh AWS Organization is a focused two days — most of it is the bucket policy and the KMS CMK key policy review. The payoff is a forensic record that survives a compromised workload account and a paging stream that contains the findings that actually matter.

Stack Harbor runs IAM cleanup, CloudWatch baseline, AWS Backup, and the cost-control rhythm as part of Managed AWS and ongoing Managed Cloud Ops. If the trail bucket sits in the same account as the workload, start with an infrastructure consult.