EventBridge Ignoring CloudShell GetFileDownloadUrls Events

Amazon EventBridge missing event GetFileDownloadUrls form CloudShell

I love working in AWS…but sometimes it makes you want to pull your hair out because it doesn’t behave the way you expect it to and a project that you think will take 10 minutes takes 3 days. That recently happened to me when I was making a contribution to the open source project AWS Security Survival Kit (ASSK).

The goal of the contribution was to add detection for the GetFileDownloadUrls action in CloudShell. This is an action that can be used by threat actors to exfiltrate potentially sensitive data from your AWS account.

I thought this would be an easy 10 minute task because I’ve created plenty of EventBridge Rules that send notifications before, and this didn’t look any different.

P.S.: if you just landed on this article and are frustrated already and just want the solution without my story behind solving the issue, scroll down to “How to make EventBridge see GetFileDownloadUrls

Why monitor GetFileDownloadUrls?

Addition of this detection was inspired by — and as far as I’m aware, first detected by — Ian Ahl from Permiso Security. It’s a data theft technique that’s actively been used by LUCR-3 (Scattered Spider) to first scrape credentials stored in Secrets Manager, save those secrets to a file in CloudShell, and then use GetFileDownloadUrls to download the file to the attacker’s device. A nifty data exfiltration that might never get picked up by your monitoring and alerting.

Enabling monitoring for GetFileDownloadUrls

After doing initial research, it sounded like AWS had support for sending CloudShell events through CloudTrail into EventBridge, and having an EventBridge rule pick them up to then send them over to SNS for notifications (or another target of your choice).

After creating an EventBridge Rule with the recommended EventPattern:

{
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.cloudshell"],
  "detail": {
    "eventSource": ["cloudshell.amazonaws.com"]
  }
}Code language: JSON / JSON with Comments (json)

I figured I could simply add an EventName to match the event I was after, and call it a day. Easy peasy…

{
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.cloudshell"],
  "detail": {
    "eventSource": ["cloudshell.amazonaws.com"],
    "eventName": ["GetFileDownloadUrls"]
  }
}Code language: JSON / JSON with Comments (json)

…just kidding. It didn’t work at all.

I would test downloading files from CloudShell, and absolutely nothing would happen. What?!

EventBridge not receiving the GetFileDownloadUrls event??

After getting nowhere, I naturally turned to complaining to anyone who would listen on Discord. My friend Georgi made the recommendation that I push the events to a CloudWatch Logs Group to see the event so I could fix my event pattern, because surely it had to be caused by a slight difference in the event versus the event pattern. I agreed, and so that’s what I did next.

Except after doing that, I saw multiple events coming through related to CloudShell, like CreateEnvironment, CreateSession, etc… but NOT GetFileDownloadUrls (literally the only one I needed!!!) Wtf?!?!

So then I went straight to CloudTrail data and the CloudWatch Logs Group receiving data from my trail, and ran queries against both of those data sources. Both of them showed the GetFileDownloadUrls event. The only thing not showing it was EventBridge.

Anyone I brought this up to was as confused as I was and we all started to think it was a bug.

How to make EventBridge see GetFileDownloadUrls

Then, one evening while playing Microsoft’s Flight Simulator (a sim I’m totally not currently obsessed with), I remembered seeing an asterisk in the AWS docs. The asterisk just says “*Non-mutating (read-only) API calls.” I couldn’t imagine what that would have to do with my problem, but I decided to search for “eventbridge read only events” in Google just in case. Lo and behold, I found this gem of an article: “Introducing support for read-only management events in Amazon EventBridge.

In order to enable read-only management events (which GetFileDownloadUrls is one as per the docs) with an EventBridge rule, you have to use the State ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS like this:

EventRuleCloudShellGetFileDownloadUrls:
    # Can detect data exfil through CloudShell (ie: scrape and download credentials stored in Secrets Manager)
    Type: "AWS::Events::Rule"
    Properties:
      Name: !Sub "${Project}-detect-cloudshell-getfiledownloadurls"
      Description: !Sub "[${Project}] Detect CloudShell GetFileDownloadUrls"
      State: "ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS"
      Targets:
        - Arn:
            Ref: CtAlertingTopic
          Id: CtAlertingTopic
      EventPattern:
        source:
          - aws.cloudshell
        detail-type:
          - "AWS API Call via CloudTrail"
        detail:
          eventSource:
            - "cloudshell.amazonaws.com"
          eventName:
            - "GetFileDownloadUrls"
Code language: PHP (php)

This currently does not appear to be possible to do via the AWS Console either, by the way. You have to do it through the CLI, API, or CloudFormation.

Also, this requires that you have a CloudTrail trail, and that it’s configured with a ReadWriteType of either All or ReadOnly. All is the default so unless you changed this, you should be good to go. If you want to double-check what it’s set to, you can do that by going to CloudTrail > Click on your trail name > scroll down to the section titled “Management events” > “API Activity” should say All or ReadOnly. But if it’s set to ReadOnly you won’t get Write events.

As soon as I changed the State to ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS, the event started firing and notifications came through.

I left feedback on the AWS documentation page because this should absolutely be mentioned on that page, but I’m not sure if they ever read that feedback or act on it, so we’ll see what comes of it.

Steps to troubleshoot missing events in EventBridge from CloudTrail

Ok so I solved this problem, but if you’re here and this doesn’t solve your problem, what steps can you take to troubleshoot and get to the root cause? Let’s break it down:

Step 1: Make sure you have a CloudTrail trail

If you don’t already have a trail enabled in your AWS account, which you can check by going to the CloudTrail service and in your Dashboard you should see at least one trail under Trails, then this is your starting point.

If you do already have a trail, go ahead and click on it so we can check settings.

Step 2: Make sure the trail is properly configured

  • If the event is happening in a different region, it may not get picked up unless you have Multi-region trail set to yes
  • I recommend pushing logs to CloudWatch Logs so you can look through the CloudTrail Logs without having to download them, so I’d recommend turning that on if you don’t already have it set up
  • Make sure API activity under Management Events is set to All if you want to get both read and write events, and that Management events is checked
  • If the data you are missing is a Data Event instead of a Management Event (learn the difference here) then you will need to enable that under Data Events on this same page. That is not enabled by default because it’s an extra cost, but that could be your issue. Look up whether the event you want is one or the other

With the above checklist completed, if you’re still not getting events in EventBridge, proceed to the next step.

Step 3: Look for the event in CloudWatch Logs and/or CloudTrail Logs

  • Go to the CloudWatch service and find the Log Group associated with your CloudTrail trai, and search for the event that you’re trying to capture and process through EventBridge. If you’re not finding it in CloudWatch (you can search using CloudWatch Insights), then it likely means CloudTrail isn’t picking it up and that’s most likely due to one of the above checklist items (it might be a data event?)
  • If you’re certain CloudTrail is picking up the event but you’re not seeing it in CloudWatch, you can always go straight to CloudTrail logs and search through those. I show a few ways of doing that in my free & short Beginner’s Guide to AWS CloudTrail for Security course if you need help
  • If the event is not showing up in CloudTrail, then either it’s not supported or your trail is misconfigured

Step 4: Send EventBridge events to CloudWatch Logs

If the event is showing up in CloudTrail and CloudWatch, but not EventBridge, then proceed to the next steps:

  1. Configure your EventBridge Rule to use a CloudWatch Logs Group as a target. That will capture all of the matched events and send them to that Logs Group, which means that if a) EventBridge is receiving the event and b) the event properly matches your rule’s Event Pattern, then it will appear in the CloudWatch Logs Group. Thanks Georgi for this tip!
  2. If you do this and the event does not appear in the CloudWatch Logs Group, chances are that your Event Pattern is incorrect for that rule. Refer to the event you are able to see from CloudTrail to make sure there are no typos or other issues (maybe your event source is wrong?)
  3. A good way to test this is to make the Event Pattern as broad as possible. For example, start by just specifying a source with no event names, and broaden even more if that doesn’t work by capturing all events (see code here for how to do that)

Step 5: Check to see if the Rule requires a different State

If you’re pretty sure that the Event Pattern is correct and EventBridge isn’t picking it up, then it means it’s not receiving it. That was my issue described above, and I solved it by using State: "ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS" (only possible to use via the CLI/API/CloudFormation at the time of writing, you won’t see this in the console) — this should only be needed if your event is a read-only management event, unless there is another weird quirk I don’t know about.

Step 6: Are you sure the event is supported by Event Bridge?

If you follow all these steps and it still doesn’t work, double-check to make sure the event you’re trying to send to EventBridge is actually supported by EventBridge. Not everything is.

In that case, an alternative would be to set up a CloudWatch Metric and a CloudWatch Alarm to send notifications.

Conclusion

Hope this helps anyone wondering why EventBridge isn’t working as expected. It’s not technically a bug, just a very well hidden ‘feature.’

If you’d like to learn more tips and tricks you can only get from people who have banged their heads against the wall for hours trying to figure out AWS’ quirks (so you don’t have to), then check out our AWS security courses.

Hope this helped.

Related Articles

Responses

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.