Mysterious S3 Access Denied Error

Posted on July 1, 2023  (Last modified on January 4, 2024 )

3 minutes  • 606 words  • Other languages:  Korean

How I discovered the error

I actually encountered the same error twice.
Neither error occurred in the course of work, but both occurred on the side projects that I had implemented a feature to test work-related functionality.
What I implemented was a Lambda function that triggered by S3 object created event.
These Lambda functions had a common logic: get the object’s key name from the received event, and look up the object information again via that key name.
However, my Lambda functions returned an Access Denied error.
Even though my lambda functions had enough permission to run GetObject from that bucket!

What was the problem

This is because the key name contains values such as Korean, special characters, or spaces.
In such cases, the value of Record.S3.Object.Key received through the event was URL encoded.
I mentioned earlier that I experienced the same problem twice, once because the name was Korean, and once because it contained special characters 😓

My guess is that this is because the object itself is in URL format.

Using the Object URL above, I tried to get the object information directly through the AWS SDK in my local code.

Then I got this error.

Failed to retrieve object: NoSuchKey: The specified key does not exist.

Not surprisingly, but one thing was different.
In my local code, it returned the correct error that the object for that key did not exist,
But my Lambda function returned Access Denied error.
Again, my Lambda has a permission to run GetObject on that Bucket, even if there’s no such key.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::nmin-access-test/*"
    }
  ]
}

I suspect this is caused by AWS not properly branching for errors that can occur when a Lambda accesses S3.
Because of this incorrect way of branching the error, I actually wasted about a week of my time trying to solve the problem by searching for “S3 Access Denied” related keywords.
So, despite the fact that it’s a simple problem, I thought I’d publish this post.

How to solve

If a Lambda function needs to look up information about an object received through an event again,
it needs to parse the key name from the event as a URL-formatted string back to the original string.

decodedObject, err := url.PathUnescape(object)
if err != nil {
  fmt.Println("Error occurred while decoding URL: ", err)
}

decodedObject = strings.Replace(decodedObject, "+", " ", -1)

In Go, there’s a built-in module called url.PathUnescape that I used.
However, that function doesn’t handle empty space value( ) being URL-encoded with special character(+), so I had to use the strings.Replace function in addition.
And finally, I was able to retrieve the object successfully even if I received a key name with Korean characters, special characters, or empty spaces.

Source code used for demonstration

+ Added on 2024/1/4

Today at work, I stumbled upon a reason as to why AWS S3 returns an Access Denied error even when the object doesn’t exist.
It’s because AWS policy doesn’t let me know if the object I am looking for via GetObject exists or not.
I did a little more research after work and found a question on StackOverflow.
To summarize, it’s designed to prevent me from exploring whether a particular key exists when I only have GetObject permission but not ListObject permission.
In the example code I implemented, I had both GetObject and ListObject permissions in my local environment because I was running it directly with my AWS credentials.
However, when I deployed it as a Lambda, I was only granted GetObject permissions, which is why I was experiencing permissions issues.