Click here to Skip to main content
15,355,296 members
Articles / Hosted Services / AWS
Article
Posted 13 Feb 2022

Stats

8.9K views
58 downloads
6 bookmarked

How to Access Private S3 Objects with AWS Cognito

Rate me:
Please Sign up or sign in to vote.
4.95/5 (31 votes)
21 May 2022CPOL10 min read
To provide a URL link to access objects in private S3 bucket through AWS Cognito User Pool (using hosted UI), Authorized API Gateway and Lambda in a Secure Way.
A scenario-based flows through Cognito, Gateway API, Lambda, S3 in AWS cloud environment. We will be looking for the answer of "How to Access/Download private S3 Objects through AWS Cognito Hosted UI, API Gateway and Lambda."

Scenario

Suppose you have been developing some applications for your client. However, there are some files such as PDF, Word, Excel, etc. that are related to the records in applications. For the simplicity of the scenario, suppose these files are stored in one private S3 bucket in AWS. Users need the ability to access any of these related files in private S3 bucket through a URL link in applications. Our solution need to run as a portable solution for any in-company software.

Introduction

The goal of this article is to show how to download files in private S3 bucket using cognito user pools. Beside Cognito, the flow from Cognito to API Gateway with Authorizer and collaboration of API Gateway with Lambda are shown. Snapshots for each steps from AWS console are shown as much as possible. There might be many snapshots to make the steps more clear for the beginners especially.

Background

To understand better what has been developed in this article, some prereading might be helpful. The following links will be good for AWS newbies especially.

What to Do

Many flows or ways could be coded for such a task. Here, we will be implementing the way as shown below. A brief description of how to do for the scenario can be shown as in the image below.

The image below shows that we need to create some items such as Cognito User Pool, S3 buckets, API Gateway Methods, Lambda Functions, etc. After creating all entities in AWS environment, we need to configure all of them properly so that they can work in collaboration all together.

Image 1

It is better to create all items in AWS environment in reverse order. For example, to use Lambda with API method, firstly Lambda function could be developed so that it can be bound easily when API Gateway method is created. Similarly, we should create S3 web bucket at Step 5 and put callback.html inside it so that we can use it in Step 6 while we are creating Cognito User Pool. Of course, this is not mandatory but this order will make the development easier. So this approach is preferred here.

Outline

We will be looking for the answers of the questions below. Remember, because all items here are created in AWS environment, you have to have an AWS account to apply all steps in this article.

  1. How to Create Private S3 Bucket
  2. How to Create a Custom Policy for the Permission to Access Objects in a Private S3 Bucket
  3. How to Create a Lambda Function to Access Objects in a Private S3 Bucket
  4. How to Create Gateway API to Use Lambda Function
  5. How to Create Public S3 Bucket to use as Web Folder
  6. How to Create Cognito User Pool and Configure Settings
  7. How to Test the Scenario

1. How to Create Private S3 Bucket

S3 is one of the region-based services in AWS. Item in S3 buckets is called object. So, in AWS, it is possible to use object and files instead of each other for S3 buckets. Keep “Block All Public Access” checkbox as checked. Here, a private S3 bucket is created. Although there are many extra config options, we are creating with default values for the simplicity of the solution.

Image 2

To test private access to S3 bucket, upload some objects into it. Later, try to access these objects with non-allowed users or possible access links. Although we now PDF, DOC, XLS, etc. as files, these all are called as objects in AWS S3 terminology.

Image 3

2. How to Create a Custom Policy for the Permission to Access Objects in a Private S3 Bucket

In AWS, IAM (Identity and Access Management) is the base of all services! Users, Groups, Roles, and Policies are the words that we have to be familiar with.

There are many built-in roles. Each role has many built-in policies that mean permissions. These are called "AWS Managed". However, it is possible to create "Customer Managed" roles and policies as well. So, a custom policy is created here.

  • Create a custom IAM policy to get object from private S3 bucket as shown below.
  • Find current policy list in AWS and create a new one to allow for “GetObject” only from your private S3 bucket as shown below:

Image 4

Create a custom policy as shown below. As service select S3 and as action select “GetObject” only as shown below:

Image 5

As resource select specific and choose your private S3 bucket so that this policy has abilities as you want.

Image 6

Give a name to your policy and create it as shown below. You can give any name, however you should remember it.

Image 7

The summary for your custom policy will look as below. It was possible to create policy using this JSON content directly.

Image 8

JavaScript
//Policy JSON definition can be copied from below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::private-s3-for-interfacing/*"
        }
    ]
}

3. How to Create a Lambda Function to Access Objects in a Private S3 Bucket

Here, NodeJS last version is used for the lambda function. Create a Lambda function and select NodeJS. It is possible to choose any supported language such as Python, Go, Java, .NET Core, etc. to use for Lambda function. Give a name to your Lambda function as shown below:

Image 9

When you create Lambda function, a sample “hello” code is shown. We need to develop our code instead.

As it is seen, Lambda development environment is looking as a web-based light IDE.

Image 10

Replace the following code with the given short example code.

New code will be as below. After changing the code, press the “Deploy” button to use Lambda function.

For the simplicity of the scenario, the bucket name is used statically. Filename is sent as parameter with name "fn". Although the default content type is accepted as pdf, it can be any file that is implemented in lambda function code. Because we will prefer to use Lambda function proxy ability in API Gateway connection, response header contains some more required data.

JavaScript
// Code for Lambda function looks like this
// This code will be returning response as blob content
// Callback-to-Download-Blob.html in attached files could be used to download

const AWS = require('aws-sdk');
const S3= new AWS.S3();
exports.handler = async (event, context) => {
    
  let fileName;
  let bucketName;
  let contentType;
  let fileExt;
    
  try {
      
    bucketName = 'private-s3-for-interfacing';
    fileName = event["queryStringParameters"]['fn']
    contentType = 'application/pdf';
    fileExt = 'pdf';
    
    //------------
  
    fileExt = fileName.split('.').pop();
    
    switch (fileExt) {
        case 'pdf':
            contentType = 'application/pdf';
            break;        
        case 'png':
            contentType = 'image/png'; 
            break;
        case 'gif':
            contentType = 'image/gif';
            break;
        case 'jpeg':
            contentType = 'image/jpeg';
            break;
        case 'jpg':
            contentType = 'image/jpeg';
            break;
        case 'svg':
            contentType = '.svg image/svg+xml';
            break;
        case 'docx':
            contentType = 
               'application/vnd.openxmlformats-officedocument.wordprocessingml.document';   
            break;
        case 'xlsx':
            contentType = 
            'Content-Type: 
             application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';  
            break;
        case 'pptx':
            contentType = 
            'Content-Type: 
             application/vnd.openxmlformats-officedocument.presentationml.presentation';    
            break;
        case 'doc':
            contentType = 'Content-Type: application/msword';    
            break;
        case 'xls':
            contentType = 'Content-Type: application/vnd.ms-excel';   
            break;
        case 'csv':
            contentType = 'Content-Type: text/csv';     
            break;    
        case 'ppt':
            contentType = 'Content-Type: application/vnd.ms-powerpoint'; 
            break;
        case 'rtf':
            contentType = 'Content-Type: application/rtf'; 
            break;
        case 'zip':
            contentType = 'Content-Type: application/zip';  
            break;   
        case 'rar':
            contentType = 'Content-Type: application/vnd.rar'; 
            break;
        case '7z':
            contentType = 'Content-Type: application/x-7z-compressed';  
            break;
        default:
            ;  
    }
    
    //------------
    
    //console.log(`Hi from Node.js ${process.version} on Lambda!`);
    const data = await S3.getObject({Bucket: bucketName, Key: fileName}).promise();
    
    return {
       headers: {
          'Content-Type': contentType,
          'Content-Disposition': 'attachment; filename=' + fileName, // key of success
          'Content-Encoding': 'base64',
          
          'Access-Control-Allow-Origin': '*',
          'Access-Control-Allow-Headers': 
          'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token', 
          'Access-Control-Allow-Methods': 'GET,OPTIONS'
      },
      body: data.Body.toString('base64'),
      isBase64Encoded: true,
      statusCode: 200
    }
  }
  catch (err) {
    return {
      statusCode: err.statusCode || 400,
      body: err.message || JSON.stringify(err.message) + 
                           ' - fileName: '+ fileName + ' - bucketName: ' + bucketName
    }
  }
}//

It was possible to use Python code in Lambda function as shown below:

Python
//The following code could be improved as NodeJS above
    
import base64
import boto3
import json
import random

s3 = boto3.client('s3')

def lambda_handler(event, context):
    try:
        
        #fileName = 'testFile.pdf'
        #bucketName = event['pathParameters']['bn']        
        #fileType = event['queryStringParameters']['ft']

        fileName = event['queryStringParameters']['fn']
        bucketName = 'private-s3-for-interfacing'        
        
        contentType = 'application/pdf'
        
        response = s3.get_object(
            Bucket=bucketName,
            Key=fileName,
        )
        
        file = response['Body'].read()
        
        return {
            'statusCode': 200,
            'headers': {  
                         'Content-Type': contentType,                            
                         'Content-Disposition': 'attachment; filename='+ fileName,
                         'Content-Encoding': 'base64'

                          #if it is required some more CORS-related code 
                          #could be added here
                        },
            'body': base64.b64encode(file).decode('utf-8'),           
            'isBase64Encoded': True
        }
    except:
        return {
            'headers': { 'Content-type': 'text/html' },
            'statusCode': 200,
            'body': 'Error occurred in Lambda!' 
        }

Another way might be creating presigned url with Lambda as shown below:

JavaScript
// This way will provide presigned url
// Callback-for-preSignedUrl.html in attached files could be used to use the 
// presigned URL link

var AWS = require('aws-sdk');
var S3 = new AWS.S3({
  signatureVersion: 'v4',
});

exports.handler =  async (event, context) => {
    
  let fileName;
  let bucketName;
  let contentType;
  let fileExt;
    
  bucketName = 'private-s3-for-interfacing';
  fileName = event["queryStringParameters"]['fn'];
  contentType = 'application/json';
    
  const presignedUrl = S3.getSignedUrl('getObject', {
    Bucket: bucketName,//'BUCKET NAME',
    Key: fileName, //'UploadedFile',
    Expires: 300 //sec
  });

  let responseBody = {'presignedUrl': presignedUrl};
  
  return {
       headers: {
          'Content-Type': contentType,
          'Access-Control-Allow-Origin': '*',
          'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,
           Authorization,X-Api-Key,X-Amz-Security-Token', 
          'Access-Control-Allow-Methods': 'GET,OPTIONS'
      },
      body: JSON.stringify(responseBody), 
      statusCode: 200
    }    
};

When Lambda function is created, a role is created with it. However, this role does not have permission to access objects in our private S3 bucket. Now, we need to attach our "Customer Managed" policy to the role that is created with Lambda function.

After creating Lambda function, we can find the role that is created automatically with Lambda function as shown below:

Image 11

Attach the custom policy you created in the previous step to this role, so that Lambda function can have limited “GetObject” access right to your S3 bucket.

Image 12

All about Lambda to access S3 bucket is done so far. It is time to create an AWS Gateway method to use our Lambda function.

4. How to Create Gateway API to Use Lambda Function

Create AWS Gateway REST API as shown below. As it is seen, there many options. However, we create a "REST" as "New API". Give a name to your API Gateway as shown.

Image 13

There are some steps to create and run AWS GW API:

  • Create API
  • Create resource
  • Create method
  • Deploy API

Create Resource for your REST API as shown below:

Image 14

The resource that is created here will be used in API’s URL later.

Image 15

Create GET method for the resource you have created as shown below:

Image 16

Any http method such as GET, POST, PUT, DELETE, etc. could be created here. For our need, we are creating GET only. Do not forget to bind Lambda function that we created in the previous steps to this method.

Lambda Proxy Integration is checked here. This approach provides us to handle all response-related content in Lambda Function.

Image 17

After creating GET method, the flow between API Gateway Method and Lambda function will be shown as below:

Image 18

Enable CORS for Gateway API as shown below. Default 4xx and Default 5xx could be checked so that even errors can return without any problem.

Image 19

After creating and configuring all about AWS Gateway method, now it is time to deploy API as shown below. API is deployed into a stage as shown. Also stage name will be used in public API URL.

Image 20

After deployment, URL will be seen as below. Now, it is possible to use this link from any application.

Image 21

We should define an Authorizer to restrict access to API gateway. We can define a Cognito Authorizer as shown below.

As it is seen in the image below, Authorization is JWT token that should be added to header of request to use authorized API method.

When Cognito Hosted UI is submitted with g Cognito user/pwd Cognito will redirect the user to Callback url by transferring id_token and additional state data.

See that the token that we should add to header is called "Authorization" under Token Source.

Image 22

After defining Cognito-based Authorizer, it can be used as below:

Image 23

On the other hand, note that, if you do not want to define Authorizers for API Gateway, it is possible to restrict the access to API URL with “Resource Policy” as shown below.

If “Resource Policy” is changed/modified/added/deleted/etc., API should be deployed. The IP that is shown as xxx.xxx.xxx.xxx can be the IP of server. When anyone tries to access the URL from a different IP, the following message will be shown.

{"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:eu-west-2:********8165:https://x9dxwctglh.execute-api.eu-west-2.amazonaws.com/apiv11/ac?fn=testFile.pdf with an explicit deny"}

Image 24

JavaScript
// Resource Policy JSON code will be as below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "*"
        },
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "*",
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": "xxx.xxx.xxx.xxx"
                }
            }
        }
    ]
}

5. How to Create Public S3 Bucket to Use as Web Folder

For the solution, we need to have two S3 buckets. The first one is created in the previous chapters. The second one is created now and will be used as web folder. The first one was used as private bucket to store all files.

Image 25

Create public S3 bucket as web folder. This bucket contains a callback.html so that it can be used as Cognito callback address.

Image 26

S3 bucket for web should be public. So, the following policy can be applied.

JavaScript
// Policy JSON will look like this

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::web-s3-for-interfacing/*"
        }
    ]
}

Content of Callback.html is as below:

  • Callback.html will receive filename and id_token as parameter.
  • FileName will be sent as url parameter to API GW method.
  • id_token will be sent to API gateway as header for Authorize-by-Cognito API GW method.

Callback.html is in the zip file below:

6. How to Create Cognito User Pool and Configure Settings

See hosted UI link below.

Add additional “state” url parameter to send parameter to hosted Cognito login page. “state” parameter will be sent to Callback.html.

Cognito Hosted UI link includes many url parameters as shown below:

https://test-for-user-pool-for-s3.auth.eu-west-2.amazoncognito.com/login?client_id=7uuggclp7269oguth08mi2ee04&response_type=token&scope=openid+profile+email&redirect_uri=https://web-s3-for-interfacing.s3.eu-west-2.amazonaws.com/Callback.html&state=fn=testFile.pdf

Domain: https://test-for-user-pool-for-s3.auth.eu-west-2.amazoncognito.com

client_id=7uuggclp7269oguth08mi2ee04
response_type=token
scope=openid+profile+email
redirect_uri=https://web-s3-for-interfacing.s3.eu-west-2.amazonaws.com/Callback.html
state=fn=testFile.pdf

state is a special url parameter. It can be sent to hosted UI page and returned to Callback.html

A client app should be created as shown below:

Image 27

App client settings can be confirmed as shown below:

Image 28

Domain name should be set so that it can be used as url for hosted UI.

Image 29

7. How to Test the Scenario

Let's see how to test API that allows restricted access using Cognito User Pool.

Any end user can click a link to start this process. Suppose that we have a web page that includes the following HTML content. As it is seen, the link for each file is URL of Cognito hosted UI.

LinkToS3Files.html in the zip file below can be used to test the scenario.

Conclusion

I hope this article has been useful for beginners of AWS cloud environment.

History

  • 14th February, 2022: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Necmettin Demir
Software Developer (Senior) NEBULACT
United Kingdom United Kingdom
Necmettin Demir is developer at NEBULACT Ltd. in London/UK.
He has BSc and MSc degrees of Computer Science. He was also graduated from MBA.
He is also trying to share his technical experience writing articles.

Comments and Discussions

 
GeneralMy vote of 5 Pin
Ștefan-Mihai MOGA22-May-22 19:32
professionalȘtefan-Mihai MOGA22-May-22 19:32 
GeneralRe: My vote of 5 Pin
Necmettin Demir22-May-22 19:39
MemberNecmettin Demir22-May-22 19:39 
GeneralMy vote of 5 Pin
bigbang202222-Mar-22 11:21
Memberbigbang202222-Mar-22 11:21 
GeneralMy vote of 5 Pin
cloudergroup22-Mar-22 0:07
Membercloudergroup22-Mar-22 0:07 
GeneralMy vote of 5 Pin
awsteach21-Mar-22 11:19
Memberawsteach21-Mar-22 11:19 
GeneralMy vote of 5 Pin
Countrycoders21-Mar-22 6:10
MemberCountrycoders21-Mar-22 6:10 
GeneralMy vote of 5 Pin
Ammycander21-Mar-22 2:33
MemberAmmycander21-Mar-22 2:33 
PraiseNice Article Pin
Member 1557275220-Mar-22 7:28
MemberMember 1557275220-Mar-22 7:28 
GeneralMy vote of 5 Pin
Minorcode20-Mar-22 4:04
MemberMinorcode20-Mar-22 4:04 
GeneralMy vote of 5 Pin
itech aqua20-Mar-22 3:07
Memberitech aqua20-Mar-22 3:07 
GeneralMy vote of 5 Pin
MuRRTeX20-Mar-22 2:56
MemberMuRRTeX20-Mar-22 2:56 
GeneralMy vote of 5 Pin
actualcloud20-Mar-22 2:47
Memberactualcloud20-Mar-22 2:47 
GeneralMy vote of 5 Pin
iskendercloud19-Mar-22 3:12
Memberiskendercloud19-Mar-22 3:12 
PraiseNice article. Thanks. ⭐⭐⭐⭐⭐ Pin
iskendercloud19-Mar-22 3:06
Memberiskendercloud19-Mar-22 3:06 
GeneralMy vote of 5 Pin
İlhan Turğut 202219-Mar-22 3:03
Memberİlhan Turğut 202219-Mar-22 3:03 
GeneralMy vote of 5 Pin
tahirturgut19-Mar-22 2:59
Membertahirturgut19-Mar-22 2:59 
GeneralMy vote of 5 Pin
Tahir Turgut19-Mar-22 2:51
MemberTahir Turgut19-Mar-22 2:51 
GeneralMy vote of 5 Pin
mehmet turgut 202219-Mar-22 2:04
Membermehmet turgut 202219-Mar-22 2:04 
GeneralMy vote of 5 Pin
Tahir Turgut17-Mar-22 10:02
MemberTahir Turgut17-Mar-22 10:02 
GeneralMy vote of 5 Pin
Tahir Turgut17-Mar-22 10:00
MemberTahir Turgut17-Mar-22 10:00 
PraisePerfect Pin
mehmet turgut 202217-Mar-22 20:44
Membermehmet turgut 202217-Mar-22 20:44 
PraiseMy vote of 5 Pin
Tahir Turgut13-Mar-22 2:24
MemberTahir Turgut13-Mar-22 2:24 
GeneralMy vote of 5 Pin
Member 152856238-Mar-22 3:12
MemberMember 152856238-Mar-22 3:12 
GeneralVery useful Pin
Member 141748247-Mar-22 16:33
MemberMember 141748247-Mar-22 16:33 
PraiseNice real world case Pin
feyyaz acet23-Feb-22 0:02
Memberfeyyaz acet23-Feb-22 0:02 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.