Homepage platformOS Logo

Uploading Files Directly to Amazon S3 Using Uppy

Last edit: Oct 21, 2020

This guide will help you to create a user-friendly image upload feature with image processing.

Note

This example uses some Twitter Boostrap classes to make it a little bit prettier.

Requirements

This is an advanced tutorial. To follow it, you should be familiar with basic platformOS concepts, HTML, JavaScript, Liquid, GraphQL, and some topics in the Get Started section.

Steps

Uploading images directly to Amazon S3 is a four-step process:

Step 1: Create Model Schema and image processing configuration

There will be only one property in Model Schema used in this example (photo), but it will generate 4 different versions of the image uploaded by the user. Apart from explicitly defined versions, there is also an original version that is always there.

modules/direct_s3_upload/public/model_schemas/uppy.yml
name: uppy
properties:
- name: photo
  type: upload
  options:
    content_length:
      lte: 2048  # maximum file size will be 2048KB
    versions:
      - name: large_bw    # version name
        output:
          format: jpeg    # output format
          quality: 70     # output quality
        resize:
          width: 1600     # width of the image - in this case max width
          height: 1200    # height of the image - in this case max height
          fit: inside     # preserve aspect ratio, do not crop the image, make sure all pixels are within viewport
          without_enlargement: false              # if image is smaller than width/height, do not enlarge it
        manipulate:
          greyscale: true   # make image greyscale (similar to css "filter: grayscale(1)")
          blur: 20          # blur using gaussian mask (similar to css "filter: blur(1.5rem)")
          flatten: red      # merge alpha transparency channel (if exists) with given color
      - name: small
        output:
          format: webp
          quality: 80
        resize:
          width: 640
          height: 480
          fit: cover
          position: top   # define the gravity direction or strategy when cropping the image using fit cover or contain
      - name: thumb
        output:
          format: png
          quality: 80
        resize:
          width: 200
          fit: contain
          background: "rgba(255, 255, 255, 0.5)"  # if resized image is not covering whole viewport, set background color
      - name: center
        output:
          format: jpeg
          quality: 100
        resize:
          height: 400
          width: 400
          fit: cover

Step 2: Get presign data and expose it using a form

To upload a file to AWS S3 directly, you will need the server-generated data called signature. This information is from GraphQL. It takes model_schema_name and property_name as arguments. Other options like versions, content_length, ACL are taken from property configuration.

modules/direct_s3_upload/public/views/pages/images.liquid

---
slug: direct-s3-upload/images
---

{% graphql data %}
mutation presign {
  property_upload_presigned_url(model_schema_name: "modules/direct_s3_upload/uppy", property_name: "my_upload") {
      upload_url
      upload_url_payload
    }
}
{% endgraphql %}
{% assign data = data.property_upload_presigned_url %}

<form action="{{ data.upload_url }}"
  data-s3-uppy="form"
  data-s3-uppy-user-id="{{ context.current_user.id }}"
  hidden>
  {% for field in data.upload_url_payload %}
    <input type="hidden" name="{{ field[0] }}" value='{{ field[1] }}'>
  {% endfor %}
</form>

Below the form, paste the code responsible for containing Uppy.

modules/direct_s3_upload/public/views/pages/images.liquid
<div class="row">
  <div class="col-5">
    <div class="card">
      <div class="card-body">
        <div id="drag-drop-area"></div> <!-- This is where Uppy will inject itself -->
      </div>
    </div>
  </div>

  <div class="col-7">
    <div class="card">
      <div class="card-body">
        <p>URLs to image returned by AWS (input):</p>
        <small>
          <ol data-s3-uppy="log"></ol>
        </small>
      </div>
    </div>
  </div>
</div>

Step 3: Configure Uppy and use presigned data

Uppy is a JavaScript widget that helps to handle uploads. With its S3 plugin on board, you can make the implementation much shorter and easier.

However, before you can use any scripts, you need to load them.

modules/direct_s3_upload/public/views/pages/images.liquid
<link href="https://transloadit.edgly.net/releases/uppy/v1.6.0/uppy.css" rel="stylesheet">
<script src="https://transloadit.edgly.net/releases/uppy/v1.6.0/uppy.js" defer></script>
<script src="https://cdn.jsdelivr.net/npm/sweetalert2@9" defer></script>
<script src="{{ 'modules/direct_s3_upload/images.js' | asset_url }}" defer></script> 

This is the final images.js file, responsible for uploading the image, logging its URL, creating the model, and showing the confirmation box.

modules/direct_s3_upload/public/assets/images.js
const _form = document.querySelector('[data-s3-uppy="form"]');
const _log = document.querySelector('[data-s3-uppy="log"]');

const uppy = Uppy.Core({
  autoProceed: true,
  restrictions: {
    maxFileSize: 2097152,  // Limit size to 2 MB on the javascript side
    maxNumberOfFiles: 3,
    allowedFileTypes: ['image/png', 'image/jpeg', 'image/webp']
  }
})
  .use(Uppy.Dashboard, {
    inline: true,
    target: '#drag-drop-area',
    note: 'Images only, up to 3 files, 2MB each',
    width: '100%',
    height: 350
  })
  .use(Uppy.DragDrop)
  .use(Uppy.GoldenRetriever)
  .use(Uppy.AwsS3, {
    getUploadParameters() {
      // 1. Get URL to post to from action attribute
      const _url = _form.getAttribute('action');
      // 2. Create Array from FormData object to make it easy to operate on
      const _formDataArray = Array.from(new FormData(_form));
      // 3. Create a JSON object from array
      const _fields = _formDataArray.reduce((acc, cur) => ({ ...acc, [cur[0]]: cur[1] }), {});

      // 4. Return resolved promise with Uppy. Uppy it will add file in file param as the last param
      return Promise.resolve({
        method: 'POST',
        url: _url,
        fields: _fields
      });
    }
  });

uppy.on('upload-success', (_file, data) => {
  const li = document.createElement('li');
  li.textContent = data.body.location;
  _log.appendChild(li);
});

uppy.on('complete', ({ failed, successful }) => {
  /*
    For every successfully uploaded image to S3, send request to the Instance
    that will create a model with the uploaded image's URL as direct_url param.
  */
  Promise.all(successful.map(({ response }) => createImage(response.body.location)))
  .then(() => {
    // Show confirmation box (SweetAlert2) after all the images has been created
    Swal.fire({
      title: 'Images uploaded',
      icon: 'success',
      text: 'Press refresh to display the results',
      confirmButtonText: 'Refresh',
      showCloseButton: true
    }).then(result => {
      if (!result.value) {
        return;
      }

      // Reload page after "Refresh" button has been clicked inside Sweet Alert2
      window.location.reload();
    });
  });
});

const createImage = imageUrl => {
  // Get logged in user id
  const userId = _form.dataset.s3UppyUserId;

  // Create model for this user with s3 image url
  return fetch('/direct-s3-upload/images/model_create', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Accept: 'application/json'
    },
    body: JSON.stringify({ direct_url: imageUrl, user_id: userId })
  }).then(response => response.json());
};

Step 4: Create page that will create the model with the image

For fetch to succeed, you need to create a page under the /direct-s3-upload/images/model_create path. It will execute the model_create mutation with params passed in the XHR request.

modules/direct_s3_upload/public/views/pages/model_create.liquid

---
slug: direct-s3-upload/images/model_create
method: post
---
{% if context.params.user_id %}
  {% graphql model, direct_url: context.params.direct_url, user_id: context.params.user_id %}
    mutation model_create($direct_url: String!, $user_id: ID!) {
      model_create(model: {
        custom_model_type_name: "modules/direct_s3_upload/uppy"
        user_id: $user_id
        properties: [{ name: "my_upload" value: $direct_url }]
      }) {
        id
      }
    }
  {% endgraphql %}
  {{ model | fetch: "model_create" }}
{% endif %}

Note

You might want to add an authorization policy to ensure that users can only add images to their own accounts.

Step 5: Show the results

To check the results of image processing, get some of them from the database and display them.

First you need a query to fetch models.

modules/direct_s3_upload/public/graph_queries/get_models.graphql
query get_models($per_page: Int = 5, $user_id: ID!) {
  models(
    per_page: $per_page
    sort: { created_at: { order: DESC } }
    filter: {
      model_schema_name: { value: "modules/direct_s3_upload/uppy" },
      user_id: { value: $user_id }
    }
  ) {
    total_entries
    results {
      id
      my_upload: property_upload(name: "my_upload") {
        url
        versions
      }
    }
  }
}

Then you can use it to display images.

modules/direct_s3_upload/public/views/pages/images.liquid

<!-- Get recently uploaded images for this user, in all versions -->
{% graphql models = "modules/direct_s3_upload/get_models", user_id: context.current_user.id | dig: "models" %}

{% if models.total_entries > 0 %}
  <hr>

  <div class="my-2 row">
    <div class="col-9">
      <h3>Last 5 processed images:</h3>
    </div>
  </div>

  <!-- Show all versions next to each other -->
  {% for model in models.results %}
    {% assign versions = model | dig: "my_upload", "versions" %}
    <div class="card my-4">
      <div class="card-body">
        <div class="row">
            <div class="col-sm-2">
              <h4>Original</h4>
              <a href="{{ model.my_upload.url }}"><img src="{{ model.my_upload.url }}" alt="Original" width="150"></a>
            </div>
          {% for version in versions %}
            <div class="col-sm-2">
              <h4>{{ version[0] }}</h4>
              <a href="{{ version[1] }}"><img src="{{ version[1] }}" alt="{{ version[0] }}" width="150"></a>
            </div>
          {% endfor %}
        </div>
      </div>
    </div>
  {% endfor %}
{% endif %}

Important notes / troubleshooting

General

  • Instead of sending an image to our server you will send a url (returned by S3 response) to the image inside the text field for given image. Image URL should look similar to https://near-me-oregon-staging.s3-us-west-2.amazonaws.com/uploads%2Fbc50d6da-c17e-477d-b143-c65422c221fd%2Ftest.3+%282%29.png
  • After form submission give our server couple of seconds to pickup your image, generate size versions, process them and upload to the new S3 location so the CDN could pick it up for you and datatabase to be updated with CDN links

If you have problems that you cannot figure out, go to live example page, inspect whats being sent to S3 and our server respectively and find differences between your requests (params, headers, method, form encoding) and those that work - the difference usually is whats broken.

AWS

  • Make sure your form is using POST method
  • Make sure your form is sending data as enctype="multipart/form-data"
  • Do not include any fields that are not listed in the upload_url_payload key, otherwise AWS will return error 403 with message Invalid according to Policy: Extra input fields: xxx
  • Make sure file input is sent last in the form
  • Remember to set your file input name to file

Read more about aws requirements.

Uppy

If you use Uppy and its S3 plugin as we did in this tutorial, you dont have to worry about most AWS notes.

Live example and source code

To play with a live example, create a developer account at https://examples.platform-os.com and go to the Direct S3 upload images page.

Source code can be found on GitHub.

Additional resources

Contribute to this page

Github Icon

Questions?

We are always happy to help with any questions you may have.

contact us