Uploading Files Directly to Amazon S3 Using Uppy

Last edit: Nov 25, 2019
  • Contributors:
  • diana-lakatos

This guide will help you to create a user-friendly image upload feature with image processing.

Note

This example uses some Twitter Boostrap classes to make it a little bit prettier.

Requirements

This is an advanced tutorial. To follow it, you should be familiar with basic platformOS concepts, HTML, JavaScript, Liquid, GraphQL, and some topics in the Get Started section.

Steps

Uploading images directly to Amazon S3 is a four-step process:

Step 1: Create Model Schema and image processing configuration

There will be only one property in Model Schema used in this example (photo), but it will generate 4 different versions of the image uploaded by the user. Apart from explicitly defined versions, there is also an original version that is always there.

modules/direct_s3_upload/public/model_schemas/uppy.yml
name: uppy
properties:
- name: photo
  type: photo
  image_versions:
    large_bw: # version name
      output:
        format: jpeg # output format
        quality: 70 # output quality
      resize:
        width: 1600 # width of the image - in this case max width
        height: 1200 # max height of the image - in this case max height
        fit: inside # preserve aspect ratio, do not crop the image, make sure all pixels are within viewport
        without_enlargement: false # if image is smaller than width/height, do not enlarge it
      manipulate:
        greyscale: true # make image greyscale (similar to css "filter: grayscale(1)")
        blur: 20 # blur using gaussian mask (similar to css "filter: blur(1.5rem)")
        flatten: red # merge alpha transparency channel (if exists) with given color
    small:
      output:
        format: webp
        quality: 80
      resize:
        width: 640
        height: 480
        fit: cover
        position: top # define the gravity direction or strategy when cropping the image using fit cover or contain
    thumb:
      output:
        format: png
        quality: 80
      resize:
        width: 200
        fit: contain
        background: "rgba(255, 255, 255, 0.5)" # if resized image is not covering whole viewport, set background color
    center:
      output:
        format: jpeg
        quality: 100
      resize:
        height: 400
        width: 400
        fit: cover

Step 2: Get presign data and expose it using a form

To upload a file to AWS S3 directly, you will need some server-generated data, called signature. This information is from GraphQL. It takes model_schema_name and property_name as arguments so it can forward image_versions from the model.

modules/direct_s3_upload/public/views/pages/images.liquid

---
slug: direct-s3-upload/images
---

{% graphql data %}
mutation presign {
  image_presign_url(
    options: {
      content_length: { gte: 0, lte: 2048 }
      model_property: {
        model_schema_name: "modules/direct_s3_upload/uppy"
        property_name: "photo"
      }
    }
  ) {
    upload_url            # url to s3 where you will post data.
    upload_url_payload    # series of fields required for the request to be valid
  }
}
{% endgraphql %}
{% assign data = data.image_presign_url %}

<form action="{{ data.upload_url }}"
  data-s3-uppy="form"
  data-s3-uppy-user-id="{{ context.current_user.id }}"
  hidden>
  {% for field in data.upload_url_payload %}
    <input type="hidden" name="{{ field[0] }}" value='{{ field[1] }}'>
  {% endfor %}
</form>

Below the form, paste the code responsible for containing Uppy.

modules/direct_s3_upload/public/views/pages/images.liquid
<div class="row">
  <div class="col-5">
    <div class="card">
      <div class="card-body">
        <div id="drag-drop-area"></div> <!-- This is where Uppy will inject itself -->
      </div>
    </div>
  </div>

  <div class="col-7">
    <div class="card">
      <div class="card-body">
        <p>URLs to image returned by AWS (input):</p>
        <small>
          <ol data-s3-uppy="log"></ol>
        </small>
      </div>
    </div>
  </div>
</div>

Step 3: Configure Uppy and use presigned data

Uppy is a JavaScript widget that helps to handle uploads. With its S3 plugin on board, you can make the implementation much shorter and easier.

However, before you can use any scripts, you need to load them.

modules/direct_s3_upload/public/views/pages/images.liquid
<link href="https://transloadit.edgly.net/releases/uppy/v1.6.0/uppy.css" rel="stylesheet">
<script src="https://transloadit.edgly.net/releases/uppy/v1.6.0/uppy.js" defer></script>
<script src="https://cdn.jsdelivr.net/npm/sweetalert2@9" defer></script>
<script src="https://uploads.prod01.oregon.platform-os.com/instances/1/assets/" defer></script>

This is the final images.js file, responsible for uploading the image, logging its URL, creating the model, and showing the confirmation box.

modules/direct_s3_upload/public/assets/images.js
const _form = document.querySelector('[data-s3-uppy="form"]');
const _log = document.querySelector('[data-s3-uppy="log"]');

const uppy = Uppy.Core({
  autoProceed: true,
  restrictions: {
    maxFileSize: 2097152,
    maxNumberOfFiles: 3,
    allowedFileTypes: ['image/png', 'image/jpeg', 'image/webp']
  }
})
  .use(Uppy.Dashboard, {
    inline: true,
    target: '#drag-drop-area',
    note: 'Images only, up to 3 files, 2MB each',
    width: '100%',
    height: 350
  })
  .use(Uppy.DragDrop)
  .use(Uppy.GoldenRetriever)
  .use(Uppy.AwsS3, {
    getUploadParameters() {
      // 1. Get URL to post to from action attribute
      const _url = _form.getAttribute('action');
      // 2. Create Array from FormData object to make it easy to operate on
      const _formDataArray = Array.from(new FormData(_form));
      // 3. Create a JSON object from array
      const _fields = _formDataArray.reduce((acc, cur) => ({ ...acc, [cur[0]]: cur[1] }), {});

      // 4. Return resolved promise with Uppy. Uppy it will add file in file param as the last param
      return Promise.resolve({
        method: 'POST',
        url: _url,
        fields: _fields
      });
    }
  });

uppy.on('upload-success', (_file, data) => {
  const li = document.createElement('li');
  li.textContent = data.body.location;
  _log.appendChild(li);
});

uppy.on('complete', ({ failed, successful }) => {
  /*
    For every successfully uploaded image to S3, send request to the Instance
    that will create a model with the uploaded image's URL as direct_url param.
  */
  Promise.all(successful.map(({ response }) => createImage(response.body.location)))
  .then(() => {
    // Show confirmation box (SweetAlert2) after all the images has been created
    Swal.fire({
      title: 'Images uploaded',
      icon: 'success',
      text: 'Press refresh to see the results',
      confirmButtonText: 'Refresh',
      showCloseButton: true
    }).then(result => {
      if (!result.value) {
        return;
      }

      // Reload page after "Refresh" button has been clicked inside Sweet Alert2
      window.location.reload();
    });
  });
});

const createImage = imageUrl => {
  // Get logged in user id
  const userId = _form.dataset.s3UppyUserId;

  // Create model for this user with s3 image url
  return fetch('/direct-s3-upload/images/model_create', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Accept: 'application/json'
    },
    body: JSON.stringify({ direct_url: imageUrl, user_id: userId })
  }).then(response => response.json());
};

Step 4: Create page that will create the model with the image

For fetch to succeed, you need to create a page under the /direct-s3-upload/images/model_create path. It will execute the model_create mutation with params passed in the XHR request.

modules/direct_s3_upload/public/views/pages/model_create.liquid

---
slug: direct-s3-upload/images/model_create
method: post
---
{% if context.params.user_id %}
{% graphql model, direct_url: context.params.direct_url, user_id: context.params.user_id %}
mutation model_create($direct_url: String!, $user_id: ID!) {
  model_create(model: {
    custom_model_type_name: "modules/direct_s3_upload/uppy"
    user_id: $user_id
    custom_images: {
      name: "photo"
      direct_url: $direct_url
    }
  }) {
    id
  }
}
{% endgraphql %}
{{ model | fetch: "model_create" | json }}
{% endif %}

Note

You might want to add a check so that users can only add images to their own accounts.

Step 5: Show the results

To see the results of image processing, get some of them from the database and display them.

First you need a query to fetch models.

modules/direct_s3_upload/public/graph_queries/get_models.graphql
query get_models($per_page: Int = 5, $user_id: ID!) {
  models(
    per_page: $per_page
    sort: { created_at: { order: DESC } }
    filter: {
      model_schema_name: { value: "modules/direct_s3_upload/uppy" },
      deleted_at: { exists: false },
      user_id: { value: $user_id }
    }
  ) {
    total_entries
    results {
      id
      images {
        original: url
        large_bw: url(version: "large_bw")
        small: url(version: "small")
        thumb: url(version: "thumb")
        center: url(version: "center")
      }
    }
  }
}

Then you can use it to display images.

modules/direct_s3_upload/public/views/pages/images.liquid

<!-- Get recently uploaded images for this user, in all versions -->
{% graphql models = "modules/direct_s3_upload/get_models", user_id: context.current_user.id | dig: "models" %}

{% if models.total_entries > 0 %}
  <hr>

  <div class="row my-2">
    <div class="col-9">
      <h3>Last 5 processed images:</h3>
    </div>
  </div>

  <!-- Show all versions next to each other -->
  {% for model in models.results %}
    {% assign versions = model | dig: "images" | first %}

    <div class="card my-4">
      <div class="card-body">
        <div class="row">
          {% for version in versions %}
            <div class="col-sm-2">
              <h4>{{ version[0] }}</h4>
              <a href="{{ version[1] }}">
                <img src="{{ version[1] }}" alt="{{ version[0] }}" width="150">
              </a>
            </div>
          {% endfor %}
        </div>
      </div>
    </div>
  {% endfor %}
{% endif %}

Important notes / troubleshooting

General

  • Instead of sending an image to our server you will send a url (returned by S3 response) to the image inside the text field for given image. Image URL should look similar to https://near-me-oregon-staging.s3-us-west-2.amazonaws.com/uploads%2Fbc50d6da-c17e-477d-b143-c65422c221fd%2Ftest.3+%282%29.png
  • After form submission give our server couple of seconds to pickup your image, generate size versions, process them and upload to the new S3 location so the CDN could pick it up for you and datatabase to be updated with CDN links

If you have problems that you cannot figure out, go to live example page, inspect whats being sent to S3 and our server respectively and find differences between your requests (params, headers, method, form encoding) and those that work - the difference usually is whats broken.

AWS

  • Make sure your form is using POST method
  • Make sure your form is sending data as enctype="multipart/form-data"
  • Do not include any fields that are not listed in the upload_url_payload key, otherwise AWS will return error 403 with message Invalid according to Policy: Extra input fields: xxx
  • Make sure file input is sent last in the form
  • Remember to set your file input name to file

Read more about aws requirements.

Uppy

If you use Uppy and its S3 plugin as we did in this tutorial, you dont have to worry about most of the above.

Live example and source code

To play with a live example, create a developer account at https://examples.platform-os.com and go to the Direct S3 upload images page.

Source code can be found on GitHub.

Additional resources

Questions?

We are always happy to help with any questions you may have.