Automate your video ingest with AWS Lambda and Bitmovin’s Cloud Video Encoder

In today’s new era of cloud computing many companies are leveraging the cloud by uploading simple tasks into on-demand and serverless cloud services. AWS Lambda abstracts the servers layer and offers pay-per-compute-time pricing. This makes it an ideal tool to run code for virtually any type of application without any cloud servers administration hassle, which makes it an ideal method to enable simple VoD encoding workflows.

This tutorial describes a simple watch-folder implementation example in Java that enables a simple VoD encode workflow by detecting when a file is uploaded into an AWS S3 bucket and then executing the encode job and finally creating the manifest files. The Java project in this tutorial contains all the dependencies and plugins needed from AWS and Bitmovin to get you started quickly. Here the Github repository project. The encode and manifest configuration in this example project are generic for a full HD encoding into four h.264 video profiles, one audio AAC and a DASH manifest. But, the encoding and manifest configuration can be changed to accommodate any of the capabilities and features that Bitmovin product supports, such as three-pass encode, per-title, GOP structure and much more.  

Below we show the high-level solution diagram, which consists of Bitmovin Cloud Encoding Service, Bitmovin Java API Client, two AWS Lambda functions and two AWS S3 buckets.

High-level sequence of events:

• User uploads VoD assets into an input S3 bucket
• AWS Lambda function catches the S3 put events
• AWS Lambda function process the event, which contains s3 input bucket name and asset key
• AWS Lambda function sends encoding job to Bitmovin Cloud Service
• Bitmovin’s Cloud Service send a webhook post containing the encode ID  to AWS API Gateway, which sends the manifest job to Bitmovin.
• AWS Lambda logs are stored in Amazon CloudWatch Logs

Description of AWS lambda functions:

• AWS Lambda function `watchfolder-java-encode`

This is the lambda function that gets triggered first. It has an S3 trigger event to receive notifications every time a file is uploaded into the S3 input bucket and execute the encoding job in Bitmovin Cloud Service. It also uses AWS CloudWatch Log resource to keep a record of the logs.

• AWS Lambda function `watchfolder-java-manifest`:

This function has an API Gateway with Lambda Proxy Integration that catches the webhook push notifications from Bitmovin cloud encode service and starts the manifest job.

bitmovinSimplewatchfolder project structure:

├── pom.xml
├── readme.md
├── src
├── pom.xml
├── readme.md
├── src
│   ├── main
│   │ ├── java
│   │ │   ├── encode
│   │ │   │ ├── EncoderManager.java
│   │ │   │ └── LambdaEncode.java
│   │ │   └── manifest
│   │ │       ├── LambdaManifest.java
│   │ │       └── ManifestManager.java
│   │ └── resources
│   └── test
│       └── java
│           ├── LambdaEncodeTest.java
│           └── LambdaManifestTest.java

1. encoding.EncoderManager: This class contains Bitmovin’s encode job configuration (Creates and starts the job), including the webhook configuration required to trigger the manifest job when the encode finishes. The encode configuration job encodes video in h.264, audio in AAC and creates 4 profiles (240p @ 400kbps, 480p @ 800kbps, 720p @ 4000kbps, 1080p @ 6500kbps) fragmented in fMP4 containers. This configuration can be changed to accommodate your specific encoding needs, for example, changing video bitrates, codec (AV1, VP9, h264, h265), GOP structure, color scheme, encryption and much more. You can use and adapt different configuration examples that we have available in Bitmovin’s Java repository here to guide you in the process. And, here there is more information about Bitmovin encoding features.
2. encoding.LambdaEncode: This class contains the handleRequest method needed to configure the `watchfolder-java-encode` lambda function. It receives S3’s ObjectCreated events and calls `handleRequest` method of `encoding.EncoderManager` class to create and start the encoding job.
3. package.ManifestManager: This class contains Bitmovin’s manifest job configuration (Creates and starts the job). The manifest job creates a DASH manifest for the encoded and fMP4 fragments encoded previously. This configuration can be changed to accommodate your specific manifest needs, for example, adding HLS, additional audio adaptations, etc. You can use and adapt different configuration examples that we have available in Bitmovin’s Java repository here to guide you in the process.
4. package.LambdaManifest: This class contains the `handleRequest` method needed to configure the `watchfolder-java-manifest` lambda function. It receives webhook post notifications thru the API Gateway interface and calls `handleRequest` method of `package.ManifestManager` class to create and start the manifest job.

How to configure the bitmovinSimpleWatchfolder project and deploy it in AWS

High-level steps:

1. Create an AWS IAM Role
2. Create the `watchfolder-java-manifest` lambda function in AWS
3. Create the `watchfolder-java-encode` lambda function in AWS
4. Update variables in project class `EncoderManager`
5. Update variables in project class `PackagerManager`
6. Build and package the project
7. Upload the JAR file to the lambda functions

STEP 1  – Create an AWS IAM Role with AWSLambdaExecute  access

Create a new Role with an AWSLambdaExecute policy. AWS console > Roles > Create New Role. Then attach an AWSLambdaExecute policy. This is a pre-made policy and can be found by simply searching for it.

STEP 2 – Create `watchfolder-java-manifest` AWS lambda function

AWS Console -> Lambda -> Functions -> Create function

Select `Author from scratch` and name the function

Select `Java 8` for runtime

Choose the role that you just created with the AWSLambdaExecute policy

Click “Create Function” this will take you to the AWS Lambda watch-folder dashboard view.

Under the “Function code” section, specify the handleRequest method path:

`manifest.LambdaManifest::handleRequest`

By default lambda functions timeout after a few seconds so it is necessary to increase the timeout setting to avoid unwanted halts.

Add an API Gateway trigger. Select `API Gateway` from the designer window

In the API Gateway configuration screen, choose “Create a new API” and select “Open” security

Click Add, then Click Save at the top right of the lambda dashboard view.

This will create the API Gateway in a proxy configuration and will give you the API endpoint, which you will need to configure your Java project by updating the NOTIFICATION_URL variable in the EncoderManager class.

STEP 3 – Create `watchfolder-java-encode` AWS lambda function

Follow the same steps for creating the `watchfolder-java-manifest` lambda function, but carry out the changes below:

Under the “Function code” section, specify the handleRequest method path:

`encode.LambdaEncode::handleRequest`

Instead of adding an API Gateway Trigger add an S3 trigger event by selecting S3 in the Designer window

Select the input bucket you want to use  as your watch-folder, select `All object create events` and leave the rest default configurations, then click Add

Click Save at the top of the page and now you have this lambda function ready. Note: See session, How to create a lambda TEST for local tests in your lambda dashboard view?

STEP 4 – Update project classes variables (API Keys, S3 bucket information, Notification URL)

Clone bitmovinSimpleWatchfolder project:

• git clone https://github.com/bitmovin/solutions-support/tree/feature/simple_aws_watchfolder/customers/DTC/public/bitmovinSimpleWatchfolder

Update Classes’ Configuration:

• Fill in the following variables: API Keys, S3 bucket information, Notification URL variables in `EncoderManager` and `PackagerManager` classes. Note: For Notification URL use the API endpoint URL of your API Gateway (see former STEP 2)

Build and package the project:

• Using the terminal, navigate to the project folder where the pom.xml is and execute `mvn package`
• Within Intellij IDEA, within the “Maven Projects” view, select “package” under Lifecycle.

Upload the JAR file to both lambda functions:

• Locate `bitmovin-simple-watchfolder-1.0-SNAPSHOT.jar` file in the project’s target folder
• Click the lambda icon in your “watchfolder-java-encode` dashboard view

•  Upload the JAR file (`bitmovin-simple-watchfolder-1.0-SNAPSHOT.jar`)

Click Save in the lambda dashboard view

Repeat the same steps for the `watchfolder-java-manifest` lambda function

Done!

At this point your watch-folder is functional. You can go ahead and drop a video file in the S3 input bucket and watch in Bitmovin’s dashboard the status and progress of your encoding job.

Troubleshooting – Where are the logs?

To see the lambda function logs go to AWS Console -> CloudWatch -> Logs -> Select the lambda function

Troubleshooting – How to create a lambda TEST for local tests in your lambda dashboard view

Create new test event

Expand the test dropdown menu and click configure test events

Under “Event Template”, select “Amazon S3 Put”

Update the `s3.bucket.name`, `s3.arn`, and `s3.object.key` properties to point to a valid file

Save and just click test next to the dropdown test menu