在Rails应用程序中将CarrierWave与Amazon Elastic Transcoder结合使用 [英] Using CarrierWave with Amazon Elastic Transcoder in a Rails app

问题描述

在此之前，我在Stack Overflow上又问了两个问题，但得到的帮助很少，我想就后代问题提出一个开放的问题。我花了很多时间来解析AWS-SDK API文档，却发现很少有直接答案可以满足我的需求。我也曾在AWS论坛上发帖，但未能在那里得到良好的回应。似乎找不到简单，全面，分步的解决方案。

I asked two additional questions before this on Stack Overflow, but got very little help and I thought I would ask an open question for posterity. I have spent time parsing the AWS-SDK API docs and found very little direct answers to my needs. I have also posted on the AWS forums and haven't been able to get a good response there. A simple, comprehensive, step-by-step solution seems impossible to find.

我已完成的工作：

• 直接用CarrierWave上载到s3。我遵循了Railscast＃383，并使其适应了我的需求。


• 我能够从s3存储桶中检索我的文件。

到目前为止，我已经做过的事情的详细信息：

Details about what I've done so far:

我使用了Carrierwave-Direct直接将其上传到s3（这利用了雾直接上传到s3）。使用Sidekiq在后台作业中处理上载。将文件放入存储桶后，我只需遍历用户上传的文件并通过s3中的上传网址来调用文件即可。

I used Carrierwave-Direct to upload direct to s3(this utilizes fog to deal with uploading directly to s3). The upload is processed in a background job with Sidekiq. After the file is put in the bucket I just retrieve it by iterating through a users uploads and call the file by the upload's url from s3.

这是我在这里丢失：

• 我需要使用AWS提供的Elastic Transcoder对视频进行转码。


• 我需要从输出存储桶中调出上传/转换的视频。如何从输出桶链接到URL？是新的URL引用还是URL与原始的上传URL相同？


• 我需要将转码后的视频从转码器集成到Cloudfront并使用JWPlayer进行显示。 / li>
  
• 如何在后台将API代码集成到我的上传过程中？

这里是到目前为止我的代码：

Here is my code so far:

我的上传器：

class VideoUploader < CarrierWave::Uploader::Base
 include CarrierWaveDirect::Uploader
end

我的初始化程序处理s3详细信息：

My initializer that handles the s3 details:

CarrierWave.configure do |config|

 config.fog_credentials = {
 provider:              'AWS',
 aws_access_key_id:     'AWS_ACCESS_KEY_ID',
 aws_secret_access_key: 'AWS_SECRET_ACCESS_KEY',
 region:                'us-west-1'}

 config.fog_directory  = 'video-input'
 config.fog_public     = false                                        # optional, defaults to true
 config.fog_attributes = { 'Cache-Control' => "max-age=#{365.day.to_i}" } # optional, defaults to {}
end

我的上传模型：

class Upload < ActiveRecord::Base
 belongs_to :user
 mount_uploader :video, VideoUploader


 after_save :enqueue_video

 def enqueue_video
  VideoWorker.perform_async(id, key) if key.present?
 end

 class VideoWorker
  include Sidekiq::Worker

  def perform(id, key)
    upload = Upload.find(id)
    upload.key = key
    video.remote_video_url = upload.video.direct_fog_url(with_path: true)
    upload.save!
  end
 end
end

我的观点：

显示所有用户上传的文件：

To show all user's uploads:

<% @user.uploads.each do |upload| %>
  <%= link_to upload.title, upload_url(upload) %>
<% end %>

显示上传内容（现在它只是一个下载链接）：

To show the upload (right now it is only a download link):

<h1><%= @upload.title %></h1>
<p><%= link_to @upload.video_url %></p>

我认为我的架构或表单不相关。

I don't think my schema or forms are relevant.

我认为代码可能如何工作的类似示例：

A similar sample of how I think the code might work:

我会将这段代码添加到Sidekiq工作者中，但是我不确定是否我做对了。我也不确定如何将我的上传连接到转换后的上传。

I would add this code into the Sidekiq worker, but I'm not sure if I'm doing this right. I'm also uncertain about how I'm going to connect my "upload" to the "converted upload".

 upload.update_column 'converted_video', 
 File.basename(upload.video.path)

transcoder = AWS::ElasticTranscoder::Client.new
transcoder.create_job(
  pipeline_id: APP_CONFIG[Rails.env][:pipeline_id],
  input: {
    key: upload.video.path,
    frame_rate: 'auto',
    resolution: 'auto',
    aspect_ratio: 'auto',
    interlaced: 'auto',
    container: 'auto'
  },
  output: {
    key: upload.converted_video.path,
    preset_id: WEB_M4_PRESET_ID,
    thumbnail_pattern: "",
    rotate: '0'
  }
)

链接到a有用的文章和有关Elastic Transcoder的文档：

Links to a helpful article and the docs about the Elastic Transcoder:

http://www.techdarkside.com/getting-started-with-the-aws-elastic-transcoder-api-in-rails

http://docs.aws.amazon.com/sdkforruby/api/Aws/ElasticTranscoder.html

推荐答案

我觉得用CarrierWave将文件导入S3是最困难的部分，而您已经做到了！下一部分看起来就像您正在爬山，但更像是在公园里漫步：）...实际上，您要做的事情少得令人惊讶，因为AWS仅能处理我们正在做的很多事情来自我们的一个输入（如果您算初次上传，则来自两个输入）是create_job请求...在开始之前，我只是说，从您的代码中引用的模块来看，您确实确实具有 gem文件中的gem'aws-sdk'，这在使用AWS资源时很重要（我们不会适当地配置SDK，因为这不在问题范围内，但是您可以按照以下说明进行操作github repo上的说明），我还要说，很明显，您已经完成了其中一些步骤，例如创建管道。尽管如此，我仍在为那些可能会偶然发现此答案的读者提供这些步骤...让我们开始吧：

I feel like getting your file into S3 with CarrierWave is the hardest part and you've already done it! The next part seems like you're climbing a mountain but it's more like a stroll in the park :)... There's surprisingly little for you to actually do because AWS is going to handle a lot of what we're doing with really only one input from us (two if you count the initial upload) being the create_job request... Before we begin, I'll just say that from the modules referenced in your code it looks like you do indeed have gem 'aws-sdk' in your gemfile, which is important when using AWS resources (we won't go in to configuring the SDK appropriately as that's outside of the scope of the question, but you can follow the instructions on the github repo), and I'll also say that it is obvious that you have been through some of these steps already, like creating a pipeline. I'm including those steps nonetheless for future readers who may stumble across this answer... Let's begin:

1. 首先。在管道将放置转码文件的位置创建一个S3存储桶。严格来说，您不需要两个存储桶，也可以只使用一个存储桶，但是两个存储桶可以使事情变得更加清洁，拥有一个存储桶本身不会花费任何额外费用（不过您需要为存储在存储桶中的存储空间付费）。

1. First things first. Create a S3 bucket where your pipeline is going to put the transcoded files. You don't strictly need two buckets, you could just use the same bucket, but two makes things much cleaner and having a bucket itself doesn't cost anything extra (you'll pay for storage in the bucket though).

创建一个CloudFront发行版以分发您的转码文件。对于原始域名，请点击输入字段，您将获得一个下拉列表，其中包括您帐户的S3存储桶；选择用于输出的S3存储桶（您在其中存储了在步骤1中创建的转码文件的存储桶）作为分发源。请注意创建发行版后将收到的唯一URL，它将类似于 https://d111111abcdef8.cloudfront.net ，这就是您要查找的地方以后再为您的文件。

Create a CloudFront distribution to distribute your transcoded files. For the Origin Domain Name click the input field and you'll get a dropdown list that includes your account's S3 buckets; select the S3 bucket for OUTPUTS, the one where you're putting your transcoded files, that you created in step 1, as the source for your distribution. Note the unique URL that you'll receive after creating your distribution, it will be something like https://d111111abcdef8.cloudfront.net and that's where you're going to look for your files later on.

在创建转码作业之前，您需要具有管道。基本上，管道是保存您的转码作业的队列。用户上传文件时，您将在此步骤中将转码作业添加到您创建的管道中。您只需创建一个管道即可，并且可以将所有要转码的作业添加到该管道。您告诉管道从哪个S3存储桶中获取作业的文件，然后告诉S3存储桶将作业中的输出文件放置在哪里。输出文件可以具有相同的名称，并且具有新的扩展名，因此，如果您上传myvideo.mp4并将其转码为.avi格式，则输出文件将为myvideo.avi（您也可以更改名称，但这很复杂事情和您的问题范围之外）。由于您知道作业中的文件名以及输出存储桶，因此只需将它们放在一起即可获取访问文件的URL（您必须确保已对​​存储桶设置了正确的访问权限以访问文件）。如果我的输出文件是myvideo.avi，并且我知道它已经输出到特定的存储桶（属于CloudFront发行版），则可以通过 myCloudFrontURL / myvideo访问它。 avi ...看起来像 https://d111111abcdef8.cloudfront.net/myvideo.avi 。由于我怀疑这将是一个标准过程（即所有上传的文件都将被转码成相同的格式，并且您的管道不会改变），因此我建议您创建管道使用GUI。您可以在此处阅读操作方法：
http： //docs.aws.amazon.com/elastictranscoder/latest/developerguide/creating-pipelines.html

Before you're able to create a transcoding job, you need to have a pipeline. A pipeline is basically the queue that holds your transcoding jobs. When a user uploads a file, you'll add a transcoding job to the pipeline you create in this step. You only have to create a pipeline once and you can add all jobs for transcoding to this pipeline. You tell the pipeline which S3 bucket to get the files for your jobs from, and you tell it the S3 bucket where it's to put the output files from the jobs. The output file can have the same name, and will have the new extension, so if you upload myvideo.mp4 and you transcode that to .avi format then the output file will be myvideo.avi (you could also change the name but that's complicating things and outside the scope of your question). Since you know the filename from the job, and you know the output bucket, you just have to put them together to get the URL to access the file (you'll have to make sure you've set the correct access permissions on the bucket in order to access the file). If my output file is myvideo.avi and I know it has been output to a specific bucket, which is part of my CloudFront distribution, I know that I'll be able to access it at myCloudFrontURL/myvideo.avi... It'll look something like https://d111111abcdef8.cloudfront.net/myvideo.avi. Since I suspect this is going to be a "standard" process (ie all the uploaded files are going to be transcoded to the same format and your pipeline isn't going to change), I'm going to suggest that you create your pipeline with the GUI. You can read how to do that here: http://docs.aws.amazon.com/elastictranscoder/latest/developerguide/creating-pipelines.html

现在我们有了一条管道，为了对我们上传的文件进行转码，我们需要在管道中创建一个作业。您正在与您的工作人员一起创建工作，这是一条正确的路，这是纯粹的香草哈希： http://docs.aws.amazon.com/sdkforruby/api/Aws/ElasticTranscoder/Types/CreateJobRequest.html 。一旦SDK将其发布，则ET管道将接管工作，该作业将添加到您的管道中，并将按照添加到ET管道中的顺序进行转码。

Now that we have a pipeline, in order to transcode our uploaded file, we need to create a job in the pipeline. You're on the right track with creating the job with your worker, it's a pure vanilla hash: http://docs.aws.amazon.com/sdkforruby/api/Aws/ElasticTranscoder/Types/CreateJobRequest.html. Once that is posted by the SDK, the ET pipeline will take over, the job will be added to your pipeline, and it will be transcoded in the order which it was added to your ET Pipeline.

EDIT 根据OP对这个答案的评论，用户可能还需要上传其他要求许多视频，您应该可以列出用户上传的所有视频。 Rails使这个超级容易。您有一个用户模型，并且有一个上载模型。在您的上传模型上，您具有 belongs_to：user 关联，这是完美的。由于用户可以上传很多文件，因此您需要将 has_many：uploads 关联添加到您的用户模型（ActiveRecord关联有两种方式）。我假设您使用Rails生成器创建了上载模型，如果这样做了，您会注意到它为您创建了一个迁移，并在数据库中创建了一个上载表。我不清楚您的模式是什么样子，但是我假设您运行的是在生成模型时创建的迁移，而没有进行任何更改（即仅生成关联表），并且您的Uploads表不包含列 user_id或 url。我们将通过在您的终端上运行 Rails g迁移AddColumnsToUploadsTable 来添加它们。从那里，我们将在 yourApp / db / migrations 文件夹中编辑迁移。我们将在迁移 add_column：uploads，：url，：string 和 add_reference：uploads，：user，：中为更改方法添加两行：索引=>正确，然后返回终端并运行 rake db：migrate 。这些列现在已添加到数据库的Upload表中。

EDIT Per the OP's comment on this answer, there could be additional requirements that Users can upload many videos and that you should be able to list all videos the user has uploaded. Rails makes this super easy. You have a User model, and you have a Upload model. You have the belongs_to :user association on your Upload model, which are perfect. Since a User can have many uploads you'll want to add the has_many :uploads association to your User model (ActiveRecord associations go two ways). I'm going to assume you used the Rails generator to create your Upload model, and if you did, you'll notice that it created a migration for you which created a Upload table in your database. I'm not clear what you schema looks like, but I'll assume that you ran the migration that was created when you generated your model without making any changes (ie just generating the associated table) and that your Uploads table does not include the columns "user_id" or "url". We'll add those by running Rails g migration AddColumnsToUploadsTable from your terminal. From there we'll go and edit our migration in the yourApp/db/migrations folder. We'll add two lines to the change method in our migration add_column :uploads, :url, :string and add_reference :uploads, :user, :index => true, and then we'll head back to our terminal and run rake db:migrate. Those columns are now added to our Upload table in the database.

5.1在步骤3中，我们创建了一个管道，在步骤4中，我们为该管道创建了一个作业。管道要求我们告诉它作业输出文件的存放位置。显然，因为我们已经告知管道将文件放置在何处，所以我们知道文件的位置。并非如此，因为我们使用CloudFront分发该文件，而不是使用我们的S3存储桶位置，所以我们将使用CloudFront网址。如果我们不使用CloudFront而仅使用S3存储桶，则将使用S3存储桶URL。继续前进，就像管道需要我们告诉它将输出文件放在何处，作业需要我们告诉它输出到哪种格式，因为我们告诉作业将代码转换为AVI格式，所以我们知道输出格式正在成为AVI。最后，由于我们知道上载的文件名，并且我们不会更改文件名，因此我们知道将要命名输出文件。 我们正在提供所有信息，并准确地告诉Elastic Transcoder 如何处理输出文件...因此，显然，我们知道存储桶的位置，文件名和文件扩展名，我们可以很容易地找出访问文件的网址：它将是 https：//< yourCloudFrontURL> /< videoName>。< extension> 。我们可以将yourCloudFrontURL，videoName和扩展名（我们知道）转换为变量，以供在其他地方使用。 urlname = https://d111111abcdef8.cloudfront.net/ \ filename = File.basename（params [：file] .original_filename，。* ） \ newextension = .avi 坚持使用标准的Ruby / Rails流程即可完成对数据库的操作：实例化一个新对象 video = Upload.new ，将新对象的user_id设置为current_user video.user_id = current_user_id 然后设置新对象 video.url = urlname + filename + newextension 的网址。我们已经设置了user_id，并设置了记录的网址，我们要做的就是用 video.save 保存它。然后，您可以按照标准方式按要求访问这些记录...要检索特定用户的所有视频列表，可以执行 @videos = Upload.where（：user_id => current_user.id）。如果user_id与current_user_id匹配，这将返回上载模型中所有对象的数组。

5.1 In step 3 we created a pipeline, and in step 4 we created a job for that pipeline. The pipeline requires us to tell it where to put the file that is output from the job. Obviously, because we've told the pipeline where to put that file, we know where it's going to be. Not that since we're using CloudFront to distribute that file, instead of using our S3 bucket location, we're going to use our CloudFront url. If we weren't using CloudFront and were just using the S3 bucket we would use the S3 bucket URL. Moving on, like the pipeline requires us to tell it where to put the output file, the job requires us to tell it which format to output to, since we tell the job to transcode to AVI format, we know that the output format is going to be AVI. Finally, since we know the name of the uploaded file, and we're not changing the name of the file, we know what the output file is going to be named. We are supplying all the information, and telling Elastic Transcoder exactly what to do with the output file... Therefore, obviously, we know the bucket location, filename, and file extension, and we can very easily figure out the url to access the file: it's going to be https://<yourCloudFrontURL>/<videoName>.<extension>. We can turn the yourCloudFrontURL, videoName and extension (which we know) in to variables for our use elsewhere. urlname = https://d111111abcdef8.cloudfront.net/ \ filename = File.basename(params[:file].original_filename, ".*") \ newextension = ".avi" Persisting that to your database can be done with the standard Ruby/Rails process: instantiate a new object video = Upload.new, set the user_id of the new object to the current_user video.user_id = current_user_id and then set the url of the new object video.url = urlname+filename+newextension. We've set the user_id and we've set the url of the record, all we need to do is save it with video.save. You can then access those records as required in the standard way... To retrieve a list of all the videos for a specific user you could do something like @videos = Upload.where(:user_id => current_user.id). This would return an array of all the objects in the Upload model were the user_id matches the current_user_id.

可选的下一步是让SNS向您的应用发送通知转码完成后。该通知对于准确了解新文件何时可用等情况可能很有用。如果您选择是否使用SNS通知，那么我们现在已经完成了转码，并且从代码的角度来看，我们不需要做很多事情，只需要创建由SDK发布到AWS资源的作业即可。

Optional next step is to have SNS send your app a notification when transcoding is complete. The notification could be useful for knowing things like exactly when the new file is available. If you chose to use the SNS notification or not, we've now completed our transcoding and we didn't have to do a whole lot from a code perspective except create a job that was posted to the AWS resource by the SDK.

下一步是通过CloudFront分发输出文件。由于我们已经将我们的发行版本的Origin设置为S3存储桶，我们在其中输出了转码后的文件，因此您无需执行其他任何操作。添加到存储桶的所有新文件都将自动添加到您的分发中。在此处阅读有关其工作原理的更多信息： http://docs.aws.amazon .com / AmazonCloudFront / latest / DeveloperGuide / AddingObjects.html 。

Your next step is to distribute the output file via CloudFront. Since we have already set Origin for our distribution as the S3 bucket that we output our transcoded files in, there's nothing else for you to do. Any new files added to the bucket(s) will automatically be added to your distribution. Read more about how that works here: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AddingObjects.html.

要播放通过JPlayer发行版分发的文件，您需要执行所需的所有操作才能使用播放器，然后通过Cloudfront URL从CDN加载文件（此格式： https://d111111abcdef8.cloudfront.net/myvideo.avi ） /玩家需要的位置。

In order to play a file you distributed via your distribution with JPlayer you'll need to do whatever it is that you need to do in order to use the player, and then load the file from the CDN via the Cloudfront URL for the file (this format: https://d111111abcdef8.cloudfront.net/myvideo.avi), when/where required by the player.

要启动转码过程（您将API代码与上传过程集成在一起），您唯一需要做的就是简单的<$ c $在Sidekiq worker上的对象上使用c> create_job 方法...一旦发生这种情况，除了等待杯中的输出文件可用时，除了抓住一杯咖啡以外，您实际上没有其他事情要做。您的CloudFront发行版。

To initiate the transcoding process (what you call integrating the API code with your uploading process) the only thing you need to do is that simple create_job method on your object on your Sidekiq worker... Once that happens there's really not a whole lot more for you to do except grab a cuppa while you wait for the output file to be available in your CloudFront distribution.

就是这样……您已经创建了输出S3存储桶，创建了发行​​版，创建了管道，并已抓取S3中的文件，使用该文件创建了一个作业，并将该作业添加到ET管道中。您已将该作业的结果输出到另一个S3存储桶，有选择地收到了作业完成的SNS通知，将该文件分发到了CloudFront CDN上，最后，您将该文件从CloudFront分发中加载到了浏览器的JPlayer中。

That's it... You've created the output S3 bucket, created the distribution, created the pipeline, grabbed a file from S3, created a job with that file and added that job to your ET pipeline. You've output the result of that job to a different S3 bucket, optionally received a SNS notification of job completion, distributed that file on the CloudFront CDN, and lastly you've loaded that file from your CloudFront distribution in to JPlayer in the browser.

希望有帮助！

这篇关于在Rails应用程序中将CarrierWave与Amazon Elastic Transcoder结合使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！