我该如何将dosObjectExist()请求批处理到Amazon S3? [英] How can I batch doesObjectExist() requests to Amazon S3?

问题描述

对于大量项目，我需要检查S3中是否存在一组密钥. (每组键都与大量项目中的一个相关.)

I need to check whether a set of keys exist in S3, for each of a large number of items. (Each set of keys relates to one of the large number of items).

我正在使用PHP SDK(v2)

I am using the PHP SDK (v2)

当前，我为每个键调用$client->doesObjectExist(BUCKET, $key)，这是一个瓶颈(每次调用到S3的往返时间).

Currently I am calling $client->doesObjectExist(BUCKET, $key) for each of the keys, which is a bottleneck (the round-trip time to S3 for each call).

我希望做类似$client->doesObjectExist(BUCKET, $batch)其中$batch = array($key1, $key2 ... $keyn)的操作，并让客户端检查所有这些键，然后返回响应数组(或其他类似结构).

I would prefer to do something like $client->doesObjectExist(BUCKET, $batch) where $batch = array($key1, $key2 ... $keyn), and for the client to check all of those keys then come back with an array of responses (or some other similar structure).

我遇到了很少 引用 "batch api"听起来很有希望，但没有具体内容.我猜想这可能仅在v1 SDK中存在.

I have come across a few references to a "batch api" which sounds promising, but nothing concrete. I'm guessing that this might have been present only in the v1 SDK.

推荐答案

您可以通过使用基础的Guzzle库功能，使用适用于PHP的AWS开发工具包进行并行请求.由于doesObjectExist方法实际上是在这种情况下执行HeadObject的操作.您可以通过执行以下操作来创建HeadObject命令组:

You can do parallel requests using the AWS SDK for PHP by taking advantage of the underlying Guzzle library features. Since the doesObjectExist method actually does HeadObject operations under that hood. You can create groups of HeadObject commands by doing something like this:

use Aws\S3\S3Client;
use Guzzle\Service\Exception\CommandTransferException;

function doObjectsExist(S3Client $s3, $bucket, array $objectKeys)
{
    $headObjectCommands = array();
    foreach ($objectKeys as $key) {
        $headObjectCommands[] = $s3->getCommand('HeadObject', array(
            'Bucket' => $bucket,
            'Key'    => $key
        ));
    }

    try {
        $s3->execute($headObjectCommands); // Executes in parallel
        return true;
    } catch (CommandTransferException $e) {
        return false;
    }
}

$s3 = S3Client::factory(array(
    'key'    => 'your_aws_access_key_id',
    'bucket' => 'your_aws_secret_key',
));
$bucket = 'your_bucket_name';
$objectKeys = array('object_key_1', 'object_key_2','object_key_3');

// Returns true only if ALL of the objects exist
echo doObjectsExist($s3, $bucket, $objectKeys) ? 'YES' : 'NO';

如果您要从响应中获取数据，而不仅仅是密钥是否存在，则可以更改try-catch块来执行类似的操作.

If you want data from the responses, other than just whether or not the keys exist, you can change the try-catch block to do something like this instead.

try {
    $executedCommands = $s3->execute($headObjectCommands);
} catch (CommandTransferException $e) {
    $executedCommands = $e->getAllCommands();
}

// Do stuff with the command objects
foreach ($executedCommands as $command) {
    $exists = $command->getResponse()->isSuccessful() ? "YES" : "NO";
    echo "{$command['Bucket']}/{$command['Key']}: {$exists}\n";
}

适用于PHP的AWS开发工具包用户指南，但我还将看看枪口批处理文档.

这篇关于我该如何将dosObjectExist()请求批处理到Amazon S3?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！