在Android中合并多个Azure的Cloud Block Blob [英] Merging multiple Azure's Cloud block blobs in Android
问题描述
我是Azure的Blob云的新手.我基本上想将视频文件从我的Android应用程序上传到Azure云，但是我不能这样做，因为一旦大小达到32MB，它就会停止，并抛出OutOfMemory异常.因此，我对如何解决此问题进行了一些研究，并提出了一种解决方案，将文件拆分为字节，然后将其上传为多个Blob.最后，将它们编译为一个blob.但是我不知道该怎么做.我尝试使用commitBlockList，但为此无法获取每个Blob的ID.
I'm new to Azure's Blob cloud. I want to basically upload a video file from my android app to Azure cloud but I can't because as soon as the size reaches 32MB it stops throws an exception as OutOfMemory. So I did a bit of research on how should I fix this problem and I came up with a solution to break a file into bytes and than upload it as multiple blobs. At the end compile them into one blob. But I don't know how to do that. I tried using commitBlockList but for that I can't get the Id's of each blobs.
try {
// Setup the cloud storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
int maxSize = 64 * Constants.MB;
// Create a blob service client
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
blobClient.getDefaultRequestOptions().setSingleBlobPutThresholdInBytes(maxSize);
CloudBlobContainer container = blobClient.getContainerReference("testing");
container.createIfNotExists();
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.CONTAINER);
container.uploadPermissions(containerPermissions);
CloudBlockBlob finalFile = container.getBlockBlobReference("1.jpg");
CloudBlob b = container.getBlockBlobReference("temp");
String Lease = b.getSnapshotID();
b.uploadFromFile(URL);
List<BlockEntry> blockEntryIterator = new ArrayList<>();
blockEntryIterator.add(new BlockEntry(Lease));
finalFile.commitBlockList(blockEntryIterator);
} catch (Throwable t) {
}
~~~更新~~~
我试图将文件分成几部分，但现在出现此错误指定的Blob或块内容无效". 公共无效splitTest(String URL)抛出IOException，URISyntaxException，InvalidKeyException，StorageException {
I tried to break the file into parts but now I have this error "The specified blob or block content is invalid". public void splitTest(String URL) throws IOException, URISyntaxException, InvalidKeyException, StorageException {
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"STARTED");
CloudBlockBlob blob = null;
List<BlockEntry> blockList = null;
try{
// get file reference
FileInputStream fs = new FileInputStream( URL );
File sourceFile = new File( URL);
// set counters
long fileSize = sourceFile.length();
int blockSize = 3 * (1024 * 1024); // 256K
int blockCount = (int)((float)fileSize / (float)blockSize) + 1;
long bytesLeft = fileSize;
int blockNumber = 0;
long bytesRead = 0;
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("testing");
String title = "Android_" + getFileNameFromUrl(URL);
// get ref to the blob we are creating while uploading
blob = container.getBlockBlobReference(title);
blob.deleteIfExists();
// list of all block ids we will be uploading - need it for the commit at the end
blockList = new ArrayList<BlockEntry>();
// loop through the file and upload chunks of the file to the blob
while( bytesLeft > 0 ) {
blockNumber++;
// how much to read (only last chunk may be smaller)
int bytesToRead = 0;
if ( bytesLeft >= (long)blockSize ) {
bytesToRead = blockSize;
} else {
bytesToRead = (int)bytesLeft;
}
// trace out progress
float pctDone = ((float)blockNumber / (float)blockCount) * (float)100;
// save block id in array (must be base64)
String x = "";
if(blockNumber<=9) {
traceLine( "blockid: 000" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "blockid000" + blockNumber;
}
else if(blockNumber>=10 && blockNumber<=99){
traceLine( "blockid: 00" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "blockid00" + blockNumber;
}
else if(blockNumber>=100 && blockNumber<=999){
traceLine( "blockid0: " + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "blockid0" + blockNumber;
}
else if(blockNumber>=1000 && blockNumber<=9999){
traceLine( "blockid: " + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "blockid" + blockNumber;
}
String blockId = Base64.encodeToString(x.getBytes(),Base64.DEFAULT).replace("\n","").toLowerCase();
traceLine( "Base 64["+x+"] -> " + blockId);
BlockEntry block = new BlockEntry(blockId);
blockList.add(block);
// upload block chunk to Azure Storage
blob.uploadBlock( blockId, fs, (long)bytesToRead);
// increment/decrement counters
bytesRead += bytesToRead;
bytesLeft -= bytesToRead;
}
fs.close();
traceLine( "CommitBlockList. BytesUploaded: " + bytesRead);
blob.commitBlockList(blockList);
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"UPLOAD COMPLETE");
return;
}
catch (StorageException storageException) {
traceLine("StorageException encountered: ");
traceLine(storageException.getMessage());
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"FAILED");
assert blockList != null;
blob.commitBlockList(blockList);
return;
} catch( IOException ex ) {
traceLine( "IOException: " + ex );
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"FAILED");
assert blockList != null;
blob.commitBlockList(blockList);
return;
} catch (Exception e) {
traceLine("Exception encountered: ");
traceLine(e.getMessage());
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"FAILED");
assert blockList != null;
blob.commitBlockList(blockList);
return;
}
}
~~~工作更新~~~
~~~WORKING UPDATE~~~
对于想要重用此方法打开文件的任何人>并逐字节读取它，都可以使用此方法.我可以上传〜1GB的文件，但之后它会给我Error 500并崩溃.如果有人有任何解决方案，请告诉我.
For anyone who wants to reuse this method to open file > and read it bytes by bytes can use this. I am able to upload ~1GB of file but after that it gives me Error 500 and crashes. If anyone has any solution please do let me know.
public boolean splitTest(String URL) throws IOException, URISyntaxException, InvalidKeyException, StorageException {
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"STARTED");
CloudBlockBlob blob = null;
List<BlockEntry> blockList = null;
try{
// get file reference
FileInputStream fs = new FileInputStream(URL);
File sourceFile = new File(URL);
// set counters
long fileSize = sourceFile.length();
int blockSize = 512 * 1024; // 256K
//int blockSize = 1 * (1024 * 1024); // 256K
int blockCount = (int)((float)fileSize / (float)blockSize) + 1;
long bytesLeft = fileSize;
int blockNumber = 0;
long bytesRead = 0;
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("testing");
String title = "Android/Android_" + getFileNameFromUrl(URL).replace("\n","").replace(" ","_").replace("-","").toLowerCase();
// get ref to the blob we are creating while uploading
blob = container.getBlockBlobReference(title);
traceLine("Title of blob -> " + title);
if(blob.exists())
blob.deleteIfExists();
blob.setStreamWriteSizeInBytes(blockSize);
// list of all block ids we will be uploading - need it for the commit at the end
blockList = new ArrayList<>();
// loop through the file and upload chunks of the file to the blob
while( bytesLeft > 0 ) {
// how much to read (only last chunk may be smaller)
int bytesToRead = 0;
if ( bytesLeft >= (long)blockSize ) {
bytesToRead = blockSize;
} else {
bytesToRead = (int)bytesLeft;
}
// trace out progress
float pctDone = ((float)blockNumber / (float)blockCount) * (float)100;
// save block id in array (must be base64)
String x = "";
if(blockNumber<=9) {
traceLine( "tempblobid0000" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "tempblobid0000" + blockNumber;
}
else if(blockNumber>=10 && blockNumber<=99){
traceLine( "tempblobid000" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "tempblobid000" + blockNumber;
}
else if(blockNumber>=100 && blockNumber<=999){
traceLine( "tempblobid00" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "tempblobid00" + blockNumber;
}
else if(blockNumber>=1000 && blockNumber<=9999){
traceLine( "tempblobid0" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "tempblobid0" + blockNumber;
}
else if(blockNumber>=10000 && blockNumber<=99999){
traceLine( "tempblobid" + blockNumber + ". " + String.format("%.0f%%",pctDone) + " done.");
x = "tempblobid" + blockNumber;
}
String blockId = Base64.encodeToString(x.getBytes(),Base64.NO_WRAP).replace("\n","").toLowerCase();
traceLine( "Base 64["+ x +"] -> " + blockId);
BlockEntry block = new BlockEntry(blockId);
blockList.add(block);
// upload block chunk to Azure Storage
blob.uploadBlock( blockId, fs, (long)bytesToRead);
notification2(a,pctDone);
//a.update(pctDone);
// increment/decrement counters
bytesRead += bytesToRead;
bytesLeft -= bytesToRead;
blockNumber++;
}
fs.close();
traceLine( "CommitBlockList. BytesUploaded: " + bytesRead + "\t total bytes -> " + fileSize + "\tBytes Left -> " + bytesLeft);
blob.commitBlockList(blockList);
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"UPLOAD COMPLETE");
return true;
}
catch (StorageException storageException) {
traceLine("StorageException encountered: ");
traceLine(storageException.getMessage());
traceLine("HTTP Status code -> " + storageException.getHttpStatusCode());
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"FAILED");
if (blob != null) {
blob.commitBlockList(blockList);
}
return false;
} catch( IOException ex ) {
traceLine( "IOException: " + ex );
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"FAILED");
if (blob != null) {
blob.commitBlockList(blockList);
}
return false;
} catch (Exception e) {
traceLine("Exception encountered: ");
traceLine(e.getMessage());
new ConversionNotificationSetup().sendNotification(a.getApplicationContext(),"FAILED");
if (blob != null) {
blob.commitBlockList(blockList);
}
return false;
}
}
推荐答案
这取决于您使用的手机类型.如果您要用尽32MB的内存，那么您是在非常小的电话上或使用许多其他进程.查看您的电话，以及有多少可用内存，例如Gaurav提及和我的其他答案提及，将阈值降低到该水平.分块处理对您要执行的操作并没有真正的帮助.
This comes down to what kind of phone you're using. If you're running out memory at 32MB you're on a very small phone or using a lot of other processes. Look at your phone and how much memory you have available and like Gaurav mentions and my other answer mentions, reduce the threshold to that level. Chunking yourself won't really help for what you're trying to do.
您要查看的两个设置是Blob本身的singleBlobPutThresholdInBytes和setStreamWriteSizeInBytes. singleBlobPutThresholdInBytes影响何时开始分块与放置整个blob，setStreamWriteSizeInBytes影响发生分块时块的大小.放置阈值默认为64MB，写入大小为4MB.请注意，如果减小写入大小，则可能无法获得最大的块Blob大小，因为块Blob限制为50k块.尝试将singleBlobPutThreshold减少到您可以处理的最大内存量，直到4MB-如果小于4MB，您确实有麻烦，还需要减小流写入大小.
The two settings you want to look at are singleBlobPutThresholdInBytes and setStreamWriteSizeInBytes on the blob itself. singleBlobPutThresholdInBytes affects when we start chunking vs putting the whole blob and setStreamWriteSizeInBytes affects the size of the blocks if chunking occurs. The put threshold defaults to 64MB and the write size to 4MB. Note that if you reduce the write size you may not be able to get the maximum block blob size as block blobs are limited to 50k blocks. Try reducing singleBlobPutThreshold to the amount of memory you can handle up until 4MB - if it's less than 4MB you're really in trouble and need to reduce the stream write size as well.
这篇关于在Android中合并多个Azure的Cloud Block Blob的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
