DriveApp.continueFileIterator(continuationToken)的正确用法 [英] Correct usage of DriveApp.continueFileIterator(continuationToken)

问题描述

我编写了一个脚本来遍历 Google Drive 文件夹中的大量文件.由于我对这些文件进行的处理超过了最大执行时间.自然地，我写入脚本以使用 DriveApp.continueFileIterator(continuationToken):令牌存储在项目属性中，当脚本运行时，它会检查是否有令牌，如果有，则从令牌创建 FileIterator，如果没有重新开始.

I've written a script to iterate through a large number of files in a Google Drive folder. Due to the processing I am doing on those files it exceeds the maximum execution time. Naturally I wrote into the script to use DriveApp.continueFileIterator(continuationToken): the token gets stored in the Project Properties and when the script runs it checks to see if there's a token, if there is it creates the FileIterator from the token if not it starts afresh.

我发现即使脚本使用继续令牌重新启动，它仍然从迭代开始，尝试再次处理相同的文件，这浪费了后续执行的时间.我是否错过了一些重要的命令或方法，使其从停止的地方开始?我应该在 while(contents.hasNext()) 循环的各个阶段更新延续令牌吗?

What have I found is even though the script restarts with the continuation token it still starts from the beginning of the iteration, trying to process the same files again which wastes time for the subsequent executions. Have I missed something vital as in a command or method to make it start from where it left off? Am I supposed to update the continuation token at various stages thoughout the while(contents.hasNext()) loop?

以下是精简的示例代码，让您了解一下:

Here's the sample code slimmed down to give you an idea:

function listFilesInFolder() {
  var id= '0fOlDeRiDg';
  var scriptProperties = PropertiesService.getScriptProperties();
  var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var lastExecution = scriptProperties.getProperty('LAST_EXECUTION');
  if (continuationToken == null) {
    // first time execution, get all files from drive folder
    var folder = DriveApp.getFolderById(id);
    var contents = folder.getFiles();
    // get the token and store it in a project property
    var continuationToken = contents.getContinuationToken();
    scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', continuationToken);
  } else {
    // we continue to import from where we left
    var contents = DriveApp.continueFileIterator(continuationToken);
  }
  var file;
  var fileID;
  var name;
  var dateCreated;

  while(contents.hasNext()) {
    file = contents.next();
    fileID = file.getId();
    name = file.getName();
    dateCreated = file.getDateCreated();
    if(dateCreated > lastExecution) {
      processFiles(fileID);
    }
  }
  // Finished processing files so delete continuation token
  scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var currentExecution = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd HH:mm:ss");
  scriptProperties.setProperty('LAST_EXECUTION',currentExecution);
};

推荐答案

就像 Jonathon 所说的那样，您错误地比较了日期.但这不是您的脚本的主要问题，也不是您提出的问题.

Like Jonathon said, you're comparing dates wrongly. But that's not the main issue with your script nor what you asked.

您错误的主要概念是在执行循环之前无法保存延续令牌.当您获得令牌时，它会保存您当时所在的位置，如果您之后继续迭代，则不会保存，您将在以后重复这些步骤，就像您正在体验一样.

The main concept you're getting wrong is that the continuation token can't be saved before you do your loop. When you get the token, it saves where you were at that moment, if you continue iterating afterwards, that's not saved and you will repeat those steps later, just like you're experiencing.

为了稍后获取令牌，您不能让脚本因错误而终止.您必须衡量在 5 分钟内可以处理多少文件，并在此之前手动停止脚本，以便有机会保存令牌.

To get the token later you cannot let your script terminate with an error. You have to measure how many files you can process under 5 minutes and stop your script manually before that, so you can have a chance at saving the token.

正确的做法如下:

function listFilesInFolder() {
  var MAX_FILES = 20; //use a safe value, don't be greedy
  var id = 'folder-id';
  var scriptProperties = PropertiesService.getScriptProperties();
  var lastExecution = scriptProperties.getProperty('LAST_EXECUTION');
  if( lastExecution === null )
    lastExecution = '';

  var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var iterator = continuationToken == null ?
    DriveApp.getFolderById(id).getFiles() : DriveApp.continueFileIterator(continuationToken);


  try { 
    for( var i = 0; i < MAX_FILES && iterator.hasNext(); ++i ) {
      var file = iterator.next();
      var dateCreated = formatDate(file.getDateCreated());
      if(dateCreated > lastExecution)
        processFile(file);
    }
  } catch(err) {
    Logger.log(err);
  }

  if( iterator.hasNext() ) {
    scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', iterator.getContinuationToken());
  } else { // Finished processing files so delete continuation token
    scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
    scriptProperties.setProperty('LAST_EXECUTION', formatDate(new Date()));
  }
}

function formatDate(date) { return Utilities.formatDate(date, "GMT", "yyyy-MM-dd HH:mm:ss"); }

function processFile(file) {
  var id = file.getId();
  var name = file.getName();
  //your processing...
  Logger.log(name);
}

无论如何，有可能在您的运行之间创建了一个文件，而您在继续迭代中没有得到它.然后，通过在上次运行后节省执行时间，您可能会在下次运行时错过它.我不知道你的用例，如果最终重新处理一些文件或遗漏一些文件是可以接受的.如果您根本没有任何一种情况，那么我看到的唯一解决方案是保存您已经处理过的所有文件的 ID.您可能需要将它们存储在驱动器文件中，因为 PropertiesService 可能太小而无法容纳太多 ID.

Anyway, it may be possible that a file gets created between your runs and you do not get it on your continued-iteration. Then, by saving the execution time after your the last run, you may miss it on your next run too. I do not know your use-case, if it's acceptable to eventually reprocess some files or to miss some. If you can't have either situations at all, then the only solution I see is to save the ids of all files you have already processed. You may need to store those on a drive file, because PropertiesService may be too small for too many ids.

这篇关于DriveApp.continueFileIterator(continuationToken)的正确用法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！