正确使用DriveApp.continueFileIterator（continuationToken） [英] Correct usage of DriveApp.continueFileIterator(continuationToken)

问题描述

我编写了一个脚本来遍历Google云端硬盘文件夹中的大量文件。由于我对这些文件所做的处理超出了最大执行时间。当然，我写入脚本使用DriveApp.continueFileIterator（continuationToken）：令牌存储在项目属性中，当脚本运行时，它会检查是否有令牌，如果存在则从令牌创建FileIterator，如果不是则重新开始。

I've written a script to iterate through a large number of files in a Google Drive folder. Due to the processing I am doing on those files it exceeds the maximum execution time. Naturally I wrote into the script to use DriveApp.continueFileIterator(continuationToken): the token gets stored in the Project Properties and when the script runs it checks to see if there's a token, if there is it creates the FileIterator from the token if not it starts afresh.

我发现的是，即使脚本重新启动了连续令牌，它仍然从迭代开始，尝试再次处理相同的文件，浪费时间进行后续处决。我是否错过了一些至关重要的命令或方法，使它从离开的地方开始？我是否应该在各个阶段更新continuation token（contents.hasNext（））循环？

What have I found is even though the script restarts with the continuation token it still starts from the beginning of the iteration, trying to process the same files again which wastes time for the subsequent executions. Have I missed something vital as in a command or method to make it start from where it left off? Am I supposed to update the continuation token at various stages thoughout the while(contents.hasNext()) loop?

以下是示例代码，以简化您的想法：

Here's the sample code slimmed down to give you an idea:

function listFilesInFolder() {
  var id= '0fOlDeRiDg';
  var scriptProperties = PropertiesService.getScriptProperties();
  var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var lastExecution = scriptProperties.getProperty('LAST_EXECUTION');
  if (continuationToken == null) {
    // first time execution, get all files from drive folder
    var folder = DriveApp.getFolderById(id);
    var contents = folder.getFiles();
    // get the token and store it in a project property
    var continuationToken = contents.getContinuationToken();
    scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', continuationToken);
  } else {
    // we continue to import from where we left
    var contents = DriveApp.continueFileIterator(continuationToken);
  }
  var file;
  var fileID;
  var name;
  var dateCreated;

  while(contents.hasNext()) {
    file = contents.next();
    fileID = file.getId();
    name = file.getName();
    dateCreated = file.getDateCreated();
    if(dateCreated > lastExecution) {
      processFiles(fileID);
    }
  }
  // Finished processing files so delete continuation token
  scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var currentExecution = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd HH:mm:ss");
  scriptProperties.setProperty('LAST_EXECUTION',currentExecution);
};

推荐答案

像Jonathon说的，你错误地比较了日期。但这不是你的脚本的主要问题，也不是你问的问题。

Like Jonathon said, you're comparing dates wrongly. But that's not the main issue with your script nor what you asked.

你错误的主要概念是在你做之前不能保存连续令牌你的循环。当你获得令牌时，它会保存那个时候的位置，如果你继续迭代，那么这些不会被保存，你将在后面重复这些步骤，就像你正在经历的那样。

The main concept you're getting wrong is that the continuation token can't be saved before you do your loop. When you get the token, it saves where you were at that moment, if you continue iterating afterwards, that's not saved and you will repeat those steps later, just like you're experiencing.

以后要获取标记，您不能让脚本以错误终止。您必须测量您可以在5分钟内处理多少个文件，并在此之前手动停止脚本，以便您可以保存令牌。

To get the token later you cannot let your script terminate with an error. You have to measure how many files you can process under 5 minutes and stop your script manually before that, so you can have a chance at saving the token.

以下是正确的这样做的方式：
$ b

Here's the correct way of doing it:

function listFilesInFolder() {
  var MAX_FILES = 20; //use a safe value, don't be greedy
  var id = 'folder-id';
  var scriptProperties = PropertiesService.getScriptProperties();
  var lastExecution = scriptProperties.getProperty('LAST_EXECUTION');
  if( lastExecution === null )
    lastExecution = '';

  var continuationToken = scriptProperties.getProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
  var iterator = continuationToken == null ?
    DriveApp.getFolderById(id).getFiles() : DriveApp.continueFileIterator(continuationToken);


  try { 
    for( var i = 0; i < MAX_FILES && iterator.hasNext(); ++i ) {
      var file = iterator.next();
      var dateCreated = formatDate(file.getDateCreated());
      if(dateCreated > lastExecution)
        processFile(file);
    }
  } catch(err) {
    Logger.log(err);
  }

  if( iterator.hasNext() ) {
    scriptProperties.setProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN', iterator.getContinuationToken());
  } else { // Finished processing files so delete continuation token
    scriptProperties.deleteProperty('IMPORT_ALL_FILES_CONTINUATION_TOKEN');
    scriptProperties.setProperty('LAST_EXECUTION', formatDate(new Date()));
  }
}

function formatDate(date) { return Utilities.formatDate(date, "GMT", "yyyy-MM-dd HH:mm:ss"); }

function processFile(file) {
  var id = file.getId();
  var name = file.getName();
  //your processing...
  Logger.log(name);
}

无论如何，文件可能会在您的跑步和您之间创建不要继续迭代。然后，通过在上次运行后保存执行时间，您可能会在下次运行时错过它。我不知道你的用例，如果最终重新处理一些文件或错过一些文件是可以接受的。如果你根本没有任何一种情况，那么我看到的唯一解决方案就是保存已处理的所有文件的ID。您可能需要将这些文件存储在驱动器文件中，因为对于太多的ID，PropertiesService可能太小。

Anyway, it may be possible that a file gets created between your runs and you do not get it on your continued-iteration. Then, by saving the execution time after your the last run, you may miss it on your next run too. I do not know your use-case, if it's acceptable to eventually reprocess some files or to miss some. If you can't have either situations at all, then the only solution I see is to save the ids of all files you have already processed. You may need to store those on a drive file, because PropertiesService may be too small for too many ids.

这篇关于正确使用DriveApp.continueFileIterator（continuationToken）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！