UrlFetchApp无法连接到公共URL [英] UrlFetchApp not connecting to public URL

问题描述

我正尝试使用操作>下载> CSV下的下载CSV链接从此网站提取数据

I'm trying to pull data from this website using the download CSV link found under Actions > Download > CSV

https://app.udot.utah.gov/apex/eom/f?p=288:300

我在UrlFetchApp行上始终收到错误302，但我不明白为什么。我使用httpexception运行记录器，并收到错误消息，提示它无法连接到服务器。该网站是公开的，可以通过任何网络访问，因此我很难理解正在发生的事情。手动输入时，URL似乎下载了CSV，但是我无法让UrlFetchApp抓取它。

I get an error 302 on the UrlFetchApp line consistently and I'm not understanding why. I ran the logger with the httpexception and got the error that it could not connect to the server. The website is public and accessible regardless of network so I'm having a hard time understanding what's going on. The URL seems to download the CSV when input manually but I cannot get the UrlFetchApp to grab it.

var EquipUrl = "https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV";
var EquipContent = UrlFetchApp.fetch(EquipUrl).getContentText();
var EquipData = Utilities.parseCsv(EquipContent);

任何帮助将不胜感激！

我只是在注释中尝试了每个建议的以下内容，并且得到了相同的错误302结果。我还尝试将重定向设置为false，并且此操作以空的EquipData数组结尾

I just tried the following per suggestion in comments and had the same Error 302 result. I also tried setting redirects to false and this proceeds and ends with an empty EquipData array

var options = {
'followRedirects' :true
];
var EquipUrl = "https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV::::";
var EquipContent = UrlFetchApp.fetch(EquipUrl, options).getContentText();
var EquipData = Utilities.parseCsv(EquipContent);

感谢您对TheMaster的帮助！我在评论中说，该代码在var cookie行上吐出了一个错误，但该错误已自行纠正，因此我认为这可能对您而言是代码的错误副本。现在的问题是，最后的Logger行吐出无法解析文本错误。

Thanks for the help so far TheMaster! I said in my comment that the code was spitting out an error on the var cookies line but that error rectified itself so I think it may have been a bad copy of your code on my part. The problem is now that the final Logger line spits out a "Could not parse text" error.

function fetchUrlWithCookie() {
  var url = 'https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV';
  var response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    followRedirects: false,
  });
  var cookie = response.getAllHeaders()['Set-Cookie'];
  response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    headers: {
      Cookie: cookie, //send the cookie we got as header
    },
  });
  Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
}

我尝试了几种其他选择，例如尝试先登录以查看是否能找到任何东西。我可以从Stackdriver日志中提取任何信息来帮助发现问题吗？

I tried a few different options like trying to log before to see if I could find anything. Is there any information I could pull from Stackdriver logging that could help identify issues?

非常感谢TheMaster！有效！如果您能发表评论并提出解决问题的过程，我将不胜感激，这样我可以帮助您了解下次发生这种情况时要注意的事情，但我知道这是很多问题。

Thank you so much TheMaster! It worked! I would love if you could comment and address your problem solving process so I can help learn what to be on the lookout for next time this happens but I know that's asking a lot.

  var url = 'https://app.udot.utah.gov/apex/eom/f?p=288:300::CSV::::';
  var response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    followRedirects: true,
  });
  var cookie = response.getAllHeaders()['Set-Cookie']; 
  response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    headers: {
      Cookie: cookie, //send the cookie we got as headerw
    },
  });
  Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV

这是功能齐全的代码的副本。再次感谢您在这几天内为解决此问题所提供的帮助。

That's a copy of the code that fully functions. Thank you again for your help over this several day period to address this issue.

推荐答案

要点：

某些网站使用 cookies 来维护状态，会话并完成请求。由于 urlFetch 在服务器端而不是浏览器中运行，因此不会在请求之间维护cookie。但是，我们可以在第一个请求中手动获取cookie，然后在后续请求中发送它。

Key Points:

Some websites use cookies for maintaining state, session and completing the request. Since urlFetch runs on server side and not in a browser, cookies are not maintained across requests. However, We can manually get the cookie in the first request and send it in subsequent requests.

function fetchUrlWithCookie() {
  var url = 'xxxx'; //Your csv url
  var response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    followRedirects: false,
  });
  var cookie = response.getAllHeaders()['Set-Cookie']; //Get cookie from header 
  response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    headers: {
      Cookie: cookie, //send the cookie we got as header
    },
  });
  Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
}

参考文献：

• RFC6265


• UrlFetchApp


• 响应标头



References:

• RFC6265
• UrlFetchApp
• Response headers

这篇关于UrlFetchApp无法连接到公共URL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！