R:从 Companies House API 获取 pdf 文档 [英] R: fetching pdf documents from Companies House API
本文介绍了R:从 Companies House API 获取 pdf 文档的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我正在尝试使用 R 从 API 中获取文档.感谢 这篇文章.我一直在按照上述步骤操作并取得部分成功，但在访问文档内容的最后一步仍然失败:
<块引用>- 查找您感兴趣的文件归档(例如，为公司提出归档历史请求 1).解析 "links" 字段中文档链接的响应:{ "document_metadata" : "link URI fragment here" }.
没问题:
库(httr)图书馆(jsonlite)图书馆(openssl)###检索归档历史####company_num = 'FC013908'键 = 'my_key'fh_path = paste0('/company/', str_to_upper(company_num), "/filing-history")fh_url <- modify_url("https://api.companieshouse.gov.uk/", path = fh_path)fh_test <- GET(fh_url, authentication(key, "")) #status_code = 200fh_parsed <- jsonlite::fromJSON(content(fh_test, "text",encoding = "utf-8"), flatten = TRUE)文档 <- fh_parsed$items完成.
<块引用>2 对于给定的文档，通过 CH Document API3 请求文档元数据.解析响应以获取可用的文档(mime)类型以及指向实际文档数据(文档 URI 片段)的链接.
这里没有问题:
md_meta_url = docs$links.document_metadata[1]key_pass <- paste0(key,":")decoded_auth <- paste0('基本', base64_encode(key_pass))md_test <- GET(md_meta_url,add_headers(Authorization = decoded_auth))md_test #status_code = 200！md_parsed <- jsonlite::fromJSON(content(md_test, "text",encoding = "utf-8"), flatten = TRUE)这样我就可以获取内容 URL:
cont_url = md_parsed$links$document<块引用>
请求实际文档9，指定 mime 类型(例如application/pdf").
我在不遵循重定向的情况下执行此操作，并且正如预期的那样，我获得了带有 location 标头的 302 状态代码:
accept = 'application/pdf'cont_test <- GET(cont_url,add_headers(授权=decoded_auth，接受=接受)，配置(跟随位置 = FALSE))final_url <- cont_test$headers$location>final_url[1] "https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/LjBouRHeXXpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/application-pdf?AWSAccessKeyId=ASIAJX7TVURFXZTY5DNQ&2957UtgS&Expires;Expires;Expires;Expires的x AMZ-安全令牌= FQoDYXdzEP％2F％2F％2F％2F％2F％2F％2F％2F％2F％2F％2FwEaDGxe7meYGe3OYhNwcSK3AwcVYJUXaUMf19oVO9s4qNPWN8AHjNNd5rrZhgE9YTkF1OmzyZSL5xHbls664kDP％2Bxd7dz9PIU5O1D％2BVxoDyoYcFiS6acDnO28KpfFE56lUZNfedf1jys％2FP0SJ8f％2F50Cbn93bfOlm0MZA9％2BQ2DYQvPfkWSvrDjMyCXHbu57gpZHjQKPNRTgzGXzUUCvFwREytGMM4eThhn4Glvvx％2FA8IiLbnsvgmEKw9iAj7KWIenhoJq3cTRytUpVeipLnQoBVLau8dFYkKdAHZaYM2Tlx0z6ObRb％2BGdm7W7eOVA1bFXuUXmUmnAHruDIwwLlgOVN2IJ9CxmJU22lY8jrEm％2BUivtrdp2oofn32PryBEJ8jJOg9cIpLbBBx％2FeOkng9zJwnZbute7Nmh％2BnaY2btsId6JjraFNsTvR％2B1qEZX9uuznUdJdqgVfTMj2gGrAmntwk0JAkILlvamzjWC％2F9vAqK7Xvt8aC6hlIMB2vdzTCU9Jf％2FrIMTClTJkk0BzBuvJ86t1l％2BXb4rF5Pab％2FegFpJ6nvZKqde％2F77wMMiTyG35EndmYx4AWqTIh9EofYwKZa9uciNvRT0E2％2BYnT5jZMo％2BdWn2QU％3D"然而，当我尝试
<块引用>再次从亚马逊请求此 URI，再次传递您想要的内容类型.我收到 400 错误:
final_test <- GET(final_url,add_headers(授权=decoded_auth，接受 = 接受))>最后一个考试回复 [https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/LjBouRHeXXpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/application-pdf?AWSAccessKeyId=ASIAJX7TVURFXZTY5DNQ&ExpiresZTY5DNQ&ExpiresZTY5DNQ&ExpiresZTY5DNQ&Expires25&7Utx6Rx6Rx7Utx6Rx7Utx7Utx6RxpXuRx7Utx7Utx6RxpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/AMZ-安全令牌= FQoDYXdzEP％2F％2F％2F％2F％2F％2F％2F％2F％2F％2F％2FwEaDGxe7meYGe3OYhNwcSK3AwcVYJUXaUMf19oVO9s4qNPWN8AHjNNd5rrZhgE9YTkF1OmzyZSL5xHbls664kDP％2Bxd7dz9PIU5O1D％2BVxoDyoYcFiS6acDnO28KpfFE56lUZNfedf1jys％2FP0SJ8f％2F50Cbn93bfOlm0MZA9％2BQ2DYQvPfkWSvrDjMyCXHbu57gpZHjQKPNRTgzGXzUUCvFwREytGMM4eThhn4Glvvx％2FA8IiLbnsvgmEKw9iAj7KWIenhoJq3cTRytUpVeipLnQoBVLau8dFYkKdAHZaYM2Tlx0z6ObRb％2BGdm7W7eOVA1bFXuUXmUmnAHruDIwwLlgOVN2IJ9CxmJU22lY8jrEm％2BUivtrdp2oofn32PryBEJ8jJOg9cIpLbBBx％2FeOkng9zJwnZbute7Nmh％2BnaY2btsId6JjraFNsTvR％2B1qEZX9uuznUdJdqgVfTMj2gGrAmntwk0JAkILlvamzjWC％2F9vAqK7Xvt8aC6hlIMB2vdzTCU9Jf％2FrIMTClTJkk0BzBuvJ86t1l％2BXb4rF5Pab％2FegFpJ6nvZKqde％2F77wMMiTyG35EndmYx4AWqTIh9EofYwKZa9uciNvRT0E2％2BYnT5jZMo％2BdWn2QU％3D]日期:2018-06-20 08:37状态:400内容类型:应用程序/xml尺寸:523 B<二元体>不用说，执行
browseURL(final_test$url)返回访问被拒绝错误.我怀疑这可能与亚马逊授权问题有关，类似于 此处.任何想法如何解决这个最后的障碍?
谢谢！
解决方案
答案由 @voracityemail 提供，以回应 我关于 Companies House Developers Hub 的问题.基本上，最终调用不需要 Authorization 标头，因此如果您为 final_test 运行以下代码:
final_test <- GET(final_url, add_headers(Accept = accept))它会返回200代码
<代码>>最后一个考试响应[https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/Rl1qKy2kNqdskHUIsqU9u0bGzH2goTfJfnCrNg4S0lg/application-pdf?AWSAccessKeyId=ASIAJMG7NTZHYC4NH3MA&Expires=1530093768&Signature=EteMSmwXS%2FqqdOFRmYY%2Fgf187Aw%3D&安培; X-AMZ-安全令牌= FQoDYXdzELf％2F％2F％2F％2F％2F％2F％2F％2F％2F％2FwEaDOMKrcNPR6jb5bnzGSK3A1yzaoVZWhgAeXYCN9WJnxx8b％2BTKCEEZyZui3aR5j0WoNWIQhW9GIQ8R4xTGVkRjwQIhzgDp％2BRCfXGQ0CfPCOfseaQri5m％2BWTEWBgjfToL7％2FMdcC1IINMTFRrih1APE％2FmmTcQaW7SvyZWv3Q4bVQB％2FtOsiX5k8rWVsT7％2FecfQmnJMljcKF0％2F3vDRTtLRURTCtrdegfnIFrSqXkelLxVVypKY9UeURBgxAgngOgoP7YhYt3wD％2BEz5rBdNfMvF1Zuv91hLGDyBaKuV4fRKMRXlymDHCwNgNZl3JeyuAmnX8pexK6PJzH7MerM8QX8LoPfge1yutvqEj0％2FjRSYEShOWUebecQ2tJqWIEOZly0Ji8fc％2BMtFDO1FWZBrMl6lXgkwTMpELnTH5％2BP4ULMdFfEz30bWSnAuTGXcAxsoFWsFTIE2uO35zgkOsAUT2un4UNGnL2S8XexWbgwq％2B％2Bhtxo9ruP9WA8mTpjBkup2Qe5EpvUiNwGX9APjThi7QFTllVWWvpKgzKTSBh％2Btua9xK8RgiNAYDgEa5k％2BH％2FmWIP56WglBE6r3HGsXgbi％2Bff8Rg8z2lVFLo8f9hVv％2BCYoptXM2QU％3D]日期:2018-06-27 10:02状态:200内容类型:应用程序/pdf大小:21.7 KB<二元体>
然后
browseURL(final_test$url)将在浏览器中打开指定的文档.胜利！
I'm trying to fetch documents from the API using R. Appreciate the clarification of the process in this post. I've been following the above steps with partial success, but still fail the last step to get access to documents' content:
- Find the document filing you're interested in (e.g. make a filing history request1 for the company). Parse the response for the link to the document in the field "links" : { "document_metadata" : "link URI fragment here" }.
No problem:
library(httr)
library(jsonlite)
library(openssl)
### retrieving filing history ####
company_num = 'FC013908'
key = 'my_key'
fh_path = paste0('/company/', str_to_upper(company_num), "/filing-history")
fh_url <- modify_url("https://api.companieshouse.gov.uk/", path = fh_path)
fh_test <- GET(fh_url, authenticate(key, "")) #status_code = 200
fh_parsed <- jsonlite::fromJSON(content(fh_test, "text",encoding = "utf-8"), flatten = TRUE)
docs <- fh_parsed$items
Done.
2 For a given document request the document metadata via CH Document API3. Parse the response to get the document (mime) types available and the link to the actual document data (document URI fragment).
No problems here:
md_meta_url = docs$links.document_metadata[1]
key_pass <- paste0(key,":")
decoded_auth <- paste0('Basic ', base64_encode(key_pass))
md_test <- GET(md_meta_url,
add_headers(Authorization = decoded_auth)
)
md_test #status_code = 200!
md_parsed <- jsonlite::fromJSON(content(md_test, "text",encoding = "utf-8"), flatten = TRUE)
This way I can obtain the content URL:
cont_url = md_parsed$links$document
Request the actual document9, specifying the mime type (e.g. "application/pdf").
I do it while NOT following the redirect and, as expected, I get the 302 status code with the location header:
accept = 'application/pdf'
cont_test <- GET(cont_url,
add_headers(Authorization = decoded_auth,
Accept = accept),
config(followlocation = FALSE)
)
final_url <- cont_test$headers$location
> final_url
[1] "https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/LjBouRHeXXpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/application-pdf?AWSAccessKeyId=ASIAJX7TVURFXZTY5DNQ&Expires=1529483765&Signature=uUQx6RTW7XBLqx4L6pYr5tOUySg%3D&x-amz-security-token=FQoDYXdzEP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGxe7meYGe3OYhNwcSK3AwcVYJUXaUMf19oVO9s4qNPWN8AHjNNd5rrZhgE9YTkF1OmzyZSL5xHbls664kDP%2Bxd7dz9PIU5O1D%2BVxoDyoYcFiS6acDnO28KpfFE56lUZNfedf1jys%2FP0SJ8f%2F50Cbn93bfOlm0MZA9%2BQ2DYQvPfkWSvrDjMyCXHbu57gpZHjQKPNRTgzGXzUUCvFwREytGMM4eThhn4Glvvx%2FA8IiLbnsvgmEKw9iAj7KWIenhoJq3cTRytUpVeipLnQoBVLau8dFYkKdAHZaYM2Tlx0z6ObRb%2BGdm7W7eOVA1bFXuUXmUmnAHruDIwwLlgOVN2IJ9CxmJU22lY8jrEm%2BUivtrdp2oofn32PryBEJ8jJOg9cIpLbBBx%2FeOkng9zJwnZbute7Nmh%2BnaY2btsId6JjraFNsTvR%2B1qEZX9uuznUdJdqgVfTMj2gGrAmntwk0JAkILlvamzjWC%2F9vAqK7Xvt8aC6hlIMB2vdzTCU9Jf%2FrIMTClTJkk0BzBuvJ86t1l%2BXb4rF5Pab%2FegFpJ6nvZKqde%2F77wMMiTyG35EndmYx4AWqTIh9EofYwKZa9uciNvRT0E2%2BYnT5jZMo%2BdWn2QU%3D"
However, when I try to
Request this URI from Amazon again passing the content type you want again. I get 400 error:
final_test <- GET(final_url,
add_headers(Authorization = decoded_auth,
Accept = accept
))
> final_test
Response [https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/LjBouRHeXXpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/application-pdf?AWSAccessKeyId=ASIAJX7TVURFXZTY5DNQ&Expires=1529483765&Signature=uUQx6RTW7XBLqx4L6pYr5tOUySg%3D&x-amz-security-token=FQoDYXdzEP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGxe7meYGe3OYhNwcSK3AwcVYJUXaUMf19oVO9s4qNPWN8AHjNNd5rrZhgE9YTkF1OmzyZSL5xHbls664kDP%2Bxd7dz9PIU5O1D%2BVxoDyoYcFiS6acDnO28KpfFE56lUZNfedf1jys%2FP0SJ8f%2F50Cbn93bfOlm0MZA9%2BQ2DYQvPfkWSvrDjMyCXHbu57gpZHjQKPNRTgzGXzUUCvFwREytGMM4eThhn4Glvvx%2FA8IiLbnsvgmEKw9iAj7KWIenhoJq3cTRytUpVeipLnQoBVLau8dFYkKdAHZaYM2Tlx0z6ObRb%2BGdm7W7eOVA1bFXuUXmUmnAHruDIwwLlgOVN2IJ9CxmJU22lY8jrEm%2BUivtrdp2oofn32PryBEJ8jJOg9cIpLbBBx%2FeOkng9zJwnZbute7Nmh%2BnaY2btsId6JjraFNsTvR%2B1qEZX9uuznUdJdqgVfTMj2gGrAmntwk0JAkILlvamzjWC%2F9vAqK7Xvt8aC6hlIMB2vdzTCU9Jf%2FrIMTClTJkk0BzBuvJ86t1l%2BXb4rF5Pab%2FegFpJ6nvZKqde%2F77wMMiTyG35EndmYx4AWqTIh9EofYwKZa9uciNvRT0E2%2BYnT5jZMo%2BdWn2QU%3D]
Date: 2018-06-20 08:37
Status: 400
Content-Type: application/xml
Size: 523 B
<BINARY BODY>
Needless to say, executing
browseURL(final_test$url)
returns Access Denied error. I suspect it may have something to do with Amazon authorization problems similar to those described here. Any ideas how to solve this final hurdle?
Thanks!
解决方案
The answer was provided by @voracityemail in response to my question on Companies House Developers Hub. Basically, the final call doesn't require the Authorization header, so if you run the following code for final_test:
final_test <- GET(final_url, add_headers(Accept = accept))
It will return 200 code
> final_test
Response [https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/Rl1qKy2kNqdskHUIsqU9u0bGzH2goTfJfnCrNg4S0lg/application-pdf?AWSAccessKeyId=ASIAJMG7NTZHYC4NH3MA&Expires=1530093768&Signature=EteMSmwXS%2FqqdOFRmYY%2Fgf187Aw%3D&x-amz-security-token=FQoDYXdzELf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDOMKrcNPR6jb5bnzGSK3A1yzaoVZWhgAeXYCN9WJnxx8b%2BTKCEEZyZui3aR5j0WoNWIQhW9GIQ8R4xTGVkRjwQIhzgDp%2BRCfXGQ0CfPCOfseaQri5m%2BWTEWBgjfToL7%2FMdcC1IINMTFRrih1APE%2FmmTcQaW7SvyZWv3Q4bVQB%2FtOsiX5k8rWVsT7%2FecfQmnJMljcKF0%2F3vDRTtLRURTCtrdegfnIFrSqXkelLxVVypKY9UeURBgxAgngOgoP7YhYt3wD%2BEz5rBdNfMvF1Zuv91hLGDyBaKuV4fRKMRXlymDHCwNgNZl3JeyuAmnX8pexK6PJzH7MerM8QX8LoPfge1yutvqEj0%2FjRSYEShOWUebecQ2tJqWIEOZly0Ji8fc%2BMtFDO1FWZBrMl6lXgkwTMpELnTH5%2BP4ULMdFfEz30bWSnAuTGXcAxsoFWsFTIE2uO35zgkOsAUT2un4UNGnL2S8XexWbgwq%2B%2Bhtxo9ruP9WA8mTpjBkup2Qe5EpvUiNwGX9APjThi7QFTllVWWvpKgzKTSBh%2Btua9xK8RgiNAYDgEa5k%2BH%2FmWIP56WglBE6r3HGsXgbi%2Bff8Rg8z2lVFLo8f9hVv%2BCYoptXM2QU%3D]
Date: 2018-06-27 10:02
Status: 200
Content-Type: application/pdf
Size: 21.7 kB
<BINARY BODY>
and then
browseURL(final_test$url)
will open the specified document in the browser. Victory!
这篇关于R:从 Companies House API 获取 pdf 文档的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
