给定一组来自​​sharepoint的文件URL，下载它们的最快方法是什么？ [英] Given a set of file URLs from sharepoint, what is the fastest way to download them all?

问题描述

我有一个SharePoint列表项URL列表和列表项附件URL。我需要从每个文件中下载所有文件。

200个sharepoint服务器场中有50,000,000个文件。它们适用于各种SharePoint环境：2010年，2013年，2016年和2019年。我还有一些我需要下载的SharePoint在线实例。



我们有限制下载到文件< = 80兆字节。并且已经删除了我们不想下载的所有文件。



所以现在我需要以最快的方式从sharepoint下载5000万个文件。



我目前正在使用多线程HTTP客户端程序来下载它们。使用单个HTTP请求下载每个文件。



这需要很长时间，因为它似乎对SharePoint Web服务器CPU造成了相当大的负担。特别是如果我将下载线程的数量设置得太高。



我在这里阅读这篇文章  https://www.itprotoday.com/5-reasons-why-you-have-sharepoint-performance-issues 它似乎说一下Blob来记录
转换的原因。但我不是肯定的。



有没有更快的方法来使用批处理操作下载所有这些文件？例如，某种方式可以从一个Web请求中同时获取多个文件而不是一次一个？zh


我在这里做了一些阅读：< a href ="https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-crawl-binary-large-objects-blobs-in-sharepoint">
https ：//docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-crawl-binary-large-objects-blobs-in-sharepoint



我们有什么办法可以直接连接数据库来批量下载内容吗？
b


或者有没有办法使用导出功能以某种方式完成此操作？
$

解决方案

这就是为什么不应该使用附件的确切原因在列表中。

我在2007年的迁移过程中经历了这一次 - > 2010年，我们在PowerShell中编写了一个脚本来下载附件，然后将其放入文档库，然后删除附件。 我希望我能保存那个剧本，但它只有
大约50行左右。或者

祝你好运，

I have a list of SharePoint list item URLs and list item attachment urls. I need to download all the files from each one.

There are 50,000,000 files over 200 sharepoint farms. They are on a variety of SharePoint environments: 2010, 2013, 2016 and soon 2019. I also have a couple SharePoint online instances I need to download from.

We have limited the download to documents <= 80 megabytes. And have eliminated all documents that we don't want to download.

So now I need the fastest way possible to download 50million files from sharepoint.

I am currently using a multi-threaded HTTP client program to download them. Each file is downloaded with a single HTTP request.

This is taking a very long time because it seems to tax the SharePoint web servers CPU pretty heavily. Especially if I turn up my number of download threads too high.

I read this article here https://www.itprotoday.com/5-reasons-why-you-have-sharepoint-performance-issues and it seems to say something about Blob to document conversion is the reason why. But I'm not positive.

Is there some faster way to download all these files using a batch operation? For example, some way to get several files at once from a single web request instead of one-at-a-time?

I was doing some reading here: https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-crawl-binary-large-objects-blobs-in-sharepoint

Is there any way we can use a direct connection to the database to bulk download content?

Or is there a way to use the export feature to somehow accomplish this?

解决方案

This IS the exact reason why attachments should NEVER be used in lists.

I went through this once during a migration from 2007 -> 2010, and we wrote a script in PowerShell to download the attachment, then put it into a document library, then delete the attachment.  I wish I would've saved that script, but it was only about 50 lines long, or so. 

Good luck,

这篇关于给定一组来自​​sharepoint的文件URL，下载它们的最快方法是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！