如何从列表中异步调用多个 URL [英] How can I call many URLs from a list asynchronously

问题描述

我有几十万个网址需要调用.这些是对应用程序服务器的调用，应用程序服务器将处理它们并将状态代码写入表.我不需要等待响应(成功/失败)，只需要服务器收到请求即可.我还希望能够指定一次可以运行多少个并发作业，因为我还没有计算出 tomcat 可以处理多少个并发请求.

I have a few hundred thousand URLs that I need to call. These are calls to an application server which will process them and write a status code to a table. I do not need to wait for a response (success/fail), only that the server got the request. I also want to be able to specify how many concurrent jobs can be running at once as I haven't worked out how many concurrent requests tomcat can handle.

这是我到目前为止所得到的，基本上取自其他人尝试做类似的事情，只是不使用 url 调用.文本文件在其自己的行中包含每个 url.网址如下所示:

Here's what I've got so far, basically taken from someone's else's attempt to do something similar, just not with url calls. The text file contains each url on its own line. The url looks like this:

http://webserver:8080/app/mwo/services/create?server=ServerName&e1user=admin&newMWONum=123456&sourceMWONum=0&tagNum=33-A-1B

和代码:

$maxConcurrentJobs = 10
$content = Get-Content -Path "C:\Temp\urls.txt"

foreach ($url in $content) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le $maxConcurrentJobs) {
        Start-Job {
             Invoke-WebRequest -UseBasicParsing -Uri $using:url
        }
    } else {
         $running | Wait-Job -Any
    }
    Get-Job | Receive-Job
}

我遇到的问题是每个作业"都会出现 2 个错误，我不知道为什么.当我转储 url 数组 $content 时，它看起来很好，当我一一运行 Invoke-WebRequest 时，它们可以正常工作.

The problems I'm having is that it is giving 2 errors per "job" and I'm not sure why. When I dump the url array $content it looks fine and when I run my Invoke-WebRequest one by one they work without error.

126    Job126          BackgroundJob   Running       True            localhost            ...                
Invalid URI: The hostname could not be parsed.
    + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodComman 
   d
    + PSComputerName        : localhost

Invalid URI: The hostname could not be parsed.
    + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodComman 
   d
    + PSComputerName        : localhost

任何帮助或替代实现将不胜感激.我愿意不使用 powershell，但我仅限于 Windows 7 桌面或 Windows 2008 R2 服务器，而且我可能会在服务器上使用 url 中的 localhost 运行最终脚本以减少网络延迟.

Any help or alternative implementations would be appreciated. I'm open to not using powershell, but I'm limited to Windows 7 Desktops or Windows 2008 R2 servers, and I'd probably be running the final script on the server itself using localhost in the url to cut down on network delays.

推荐答案

使用 Job 会产生大量开销，因为每个新 Job 都会产生一个新进程.

With Jobs you incur a large amount of overhead, because each new Job spawns a new process.

使用运行空间代替！

$maxConcurrentJobs = 10
$content = Get-Content -Path "C:\Temp\urls.txt"

# Create a runspace pool where $maxConcurrentJobs is the 
# maximum number of runspaces allowed to run concurrently    
$Runspace = [runspacefactory]::CreateRunspacePool(1,$maxConcurrentJobs)

# Open the runspace pool (very important)
$Runspace.Open()

foreach ($url in $content) {
    # Create a new PowerShell instance and tell it to execute in our runspace pool
    $ps = [powershell]::Create()
    $ps.RunspacePool = $Runspace

    # Attach some code to it
    [void]$ps.AddCommand("Invoke-WebRequest").AddParameter("UseBasicParsing",$true).AddParameter("Uri",$url)

    # Begin execution asynchronously (returns immediately)
    [void]$ps.BeginInvoke()

    # Give feedback on how far we are
    Write-Host ("Initiated request for {0}" -f $url)
}

如链接的 ServerFault 帖子所述，您还可以使用更通用的解决方案，例如 Invoke-Parallel，它基本上可以完成上述操作

As noted in the linked ServerFault post, you can also use a more generic solution, like Invoke-Parallel, which basically does the above

这篇关于如何从列表中异步调用多个 URL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！