错误使用的Apache Mahout中使用HDInsight运行电影推荐 [英] Error in running movie recommendations by using Apache Mahout with HDInsight
问题描述
我跑到下面的code,但收到一个错误...
I ran the following code but receiving an error...
# The HDInsight cluster name.
$clusterName = "my-cluster-name"
Use-AzureHDInsightCluster $clusterName
# NOTE: The version number portion of the file path
# may change in future versions of HDInsight.
# So dynamically grab it using Hive.
$mahoutPath = Invoke-Hive -Query '!${env:COMSPEC} /c dir /b /s ${env:MAHOUT_HOME}\examples\target\*-job.jar' | where {$_.startswith("C:\apps\dist")}
$mahoutPath = $mahoutPath -replace "\\", "/"
$jarFile = "file:///$mahoutPath"
#
# If you are using an earlier version of HDInsight,
# set $jarFile to the jar file you
# uploaded.
# For example,
# $jarFile = "wasb:///example/jars/mahout-core-0.9-job.jar"
# The arguments for this job
# * input - the path to the data uploaded to HDInsight
# * output - the path to store output data
# * tempDir - the directory for temp files
$jobArguments = "-s", "SIMILARITY_COOCCURRENCE",
"--input", "wasb:///u.data",
"--output", "wasb:///example/out",
"--tempDir", "wasb:///temp/mahout"
# Create the job definition
$jobDefinition = New-AzureHDInsightMapReduceJobDefinition `
-JarFile $jarFile `
-ClassName "org.apache.mahout.cf.taste.hadoop.item.RecommenderJob" `
-Arguments $jobArguments
# Start the job
$job = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $jobDefinition
# Wait on the job to complete
Write-Host "Wait for the job to complete ..." -ForegroundColor Green
Wait-AzureHDInsightJob -Job $job
# Write out any error information
Write-Host "STDERR"
Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $job.JobId -StandardError
我已经上传使用Azure存储资源管理器包含hdinsight文件容器..
I have uploaded the u.data file using azure storage explorer to the root of the container which contains the hdinsight files..
我收到错误的行。
PS C:> $ =工作启动AzureHDInsightJob -cluster $ CLUSTERNAME -JobDefinition $ jobDefinition
PS C:> $job = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $jobDefinition
错误:
启动AzureHDInsightJob:请求失败,code:InternalServerError
内容:{错误:空}
在行:1字符:8
+ $ =工作启动AzureHDInsightJob -cluster $ CLUSTERNAME -JobDefinition $ jobDefiniti ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo:NotSpecified:(:) [起动AzureHDInsightJob] HttpLayerException
+ FullyQualifiedErrorId:Microsoft.WindowsAzure.Management.HDInsight.Framework.Core.Library.WebRequest.HttpLayerE
xception,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.StartAzureHDInsightJobCmdlet
Start-AzureHDInsightJob : Request failed with code:InternalServerError Content:{"error":null} At line:1 char:8 + $job = Start-AzureHDInsightJob -Cluster $clusterName -JobDefinition $jobDefiniti ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Start-AzureHDInsightJob], HttpLayerException + FullyQualifiedErrorId : Microsoft.WindowsAzure.Management.HDInsight.Framework.Core.Library.WebRequest.HttpLayerE xception,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.StartAzureHDInsightJobCmdlet
任何帮助表示衷心appeciated ..
Any help is sincerely appeciated..
感谢
推荐答案
这看起来像一个最近更改为蜂巢/邓普顿的HDInsight集群,它现在返回的文件路径末尾的CRLF上。我改变了脚本以下解决它:
This looks like a recent change to either Hive/Templeton on the HDInsight cluster, where it now returns a CRLF on the end of the file path. I changed the script to the following to fix it:
$mahoutPath = Invoke-Hive -Query '!${env:COMSPEC} /c dir /b /s ${env:MAHOUT_HOME}\examples\target\*-job.jar' | where {$_.startswith("C:\apps\dist")}
$noCRLF = $mahoutPath -replace "`r`n", ""
$cleanedPath = $noCRLF -replace "\\", "/"
$jarFile = "file:///$cleanedPath"
这篇关于错误使用的Apache Mahout中使用HDInsight运行电影推荐的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!