从HttpResponseStream读取失败 [英] Reading from a HttpResponseStream fails
问题描述
我遇到了一个问题，其中从HttpResponseStream读取失败，因为我要包装的StreamReader读取速度更快，因此Response Stream获得实际响应.我正在检索一个较小的文件(大约60k)，但是将响应处理为实际对象的解析器失败，因为它遇到了意外的字符(代码65535)，根据我的经验，该字符是您从某个字符读取时产生的字符StreamReader，并且没有其他可用字符.
I'm running into an issue where reading from a HttpResponseStream fails because the StreamReader that I'm wrapping around in reads faster that the Response Stream gets the actual response. I'm retrieving a reasonably small sized file (around 60k) but the Parser which processes the response into an actual object fails because it hits an unexpected character (Code 65535) which from experience I know to be the character produced when you read from a StreamReader and there are no further characters available.
出于记录，我知道返回的内容是有效的，并且可以正确解析，因为每次我运行代码时，失败都发生在文件的不同位置.失败的是以下的parser.Load()行.
For the record I know that the content being returned is valid and will parse correctly since the failure occurs at different points in the file each time I run the code. It's the parser.Load() line in the following where it fails.
是否有一种方法可以确保在尝试解析内容之前先读取所有内容，而无需将响应流复制到MemoryStream或字符串中然后进行处理?
Is there a way to ensure I've read all the content before attempting to parse it short of copying the response stream into a MemoryStream or string and then processing it?
/// <summary>
/// Makes a Query where the expected Result is an RDF Graph ie. CONSTRUCT and DESCRIBE Queries
/// </summary>
/// <param name="sparqlQuery">SPARQL Query String</param>
/// <returns>RDF Graph</returns>
public Graph QueryWithResultGraph(String sparqlQuery)
{
try
{
//Build the Query URI
StringBuilder queryUri = new StringBuilder();
queryUri.Append(this._endpoint.ToString());
queryUri.Append("?query=");
queryUri.Append(Uri.EscapeDataString(sparqlQuery));
if (!this._defaultGraphUri.Equals(String.Empty))
{
queryUri.Append("&default-graph-uri=");
queryUri.Append(Uri.EscapeUriString(this._defaultGraphUri));
}
//Make the Query via HTTP
HttpWebResponse httpResponse = this.DoQuery(new Uri(queryUri.ToString()),false);
//Set up an Empty Graph ready
Graph g = new Graph();
g.BaseURI = this._endpoint;
//Parse into a Graph based on Content Type
String ctype = httpResponse.ContentType;
IRDFReader parser = MIMETypesHelper.GetParser(ctype);
parser.Load(g, new StreamReader(httpResponse.GetResponseStream()));
return g;
}
catch (UriFormatException uriEx)
{
//URI Format Invalid
throw new Exception("The format of the URI was invalid", uriEx);
}
catch (WebException webEx)
{
//Some sort of HTTP Error occurred
throw new Exception("A HTTP Error occurred", webEx);
}
catch (RDFException)
{
//Some problem with the RDF or Parsing thereof
throw;
}
catch (Exception)
{
//Other Exception
throw;
}
}
/// <summary>
/// Internal Helper Method which executes the HTTP Requests against the SPARQL Endpoint
/// </summary>
/// <param name="target">URI to make Request to</param>
/// <param name="sparqlOnly">Indicates if only SPARQL Result Sets should be accepted</param>
/// <returns>HTTP Response</returns>
private HttpWebResponse DoQuery(Uri target, bool sparqlOnly)
{
//Expect errors in this function to be handled by the calling function
//Set-up the Request
HttpWebRequest httpRequest;
HttpWebResponse httpResponse;
httpRequest = (HttpWebRequest)WebRequest.Create(target);
//Use HTTP GET/POST according to user set preference
if (!sparqlOnly)
{
httpRequest.Accept = MIMETypesHelper.HTTPAcceptHeader();
//For the time being drop the application/json as this doesn't play nice with Virtuoso
httpRequest.Accept = httpRequest.Accept.Replace("," + MIMETypesHelper.JSON[0], String.Empty);
}
else
{
httpRequest.Accept = MIMETypesHelper.HTTPSPARQLAcceptHeader();
}
httpRequest.Method = this._httpMode;
httpRequest.Timeout = this._timeout;
//HTTP Debugging
if (Options.HTTPDebugging)
{
Tools.HTTPDebugRequest(httpRequest);
}
httpResponse = (HttpWebResponse)httpRequest.GetResponse();
//HTTP Debugging
if (Options.HTTPDebugging)
{
Tools.HTTPDebugResponse(httpResponse);
}
return httpResponse;
}
修改
要弄清我已经说过的这不是解析器中的错误，这是StreamReader读取速度比Response Stream提供数据快的问题.我可以通过执行以下操作来解决此问题，但希望提出更好或更优雅的解决方案的建议:
To clarify what I already stated this is not a bug in the Parser, this is an issue of the StreamReader reading faster than the Response Stream provides data. I can get around this by doing the following but would like suggestions of better or more elegant solutions:
//Parse into a Graph based on Content Type
String ctype = httpResponse.ContentType;
IRDFReader parser = MIMETypesHelper.GetParser(ctype);
Stream response = httpResponse.GetResponseStream();
MemoryStream temp = new MemoryStream();
Tools.StreamCopy(response, temp);
response.Close();
temp.Seek(0, SeekOrigin.Begin);
parser.Load(g, new StreamReader(temp));
编辑2
按照Eamon的建议使用BlockingStreamReader类:
BlockingStreamReader class as per Eamon's suggestion:
/// <summary>
/// A wrapper to a Stream which does all its Read() and Peek() calls using ReadBlock() to handle slow underlying streams (eg Network Streams)
/// </summary>
public sealed class BlockingStreamReader : StreamReader
{
private bool _peeked = false;
private int _peekChar = -1;
public BlockingStreamReader(StreamReader reader) : base(reader.BaseStream) { }
public BlockingStreamReader(Stream stream) : base(stream) { }
public override int Read()
{
if (this._peeked)
{
this._peeked = false;
return this._peekChar;
}
else
{
if (this.EndOfStream) return -1;
char[] cs = new char[1];
base.ReadBlock(cs, 0, 1);
return cs[0];
}
}
public override int Peek()
{
if (this._peeked)
{
return this._peekChar;
}
else
{
if (this.EndOfStream) return -1;
this._peeked = true;
char[] cs = new char[1];
base.ReadBlock(cs, 0, 1);
this._peekChar = cs[0];
return this._peekChar;
}
}
public new bool EndOfStream
{
get
{
return (base.EndOfStream && !this._peeked);
}
}
}
编辑3
这是一个经过改进的解决方案，可以包装任何TextReader并提供EndOfStream属性.它使用内部缓冲区，该缓冲区是通过在包装的TextReader上使用ReadBlock()填充的.可以使用此缓冲区定义读取器的所有Read()方法，缓冲区大小是可配置的:
Here is a much improved solution which can wrap any TextReader and provide an EndOfStream property. It uses an internal buffer which is filled by using ReadBlock() on the wrapped TextReader. All the Read() methods of the reader can the be defined using this buffer, buffer size is configurable:
/// <summary>
/// The BlockingTextReader is an implementation of a <see cref="TextReader">TextReader</see> designed to wrap other readers which may or may not have high latency.
/// </summary>
/// <remarks>
/// <para>
/// This is designed to avoid premature detection of end of input when the input has high latency and the consumer tries to read from the input faster than it can return data. All methods are defined by using an internal buffer which is filled using the <see cref="TextReader.ReadBlock">ReadBlock()</see> method of the underlying <see cref="TextReader">TextReader</see>
/// </para>
/// </remarks>
public sealed class BlockingTextReader : TextReader
{
private char[] _buffer;
private int _pos = -1;
private int _bufferAmount = -1;
private bool _finished = false;
private TextReader _reader;
public const int DefaultBufferSize = 1024;
public BlockingTextReader(TextReader reader, int bufferSize)
{
if (reader == null) throw new ArgumentNullException("reader", "Cannot read from a null TextReader");
if (bufferSize < 1) throw new ArgumentException("bufferSize must be >= 1", "bufferSize");
this._reader = reader;
this._buffer = new char[bufferSize];
}
public BlockingTextReader(TextReader reader)
: this(reader, DefaultBufferSize) { }
public BlockingTextReader(Stream input, int bufferSize)
: this(new StreamReader(input), bufferSize) { }
public BlockingTextReader(Stream input)
: this(new StreamReader(input)) { }
private void FillBuffer()
{
this._pos = -1;
if (this._finished)
{
this._bufferAmount = 0;
}
else
{
this._bufferAmount = this._reader.ReadBlock(this._buffer, 0, this._buffer.Length);
if (this._bufferAmount == 0 || this._bufferAmount < this._buffer.Length) this._finished = true;
}
}
public override int ReadBlock(char[] buffer, int index, int count)
{
if (count == 0) return 0;
if (buffer == null) throw new ArgumentNullException("buffer");
if (index < 0) throw new ArgumentException("index", "Index must be >= 0");
if (count < 0) throw new ArgumentException("count", "Count must be >= 0");
if ((buffer.Length - index) < count) throw new ArgumentException("Buffer too small");
if (this._bufferAmount == -1 || this._pos >= this._bufferAmount)
{
if (!this._finished)
{
this.FillBuffer();
if (this.EndOfStream) return 0;
}
else
{
return 0;
}
}
this._pos = Math.Max(0, this._pos);
if (count <= this._bufferAmount - this._pos)
{
//If we have sufficient things buffered to fufill the request just copy the relevant stuff across
Array.Copy(this._buffer, this._pos, buffer, index, count);
this._pos += count;
return count;
}
else
{
int copied = 0;
while (copied < count)
{
int available = this._bufferAmount - this._pos;
if (count < copied + available)
{
//We can finish fufilling this request this round
int toCopy = Math.Min(available, count - copied);
Array.Copy(this._buffer, this._pos, buffer, index + copied, toCopy);
copied += toCopy;
this._pos += toCopy;
return copied;
}
else
{
//Copy everything we currently have available
Array.Copy(this._buffer, this._pos, buffer, index + copied, available);
copied += available;
this._pos = this._bufferAmount;
if (!this._finished)
{
//If we haven't reached the end of the input refill our buffer and continue
this.FillBuffer();
if (this.EndOfStream) return copied;
this._pos = 0;
}
else
{
//Otherwise we have reached the end of the input so just return what we've managed to copy
return copied;
}
}
}
return copied;
}
}
public override int Read(char[] buffer, int index, int count)
{
return this.ReadBlock(buffer, index, count);
}
public override int Read()
{
if (this._bufferAmount == -1 || this._pos >= this._bufferAmount - 1)
{
if (!this._finished)
{
this.FillBuffer();
if (this.EndOfStream) return -1;
}
else
{
return -1;
}
}
this._pos++;
return (int)this._buffer[this._pos];
}
public override int Peek()
{
if (this._bufferAmount == -1 || this._pos >= this._bufferAmount - 1)
{
if (!this._finished)
{
this.FillBuffer();
if (this.EndOfStream) return -1;
}
else
{
return -1;
}
}
return (int)this._buffer[this._pos + 1];
}
public bool EndOfStream
{
get
{
return this._finished && (this._pos >= this._bufferAmount - 1);
}
}
public override void Close()
{
this._reader.Close();
}
protected override void Dispose(bool disposing)
{
this.Close();
this._reader.Dispose();
base.Dispose(disposing);
}
}
推荐答案
在不知道您使用的解析器的具体细节的情况下，我只能猜测该错误，但是将.NET框架I/作为该错误相当容易O libs几乎鼓励您做...
Without knowing the specifics the parser you're using, I can only guess at the bug, but there's a fairly easy to make bug the .NET framework I/O libs almost encourage you to make...
您是否知道Streams和TextReader读取的字节/字符数可能少于请求的事实?
Are you aware of the fact that Streams and TextReaders may read fewer bytes/characters than requested?
尤其是TextReader.Read(char []缓冲区，int索引，int计数)的文档说:
In particular, TextReader.Read(char[] buffer, int index, int count)'s docs say:
返回值
类型:System .. ::.Int32
Type: System..::.Int32
已读取的字符数.该数字将 小于或等于计数，具体取决于数据流中是否可用.如果没有更多的字符可读取，则调用此方法将返回零.
The number of characters that have been read. The number will be less than or equal to count, depending on whether the data is available within the stream. This method returns zero if called when no more characters are left to read.
强调我的.
例如，如果您调用reader.Read(buffer，0，100)，则不能假定已读取100个字符.
For example, if you call reader.Read(buffer, 0, 100) you cannot assume that 100 characters have been read.
解析器很可能确实假定了这一点.并且这解释了您观察到的行为:如果您将流完全缓存在MemoryStream中，则始终会有足够的字符来满足请求-但如果不这样做，解析器将收到的字符少于请求的字符当基础流变慢"时，在不可预知的时间.
It's very likely that the parser does assume this; and this explains your observed behavior: if you fully cache the stream in a MemoryStream, there will always be enough characters to fullfill the request - but if you don't, the parser will receive fewer characters than requested at unpredictable times whenever the underlying stream is "slow".
Edit2:您可以通过将解析器中的所有TextReader.Read()实例替换为TextReader.ReadBlock()来修复错误.
You can fix your bug by replacing all instances of TextReader.Read() in the parser with TextReader.ReadBlock().
这篇关于从HttpResponseStream读取失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
