4GB的stringstream溢出 [英] stringstream overflow at 4GB
问题描述
即使stringstream运行在具有足够内存的64位linux机器上，我也无法超出stringstream的4GB限制.
I'm having trouble getting beyond 4GB limitation for stringstream, even though it is running on a 64bit linux box with enough memory.
下面的测试代码(阅读您的评论后进行了修订)在4GB之后发生了核心转储.
The test code below (revised after reading your comments) core dump after 4GB.
在gdb跟踪中，stringstream使用默认的std :: char_traits，其中int_type设置为32bit int，而不是64bit size_t.有什么建议可以解决吗?
From the gdb trace, stringstream uses the default std::char_traits where int_type is set to 32bit int, rather than 64bit size_t. Any suggestion to work around?
#include <stdint.h>
#include <iostream>
#include <sstream>
#include <math.h>
using namespace std;
int main(int narg, char** argv)
{
string str;
stringstream ss;
const size_t GB = (size_t)pow(2, 30);
str.resize(GB, '1');
cerr << "1GB=" << str.size()
<< ", string::max_size=" << str.max_size()/GB << "GB"
<< ", sizeof(int)=" << sizeof(int)
<< ", sizeof(int64_t)=" << sizeof(int64_t)
<< ", sizeof(size_t)=" << sizeof(size_t)
<< endl;
string().swap(str);
str.resize(6*GB, '6');
cerr << "str.size()=" << (str.size() / GB) << "GB allocated successfully"
<< ", ended with " << str.substr(str.size()-5, 5) << endl;
string().swap(str);
str.resize(GB/4, 'Q');
cerr << "writing to stringstream..." << std::flush;
for (int i = 0; i < 30; ++i) {
ss << str << endl;
cerr << double(ss.str().size())/GB << "GB " << std::flush;
}
cerr << endl;
exit(0);
}
输出为:
1GB=1073741824, string::max_size=4294967295GB, sizeof(int)=4, sizeof(int64_t)=8, sizeof(size_t)=8
str.size()=6GB allocated successfully, ended with 66666
writing to stringstream...0.25GB 0.5GB 0.75GB 1GB 1.25GB 1.5GB 1.75GB 2GB 2.25GB 2.5GB 2.75GB 3GB 3.25GB 3.5GB 3.75GB Segmentation fault (core dumped)
gdb堆栈跟踪为:
(gdb) where
#0 0x00002aaaaad5e0c1 in std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) () from /usr/lib64/libstdc++.so.6
#1 0x00002aaaaad62cbd in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) () from /usr/lib64/libstdc++.so.6
#2 0x00002aaaaad5657d in std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib64/libstdc++.so.6
#3 0x000000000040112b in main ()
二进制文件似乎是64位的.
The binary seems to be 64bits.
$ file a.out
a.out: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
尼克，检查预处理器输出的好点. std :: char_straits会以某种方式覆盖__gnu_cxx :: char_straits，并将int_type重新定义为int，而不是像__gnu_cxx :: char_straits一样是无符号的.这真是令人惊讶！
Nick, good point on checking the preprocessor output. Somehow std::char_straits overwrites __gnu_cxx::char_straits, and redefines int_type to be int, rather than unsigned long as in __gnu_cxx::char_straits. This is very surprising!
namespace __gnu_cxx
{
# 61 "/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/char_traits.h" 3
template <class _CharT>
struct _Char_types
{
typedef unsigned long int_type;
typedef std::streampos pos_type;
typedef std::streamoff off_type;
typedef std::mbstate_t state_type;
};
# 86 "/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/char_traits.h" 3
template<typename _CharT>
struct char_traits
{
typedef _CharT char_type;
typedef typename _Char_types<_CharT>::int_type int_type;
typedef typename _Char_types<_CharT>::pos_type pos_type;
typedef typename _Char_types<_CharT>::off_type off_type;
typedef typename _Char_types<_CharT>::state_type state_type;
....
};
namespace std
{
# 224 "/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/char_traits.h" 3
template<class _CharT>
struct char_traits : public __gnu_cxx::char_traits<_CharT>
{ };
template<>
struct char_traits<char>
{
typedef char char_type;
typedef int int_type;
typedef streampos pos_type;
typedef streamoff off_type;
typedef mbstate_t state_type;
...
};
下面来自STL网站 http://www.cplusplus.com/reference/string/char_traits/其中char_traits专业化使用int.
below from the STL website http://www.cplusplus.com/reference/string/char_traits/ where char_traits specialization uses int.
typedef INT_T int_type;
Where INT_T is a type that can represent all the valid characters representable by a char_type plus an end-of-file value (eof) which is compatible with iostream class member functions.
For char_traits<char> this is int, and for char_traits<wchar_t> this is wint_t
推荐答案
可以发布经过预处理的源文件.
Can you post a preprocessed source file.
已删除对_Char_types的引用，因为它具有误导性.
Reference to _Char_types removed because it was misleading.
我不确定int_type是否相关，char的专业化确实确实将int用于int_type.
I'm not sure the int_type is related, the specialisation for char does indeed use int for int_type.
int_type是用于保存单个字符的typedef(即，对于ascii至少一个字节，对于wchar_t更高).它不用于存储一定范围的字符(请参见下面的std :: streamoff).
int_type is a typedef to hold an individual character (i.e. at least one byte for ascii, higher for wchar_t). It is not used to store a range of characters (see std::streamoff below).
从您的预处理输出中，我怀疑std :: streamoff可能是罪魁祸首.从我系统上的标题中:
From your preprocessed output I suspect that std::streamoff could be the culprit. From the headers on my system:
# 90 "/usr/include/c++/4.6/bits/postypes.h" 3
typedef long streamoff;
std :: streamoff用于std :: streampos，如果定义不正确，我认为这会导致字符串流调用溢出函数.
std::streamoff is used in std::streampos which if defined incorrectly I think would lead to the stringstream calling the overflow function.
您检查是否包含正确的标题吗?
Have you checked that the correct headers are included?
# 61 "/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/char_traits.h" 3
是以下路径:
/usr/include/c++/4.1.2/bits/char_traits.h
-尼克
这篇关于4GB的stringstream溢出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
