sizeof(string) 不等于字符串长度 [英] sizeof(string) not equal to string length

问题描述

我曾经认为每个字符都是一个字节(至少在 c/c++ 中是这样)所以字符串的大小应该等于 len(string) 字节.然而，一个简单的实验告诉我，在 python 中并非如此:

导入字符串，系统，随机# 抽象代码，去除不必要的部分def loadKeyLength(db,key,N):val = key[:5] + ''.join(random.choice(string.ascii_letters + string.digits) for _ in xrange(N-5))打印 sys.getsizeof(val), len(val),valdef loadKeysSize(s):r=0键=str(s).zfill(5)+str(r).zfill(5)loadKeyLength(None,key,s)对于范围内的 i (80,120,3):loadKeysSize(i)

Ubuntu 14.04 上的输出:

117 80 000802qdxV2TY3qjGpe6F35hLczQNE2h7bWRWpHEMxqcyrb01sI2A6gcTLKLxdQSFjGMFtWPZJDOtKe4120 83 00083oX6FouwlAyUkmZZCLWuIWnDMAKZDlNvO4ElHTK4x6vjka42APnwOcEMFHDLXbTZg9Cupd5ALqveowX123 86 000860z1yhl3i1mKYFMhY4D2kWKA6Bvpfw91VeI7gXyP52PrVbLoP95ykgkz47k3KhCgmgrHq3CBCEdV14aiOa126 89 00089xfcmZyf8RrftFbxvx9qvJUd8bvG5FKH2Ydz7aN5EsnaBpQkvrTLIsAKNRADeF1M74Ghvk1opzRs28IokPVhS129 92 00092COlhIGMXrQ4Zl7e6GPlVz43BVWLbnvC3ymtoZ6Itus8KWuM1I31xGPU5Y4vggpcq2g4c6uSvnmUjsAgpYkNoX1u132 95 00095IrjvnSVC8ECKf2cNUsBkzrSfuTNobIUmAD9BktiMfQSoCBLkwPOa2QmovhnUEpYyAsCKdM2haVqb53PggDviQHseex135 98 00098DsLvbvWmqgyuWsnQd0DillNmd3LyTSJ98XjKUDhbqBSxhVRoXyv0IkOjWAbZIEb5lmrnISWS28WS4OpisoJYPCIfnB4bw138 101 00101JNfUNutpjBFhFlhyNhFae2gulYTIfBpfoBbnLl881LPeZNGQkwYF49pbDEvnqYkPSleFUrZ1tEfO4AokI7ka3Gcn8KkTmWWg141 104 00104lbmN2zeZeUMS6xGQfjtImCkwQwmewbXxsxj0NGETdBGwAfnhBmXOSew8LMdULQYCEA3Nz8ny6OlGfOUP3zjf5lZXlNC8Cn89Il4144 107 00107UwQCaa4szAYj9if1oIPleauAvyWVkyDzbtZSt0SiKfJgNG7avZLe4TSTWXuEZBOUICfTAjIzVlShwXJ54Oz14rZlBrQL0w05FJsckY147 110 00110lNZA1HsGmdFZke0Up1PwxPtpt2RFDM9EdOljQ6K3oao44Q6CNsBZHxo56n63Lny5l6k5ny7rhgWtkEGoJS7JbeNBg9ACXApfX4seWi150 113 00113KJ1sSGNZfZx2xl0MBXY0yf6ybPNjpmYBYiHi0ZsBb9GFuE9hIQgR0TssgbdE9sqq1m90YlS1ZWHSwwElaCkNOT05GbbIt3AflJAfjA153 116 001167jYYE6oyKM7qKQdzjpV1xUVUb85hpHpliNZRyiX7r6vJJ4n2FSe9tLUJ4W0ecUEALEemAZ0mUSkSROPl3AdQJ9AFdUAWvT5v4WTbNUZlxlX2156 119 00119ehpukL2CAOfCDbdtvuEvROVZJUvg044u8YS3d81SQ1FQqZDoVe55F8zCi7ikH1rEk2MWGUQLrmdJkMDKCXrtoeuZBDpo7pJOcmLRYZiWLiCx

在 Windows10 上:

101 80 00080Ra86ljAznn9AM17OtGUuFmxdYd7lU1hkInjZoPQJ4C2g3itkqn7wV0thhxPgpxrDimJwUElXzL2104 83 00083nxTNohavc5sQfvlmnPnGOQNpzn2PKQJTYeDq4I9lMkuMKAxOhdOm1l3KAxmyCNOhlCKla9KMp8XYit107 86 00086NBNeIqBFwWPbGzvtEihHvFnO3XyfPUEL0izlzF45P0NdfNTCyCDHvO6xa6BX4TyybChCEllhPOFXWpWd4110 89 00089PmSiJYRGmI2AlXkbFUwcO1Ipr0bFvCmA2Se1A9JGMRTcg7617mXmG7fNCmDZWWFwI5DgTNHtqRDvTzrrmf08113 92 00092DbsiuxTEZJT8DznuF3mtpdRP4LP4Nboj8tpCbgZkfeeP925U8N7v34qQpT69bw26Lfwp1jJXhkcb1o0wGUsgSIt116 95 000958GexIZILU3le53WUGzTC6sRLK3vQVCsNI1yOuFt1HdW6QHZm05n5XGGMsluSamrKINAoBPxuQ5yrYSQHE7BlrWI6Jl119 98 00098suYJVfpHKDkjHmnXwevRUOskhnCfF5Zp2jcN4avlg7ZN9g98G3vFeMpoXrulM7g5VfOQKI7UudzNfqkGDSaNfSuDvfyEE122 101 00101doDJHZQEJre8aDWDGIPeKzN2aFXKZxYH9w6o9ZxgAXXozc75KMMwQ23YN0N27IMdAY5Oe9WLQSUgIf6AkfSNjWTFOODBvXeg125 104 00104lhRSLYEXD1waMMkSVKct8jnb8M97rRQl582dlnzRr8hGM00jJLxrhHVvq1Kbu59dtcCSb4vm0KzbXKGiIDarDakNVLuCWYA9Mrh128 107 00107y11AyDhwM8BZzT73VhkYu9U4ogvbw10ZPmnA824MzAznGbhLbHPDJjnl2NfquwH9XEOTx4vJjz74HC7I8GceZsQtWEigt131 110 00110GD8f97kpurShkyrXYI40UvWlWOvskqRrDbUNjR2x6cZcg4NywVe1UAecrehoTJU5WUqZvvxseD16fYFvzaTKv7Jdwn1yQOZXSKheH134 113 001130coEBOEdkY1raJ65VK77UPU7eRraN2dz9mibcbu4khQwFQWf9WVBPUwTjlddveJKGLKS4gtNLWNeN4U720DW8XmHdpqhkXxqGBouZ2ARYfU137 116 001164x8ALzFvQKijeOIcDz6DCBnqzcMPQiR7rLaMBNuNFBYULSJ2xWIcGdyHHZw2lqW817fYo56Yg5hibAO7NzOyehOyxxUA865lQdOffcff140 119 00119Sak3ByDRCFDMYpzpNEIKU5yNEWbWdL0popfhspb8cjE9sEBpMNxyGj5wjofhdois8DYQUTumJ3Xy7nzR04xGCG3mNQkVzKw1d97XP5RwN9FY

这是什么意思?如果我想生成一个以字节为单位的特定大小的随机字符串，我该怎么办?为什么在两个不同的平台上大小会有所不同(我确信某些 Python 内部结构)?

注意:我的主要目的是生成随机字符串，其大小遵循内存研究问题的特定分布，但这在这里无关紧要.

解决方案

Python 字符串对象包含更多信息，而不仅仅是字符.它们包含引用计数、对类型定义的引用、字符串的长度、缓存的散列和驻留状态.请参阅PyStringObject 结构，以及 PyObject_VAR_HEAD 结构 引用.

因此，空字符串也有内存大小:

<预><代码>>>>导入系统>>>sys.getsizeof('')37

这个大小是平台相关的，因为指针和 C 整数在不同的平台上有不同的大小.37 是 Mac OS X 上 Python 2 str 对象的大小.

对于unicode 对象，图片更加失真；Python 2 可以使用 2 或 4 个字节每个代码点，具体取决于编译时的选择.最新的 Python 3 版本对 Unicode 文本使用可变字节数，每个代码点在 1 到 4 个字节之间，具体取决于文本中的最高代码点要求.

因此，sys.getsizeof() 返回一个不同的、更高的值是正常.sys.getsizeof() 是不是获取字符串长度的函数.使用 len() .

如果你想知道其他软件为一个字符串使用了多少内存，你绝对不能使用sys.sizeof()代码>值；其他软件会对如何存储文本做出不同的选择，并且会有不同的开销.encoded 文本的 len() 值可能是一个起点，但您必须与该其他软件的文档或开发人员核对以查看他们可以告诉你一段给定的文本需要多少内存.

I used to think that each character is one byte (at least that is the case in c/c++) so the size of string should be equal to len(string) bytes. However, a simple experiment tells me that it is not the case in python:

import string, sys, random
# abstracted code, removed unnecessary parts
def loadKeyLength(db,key,N):
    val = key[:5] + ''.join(random.choice(string.ascii_letters + string.digits) for _ in xrange(N-5))
    print sys.getsizeof(val), len(val),val

def loadKeysSize(s): 
    r=0
    key=str(s).zfill(5)+str(r).zfill(5)
    loadKeyLength(None,key,s)

for i in range(80,120,3):
    loadKeysSize(i)

output on Ubuntu 14.04:

117 80 000802qdxV2TY3qjGpe6F35hLczQNE2h7bWRWpHEMxqcyrb01sI2A6gcTLKLxdQSFjGMFtWPZJDOtKe4
120 83 00083oX6FouwlAyUkmZZCLWuIWnDMAKZDlNvO4ElHTK4x6vjka42APnwOcEMFHDLXbTZg9CUpd5ALqveowX
123 86 000860z1yhl3i1mKYFMhY4D2kWKA6Bvpfw91VeI7gXyP52PrVbLoP95ykgkz47k3KhCgmgrHq3CBCEdV14aiOa
126 89 00089xfcmZyf8RrftFbxvx9qvJUd8bvG5FKH2Ydz7aN5EsnaBpQkvrTLIsAKNRADeF1M74Ghvk1opzRs28IokPVhS
129 92 00092COlhIGMXrQ4Zl7e6GPlVz43BVWLbnvC3ymtoZ6Itus8KWuM1I31xGPU5Y4vggpcq2g4c6uSvnmUjsAgpYkNoX1u
132 95 00095IrjvnSVC8ECKf2cNUsBkzrSfuTNobIUmAD9BktiMfQSoCBLkwPOa2QmovhnUEpYyAsCKdM2haVqb53PggDviQHseex
135 98 00098DsLvbvWmqgyuWsnQd0DillNmd3LyTSJ98XjKUDhbqBSxhVRoXyv0IkOjWAbZIEb5lmrnISWS28WS4OpisoJYPCIfnB4bw
138 101 00101JNfUNutpjBFhFlhyNhFae2gulYTIfBpfoBbnLl881LPeZNGQkwYF49pbDEvnqYkPSleFUrZ1tEfO4AokI7ka3Gcn8KkTmWWg
141 104 00104lbmN2zeZeUMS6xGQfjtImCkwQwmewbXxsxj0NGETdBGwAfnhBmXOSew8LMdULQYCEA3Nz8ny6OlGfOUP3zjf5lZXlNC8Cn89Il4
144 107 00107UwQCaa4szAYj9if1oIPleauAvyWVkyDzbtZSt0SiKfJgNG7avZLe4TSTWXuEZBOUICfTAjIzVlShwXJ54Oz14rZlBrQL0w05FJsckY
147 110 00110lNZA1HsGmdFZke0Up1PwxPtpt2RFDM9EdOljQ6K3oao44Q6CNsBZHxo56n63Lny5l6k5ny7rhgWtkEGoJS7JbeNBg9ACXApfz4seWiZrX
150 113 00113KJ1sSGNZfZx2xl0MBXY0yf6ybPNjpmYBYiHi0ZsBb9GFuE9hIQgR0TssgbdE9sqq1m90YlS1ZWHSwwElaCkNOT05GbbIt3AfZAzzlpul5jEJ
153 116 001167jYYE6oyKM7qKQdzjpV1xUVUb85hpHpliNZRyiX7r6vJJ4n2FSe9tLUJ4W0ecUEALEemAZ0mUSkSROPl3AdQJ9AFdUAWvT5v4WTbNUZlFk2x0JX
156 119 00119ehpukL2CAOfCDbdtvuEvROVZJUvg044u8YS3d81SQ1FQqZDoVe55F8zCi7ikH1rEk2MWGUQLrmdJkMDKCXrtoeuZBDpo7pJOcmLRYZMLcLiC37iWXx

On Windows10:

101 80 00080Ra86ljAznn9AM17OtGUuFmxdYd7lU1hkInjZoPQJ4C2g3itkqn7wV0thhxPgpxrDimJwUElXzL2
104 83 00083nxTNohavc5sQfvlmnPnGOQNpzn2PKQJTYeDq4I9lMkuMKAxOhdOm1l3KAxmyCNOhlCKla9KMp8XYit
107 86 00086NBNeIqBFwWPbGzvtEihHvFnO3XyfPUEL0izlzF45P0NdfNTCyCDHvO6xa6BX4TyybChCEllhPOFXWpWd4
110 89 00089PmSiJYRGmI2AlXkbFUwcO1Ipr0bFvCmA2Se1A9JGMRTcg7617mXmG7fNCmDZWWFwI5DgTNHtqRDvTzrrmf08
113 92 00092DbsiuxTEZJT8DznuF3mtpdRP4LP4Nboj8tpCbgZkfeeP925U8N7v34qQpT69bw26Lfwp1jJXhkcb1o0wGUsgSIt
116 95 000958GexIZILU3le53WUGzTC6sRLK3vQVCsNI1yOuFt1HdW6QHZm05n5XGGMsluSamrKINAoBPxuQ5yrYSQHE7BlrWI6Jl
119 98 00098suYJVfpHKDkjHmnXwevRUOskhnCfF5Zp2jcN4avlg7ZN9g98G3vFeMpoXrulM7g5VfOQKI7UudzNfqkGDSaNfSuDvfyEE
122 101 00101doDJHZQEJre8aDWDGIPeKzN2aFXKZxYH9w6o9ZxgAXXozc75KMMwQ23YN0N27IMdAY5Oe9WLQSUgIf6AkfSNjWTFOODBvXeg
125 104 00104lhRSLYEXD1waMMkSVKct8jnb8M97rRQl582dlnzRr8hGM00jJLxrhHVvq1Kbu59dtcCSb4vm0KzbXKGiIdarDakNVLuCWYA9Mrh
128 107 00107y11AyDhwM8BZzT73VhkYu9U4ogvbw10ZPmnA824MzAznGbhLbHPDJjnl2NfquwH9XEOTx4vJjz74HC7I8GceZsCTlIQE4tQtWEtmig
131 110 00110GD8f97kpuRShkyrXYI40UvWlWOvskqRrDbUNjR2x6cZcg4NywVe1UAecrehoTJU5WUqZvvxseD16fYFvzaTKv7Jdwn1yQOazXSKheHORZ
134 113 001130coEBOEdkY1raJ65VK77UPU7eRraN2dz9mibcbu4khQwFQWf9WVBPUwTjlddveJKGLKS4gtNLWNeN4U720DW8XmHdpqhkXxqGBouZ2ARYfU6
137 116 001164x8ALzFvQKijeOIcDz6DCBnqzcMPQiR7rLaMBNuNFBYULSJ2xWIcGdyHHZw2lqW817fYo56Yg5hibAO7NzOyehOyxxUA865lQUjiP8LwmffCdnO
140 119 00119Sak3ByDRCFDMYpzpNEIKU5yNEWbWdL0popfhspb8cjE9sEBpMNxyGj5wjofhdois8DYQUTumJ3Xy7nzR04xGCG3mNQkVzKw1d97XP5RwN99Yac6I5F

So what does this mean? What can I do if I want to generate a random string of specific size in bytes? Why is there a difference in size on two different platforms (some python internals I am sure) ?

Note: My main purpose is to generate random strings whose size follows a specific distribution for a memory research problem but that is irrelevant here.

解决方案

Python string objects contain more information than just the characters. They contain a reference count, a reference to the type definition, the length of the string, the cached hash and the interning state. See the PyStringObject struct, as well as the PyObject_VAR_HEAD struct referenced.

As a result, an empty string has a memory size too:

>>> import sys
>>> sys.getsizeof('')
37

This size is platform dependent, because pointers and C integers have different sizes on different platforms. 37 is the size of a Python 2 str object on Mac OS X.

For unicode objects the picture is even more distorted; Python 2 can use either 2 or 4 bytes per codepoint, depending on a compilation-time choice. The most recent Python 3 versions use a variable number of bytes for Unicode text, between 1 and 4 bytes per codepoint depending on the highest codepoint requirements in the text.

As such, it is normal for sys.getsizeof() to return a different, higher value. sys.getsizeof() is not a function to get a string length. Use len() for that.

If you are want to know how much memory other software uses for a string, you definitely can't use the sys.sizeof() value; other software will make different choices about how to store text, and will have different overheads. The len() value of the encoded text may be a starting point, but you'll have to check with the documentation or developers for that other piece of software to see what they can tell you how much memory is required for a given piece of text.

这篇关于sizeof(string) 不等于字符串长度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！