Python:在字边界上分割unicode字符串 [英] Python: Split unicode string on word boundaries

问题描述

我需要取一个字符串，并将其缩短为140个字符.

I need to take a string, and shorten it to 140 characters.

目前我正在做

if len(tweet) > 140:
    tweet = re.sub(r"\s+", " ", tweet) #normalize space
    footer = "… " + utils.shorten_urls(post['url'])
    avail = 140 - len(footer)
    words = tweet.split()
    result = ""
    for word in words:
        word += " "
        if len(word) > avail:
            break
        result += word
        avail -= len(word)
    tweet = (result + footer).strip()
    assert len(tweet) <= 140

因此，这对于英语以及像字符串一样的英语非常有用，但对于中文字符串则无效，因为tweet.split()仅返回一个数组:

So this works great for English, and English like strings, but fails for a Chinese string because tweet.split() just returns one array:

>>> s = u"简讯：新華社報道，美國總統奧巴馬乘坐的「空軍一號」專機晚上10時42分進入上海空域，預計約30分鐘後抵達浦東國際機場，開展他上任後首次訪華之旅。"
>>> s
u'\u7b80\u8baf\uff1a\u65b0\u83ef\u793e\u5831\u9053\uff0c\u7f8e\u570b\u7e3d\u7d71\u5967\u5df4\u99ac\u4e58\u5750\u7684\u300c\u7a7a\u8ecd\u4e00\u865f\u300d\u5c08\u6a5f\u665a\u4e0a10\u664242\u5206\u9032\u5165\u4e0a\u6d77\u7a7a\u57df\uff0c\u9810\u8a08\u7d0430\u5206\u9418\u5f8c\u62b5\u9054\u6d66\u6771\u570b\u969b\u6a5f\u5834\uff0c\u958b\u5c55\u4ed6\u4e0a\u4efb\u5f8c\u9996\u6b21\u8a2a\u83ef\u4e4b\u65c5\u3002'
>>> s.split()
[u'\u7b80\u8baf\uff1a\u65b0\u83ef\u793e\u5831\u9053\uff0c\u7f8e\u570b\u7e3d\u7d71\u5967\u5df4\u99ac\u4e58\u5750\u7684\u300c\u7a7a\u8ecd\u4e00\u865f\u300d\u5c08\u6a5f\u665a\u4e0a10\u664242\u5206\u9032\u5165\u4e0a\u6d77\u7a7a\u57df\uff0c\u9810\u8a08\u7d0430\u5206\u9418\u5f8c\u62b5\u9054\u6d66\u6771\u570b\u969b\u6a5f\u5834\uff0c\u958b\u5c55\u4ed6\u4e0a\u4efb\u5f8c\u9996\u6b21\u8a2a\u83ef\u4e4b\u65c5\u3002']

该如何处理I18N?这在所有语言中都有意义吗?

How should I do this so it handles I18N? Does this make sense in all languages?

如果重要的话，我使用的是python 2.5.4.

I'm on python 2.5.4 if that matters.

推荐答案

与一些以粤语，普通话和日语为母语的人交谈后，看来正确的做法很困难，但我目前的算法仍然对他们有意义互联网帖子的背景.

After speaking with some native Cantonese, Mandarin, and Japanese speakers it seems that the correct thing to do is hard, but my current algorithm still makes sense to them in the context of internet posts.

意思是，它们习惯于在空间上分割并在末尾添加…"处理.

Meaning, they are used to the "split on space and add … at the end" treatment.

所以我会懒惰并坚持下去，直到收到不了解它的人的抱怨为止.

So I'm going to be lazy and stick with it, until I get complaints from people that don't understand it.

对我的原始实现的唯一更改是不要在最后一个单词上强制使用空格，因为在任何语言中都不需要空格(并使用unicode字符…&#x2026而不是... three dots来保存2个字符)

The only change to my original implementation would be to not force a space on the last word since it is unneeded in any language (and use the unicode character … &#x2026 instead of ... three dots to save 2 characters)

这篇关于Python:在字边界上分割unicode字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！