Ruby 1.9:带有未知输入编码的正则表达式 [英] Ruby 1.9: Regular Expressions with unknown input encoding
问题描述
有没有一种被接受的方式来处理Ruby 1.9中的正则表达式,其输入的编码是未知的?假设我的输入是UTF-16编码的:
Is there an accepted way to deal with regular expressions in Ruby 1.9 for which the encoding of the input is unknown? Let's say my input happens to be UTF-16 encoded:
x = "foo<p>bar</p>baz"
y = x.encode('UTF-16LE')
re = /<p>(.*)<\/p>/
x.match(re)
=> #<MatchData "<p>bar</p>" 1:"bar">
y.match(re)
Encoding::CompatibilityError: incompatible encoding regexp match (US-ASCII regexp with UTF-16LE string)
我目前的方法是在内部使用UTF-8并在必要时对输入进行重新编码(复制):
My current approach is to use UTF-8 internally and re-encode (a copy of) the input if necessary:
if y.methods.include?(:encode) # Ruby 1.8 compatibility
if y.encoding.name != 'UTF-8'
y = y.encode('UTF-8')
end
end
y.match(/<p>(.*)<\/p>/u)
=> #<MatchData "<p>bar</p>" 1:"bar">
但是,这对我来说有点尴尬,我想问一个更好的方法做它。
However, this feels a little awkward to me, and I wanted to ask if there's a better way to do it.
推荐答案
据我所知,没有更好的方法使用。但是,我可以建议稍微改动吗?
As far as I am aware, there is no better method to use. However, might I suggest a slight alteration?
为了不改变正则表达式的编码,而不是改变输入的编码?
Rather than changing the encoding of the input, why not change the encoding of the regex? Translating one regex string every time you meet a new encoding is a lot less work than translating hundreds or thousands of lines of input to match the encoding of your regex.
# Utility function to make transcoding the regex simpler.
def get_regex(pattern, encoding='ASCII', options=0)
Regexp.new(pattern.encode(encoding),options)
end
# Inside code looping through lines of input.
# The variables 'regex' and 'line_encoding' should be initialized previously, to
# persist across loops.
if line.methods.include?(:encoding) # Ruby 1.8 compatibility
if line.encoding != last_encoding
regex = get_regex('<p>(.*)<\/p>',line.encoding,16) # //u = 00010000 option bit set = 16
last_encoding = line.encoding
end
end
line.match(regex)
在病理情况下(输入编码每行改变) ,因为你通过循环每次重新编码正则表达式。但是在99.9%的情况下,编码对于数百或数千行的整个文件是不变的,这将导致重新编码的大幅减少。
In the pathological case (where the input encoding changes every line) this would be just as slow, since you're re-encoding the regex every single time through the loop. But in 99.9% of situations where the encoding is constant for an entire file of hundreds or thousands of lines, this will result in a vast reduction in re-encoding.
这篇关于Ruby 1.9:带有未知输入编码的正则表达式的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!