Java中的字符串标记化(大文本) [英] String tokenization in java (LARGE text)

问题描述

我有这么大的文字(读大).我需要标记每个单词，在每个非字母上定界.我使用StringTokenizer一次读取一个单词.但是，正如我在研究如何编写定界符字符串(每个非字母")而不是执行以下操作一样:

I have this large text (read LARGE). I need to tokenize every word, delimit on every non-letter. I used StringTokenizer to read one word at a time. However, as I was researching how to write the delimiter string ("every non-letter") instead of doing something like:

new StringTokenizer(text, "\" ();,.'[]{}!?:""…\n\r0123456789 [etc etc]");

我发现每个人基本上都讨厌StringTokenizer(为什么?).

I found that everyone basically hates StringTokenizer (why?).

那么，我该怎么用呢?不要建议使用String.split，因为它会复制我的大文本.我需要逐字检查文本，并在每个非字母上定界.自己建造东西会更容易，还是有一些最佳实践来解决这个问题?

So, what can I use instead? Dont suggest String.split as it will duplicate my large text. I need to go through the text word by word and delimit on every non-letter. Is it easier to build something on my own or is there some best practice way to confront this problem?

提前谢谢！

推荐答案

您可以使用灵活的字符串 Splitter 类"class ="post-tag" title =显示标记为'guava'的问题"rel =" tag>番石榴库.

You can use the flexible string Splitter class from Google's guava library.

如果您需要更强大的功能，请查看

If you need something more powerful, have a look at StandardTokenizer from Apache Lucene. From the docs:

对于大多数欧洲语言文档来说，这应该是一个很好的标记器:

This should be a good tokenizer for most European-language documents:

• 在标点符号处分割单词，删除标点符号.但是，不带空格的点被认为是 令牌.
• 除非连字符中有数字，否则用连字符将单词分开，在这种情况下，整个令牌被解释为产品编号，并且 不分裂.
• 将电子邮件地址和互联网主机名识别为一个令牌.

• Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
• Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
• Recognizes email addresses and internet hostnames as one token.

这篇关于Java中的字符串标记化(大文本)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！