将文本拆分为单个单词 [英] Split a text into single words

问题描述

我想使用 PHP 将文本拆分为单个单词.您知道如何实现这一目标吗?

I would like to split a text into single words using PHP. Do you have any idea how to achieve this?

我的方法:

function tokenizer($text) {
    $text = trim(strtolower($text));
    $punctuation = '/[^a-z0-9äöüß-]/';
    $result = preg_split($punctuation, $text, -1, PREG_SPLIT_NO_EMPTY);
    for ($i = 0; $i < count($result); $i++) {
        $result[$i] = trim($result[$i]);
    }
    return $result; // contains the single words
}
$text = 'This is an example text, it contains commas and full-stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
print_r(tokenizer($text));

这是一个好方法吗?你有什么改进的想法吗?

Is this a good approach? Do you have any idea for improvement?

提前致谢！

推荐答案

使用匹配任何 unicode 标点符号的类 p{P}，结合 s 空白类.

Use the class p{P} which matches any unicode punctuation character, combined with the s whitespace class.

$result = preg_split('/((^p{P}+)|(p{P}*s+p{P}*)|(p{P}+$))/', $text, -1, PREG_SPLIT_NO_EMPTY);

这将拆分为一组一个或多个空白字符，但也会吸收任何周围的标点符号.它还匹配字符串开头或结尾的标点字符.这会区分诸如不要"和他说‘哎哟！’"之类的情况

This will split on a group of one or more whitespace characters, but also suck in any surrounding punctuation characters. It also matches punctuation characters at the beginning or end of the string. This discriminates cases such as "don't" and "he said 'ouch!'"

这篇关于将文本拆分为单个单词的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！