NLP中的英语分词? [英] English word segmentation in NLP?
问题描述
我是NLP领域的新手，但我目前的研究需要从URL地址进行一些文本解析(或称为关键字提取)，例如伪造的网址，
I am new in the NLP domain, but my current research needs some text parsing (or called keyword extraction) from URL addresses, e.g. a fake URL,
http://ads.goole.com/appid/heads
我的解析有两个约束，
-
第一个"ads"和最后一个"heads"应该是不同的，因为"heads"中的"ads"意味着更多的后缀而不是广告.
The first "ads" and last "heads" should be distinct because "ads" in the "heads" means more suffix rather than an advertisement.
"appid"可以分为两部分；分别是"app"和"id"，它们在互联网上都具有语义含义.
The "appid" can be parsed into two parts; that is 'app' and 'id', both taking semantic meanings on the Internet.
我尝试了 Stanford NLP 工具包和Google搜索引擎.前者试图将每个单词的语法含义分类，这是我的期望. Google引擎显示出对"appid"的更多了解，从而为我提供了有关"app id"的建议.
I have tried the Stanford NLP toolkit and Google search engine. The former tries to classify each word in a grammar meaning which is under my expectation. The Google engine shows more smartness about "appid" which gives me suggestions about "app id".
我无法查看Google搜索中的搜索历史记录，因此它给了我应用程序ID"，因为有很多人搜索了这些单词.我可以使用一些离线方法来执行类似的解析吗?
I can not look over the reference of search history in Google search so that it gives me "app id" because there are many people have searched these words. Can I get some offline line methods to perform similar parsing??
更新:
请跳过正则表达式建议，因为即使在简单的URL中，诸如"appid"之类的单词组成的数量也可能未知.
Please skip the regex suggestions because there is a potentially unknown number of compositions of words like "appid" in even simple URLs.
谢谢
Jamin
推荐答案
听起来不是真正的分词，而是分词.例如，这是一种理解没有空间的方式的方法.
Rather than tokenization, what it sounds like you really want to do is called word segmentation. This is for example a way to make sense of asentencethathasnospaces.
我还没有读完整个教程，但这应该可以帮助您入门.他们甚至提供url作为潜在的用例.
I haven't gone through this entire tutorial, but this should get you started. They even give urls as a potential use case.
http://jeremykun.com/2012/01/15/word-segmentation/
这篇关于NLP中的英语分词?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
