URL路径相似度/字符串相似度算法 [英] URL path similarity/string similarity algorithm

问题描述

我的问题是，我需要比较URL路径并推断出它们是否相似。下面我提供了要处理的示例数据：

My problem is that I need to compare URL paths and deduce if they are similar. Below I provide example data to process:

# GROUP 1
/robots.txt

# GROUP 2
/bot.html

# GROUP 3
/phpMyAdmin-2.5.6-rc1/scripts/setup.php
/phpMyAdmin-2.5.6-rc2/scripts/setup.php
/phpMyAdmin-2.5.6/scripts/setup.php
/phpMyAdmin-2.5.7-pl1/scripts/setup.php
/phpMyAdmin-2.5.7/scripts/setup.php
/phpMyAdmin-2.6.0-alpha/scripts/setup.php
/phpMyAdmin-2.6.0-alpha2/scripts/setup.php

# GROUP 4
//phpMyAdmin/

我尝试用Levenshtein距离进行比较，但是对我来说还不够准确。我不需要100％准确的算法，但是我认为必须达到90％以上。

I tried Levenshtein distance to compare, but for me is not enough accurate. I do not need 100% accurate algorithm, but I think 90% and above is a must.

我认为我需要某种分类器，但问题是新数据的每个部分都可以包含应该分类为新的未知类的路径。

I think that I need some sort of classifier, but the problem is that each portion of new data can containt path that should be classified to the new unknown class.

您能否将我定向到正确的位置？

Could you please direct me to the right thoutht?

谢谢

推荐答案

在检查@ jakub.gieryluk建议时，我偶然发现了令我满意的解决方案- 霍博姆聚类算法，最初旨在减少生物序列数据集的冗余。

When checking @jakub.gieryluk suggestion I accidentally have found solution that satisfy me - "Hobohm clustering algorithm, originally devised to reduce redundancy of biological sequence data sets."

由 Bruno Vecchi 给了我很好的结果。唯一的问题是我需要Python实现，但是我相信我既可以在Internet上找到一个代码，也可以自己重新实现代码。

Tests of PERL library implemented by Bruno Vecchi gave me really good results. The only problem is that I need Python implementation, but I belive that I can either find one on the Internet or reimplement code by myself.

接下来的事情是我还没有检查了该算法的主动学习能力；）

Next thing is that I have not checked active learning ability of this algorithm yet ;)

这篇关于URL路径相似度/字符串相似度算法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！