爬网时显式显示特殊字符 [英] Explicit special characters from crawling

问题描述

使用Storm Crawler 1.13和弹性搜索6.5.2.如何限制搜寻器不搜寻/索引特殊字符� � � � � ��� �� � •

Working on Storm Crawler 1.13 and elastic search 6.5.2. How to restrict the crawler not to crawl/index the special characters � � � � � ��� �� � •

推荐答案

一种简单的方法是编写

        ParseData pd = parse.get(URL);
        String text = pd.getText();
        // remove chars
        pd.setText(text);

这将被JSoup或Tika解析的文档调用. 请查看存储库中的解析过滤器以获取示例.

This will get called on documents parsed by JSoup or Tika. Have a look at the parse filters in the repository for examples.

这篇关于爬网时显式显示特殊字符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！