保持RSS提要在SQL数据库中唯一的最佳实践 [英] Best practice to keep RSS feeds unique in sql database

问题描述

我正在一个项目中，该项目显示了来自不同站点的rss feed.我将它们保存在数据库中，每隔3个小时我的程序就会提取一次并将它们插入到sql数据库中.我希望提供者的唯一记录不要显示重复的内容.

I am working on a project which shows rss feeds from different sites. I keep them in the database, every 3 hours my program fetches and inserts them into sql database. I want unique records for providers not to show duplicate content.

但是问题是有些提供程序没有提供GUID字段，而另一些提供程序没有提供GUID字段，但是没有发布日期.还有一些提供程序甚至没有提供GUID或PubDate标题和链接.

But problem is some providers do not give GUID field, and some others gives GUID field but not pubdate.. And some others does not even give GUID or PubDate just title and link.

因此，要使rss feed在sql server中保持唯一性，最好的方法是什么?

So to keep rss feeds uniqe in sql server what would be the best way?

我应该先检查GUID，然后是pubbdate，然后是链接，然后是标题吗?比较SQL中的链接字段以检查唯一性是一种好习惯吗?

Should I check for first guid, then pubbdate, then link, then title? Will it be to good practice to compare link fields in SQL to check uniqueness?

谢谢.

推荐答案

我将开发一个例程，该例程采用某些关键参数(例如标题，源和正文)，然后将它们组合以创建CRC哈希.然后将哈希值与Feed一起存储为属性，并在添加新的Feed之前检查匹配的哈希值.

I would develop a routine that takes certain key parameters like the title, source and body and then combines them to create a CRC hash. Then store the hash as an attribute with the feed and check for a matching hash before adding a new feed.

我不确定您的环境有何限制，但这是在C#中计算CRC-32的示例: http://damieng.com/blog/2006/08/08/calculating_crc32_in_c_and_net

I'm not sure what your environment contraints are but here is an example for calculating CRC-32 in C#: http://damieng.com/blog/2006/08/08/calculating_crc32_in_c_and_net

这篇关于保持RSS提要在SQL数据库中唯一的最佳实践的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！