如何在 hadoop 集群上处理 id 生成? [英] How to handle id generation on a hadoop cluster?

问题描述

我正在 hadoop 集群上构建字典，需要为每个令牌生成一个 数字 id.我该怎么做?

I am building a dictionary on a hadoop cluster and need to generate a numeric id for each token. How should I do it?

推荐答案

您有两个问题.首先，您要确保为每个令牌准确分配一个 ID.为此，您应该按标记对记录进行排序和分组，并在减速器中进行分配.一旦确定为每个标记只调用了一次 reducer 方法，您就可以使用上下文中的分区号和由 reducer 维护的唯一数字 ID(每个分区一个实例) - 只需使用初始化为 1 的实例变量在 setup 方法中并在 reduce 方法中递增.

You have two problems. First you want to make sure that you assign exactly one id for each token. To do that you should sort and group records by token and make the assignment in a reducer. Once you've made sure that the reducer method is called exactly once for each token you can use the partition number from the context and a unique numeric id maintained by the reducer (one instance per partition) - just use an instance variable initialized to 1 in the setup method and incremented in the reduce method.

这篇关于如何在 hadoop 集群上处理 id 生成?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！