HBase，Hadoop:如何估算HBase表或Hadoop文件系统路径的大小? [英] HBase, Hadoop : How can I estimate the size of a HBase table or Hadoop File System Paths?

问题描述

我有多个HBase表，如何估计在Java中使用的表的大致大小?

I have multiple HBase tables, how can I estimate the approximate size of the tables using in java?

推荐答案

一种方法是，您必须使用Java客户端(通常在/hbase文件夹下)访问hdfs 所有表信息.将存在.

One way is you have to access hdfs using java client usually under /hbase folder all the tables info. will be present.

您可以使用hadoop fs -du -h **path to hbase**/hbase

在/hbase下，每个表又占用一个文件夹...

under /hbase each table occupies one more folder...

hadoop fs -ls -R **path to hbase**/hbase

hadoop fs -du -h **path to hbase**/hbase/tablename

同样，您可以通过在hbase根目录下传递每个表路径来使用Java hdfs客户端，如下所示... 检查getSizeOfPaths& getSizeOfDirectory方法

Same thing you can use java hdfs client by passing each table path under hbase root dir like below ... Check getSizeOfPaths & getSizeOfDirectory methods

public class HdfsUtil {
    /**
     * Estimates the number of splits by taking the size of the paths and dividing by the splitSize.
     *
     * @param paths
     * @param configuration
     * @param splitSize
     * @return
     * @throws IOException
     */
    public static long getNumOfSplitsForInputs(Path[] paths, Configuration configuration, long splitSize) throws IOException
    {
        long size = getSizeOfPaths(paths, configuration);
        long splits = (int) Math.ceil( size / (splitSize)) ;
        return splits;
    }

    public static long getSizeOfPaths(Path[] paths, Configuration configuration) throws IOException
    {
        long totalSize = 0L;

        for(Path path: paths)
        {
           totalSize += getSizeOfDirectory(path, configuration);
        }
        return totalSize;
    }
// here you can give hbase path folder which was described through shell
        public static long getSizeOfDirectory(Path path, Configuration configuration) throws IOException {
            //Get the file size of the unannotated Edges
            FileSystem fileSystem = FileSystem.get(configuration);
            long size  = fileSystem.getContentSummary(path).getLength();
/**static String    byteCountToDisplaySize(BigInteger size)
Returns a human-readable version of the file size, where the input represents a specific number of bytes.**/
System.out.println(FileUtils.byteCountToDisplaySize(size))
            return size;
        }
    }

这篇关于HBase，Hadoop:如何估算HBase表或Hadoop文件系统路径的大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！