如何在Google Cloud Bigtable中设置将来的插入日期?尝试使用TTL计算 [英] How to set a future insert date in Google Cloud Bigtable? Trying to calculate it using TTL

问题描述

我的表只有一个列族，此列的TTL为172800秒(2天)，我需要在截止日期之前删除一些数据.如果我希望该值在5分钟后过期，我将计算过期时间并将插入日期设置为过期时间之前的5分钟.

I have a table with only one column family, this column has a TTL of 172800 SECONDS (2 DAYS), I need some data to be deleted before the deadline. If I want the value to expire in 5mins, I calculate the expiry time and set the insert date to be 5 mins before expiry time.

我正在使用Java的HBase客户端来完成此操作.

I am using the HBase Client for Java to do this.

但是该值似乎没有过期.有什么建议吗?

But the value doesn't seem to expire. Any suggestions on the same?

我使用cbt创建表:

cbt createtable my_table families=cf1:maxage=2d

HColumnDescriptor:

HColumnDescriptor:

{NAME => 'cf1', BLOOMFILTER => 'ROW', VERSIONS => '2147483647', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '172800 SECONDS (2 DAYS)', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}

Java代码:

import com.google.cloud.bigtable.hbase.BigtableConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;

import java.io.IOException;
import java.util.Calendar;
import java.util.Date;

public class BigTable {
    public static void main(String... args) {
        String projectId = "my-gcp-project-id";
        String instanceId = "my-bigtable-instance-id";
        String tableId = "my-table";    // my-bigtable-table-id

        try (Connection connection = BigtableConfiguration.connect(projectId, instanceId)) {
            try (Table table = connection.getTable(TableName.valueOf(tableId))) {

                HTableDescriptor hTableDescriptor = table.getTableDescriptor();
                hTableDescriptor.setCompactionEnabled(true);

                byte[] cf1 = Bytes.toBytes("cf1");
                byte[] rk1 = Bytes.toBytes("rowkey1");
                byte[] q1 = Bytes.toBytes("q1");

                HColumnDescriptor cfDescriptor1 = hTableDescriptor.getFamily(cf1);
                System.out.println("\n " + cfDescriptor1);

                Calendar now = Calendar.getInstance();
                Calendar now1 = Calendar.getInstance();
                now1.setTime(now.getTime());

                long nowMillis = now.getTimeInMillis(); // Current time

                now.add(Calendar.SECOND, cfDescriptor1.getTimeToLive()); // Adding 172800 SECONDS (2 DAYS) to current time
                long cfTTLMillis = now.getTimeInMillis(); // Time the values in the column family will expire at

                now1.add(Calendar.SECOND, 300); // Adding 300 secs (5mins)
                long expiry = now1.getTimeInMillis(); // Time the value should actually live

                long creationTime = nowMillis + cfTTLMillis - expiry;

                System.out.println("\n Date nowMillis:\t" + new Date(nowMillis) + "\n Date creationTime:\t" + new Date(creationTime) + "\n Date cfTTLMillis:\t" + new Date(cfTTLMillis));

                //Add Data
                Put p = new Put(rk1, creationTime);
                p.addColumn(cf1, q1, Bytes.toBytes("CFExpiry_2d_ExpTime_5mins"));
                //p.setTTL(creationtime); // What does this do?
                table.put(p);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }}

计算日期:

 Date nowMillis:    Wed Oct 03 10:34:15 EDT 2018
 Date creationTime: Fri Oct 05 10:29:15 EDT 2018
 Date cfTTLMillis:  Fri Oct 05 10:34:15 EDT 2018

已正确插入带有正确计算日期的值.但是似乎还没到期吗?如果错误，请更正我的概念.

The Value is inserted correctly with the correct calculated dates. But doesn't seem to expire? Please correct my concepts if wrong.

在日期计算中进行以下更正后，这些值确实会过期.

After the below correction in date calculation, the values do expire.

long nowMillis = System.currentTimeMillis() / 1000;
long cfTTLMillis = nowMillis - cfDescriptor1.getTimeToLive();
long creationTime = (cfTTLMillis + 300) * 1000;

推荐答案

在压缩之前，Cloud Bigtable不会对行进行垃圾收集.这可能会在预期的到期时间后数小时(或几天)内发生.

Cloud Bigtable does not garbage collect rows until a compaction occurs. That may happen hours (or possibly a few days) after the expected expiration.

如果要确保不读取应已过期的数据，请对读取的数据设置一个时间戳范围过滤器，以使查询中不会返回超出允许范围的值.

If you want to make sure to not read data that should have expired, please set a timestamp range filter on the data read so that values outside of the allowed range aren't returned in the query.

或者，您必须在返回数据之后将其过滤掉，但是将其过滤到服务器端效率更高，从而客户端不必下载或处理它.

Alternatively, you'll have to filter them out after the data is returned, but it's much more efficient to filter it out server-side so that the client does not have to download or process it.

这篇关于如何在Google Cloud Bigtable中设置将来的插入日期?尝试使用TTL计算的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！