如何使用Java API在Google Dataproc Cluster上设置可选属性? [英] How to set optional properties on Google Dataproc Cluster using Java API?

问题描述

我正在尝试按照本文档使用Java API创建Dataproc集群

可以使用以下Google Cloud SDK添加这些属性

  gcloud dataproc集群创建my-cluster \--region = region \--properties = spark:spark.executor.memory = 4g \...其他参数... 

如何使用Java API进行设置.同样，对于Labels也完全一样，我们如何使用Java API在集群上设置标签.

解决方案

您可以检查完整的https://cloud.google.com/dataproc/docs/quickstarts/quickstart-lib

Sample code is as below

  public static void createCluster() throws IOException, InterruptedException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String region = "your-project-region";
    String clusterName = "your-cluster-name";
    createCluster(projectId, region, clusterName);
  }

  public static void createCluster(String projectId, String region, String clusterName)
      throws IOException, InterruptedException {
    String myEndpoint = String.format("%s-dataproc.googleapis.com:443", region);

    // Configure the settings for the cluster controller client.
    ClusterControllerSettings clusterControllerSettings =
        ClusterControllerSettings.newBuilder().setEndpoint(myEndpoint).build();

    // Create a cluster controller client with the configured settings. The client only needs to be
    // created once and can be reused for multiple requests. Using a try-with-resources
    // closes the client, but this can also be done manually with the .close() method.
    try (ClusterControllerClient clusterControllerClient =
        ClusterControllerClient.create(clusterControllerSettings)) {
      // Configure the settings for our cluster.
      InstanceGroupConfig masterConfig =
          InstanceGroupConfig.newBuilder()
              .setMachineTypeUri("n1-standard-1")
              .setNumInstances(1)
              .build();
      InstanceGroupConfig workerConfig =
          InstanceGroupConfig.newBuilder()
              .setMachineTypeUri("n1-standard-1")
              .setNumInstances(2)
              .build();
      ClusterConfig clusterConfig =
          ClusterConfig.newBuilder()
              .setMasterConfig(masterConfig)
              .setWorkerConfig(workerConfig)
              .build();
      // Create the cluster object with the desired cluster config.
      Cluster cluster =
          Cluster.newBuilder().setClusterName(clusterName).setConfig(clusterConfig).build();

      // Create the Cloud Dataproc cluster.
      OperationFuture<Cluster, ClusterOperationMetadata> createClusterAsyncRequest =
          clusterControllerClient.createClusterAsync(projectId, region, cluster);
      Cluster response = createClusterAsyncRequest.get();

      // Print out a success message.
      System.out.printf("Cluster created successfully: %s", response.getClusterName());

    } catch (ExecutionException e) {
      System.err.println(String.format("Error executing createCluster: %s ", e.getMessage()));
    }
  }
}

So based on the documentation i am able to create it successfully but there are few optional properties which i am not able to figure out how to set it here, for reference below screenshot of it which can be done using the Google Cloud console.

These properties can be added using Google Cloud SDK as below

gcloud dataproc clusters create my-cluster \
    --region=region \
    --properties=spark:spark.executor.memory=4g \
    ... other args ...

How to set this using Java API. Also the exact same thing for Labels as well, how can we set the labels on the cluster using Java API.

解决方案

You can check complete API reference for Dataproc Java client library.

Specifically, to set properties you want to look at SoftwareConfig.Builder. Similarly you can associate a label to a Cluster with Cluster.Builder.

这篇关于如何使用Java API在Google Dataproc Cluster上设置可选属性?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！