scala spark中基于时间戳的文件夹创建 [英] Timestamp based folder creation in scala spark

问题描述

我正在尝试读取基于时间戳的文件夹结构.如果我传递时间戳，那么它会根据输入路径读取文件夹结构.同样，我需要创建一个基于时间戳的文件夹结构来写入输出路径.

I am trying to read a timestamp based folder structure. If I pass the timestamp then it reads the folder structure according to the input path. Similarly, I need to create a timestamp based folder structure to write a output path.

This is my input path 
/Desktop/user/outFiles6/test1/2017/06/09/15

Similarly my output path should be created.

我是这样试的

  def buildPaths(date_key: DateTime, sc:SparkContext): (Path,Path) = {
  val (year, month, day,hour) = (date_key.toString("YYYY"), date_key.toString("MM"), date_key.toString("dd"),date_key.toString("HH"))
  val inpath_tag = new Path(
    makePath("/", Some("") :: Some("/home/user/Desktop/SparK-op/") :: Some(year) :: Some(month) :: Some(day) :: Some(hour) :: Nil)
  )

  val outpath = new Path(
    makePath("/", Some("") :: Some("/home/user/Desktop/SparK-op/") :: Some(year) :: Some(month) :: Some(day) ::  Some(hour) :: Nil)
  )
  //queryHDFS(sc, inpath_tag);
  //queryHDFS(sc, inpath_sens);
  (inpath_tag, outpath)
   }
  def makePath(char:String, components: List[Option[String]]) = components.flatten mkString char;

  } 
  }

我不知道如何进行.如果您能帮助构建逻辑，我们将不胜感激.

I don't know how to proceed.any help to construct a logic would be appreciated.

推荐答案

您需要导入库 File 才能使其工作.基本上需要创建您的目录并检查是否已经存在任何目录.下面是递归创建目录的代码.

You will need to import the library File for it to work. Basically it is needed to create your directory and also for checking if any directory is already present. Below is the code to create directory recursively.

import java.io.File

def makePath(char:String, components: List[Option[String]]) = {
   var path = ""
   var pathstring = components.flatten mkString char
   var pathArray = pathstring.split("/")
   for (directory <- pathArray) {
         if( directory != ""){
             path += directory 
             //Check if path present else create
             var file = new File(path)
             if(!file.exists){
                file.mkdir
                path += "/" // separator for creating next directory
             }
        }
    }
}

希望有帮助！

这篇关于scala spark中基于时间戳的文件夹创建的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！