联接后如何在Pyspark数据框中选择和排序多个列 [英] How to select and order multiple columns in a Pyspark Dataframe after a join

问题描述

我想从现有数据框(在连接后创建)中选择多个列，并想将文件排序为我的目标表结构.怎么做到呢 ?我使用的方法如下.在这里，我可以选择所需的必要列，但不能按顺序进行.

I want to select multiple columns from existing dataframe (which is created after joins) and would like to order the fileds as my target table structure. How can it be done ? The approached I have used is below. Here I am able to select the necessary columns required but not able to make in sequence.

Required (Target Table structure) :
hist_columns = ("acct_nbr","account_sk_id", "zip_code","primary_state", "eff_start_date" ,"eff_end_date","eff_flag")

account_sk_df = hist_process_df.join(broadcast(df_sk_lkp) ,'acct_nbr','inner' )
account_sk_df_ld = account_sk_df.select([c for c in account_sk_df.columns if c in hist_columns])

>>> account_sk_df
DataFrame[acct_nbr: string, primary_state: string, zip_code: string, eff_start_date: string, eff_end_date: string, eff_flag: string, hash_sk_id: string, account_sk_id: int]


>>> account_sk_df_ld
DataFrame[acct_nbr: string, primary_state: string, zip_code: string, eff_start_date: string, eff_end_date: string, eff_flag: string, account_sk_id: int]

account_sk_id必须位于第二位.最好的方法是什么?

The account_sk_id need to be in 2nd place. What's the best way to do this ?

推荐答案

尝试仅通过提供列表来选择列，而不是通过迭代现有列或排序应该可以:

Try selecting columns by just giving a list, not by iterating existings columns or ordering should be OK:

account_sk_df_ld = account_sk_df.select(*hist_columns)

这篇关于联接后如何在Pyspark数据框中选择和排序多个列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！