如何解决csv.DictWriter覆盖csv中的数据? [英] How to resolve csv.DictWriter overwriting data in the csv?

问题描述

我试图抓取Twitter，以获取某些用户的关注者/朋友计数.我有大量要签出的用户.我实际上想将输出收集到字典中，然后将输出写入CSV文件.我同时尝试了大熊猫(dict-> dataframe-> csv)和(dict-> CSV)路由，但写操作一直失败.

I am trying to scrape Twitter in order to get the follower/friend counts of certain user. I have a large list of users to check out. I actually want to collect the output into a dictionary and then write the output into a CSV file. I tried both the pandas (dict -> dataframe -> csv) and (dict -> CSV) routes but I keep getting failed writing.

我的代码如下:

# Writing directly from Dictionary to CSV  

auth = tweepy.OAuthHandler(api_key, api_secret_key)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True,
    wait_on_rate_limit_notify=True)

# *Just a sample of the large user list I want to check out*
z =['Schwarzenegger', 'drdrew', 'NPR', 'billboard', 'SenJohnMcCain', 'LaurenJauregui', 'MarkRuffalo']

for i in z:
    user_dict = {}
    follower_count = api.get_user(i).followers_count
    friend_count = api.get_user(i).friends_count
    # print(i, follower_count, friend_count)

    # create a dictionary to hold values
    user_dict[i] = follower_count, friend_count

    # Write dictionary into csv file
    cols = ["username", "followers_count"]
    try:
        with open('details.csv', 'w', newline='', encoding='utf8') as f:
            writer = csv.DictWriter(f, fieldnames=cols)
            writer.writeheader()
            for data,val in user_dict.items():
                writer.writerows([{"username": data, "followers_count": val}])
    except IOError:
        print("I/O error")

#Notify me when operation is completed
print("file write completed")

输出 >>> 文件仅包含最后一个条目:

MarkRuffalo,"(6674117, 1852)"

Dict-> DF-> csv路由还生成了一个仅包含标题但内容为空的文件:

The Dict -> DF -> csv route also produced a file that only has headings but empty contents:

df = pd.DataFrame(user_dict, columns = ["follower_count","friend_count"])
print(df)
df.to_csv('user_files.csv', header=True)

请确保我将所有词典条目都写入文件中，该怎么办.谢谢你. 附注:我不熟悉所有这些内容，因此我的写作可能很尴尬.

Please what can I do to ensure all the dictionary entries are written into the file. Thank you. P.S: I am new to all of these, so my writing may be awkward.

推荐答案

1. 在open()语句后的for循环内放置"cols"
2. 在writeheader()语句后的"try"中放入for循环(对于z中的i而言)
3. 删除此行:对于数据，user_dict.items()中的val:"
4. 在您的writerow变量中使用API​​变量(来自for循环) ("writerow"不是复数-删除最后的"s")

1. Place "cols" inside the for loop after the open() statement
2. Put the for loop (for i in z:) inside your "try" after the writeheader() statement
3. Remove this line: "for data,val in user_dict.items():"
4. Use the API variables (from the for loop) in your writerow variables ("writerow" is not plural - remove the "s" at the end)

这些资源将为您提供帮助:

使用Python遍历字典: https://realpython.com/iterate-through-dictionary-python/

阅读和阅读编写CSV文件: https://realpython.com/python-csv/

我最终尝试了它，但是效果很好.很抱歉缩进可能会消失

I tried it on my end and it worked. I apologize for the indenting it may be off

# Write dictionary into csv file

try:
    with open('details.csv', node='w') as f:
    cols = ["username", "followers_count","friends_count"]
    writer = csv.DictWriter(f, fieldnames=cols)

    writer.writeheader()
    for i in z:
        user_dict = {}
        follower_count = api.get_user(i).followers_count
        friend_count = api.get_user(i).friends_count
        # print(i, follower_count, friend_count)

        # assign values
        user_dict[i] = follower_count, friend_count

        #write to each row
        writer.writerow({cols[0]:i, cols[1]:follower_count, cols[2]:friend_count})

except IOError:
    print("I/O error")

#Notify me when operation is completed
print("file write completed")

对于Panda DataFrame: 我使用下面的代码来工作-但没有标题 在单独的列中显示字典键和值

For the Panda DataFrame: I got it to work using the below - but there are no headers displays dictionary key+value(s) in separate columns

df = pd.DataFrame(data=user_dict)
print(df)
df.to_csv('user_files.csv', header=True)

第三个示例-现在使用转置在单独的行上显示字典键和值

A third example - now using Transpose to display dictionary key+values(s) on separate rows

df = pd.DataFrame(data = user_dict)
df = df.T
print(df)
df.to_csv('user_files2.csv', header=True)

您将不得不使用这些标题的列标题

You will have to play around with the column headers on these ones

我的资源: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html

这篇关于如何解决csv.DictWriter覆盖csv中的数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！