处理大量CSV数据的最佳方式？ [英] Best way to process large amounts of CSV data?

问题描述

我是编码新手，目前负责以xlsx格式处理大型csv文件。需要分析的数据是调查的呼叫详细记录。我想开发和应用Windows平台，允许普通用户输入这些记录，并在后处理时显示以下结果：所有数字按递减顺序，日期和时间范围进行通信，以便根据导入活动的时间范围，显示所有纬度和长度坐标。我目前正在使用MS Excel，还没有实现我希望的结果。我不是一个经验丰富的程序员，正在努力提高我的技能，并认为这将是一个很好的起点。



提前致谢....



我尝试了什么：



Excel，虽然我还没弄明白一旦我开始分析我的数据，如何自动化该过程。我也试过学习python但是没有找到一个简单的项目让我学到更多东西。一个.net开发人员告诉我，我应该尝试使用C＃但是我还没有开始学习那种语言。

I am new to coding and currently am responsible for processing large csv files in xlsx form. The data that needs to be analyzed are call detail records for investigations. I would like to develop and application for a Windows platform that would allow an average user to input these records and, post processing, display the following results: all numbers being communicated with in descending order, dates and times ranges to be manually selected based upon time frame of activity imported, ma display of all lat and long coordinates. I am currently using MS Excel for this and have not yet realized the results I was hoping for. I am not an experienced programmer and am working toward improving my skills and figured this would be a great place to start.

Thanks in advance....

What I have tried:

Excel, although I haven't figured out how to automate the process once I start analyzing the data I have. I have also tried to learn python but have not found a project simple enough for me to learn more. A .net developer told me I should try C# but I have yet to start learning that language as of yet.

推荐答案

请阅读我对这个问题的评论。



如果你想能够操作来自csv文件的大量数据，你必须专注于C＃。当然，这取决于大量数据在您的情况下意味着什么。



有两种 - 至少 - 操纵csv数据的方法：

1）使用 ADO.Net [< a href =https://msdn.microsoft.com/en-us/library/e80y5yhx(v=vs.110).aspx\"target =_ blanktitle =New Window> ^ ] with OleDb [ ^ ]

读取文本文件特定列 [ ^ ]

OLE数据库教程（C＃） [ ^ ]

关于文本文件的大量ADO [ ^ ]

Schema.ini文件（文本文件驱动程序）| Microsoft Docs [ ^ ]

如何：向文本文件数据源添加架构定义 [ ^ ]

读取文本文件（txt，csv，日志，标签，固定长度） [ ^ ]



2）使用自定义解析器

CSV文件解析器 [ ^ ]

CSV Parser（C＃） [ ^ ]

使用LINQ读取分隔的文本文件 [ ^ ]

Please, read my comments to the question.

You have to focus on C# if you want to be able to operate large amount of data comming from csv files. Of course, it depends of what "large amount of data" means in your case.

There are two - at least - ways to manipulate csv data:
1) using ADO.Net[^] with OleDb[^]
Read Text File Specific Columns[^]
OLE DB Tutorial (C#)[^]
Much ADO About Text Files[^]
Schema.ini File (Text File Driver) | Microsoft Docs[^]
How to: Add a Schema Definition to a Text File Data Source[^]
Read Text File (txt, csv, log, tab, fixed length)[^]

2) using custom parser
CSV File Parser[^]
CSV Parser (C#)[^]
Using LINQ to read delimited text files[^]

Thanks for the input so far.  The avg file size I would need to process would be between 2.5 and 10 mb.  Being a forensic examiner I read from many sources that python is a great beginner language so I am following the pack on that.  Macros seems to be the easiest solution to learn while cutting my teeth on coding "hello world" and what is your age programs.

这篇关于处理大量CSV数据的最佳方式？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！