更快的数据处理。 [英] Faster data processing.

问题描述

目前我正在研究telcom项目，我必须在minitue中处理一些数量较少的cdr文件（cdr是一个包含详细信息的文本文件） \\ / $
与逗号分隔格式的调用有关。每行调用代表每个调用。该文件的大部分包含多达8k行）。我们必须处理每一行（格式化，评级等）并插入表中的那些值。我们正在使用

c＃用于前端和sql用于后端处理。每个minitue的一些处理好的cdr文件正常工作..但根据当前的要求我们必须处理几千我们当前的流程都失败了。

非常可怕。

所以有更快的数据处理方法。我听说过大数据（hadoop）。但是cdrs是完美结构的数据。这不是大数据应用的理想情况。请建议任何其他选择..



我尝试过：



我们逐行插入每个数据。我试图将其更改为批量插入但它没有用，因为我们要插入数据的表是也被其他服务使用（例如报告服务），所以我们不能锁定表超过100ms

Hi,

currently i am working on telcom project where i have to process few hundered cdr files in a minitue( cdr is a text file which contains details\
related with a call in comma seperated format.each line represent each call.most of the file contains up to 8k lines).we have to process each line(formating,rating etc etc) and insert those values in a table. we were using
c# for for the front end and sql for back end processing.everything working fine for a few hundered cdr files per minitue..but as per the current requiremnt we have to process a few thousand files at a time.n our current process fails
terribly.
so is there any alternative for more faster data processing.i heard about big data(hadoop).but cdrs are perfectly structed data.which is not the ideal case for big data applications.please suggest any other alternatives..

What I have tried:

we were inserting each data on row by row basis.i tried to change it to bulk insert but it didnt work because the table to which we are inserting data is used by other services too(such as reporting service),so we cannot lock table for for than 100ms

推荐答案

在我看来你首先找出来你的程序失败的地方，这就是瓶颈所在。例如，它可以是CDR格式解析或数据库内部的数据插入（或两者）。

一旦找到瓶颈，你可以尝试提高性能，（例如优化解析算法）。

In my opinion you have first to find out where your program is failing, that is where is the bottleneck. It could be, for instance, either in the CDR format parsing or in the data insertion inside the database (or both).
Once you located the bottleneck you could try to improve the performance, (e.g. optimizing the parsing algorithm).

这篇关于更快的数据处理。的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！