断开连接的数据集有多大？ [英] how big can disconnected dataset be?

问题描述

您好，


这是很多I / O.我正在使用VB6（需要

小时）。我从其对象库中使用

API连接到外部数据源（而不是ODBC，其中

情况我只是使用DTS来提取此数据并且只是

循环。我在想用VB.Net我可以把这个数据读成内存中的数据集（目前有4个演出

的内存 - 如果他们让它可以升级到16演出我有win2K

高级服务器 - 目前2个处理器1个演出每个）。我的想法是，如果我能将数据读入内存（

数据集），那么写入每条记录要快得多

（每个记录180个字段）。然后，只要

数据全部在本地内存中，我就可以执行

dataAdapter.Update或InsertInto到Sql Server。有什么想法吗？


当我在这里时，记录来自4个不同的

来源。我正在考虑使用多线程和

同时提取数据。我知道比Sql

Server2k只支持4演出一个mem。但是，如果我有超过4 GB的数据，那么一个VB.Net应用程序可以在
内管理数据集超过4 GB的内存吗？一旦我填充了我的数据集，我就会一次只执行一次InsertInto
。 VB.Net如何实现
多线程？再次，在VB6中，我调用4个独立的应用程序

，每晚同时提取数据。他们将
写入Sql Server2k中的4个独立表。我真的很喜欢

将所有这些放在一个应用程序中并直接读/写到
内存。 VB.Net是这样做的吗？


谢谢，

Rich

Hello,

I have to read and write around one million records from
an external data source to Sql Server2k every night.
That''s a lot of I/O. I am using VB6 for this (takes
hours). I am connecting to the external data source with
API''s from its object library (as opposed to ODBC in which
case I would just use DTS to pull this data) and just
looping. I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs
of mem - may upgrade to 16 gigs if they let me have win2K
Advanced Server - currently 2 processors 1 gig each). My
idea is that if I could just read the data into memory (a
dataset) it would be much faster than writing each record
(180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

While I''m at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in
more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for
multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write
to 4 separate table in Sql Server2k. I would really like
to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Thanks,
Rich

推荐答案

">

">

我必须每晚从外部数据源读取和写入大约一百万条记录到Sql Server2k。<这是很多I / O.


是的... =）

我正在使用VB6（需要
小时）。我使用其对象库中的API来连接外部数据源（而不是ODBC，其中我将使用DTS来提取此数据）并且只是循环。


我假设这是一种专有的数据格式？

我在想用VB.Net我可以把这个数据读进去内存中的数据集（目前有4个演出内存 - 如果他们让我拥有win2K高级服务器 - 目前每个处理器2个演出，可升级到16演出）。我的想法是，如果我可以将数据读入内存（
数据集），它将比编写每个记录要快得多（每个记录180个字段）。然后，只要
数据全部在本地内存中，我就可以对Sql Server执行一次
dataAdapter.Update或InsertInto。有什么想法吗？


您的更新/插入速度仍然会下降。我不是

确定它是否会将数据缓存在本地磁盘上，具体取决于内存量
。但是你仍然会受到SQL

服务器速度的影响。所以你可以让它排队等候，但它不一定会变得更快。

当我在它的时候，记录来自4个不同的<消息来源。我正在考虑使用多线程和同时拉取数据。我知道比Sql
Server2k只支持4演出一个mem。但是，如果我有超过4演出的数据，一个VB.Net应用程序可以管理超过4演出内存的数据集吗？一旦我填充了我的数据集，我就会一次只做一个InsertInto。 VB.Net如何实现多线程？再次，在VB6中，我调用4个独立的应用程序
，每晚同时拉取数据。他们在Sql Server2k中写入4个单独的表。我真的希望
在一个应用程序中拥有所有这些并直接读/写内存。 VB.Net是这样做的吗？


很多方面，我只是不知道你给我们的好处。

是的，您的读取和加载速度很快，但是这个程序将在SQL服务器本身上运行

吗？如果你想减少你的网络数量

流量，是的，这将解决你的问题，如果你想加速注入你的SQL服务器，不会这样做... =（


您可以为SQL Server使用更新的OLEDB驱动程序，由于优化的编程可能会刮掉一些时间，但是，你不知道如何获得实际获得的收益。


-CJ

谢谢，
Rich

I have to read and write around one million records from
an external data source to Sql Server2k every night.
That''s a lot of I/O.
Yes it is... =)
I am using VB6 for this (takes
hours). I am connecting to the external data source with
API''s from its object library (as opposed to ODBC in which
case I would just use DTS to pull this data) and just
looping.
I''m assuming this is a proprietary data format?
I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs
of mem - may upgrade to 16 gigs if they let me have win2K
Advanced Server - currently 2 processors 1 gig each). My
idea is that if I could just read the data into memory (a
dataset) it would be much faster than writing each record
(180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

Your still going to have a downturn on speed on the update/insert. I''m not
sure if it will cache data on your local disk or not depending on the amount
of memory. But you are still going to be subject to the speed of your SQL
Server. So you can have it all queued up, but its not necessarily going to
make it faster.
While I''m at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in
more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for
multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write
to 4 separate table in Sql Server2k. I would really like
to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Many ways, I just don''t know the benefit of it from what your giving us.
Yeah your reads and loading will be fast, but is this program going to run
on the SQL server itself? If your looking to reduce your amount of network
traffic, yeah this will solve your problem, if your looking to speed up
injection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server, which may shave
some time off because of optimized programming, but again, don''t know how
much of a gain you will actually get.

-CJ
Thanks,
Rich

谢谢。我忘记提及，是的，减少网络

流量，或者更确切地说，获取所有数据到本地服务器

并关闭连接对于

实现是一件好事。其他数据源是Lotus Notes R5。我有

从4个单独的Notes DB（nsf'）中提取数据。不是我的

设计。我可以接受或离开它。


所以你今天可以填充数据集

（光盘onnected数据集）这种方式？在外面思考，我在Bql Server中有4个表，并且在我的VB.net应用程序中有4个数据集。然后，通过多线程，我会开始

同时将数据拉入每个数据集。

我现在有4场演讲。这笔交易是旧的

方式，我只是将数据直接写入Sql

表。在这里，我尽可能快地将数据加载到内存中，

关闭我与Lotus Notes的连接（使用Domino Object

Library），然后开始将数据推送到Sql

来自数据集的服务器。这比使用基于4 com的应用程序阅读和书写更有效，更有效吗？
数据250,000+次？

Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB''s (nsf''s). Not my
design. I could take it or leave it.

So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app. Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn''t this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?

-----原始消息-----

">

-----Original Message-----

">

我必须读写大约一百万条记录从
一个外部数据源到Sql Server2k每晚。
这是很多I / O.
是的它是...... =）

I have to read and write around one million records from
an external data source to Sql Server2k every night.
That''s a lot of I/O.
Yes it is... =)

我正在使用VB6（需要
小时）。我从其对象库连接到外部数据源
和API'（而不是
中的ODBC，我只使用DTS来提取这些数据）并且只是循环。

I am using VB6 for this (takes
hours). I am connecting to the external data source with API''s from its object library (as opposed to ODBC in which case I would just use DTS to pull this data) and just
looping.

我假设这是一种专有的数据格式？

I''m assuming this is a proprietary data format?

我在想用VB.Net我可以读到 win2K高级服务器 - 目前每个处理器1个处理器，可以升级到16个演出）。
我的想法是，如果我可以将数据读入内存
（数据集），那么写入每个
记录（每个记录180个字段）要快得多。然后，只要
数据全部在本地内存中，我就可以对Sql Server执行一次
dataAdapter.Update或InsertInto。有什么想法吗？

I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs of mem - may upgrade to 16 gigs if they let me have win2K Advanced Server - currently 2 processors 1 gig each). My idea is that if I could just read the data into memory (a dataset) it would be much faster than writing each record (180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

你仍然会在

Your still going to have a downturn on speed on the

更新/插入时速度下降。我不确定它是否会在本地磁盘上缓存数据，或者不是
，具体取决于内存量。但是你仍然需要服从SQLServer的
速度。所以你可以让它排队等候，但它不是
必然会让它更快。

update/insert. I''m notsure if it will cache data on your local disk or not depending on the amountof memory. But you are still going to be subject to the speed of your SQLServer. So you can have it all queued up, but its not necessarily going tomake it faster.

当我在它时，记录来自4个不同的 Server2k只支持4演出一个mem。但是，如果我有超过4 GIG的数据，那么一个VB.Net应用程序可以在超过4 GB的内存中管理数据集
？一旦我填充了我的数据集，我就会一次只做一个InsertInto。 VB.Net
如何用于多线程？再次，在VB6中，我调用4个独立的应用程序
，每晚同时拉取数据。他们
写入Sql Server2k中的4个单独的表。我真的想要在一个应用程序中拥有所有这些并直接读/写内存。 VB.Net是一种这样做的方法吗？

很多方面，我只是不知道你给我们的是什么

While I''m at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write to 4 separate table in Sql Server2k. I would really like to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Many ways, I just don''t know the benefit of it from what

。是的，你的读取和加载会很快，但这个
程序是否会运行SQL服务器本身？如果您希望减少
的网络流量，是的，这将解决您的问题，如果您的
希望加速注入您的SQL服务器，不会这样做... =（
<你可以使用更新的OLEDB驱动程序用于SQL Server，
由于优化编程可能会浪费时间，但再次
，不知道你实际上会获得多少收益。

-CJ

your giving us.Yeah your reads and loading will be fast, but is this program going to runon the SQL server itself? If your looking to reduce your amount of networktraffic, yeah this will solve your problem, if your looking to speed upinjection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server, which may shavesome time off because of optimized programming, but again, don''t know howmuch of a gain you will actually get.

-CJ

谢谢，
Rich

Thanks,
Rich

。

.

" Rich"< an ******* @ discussion.microsoft.com>写在留言中

新闻：14 ***************************** @ phx.gbl ...

"Rich" <an*******@discussions.microsoft.com> wrote in message
news:14*****************************@phx.gbl...

谢谢。我忘了提，是的，减少网络流量，或者更确切地说，将所有数据都提供给本地服务器
并关闭连接将是一件好事。
实现。其他数据源是Lotus Notes R5。我有从4个独立的Notes DB中提取数据's（nsf'）。不是我的
设计。我可以接受或离开它。


我编码了两年的笔记（4.5和5）......老实说，我有

从不看了一个项目并说你知道什么是好的

解决方案......注意......


我知道它有它的优点，但从开发人员的立场来看，它有点糟透了，而且非常慢......

所以你有一天可以填充数据集
（断开连接的数据集）这种方式？在外面思考，我在Sql Server中有4个表，在我的VB.net应用程序中有4个数据集。


是的，那没关系。我确定Notes API会降低你的速度，因为它是b $ b ...所以这会降低你的整体网络流量（好吧，

的时间用于网络流量，你会在一开始就爆发一次大爆炸，然后什么都没有。

那么，多线程，我会开始拉动同时将数据输入每个数据集。
我现在有4个演出。这笔交易是旧的方式，我只是将数据直接写入Sql
表。在这里，我尽可能快地将数据加载到内存，
关闭我与Lotus Notes的连接（在这里使用Domino Object
库），然后开始从数据集将数据推送到Sql
服务器。与基于4个com的应用程序读取和写入数据的250,000次以上相比，这不是更好，更有效吗？


对于网络流量，是的，它会更好。我不认为你会在你的SQL服务器上看到一个巨大的性能提升，但我相信你会看到一些

（因为你现在不是从每个记录的外国来源请求数据）。


尽管如此...... 180个字段，100万条记录，仍然需要花费时间

etime to进到那里去。 =）


-CJ

Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB''s (nsf''s). Not my
design. I could take it or leave it.

I coded notes (4.5 and 5) for 2 years... I can honestly say that I have
never looked on a project and said "You know what would be a good
solution... Notes..."

I know it has its advantages, but coming from a developer standpoint it
kinda sucks and is INCREDIBLY slow...
So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app.
Yeah, that would be fine. I''m sure the Notes API is slowing you down as it
is... So this will reduce your overall network traffic (well, amount of
time used for network traffic, you''ll have a big burst at the beginning and
then nothing).
Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn''t this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?
For network traffic, yes, it would be better. I don''t think you will see a
huge performance increase on your SQL server, but I''m sure you will see some
(as you now are not requesting data from a foreign source for each record).

Nonetheless... 180 fields, 1 million records, still going to take al ittl
etime to get in there. =)

-CJ

-----原始消息-----

">

-----Original Message-----

">

我必须每天晚上从外部数据源读取和写入大约一百万条记录到Sql Server2k。
这就是我的很多/ O。

I have to read and write around one million records from
an external data source to Sql Server2k every night.
That''s a lot of I/O.

是的，它是... =）

Yes it is... =)

我正在使用VB6（需要
小时）。我从其对象库连接到外部数据源（与ODBC相反，在这种情况下我只使用DTS来提取这些数据）并且只是循环。

I am using VB6 for this (takes
hours). I am connecting to the external data source with API''s from its object library (as opposed to ODBC in which case I would just use DTS to pull this data) and just
looping.

我假设这是一种专有数据格式？

I''m assuming this is a proprietary data format?

我在想用VB.Net我可以将这些数据读入数据集中内存（目前有4演出的内存 - 如果他们让我拥有win2K高级服务器 - 目前有2个处理器1个演出，可以升级到16演出）。我的想法是，如果我可以将数据读入内存（数据集），它将比编写每条记录（每条记录180个字段）快得多。然后，只要
数据全部在本地内存中，我就可以对Sql Server执行一次
dataAdapter.Update或InsertInto。有什么想法吗？

I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs of mem - may upgrade to 16 gigs if they let me have win2K Advanced Server - currently 2 processors 1 gig each). My idea is that if I could just read the data into memory (a dataset) it would be much faster than writing each record (180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

Your still going to have a downturn on speed on the

更新/插入你的速度仍然会有所下降。我不是

update/insert. I''m not

确定它是否会在本地磁盘上缓存数据

sure if it will cache data on your local disk or not

，具体取决于内存的

depending on the amount

的数量。但是你仍然会受到SQL

of memory. But you are still going to be subject to the

Server的

速度的影响。所以你可以让它排队等候，但它不是

Server. So you can have it all queued up, but its not

必然要去

necessarily going to

让它更快。

make it faster.

虽然我'对此，记录来自4个不同的来源。我正在考虑使用多线程和同时拉取数据。我知道比Sql
Server2k只支持4演出一个mem。但是，如果我有超过4个数据的数据，一个VB.Net应用程序可以管理4个以上内存中的数据集吗？一旦我填充了我的数据集，我就会一次只做一个InsertInto。 VB.Net如何进行多线程处理？再次，在VB6中，我调用4个独立的应用程序
，每晚同时拉取数据。他们在Sql Server2k中写入4个单独的表。我真的希望在一个应用程序中拥有所有这些并直接读/写内存。 VB.Net是一种这样做的方法吗？

While I''m at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write to 4 separate table in Sql Server2k. I would really like to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

很多方面，我只是不知道你给我们的

Many ways, I just don''t know the benefit of it from what

带来的好处。

your giving us.

是的，您的读取和加载速度很快，但这个

Yeah your reads and loading will be fast, but is this

程序是否会在SQL服务器本身上运行

program going to run

？如果你想减少你的

数量的网络

amount of network

流量，是的，这将解决你的问题，如果你的

traffic, yeah this will solve your problem, if your

希望加快

looking to speed up

注入你的SQL服务器，不会这样做... =（

你可以使用更新的OLEDB驱动程序用于SQL Server，

injection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server,

由于优化编程可能会刮掉

which may shave

一段时间，但

再次，不知道

again, don''t know how

你实际上会获得多少收益得到。

-CJ

much of a gain you will actually get.

-CJ

谢谢，
Rich

Thanks,
Rich

。

.

这篇关于断开连接的数据集有多大？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！