为批处理数据实现自定义迭代器/枚举器 [英] Implementing custom iterator/enumerator for batch data

问题描述

我有一个对对象建模的C ++类.该对象具有一个数据列表，该列表可能非常大.而且，检索这些数据可能会很昂贵.因此，当我需要访问列表中的项目时，我的对象将一次返回一批(例如2Kb的数据).

有没有一种方法可以编写自定义的迭代器或枚举器，这样我就可以遍历所有项目并让迭代器/枚举器处理每个批处理?我不确定是否将迭代器用于此目的，我对标准C ++迭代器没有太多经验.

基本上，我希望能够做这样的事情:

I have a C++ class which models an object. This object has a list of data, which can be quite large. And retrieving this data can be expensive. So when i need to access the items in the list, my object will return one batch at a time (say, 2Kb of data).

Is there a way to write a custom iterator or enumerator so i can loop over all items and have the iterator/enumerator handle getting each batch? I''m not sure if iterators are used for this purpose, i don''t have much experience with the standard C++ iterators.

Basically, i want to be able to do something like this:

Model::custom_iterator i;
for (i = model.items.begin(); i != model.items.end(); i++) {
    process item *i;
}

在Model类(或迭代器类)中，将有一些代码将获取一批数据并将其缓存，然后在需要更多数据时获取下一批数据.然后，迭代器将能够访问列表中的项目，并在需要时请求下一批.


我还想使用类似的技术来遍历生成成本高的列表(具有父/子关系的模型的祖先(读取:树))，这样我就不必在创建列表之前我遍历它.

And in the Model class (or the iterator class), there would be some code that would get a batch of data and cache it, then when more data is required it gets the next batch. The iterator would then be able to access the items in the list, and request the next batch when it needs it.


I would also like to use a similar technique for iterating over a list that is expensive to generate (ancestors of a model with a parent/child relationship (read: tree)), so that i don''t have to create the list before i iterate over it.

推荐答案

您可能会发现这很有帮助 ^ ]-这将有助于正确实现迭代器.

如果您的数据来自数据库，则可以将结果/游标"句柄包装在迭代器中.像OTL一样， Oracle，Odbc和DB2-CLI模板库 [
问候
Espen Harlinn

You may find this helpful The Boost.Iterator Library[^] - it will help to implement an iterator correctly.

If your data is comming from a database you can wrap the "result/cursor" handle in an iterator. Like OTL, the Oracle, Odbc and DB2-CLI Template Library[^]

One approach could be to use a vector for the "batches", wrap the iterator for the vector in a new custom iterator and get next batch each time the inner itertor reaches the end of the vector.

Regards
Espen Harlinn

一个复杂问题的简单答案:是的，毕竟这是迭代器的作用.您可以构建一个枚举器，该枚举器将有效地返回N或斐波那契数列或教师功能的全部.实际上消耗了该系列的多少取决于用户运行的代码(当然，以及涉及的数据类型，因为您可能会达到极限).

以下网站使用C#枚举，但也适用于C ++: http://thoughtfulcode.wordpress.com/2011/01/14/ienumerable-is-lazy-and-thats-cool/ [

A simple answer to a complex question: Yes that is what iterators are here for after all. You can build an enumerator that would effectively return all of N or the fibonacci series or the faculty function. How much of that series is in fact consumed depends on the code the user runs (and the data type involved of course as you might hit a limit there).

The following site deals with an enumerator in C# but applies also to C++:http://thoughtfulcode.wordpress.com/2011/01/14/ienumerable-is-lazy-and-thats-cool/[^]

Modification:

XTAL256写道:

XTAL256 wrote:

会有一些代码将获取一批数据并将其缓存，然后在需要更多数据时获取下一批

there would be some code that would get a batch of data and cache it, then when more data is required it gets the next batch

我实际上错过了您的问题的这一部分，并想详细说明枚举器为您提供的可能性:由于您完全控制了枚举器在内部所做的工作，因此您还可以实现某种分页机制，该机制将在您放入时提取批量数据它.只要您了解并实现所有IEmumerable接口，所有这些对于代码用户都是透明的.为什么我说可以对用户透明"?
这意味着您可以向用户隐藏所有幕后"细节，或者通过为您的IEnumerable实现类定义一些构造函数(让他指定获取的批处理大小等)来给他一些控制权.

最终修改

最好的问候，
曼弗雷德(Manfred)

I actually missed this part in your question and want to elaborate a bit on the possibilities enumerators give you: Since you have total control over what the enumerators do internally you can also implement some kind of paging mechanism that will fetch batches of data as you put it. As long as you understand and implement all of the IEmumerable interface all of this can be transparent to the user of your code. Why did I say "can be transparent to the user"?
It means you can hide all the "behind the curtain" details from the user or else give him some control by defining some constructors for your IEnumerable implementing class that let him specify fetched batch size etc.

End Modification

Best Regards,
Manfred

这篇关于为批处理数据实现自定义迭代器/枚举器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！