将所有导航属性加载（懒惰或渴望）放入内存之前过滤 [英] Filter all navigation properties before they are loaded (lazy or eager) into memory

问题描述

对于未来的访问者：对于EF6，您最好使用过滤器，例如通过此项目： https ：//github.com/jbogard/EntityFramework.Filters

For future visitors: for EF6 you are probably better off using filters, for example via this project: https://github.com/jbogard/EntityFramework.Filters

在我们构建的应用程序中，我们应用每个类都有的软删除模式一个已删除的布尔。实际上，每个类都从这个基类继承：

In the application we're building we apply the "soft delete" pattern where every class has a 'Deleted' bool. In practice, every class simply inherits from this base class:

public abstract class Entity
{
    public virtual int Id { get; set; }

    public virtual bool Deleted { get; set; }
}

给出一个简单的例子，假设我有类 GymMember 和锻炼：

To give a brief example, suppose I have the classes GymMember and Workout:

public class GymMember: Entity
{
    public string Name { get; set; }

    public virtual ICollection<Workout> Workouts { get; set; }
}

public class Workout: Entity
{
    public virtual DateTime Date { get; set; }
}

当我从数据库中获取健身房成员名单时，我可以确保没有一个删除的健身房成员被取出，如下所示：

When I fetch the list of gym members from the database, I can make sure that none of the 'deleted' gym members are fetched, like this:

var gymMembers = context.GymMembers.Where(g => !g.Deleted);

然而，当我通过这些健身房会员时，他们的锻炼从数据库加载，而不考虑他们的已删除标志。虽然我不能怪实体框架没有拿起来，我想配置或截取延迟属性加载某种方式，以便删除的导航属性永远不会加载。

However, when I iterate through these gym members, their Workouts are loaded from the database without any regard for their Deleted flag. While I cannot blame Entity Framework for not picking up on this, I would like to configure or intercept lazy property loading somehow so that deleted navigational properties are never loaded.

我一直在浏览我的选项，但看起来很差：

I've been going through my options, but they seem scarce:

• 前往数据库首先并使用每个对象对于每一对多属性的条件映射。

• Going to Database First and use conditional mapping for every object for every one-to-many property.

这根本不是一个选项，因为这将是太多的手动工作。 （我们的应用是巨大的，每天都越来越拥抱）。我们也不想放弃使用Code First（其中有很多）的优势。

This is simply not an option, since it would be too much manual work. (Our application is huge and getting huger every day). We also do not want to give up the advantages of using Code First (of which there are many)

• 始终热切加载导航属性。

• Always eagerly loading navigation properties.

再次，不是一个选项。此配置仅适用于每个实体。一直热心加载实体也会造成严重的性能损失。

Again, not an option. This configuration is only available per entity. Always eagerly loading entities would also impose a serious performance penalty.

• 应用自动注入的表达式访问者模式。 （e =>！e.Deleted）在任何地方找到 IQueryable< Entity> ，如此处和此处。

• Applying the Expression Visitor pattern that automatically injects .Where(e => !e.Deleted) anywhere it finds an IQueryable<Entity>, as described here and here.

我在一个概念应用证明中实际测试了这个，它的工作非常好。
这是一个非常有趣的选项，但是，它无法将过滤应用于延迟加载的导航属性。这是显而易见的，因为这些惰性属性不会出现在表达式/查询中，因此无法替换。我不知道Entity Framework是否允许在他们的 DynamicProxy 类中的某个位置加载懒惰属性的注入点。
我也担心会有其他后果，例如在EF中打破 Include 机制的可能性。

I actually tested this in a proof of concept application, and it worked wonderfully. This was a very interesting option, but alas, it fails to apply filtering to lazily loaded navigation properties. This is obvious, as those lazy properties would not appear in the expression/query and as such cannot be replaced. I wonder if Entity Framework would allow for an injection point somewhere in their DynamicProxy class that loads the lazy properties. I also fear for for other consequences, such as the possibility of breaking the Include mechanism in EF.

• 编写实现ICollection的自定义类，然后自动过滤删除实体。 / li>
  

• Writing a custom class that implements ICollection but filters the Deleted entities automatically.

这实际上是我的第一个方法。这个想法将是为内部使用自定义Collection类的每个collection属性使用backing属性：

This was actually my first approach. The idea would be to use a backing property for every collection property that internally uses a custom Collection class:

public class GymMember: Entity
{
    public string Name { get; set; }

    private ICollection<Workout> _workouts;
    public virtual ICollection<Workout> Workouts 
    { 
        get { return _workouts ?? (_workouts = new CustomCollection()); }
        set { _workouts = new CustomCollection(value); }
     }

}

虽然这种方法其实不是不好，我还有一些问题：

While this approach is actually not bad, I still have some issues with it:

• 它仍然加载所有的锻炼 s进入内存，并在属性设置器被击中时过滤已删除。在我谦虚的观点下，这太晚了。

• It still loads all the Workouts into memory and filters the Deleted ones when the property setter is hit. In my humble opinion, this is much too late.

执行的查询和加载的数据之间存在逻辑上的不匹配。

There is a logical mismatch between executed queries and the data that is loaded.

形成一个我想要上周进行锻炼的健身房成员名单的场景：

Image a scenario where I want a list of the gym members that did a workout since last week:

var gymMembers = context.GymMembers.Where(g => g.Workouts.Any(w => w.Date >= DateTime.Now.AddDays(-7).Date));

此查询可能会返回仅具有已删除锻炼但也满足谓词的健身会员。一旦它们被装入内存，看起来好像这个健身房的成员根本没有锻炼！
您可以说开发人员应该注意已删除，并始终将其包含在他的查询中，但这是我真正希望避免的事情。也许ExpressionVisitor可以再次提供答案。

This query might return a gym member that only has workouts that are deleted but also satisfy the predicate. Once they are loaded into memory, it appears as if this gym member has no workouts at all! You could say that the developer should be aware of the Deleted and always include it in his queries, but that's something I would really like to avoid. Maybe the ExpressionVisitor could offer the answer here again.

• 实际上不可能将导航属性标记为已删除当使用CustomCollection。



想象这种情况：

var gymMember = context.GymMembers.First();
gymMember.Workouts.First().Deleted = true;
context.SaveChanges();`

你会期望适当的锻炼记录在数据库中更新，你会错的！由于 gymMember 正在由 ChangeTracker 进行检查，以进行任何更改，资源 gymMember.Workouts 将突然返回1次锻炼。那是因为CustomCollection自动过滤删除的实例，记得吗？所以现在Entity Framework认为需要删除训练，EF会尝试将FK设置为​​null，或者实际上删除该记录。 （取决于DB的配置方式）。这是我们试图避免的软删除模式开始!!!

You would expect that the appropriate Workout record is updated in the database, and you would be wrong! Since the gymMember is being inspected by the ChangeTracker for any changes, the property gymMember.Workouts will suddenly return 1 fewer workout. That's because CustomCollection automatically filters deleted instances, remember? So now Entity Framework thinks the workout needs to be deleted, and EF will try to set the FK to null, or actually delete the record. (depending on how your DB is configured). This is what we were trying to avoid with the soft delete pattern to begin with!!!

我偶然发现一个有趣的博客帖子覆盖了默认的 SaveChanges DbContext ，以便将具有 EntityState.Deleted 的所有条目更改回 EntityState.Modified 但这再次感觉黑客，而不是安全。但是，如果解决问题而没有任何意外的副作用，我愿意尝试一下。

I stumbled upon an interesting blog post that overrides the default SaveChanges method of the DbContext so that any entries with an EntityState.Deleted are changed back to EntityState.Modified but this again feels 'hacky' and rather unsafe. However, I'm willing to try it out if it solves problems without any unintended side effects.

所以这里我是StackOverflow。我已经很广泛地研究了我的选择，如果我可以自己说，我在我的智慧结束。所以现在我转向你。您如何在企业应用程序中实施软删除？

So here I am StackOverflow. I've researched my options quite extensively, if I may say so myself, and I'm at my wits end. So now I turn to you. How have you implemented soft deletes in your enterprise application?

要重申，这些是我要查找的要求：

To reiterate, these are the requirements I'm looking for:

• 查询应自动排除数据库级别的已删除实体


• 删除一个实体调用SaveChanges应该只是更新相应的记录，没有其他的副作用。


• 加载导航属性时，无论是懒惰还是渴望，$ code>已删除应该被自动排除。

• Queries should automatically exclude the Deleted entities on the DB level
• Deleting an entity and calling 'SaveChanges' should simply update the appropriate record and have no other side effects.
• When navigational properties are loaded, whether lazy or eager, the Deleted ones should be automatically excluded.

我期待着任何和所有的建议，谢谢你提前。 p>

I am looking forward to any and all suggestions, thank you in advance.

推荐答案

经过多次研究，我终于找到了一种实现我想要的方法。
它的要点是我使用对象上下文中的事件处理程序拦截物化实体，然后在我可以找到的每个集合属性（与反射）中注入我的自定义集合类。

After much research, I've finally found a way to achieve what I wanted. The gist of it is that I intercept materialized entities with an event handler on the object context, and then inject my custom collection class in every collection property that I can find (with reflection).

最重要的部分是截取DbCollectionEntry，负责加载相关集合属性的类。通过在实体和DbCollectionEntry之间摆动自己，我可以充分控制什么时候加载什么和如何。唯一的缺点是这个DbCollectionEntry类几乎没有公共成员，这需要我使用反射来操纵它。

The most important part is intercepting the "DbCollectionEntry", the class responsible for loading related collection properties. By wiggling myself in between the entity and the DbCollectionEntry, I gain full control over what's loaded when and how. The only downside is that this DbCollectionEntry class has little to no public members, which requires me to use reflection to manipulate it.

这是我的自定义集合类，实现ICollection并包含对适当的DbCollectionEntry的引用：

Here is my custom collection class that implements ICollection and contains a reference to the appropriate DbCollectionEntry:

public class FilteredCollection <TEntity> : ICollection<TEntity> where TEntity : Entity
{
    private readonly DbCollectionEntry _dbCollectionEntry;
    private readonly Func<TEntity, Boolean> _compiledFilter;
    private readonly Expression<Func<TEntity, Boolean>> _filter;
    private ICollection<TEntity> _collection;
    private int? _cachedCount;

    public FilteredCollection(ICollection<TEntity> collection, DbCollectionEntry dbCollectionEntry)
    {
        _filter = entity => !entity.Deleted;
        _dbCollectionEntry = dbCollectionEntry;
        _compiledFilter = _filter.Compile();
        _collection = collection != null ? collection.Where(_compiledFilter).ToList() : null;
    }

    private ICollection<TEntity> Entities
    {
        get
        {
            if (_dbCollectionEntry.IsLoaded == false && _collection == null)
            {
                IQueryable<TEntity> query = _dbCollectionEntry.Query().Cast<TEntity>().Where(_filter);
                _dbCollectionEntry.CurrentValue = this;
                _collection = query.ToList();

                object internalCollectionEntry =
                    _dbCollectionEntry.GetType()
                        .GetField("_internalCollectionEntry", BindingFlags.NonPublic | BindingFlags.Instance)
                        .GetValue(_dbCollectionEntry);
                object relatedEnd =
                    internalCollectionEntry.GetType()
                        .BaseType.GetField("_relatedEnd", BindingFlags.NonPublic | BindingFlags.Instance)
                        .GetValue(internalCollectionEntry);
                relatedEnd.GetType()
                    .GetField("_isLoaded", BindingFlags.NonPublic | BindingFlags.Instance)
                    .SetValue(relatedEnd, true);
            }
            return _collection;
        }
    }

    #region ICollection<T> Members

    void ICollection<TEntity>.Add(TEntity item)
    {
        if(_compiledFilter(item))
            Entities.Add(item);
    }

    void ICollection<TEntity>.Clear()
    {
        Entities.Clear();
    }

    Boolean ICollection<TEntity>.Contains(TEntity item)
    {
        return Entities.Contains(item);
    }

    void ICollection<TEntity>.CopyTo(TEntity[] array, Int32 arrayIndex)
    {
        Entities.CopyTo(array, arrayIndex);
    }

    Int32 ICollection<TEntity>.Count
    {
        get
        {
            if (_dbCollectionEntry.IsLoaded)
                return _collection.Count;
            return _dbCollectionEntry.Query().Cast<TEntity>().Count(_filter);
        }
    }

    Boolean ICollection<TEntity>.IsReadOnly
    {
        get
        {
            return Entities.IsReadOnly;
        }
    }

    Boolean ICollection<TEntity>.Remove(TEntity item)
    {
        return Entities.Remove(item);
    }

    #endregion

    #region IEnumerable<T> Members

    IEnumerator<TEntity> IEnumerable<TEntity>.GetEnumerator()
    {
        return Entities.GetEnumerator();
    }

    #endregion

    #region IEnumerable Members

    IEnumerator IEnumerable.GetEnumerator()
    {
        return ( ( this as IEnumerable<TEntity> ).GetEnumerator() );
    }

    #endregion
}

如果您可以浏览它，您会发现最重要的部分是Entities属性，它将延迟加载实际值。在FilteredCollection的构造函数中，我传递了一个可选的ICollection，用于场景中的集合已经被热切的加载。

If you skim through it, you'll find that the most important part is the "Entities" property, which will lazy load the actual values. In the constructor of the FilteredCollection I pass an optional ICollection for scenario's where the collection is already eagerly loaded.

当然，我们仍然需要配置Entity Framework，以便我们的FilteredCollection被用于有收集属性的地方。这可以通过挂接到Entity Framework的基础ObjectContext的ObjectMaterialized事件来实现：

Of course, we still need to configure Entity Framework so that our FilteredCollection is used everywhere where there are collection properties. This can be achieved by hooking into the ObjectMaterialized event of the underlying ObjectContext of Entity Framework:

(this as IObjectContextAdapter).ObjectContext.ObjectMaterialized +=
    delegate(Object sender, ObjectMaterializedEventArgs e)
    {
        if (e.Entity is Entity)
        {
            var entityType = e.Entity.GetType();
            IEnumerable<PropertyInfo> collectionProperties;
            if (!CollectionPropertiesPerType.TryGetValue(entityType, out collectionProperties))
            {
                CollectionPropertiesPerType[entityType] = (collectionProperties = entityType.GetProperties()
                    .Where(p => p.PropertyType.IsGenericType && typeof(ICollection<>) == p.PropertyType.GetGenericTypeDefinition()));
            }
            foreach (var collectionProperty in collectionProperties)
            {
                var collectionType = typeof(FilteredCollection<>).MakeGenericType(collectionProperty.PropertyType.GetGenericArguments());
                DbCollectionEntry dbCollectionEntry = Entry(e.Entity).Collection(collectionProperty.Name);
                dbCollectionEntry.CurrentValue = Activator.CreateInstance(collectionType, new[] { dbCollectionEntry.CurrentValue, dbCollectionEntry });
            }
        }
    };

这一切看起来都比较复杂，但它实际上是扫描物质化类型的集合属性和更改过滤集合的值。它也将DbCollectionEntry传递给过滤的集合，以便它可以起作用。

It all looks rather complicated, but what it does essentially is scan the materialized type for collection properties and change the value to a filtered collection. It also passes the DbCollectionEntry to the filtered collection so it can work its magic.

这涵盖了整个加载实体部分。到目前为止，唯一的缺点是，热切加载的集合属性仍将包括已删除的实体，但它们将在FilterCollection类的添加方法中过滤掉。这是一个可以接受的缺点，虽然我还没有做一些测试，如何影响SaveChanges（）方法。

This covers the whole 'loading entities' part. The only downside so far is that eagerly loaded collection properties will still include the deleted entities, but they are filtered out in the 'Add' method of the FilterCollection class. This is an acceptable downside, although I have yet to do some testing on how this affects the SaveChanges() method.

当然，这还有一个问题：有没有自动过滤查询。如果你想在过去一周里去锻炼身体的健身房，你想自动排除已删除的锻炼。

Of course, this still leaves one issue: there is no automatic filtering on queries. If you want to fetch the gym members who did a workout in the past week, you want to exclude the deleted workouts automatically.

这是通过一个ExpressionVisitor来实现的，该表达式可以为给定表达式中可以找到的每个IQueryable自动应用.Where（e =>！e.Deleted）过滤器。

This is achieved through an ExpressionVisitor that automatically applies a '.Where(e => !e.Deleted)' filter to every IQueryable it can find in a given expression.

这是代码：

public class DeletedFilterInterceptor: ExpressionVisitor
{
    public Expression<Func<Entity, bool>> Filter { get; set; }

    public DeletedFilterInterceptor()
    {
        Filter = entity => !entity.Deleted;
    }

    protected override Expression VisitMember(MemberExpression ex)
    {
        return !ex.Type.IsGenericType ? base.VisitMember(ex) : CreateWhereExpression(Filter, ex) ?? base.VisitMember(ex);
    }

    private Expression CreateWhereExpression(Expression<Func<Entity, bool>> filter, Expression ex)
    {
        var type = ex.Type;//.GetGenericArguments().First();
        var test = CreateExpression(filter, type);
        if (test == null)
            return null;
        var listType = typeof(IQueryable<>).MakeGenericType(type);
        return Expression.Convert(Expression.Call(typeof(Enumerable), "Where", new Type[] { type }, (Expression)ex, test), listType);
    }

    private LambdaExpression CreateExpression(Expression<Func<Entity, bool>> condition, Type type)
    {
        var lambda = (LambdaExpression) condition;
        if (!typeof(Entity).IsAssignableFrom(type))
            return null;

        var newParams = new[] { Expression.Parameter(type, "entity") };
        var paramMap = lambda.Parameters.Select((original, i) => new { original, replacement = newParams[i] }).ToDictionary(p => p.original, p => p.replacement);
        var fixedBody = ParameterRebinder.ReplaceParameters(paramMap, lambda.Body);
        lambda = Expression.Lambda(fixedBody, newParams);

        return lambda;
    }
}

public class ParameterRebinder : ExpressionVisitor
{
    private readonly Dictionary<ParameterExpression, ParameterExpression> _map;

    public ParameterRebinder(Dictionary<ParameterExpression, ParameterExpression> map)
    {
        _map = map ?? new Dictionary<ParameterExpression, ParameterExpression>();
    }

    public static Expression ReplaceParameters(Dictionary<ParameterExpression, ParameterExpression> map, Expression exp)
    {
        return new ParameterRebinder(map).Visit(exp);
    }

    protected override Expression VisitParameter(ParameterExpression node)
    {
        ParameterExpression replacement;

        if (_map.TryGetValue(node, out replacement))
            node = replacement;

        return base.VisitParameter(node);
    }
}

我运行时间有点短，所以我稍后会有更多的细节回到这个帖子，但是它的要点是写下来的，而对于那些渴望尝试一切的人来说，我在此发布了完整的测试应用程序：​​ https://github.com/amoerie/TestingGround

I am running a bit short on time, so I'll get back to this post later with more details, but the gist of it is written down and for those of you eager to try everything out; I've posted the full test application here: https://github.com/amoerie/TestingGround

但是，仍然可能会有一些错误，因为这是一个非常有用的工作。概念性的想法是声音，我希望它可以很快完成功能，一旦我完整地整理了一切，并找到时间来为此做一些测试。

However, there might still be some errors, as this is very much a work in progress. The conceptual idea is sound though, and I expect it to fully function soon once I've refactored everything neatly and find the time to write some tests for this.

这篇关于将所有导航属性加载（懒惰或渴望）放入内存之前过滤的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！