星火一起使用Spring [英] Use Spring together with Spark

问题描述

我正在开发一个Spark应用程序，我已经习惯了Spring作为一个依赖注入框架。现在，我坚持了这个问题，该处理部分使用了Spring的@Autowired功能，但它是序列化和反序列化星火

I'm developing a Spark Application and I'm used to Spring as a Dependency Injection Framework. Now I'm stuck with the problem, that the processing part uses the @Autowired functionality of Spring, but it is serialized and deserialized by Spark.

所以下面code使我陷入麻烦：

So the following code gets me into trouble:

Processor processor = ...; // This is a Spring constructed object
                           // and makes all the trouble
JavaRDD<Txn> rdd = ...; // some data for Spark
rdd.foreachPartition(processor);

处理器看起来像：

The Processor looks like that:

public class Processor implements VoidFunction<Iterator<Txn>>, Serializeable {
    private static final long serialVersionUID = 1L;

    @Autowired // This will not work if the object is deserialized
    private transient DatabaseConnection db;

    @Override
    public void call(Iterator<Txn> txns) {
        ... // do some fance stuff
        db.store(txns);
    }
}

所以我的问题是：它甚至有可能使用类似弹簧结合星火？如果没有，什么是最优雅的方式做这样的事情？任何帮助AP preciated！

So my question is: Is it even possible to use something like Spring in combination with Spark? If not, what is the most elegant way to do something like that? Any help is appreciated!

推荐答案

的补充：的要直接干预反序列化的部分，而无需修改自己的类使用以下的弹簧星火项目。该项目autowires当它被弹簧反序列化你的bean。

Added: To interfere the deserialization part directly without modifying your own classes use the following spring-spark project. This projects autowires your bean when it gets deserialized by spring.

另一个可能的黑客攻击将迫使春注入的情况下，你的分贝数据库连接它的无效。

Another possible hack would be to force Spring to inject your db database connection in case it's null.

@Autowired // This will not work if the object is deserialized
private transient DatabaseConnection db;

@Override
public void call(Iterator<Txn> txns) {
    ... // do some fance stuff
    getDb().store(txns);
}

private DatabaseConnection getDb() {
    if(db == null) {
        WebApplicationContext webApplicationContext = WebApplicationContextUtils.getWebApplicationContext(MySpringUtils.getServletContext());
        AutowireCapableBeanFactory autowireCapableBeanFactory = webApplicationContext.getAutowireCapableBeanFactory();
        autowireCapableBeanFactory.configureBean(this, "databaseConnection");
    }
    return db;
}

和则需要

@Component
public class MySpringUtils {
     @Autowired
     private ServletContext servletContext;

     private static ServletContext staticContext;

     @PostConstruct
     public void init() {
         this.staticContext = servletContext;
     }

     public static ServletContext getServletContext() {
         return staticContext;
     }
}

但是，这听起来像一个黑客，因为分贝似乎被弹簧非托管，如这不是一个数据源 - 这似乎有些奇怪。因此，这可能不是最佳的解决方案。

But this sounds like a hack, because db seems unmanaged by Spring, as in that is not a DataSource - which seems somewhat odd. Therefore, this might not be the best possible solution.

这篇关于星火一起使用Spring的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！