结合使用带有火花的DEAP(遗传算法库) [英] Using DEAP (genetic algorithm library) with spark

问题描述

是否可以使用DEAP( http://deap.readthedocs.io/en/master /)与火花簇来映射适应性评估功能.我想运行GA，但是适应度函数相当长，我正计划将其分布在Spark集群上.

Is IT possible to use DEAP ( http://deap.readthedocs.io/en/master/) with a spark cluster to map the fitness evaluation function. I would like to run a GA but the fitness function is rather long and I was planning on distributing it on a spark cluster.

推荐答案

您应该查看使用DEAP文档中的多个处理器部分，并且位于示例.他们介绍了如何替换DEAP ma​​p 函数"rel ="nofollow noreferrer">工具箱，由您选择的地图功能完成.

You should look at the Using Multiple Processors section in the DEAP documentation and at this example. They explain how to replace the map function in the DEAP toolbox by a map function of your choice.

要使用pyspark映射适应性评估功能，您可以执行以下操作:

To use pyspark to map the fitness evaluation function, you could do something like that:

from pyspark import SparkContext

sc = SparkContext(appName="DEAP")

def sparkMap(algorithm, population):
    return sc.parallelize(population).map(algorithm)

toolbox.register("map", sparkMap)

这篇关于结合使用带有火花的DEAP(遗传算法库)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！