• spark-sklearn(spark扩展scikitlearn)


    (1)官方规定安装条件:此包装具有以下要求:

    -*最新版本的scikit学习。 版本0.17已经过测试,旧版本也可以使用。
    - *Spark> = 2.0。 Spark可以从对应官网下载
    [Spark官方网站](http://spark.apache.org/)

    -*为了使用spark-sklearn,您需要使用pyspark解释器或其他Spark兼容的python解释器。

    有关详细信息,请参阅[Spark指南](https://spark.apache.org/docs/latest/programming-guide.html#overview)。
    - (https://nose.readthedocs.org)(仅测试依赖关系)

    英文原文:This package has the following requirements:
    - a recent version of scikit-learn. Version 0.17 has been tested, older versions may work too.
    - Spark >= 2.0. Spark may be downloaded from the
    [Spark official website](http://spark.apache.org/) In order to use spark-sklearn, you need to use the pyspark interpreter or another Spark-compliant python interpreter. See the [Spark guide](https://spark.apache.org/docs/latest/programming-guide.html#overview) for more details.
    - [nose](https://nose.readthedocs.org) (testing dependency only)

    (2)首先安装pyspark:

    参考为的博客:http://www.cnblogs.com/jackchen-Net/p/6667205.html#_label5

    (3)访问网址:https://pypi.python.org/pypi/spark-sklearn

    目前Spark集成了Scikit-learn包,这样可以极大的简化了python数据科学家们的工作,这个包可以在Spark集群上自动分配模型参数优化计算任务

     (4)官方文档的例子测试

     1 ## Example
     2 
     3 Here is a simple example that runs a grid search with Spark. See the [Installation](#Installation) section on how to install spark-sklearn.
     4 
     5 ```python
     6 from sklearn import svm, grid_search, datasets
     7 from spark_sklearn import GridSearchCV
     8 iris = datasets.load_iris()
     9 parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
    10 svr = svm.SVC()
    11 clf = GridSearchCV(sc, svr, parameters)
    12 clf.fit(iris.data, iris.target)
    13 ```
    14 
    15 This classifier can be used as a drop-in replacement for any scikit-learn classifier, with the same API.

    END~

  • 相关阅读:
    Python自动化之面向对象进阶
    Python自动化之pickle和面向对象初级篇
    Python自动化之常用模块
    剑指offer 顺时针打印矩阵
    剑指 offer 树的子结构
    剑指offer 合并两个排序的链表
    剑指offer 调整数组顺序使奇数位于偶数前面
    剑指offer 数值的整数次方
    变态跳台阶
    剑指offer 重建二叉树
  • 原文地址:https://www.cnblogs.com/jackchen-Net/p/7297555.html
Copyright © 2020-2023  润新知