hive中标准偏差函数stddev()详细讲解

1.标准偏差概念

标准偏差(Std Dev,Standard Deviation) -统计学名词。一种度量数据分布的分散程度之标准，用以衡量数据值偏离算术平均值的程度。标准偏差越小，这些值偏离平均值就越少，反之亦然。标准偏差的大小可通过标准偏差与平均值的倍率关系来衡量。

例如，A、B两组各有6位学生参加同一次语文测验，A组的分数为95、85、75、65、55、45，B组的分数为73、72、71、69、68、67。这两组的平均数都是70，但A组的标准差应该是17.078分,B组的标准差应该是2.160分，说明A组学生之间的差距要比B组学生之间的差距大得多。

标准偏差又分为总体标准偏差与样本标准偏差

总体标准偏差：针对总体数据的偏差，所以要平均，

 

样本标准偏差，也称实验标准偏差：针对从总体抽样，利用样本来计算总体偏差，为了使算出的值与总体水平更接近，就必须将算出的标准偏差的值适度放大，即，

 

 

2.标准偏差计算公式：

 

样本标准偏差

  

，

  

代表所采用的样本X1,X2,...,Xn的均值。

总体标准偏差

  

，

  

代表总体X的均值。

例：有一组数字分别是200、50、100、200，求它们的样本标准偏差。

 

= (200+50+100+200)/4 = 550/4 = 137.5

 

= [(200-137.5)^2+(50-137.5)^2+(100-137.5)^2+(200-137.5)^2]/(4-1)

样本标准偏差 S = Sqrt(S^2)=75， 注：八年级（下册）上海科学技术出版 21.2数据的离散程度中的标准差是总体标准差

 

3.hive中的标准偏差函数 stddev_pop(),stddev_samp(),stddev()

stddev_pop()  总体标准方差，stddev_samp() 样本标准方差

 

(1) hive引擎计算标准偏差

select col, stddev_pop(num),stddev_samp(num),stddev(num) as stddev_col
from (
    select 'A' as col, '1' as num
    union all
    select 'A' as col, '2' as num
    union all
    select 'A' as col, '3' as num
    union all
    select 'B' as col, '1' as num
    union all
    select 'B' as col, '2' as num
) as a
group by col
;

查询结果：

 （2）spark引擎查询标准偏差

select col, stddev_pop(num),stddev_samp(num),stddev(num) as stddev_col
from (
    select 'A' as col, '1' as num
    union all
    select 'A' as col, '2' as num
    union all
    select 'A' as col, '3' as num
    union all
    select 'B' as col, '1' as num
    union all
    select 'B' as col, '2' as num
) as a
group by col

查询结果

由上可看出，hive中stddev()函数默认计算总体标准偏差，spark 中stddev()函数默认计算样本标准偏差

 

4.stddev()也可用于窗口函数

select col, stddev(num) over(partition by col) as stddev_col
from (
    select 'A' as col, '1' as num
    union all
    select 'A' as col, '2' as num
    union all
    select 'A' as col, '3' as num
    union all
    select 'B' as col, '1' as num
    union all
    select 'B' as col, '2' as num
) as a

查询结果：

5. 当计算的输入数据只有一行时 ，hive和spark计算标准方差的结果

（1）hive

select col, stddev_pop(num),stddev_samp(num),stddev(num) as stddev_col
from (
    select 'A' as col, '1' as num
    union all
    select 'B' as col, '2' as num
) as a
group by col
;

  查询结果：

（2）spark

select col, stddev_pop(num),stddev_samp(num),stddev(num) as stddev_col
from (
    select 'A' as col, '1' as num
    union all
    select 'B' as col, '2' as num
) as a
group by col
;

 查询结果：