• pandas模块


    pandas

    引入约定

    >>> from pandas import Series,DataFrame
    >>> import pandas as pd

    Series

    类似于一维数组的对象,由一组数据和相关的数据标签(索引)组成

    >>> obj=Series([4,7,-5,3])
    >>> obj
    0    4
    1    7
    2   -5
    3    3
    dtype: int64

    通过values和index属性获取其数组表示形式和索引对象

    >>> obj.values
    array([ 4,  7, -5,  3])
    >>> obj.index
    RangeIndex(start=0, stop=4, step=1)

    对各个数据点进行标记的索引

    >>> obj2=Series([4,7,-5,3],index=['d','b','a','c'])
    >>> obj2
    d    4
    b    7
    a   -5
    c    3
    dtype: int64
    
    >>> obj2.index
    Index([u'd', u'b', u'a', u'c'], dtype='object')

    与普通Numpy数组相比,可以通过索引的方式选取Series中的单个或一组值

    >>> obj2['a']
    -5
    
    >>> obj2[['a','b','c']]
    a   -5
    b    7
    c    3
    dtype: int64

    将Series看成一个定长的有序字典

    >>> 'b' in obj2
    True
    >>> 'e' in obj2
    False

    如果数据被存放在一个python字典中,可以直接通过这个字典创建Series

    >>> sdata={'Ohio':35000,'Texas':71000,'Oregon':16000,'Utah':5000}
    >>> obj3=Series(sdata)
    >>> obj3
    Ohio      35000
    Oregon    16000
    Texas     71000
    Utah       500

    如果只传入一个字典,则结果Series中的索引就是原字典的键

    sdate中跟states索引相匹配,按照传入的states顺序进行排列

    >>> states=['California','Ohio','Oregon','Texas']
    >>> obj4=Series(sdata,index=states)
    >>> obj4
    California        NaN
    Ohio          35000.0
    Oregon        16000.0
    Texas         71000.0
    dtype: float64

    pandas的isnull和notnull函数用于检查缺失数据

    >>> pd.isnull(obj4)
    California     True
    Ohio          False
    Oregon        False
    Texas         False
    dtype: bool

    Series也有类似的实例方法

    >>> obj4.isnull()
    California     True
    Ohio          False
    Oregon        False
    Texas         False
    dtype: bool

    Series最重要的功能是---算术运算中会自动对齐不同索引的数据;数据对齐功能

    >>> obj3+obj4
    California         NaN
    Ohio           70000.0
    Oregon         32000.0
    Texas         142000.0
    Utah               NaN
    dtype: float64

    Series对象本身及其索引都有一个name属性

    >>> obj4.name='population'
    >>> obj4.index.name='state'
    >>> obj4
    state
    California        NaN
    Ohio          35000.0
    Oregon        16000.0
    Texas         71000.0
    Name: population, dtype: float64

    Series的索引可以通过赋值的方式就地修改

    >>> obj.index=['Bob','Steve','Jeff','Ryan']
    >>> obj
    Bob      4
    Steve    7
    Jeff    -5
    Ryan     3
    dtype: int64

    DataFrame

    构建DataFrame,直接传入一个由等长列表或Numpy数组组成的字典

    自动加上索引,且全部列会被有序排列

    >>> data={'state':['Ohio','Ohio','Ohio','Nevada','Nevada'],'year':[2000,2001,2002,2001,2002],'pop':[1.5,1.7,3.6,2.4,2.9]}
    >>> frame=DataFrame(data)
    >>> frame
       pop   state  year
    0  1.5    Ohio  2000
    1  1.7    Ohio  2001
    2  3.6    Ohio  2002
    3  2.4  Nevada  2001
    4  2.9  Nevada  2002

    如果指定了列序列,则DataFrame的列就会按照指定顺序进行排列

    >>> DataFrame(data,columns=['year','state','pop'])
       year   state  pop
    0  2000    Ohio  1.5
    1  2001    Ohio  1.7
    2  2002    Ohio  3.6
    3  2001  Nevada  2.4
    4  2002  Nevada  2.9

    跟Series一样,如果传入的列在数据中找不到,就会产生NA值

    >>> frame2=DataFrame(data,columns=['year','state','pop','debt'],index=['one','two','three','four','five'])
    >>> frame2
           year   state  pop debt
    one    2000    Ohio  1.5  NaN
    two    2001    Ohio  1.7  NaN
    three  2002    Ohio  3.6  NaN
    four   2001  Nevada  2.4  NaN
    five   2002  Nevada  2.9  NaN
    
    >>> frame2.columns
    Index([u'year', u'state', u'pop', u'debt'], dtype='object')

    通过类似字典标记的方式或属性,可以将DataFrame的列获取为一个Series,拥有原DataFrame相同的索引,其name属性已经被相应地设置好

    >>> frame2['state']
    one        Ohio
    two        Ohio
    three      Ohio
    four     Nevada
    five     Nevada
    Name: state, dtype: object
    
    >>> frame2.year
    one      2000
    two      2001
    three    2002
    four     2001
    five     2002
    Name: year, dtype: int64

    行也可以通过位置或名称的方式进行获取,比如用索引字段ix

    >>> frame2.ix['three']
    year     2002
    state    Ohio
    pop       3.6
    debt      NaN
    Name: three, dtype: object

    列可以通过赋值的方式进行修改

    >>> frame2['debt']=16.5
    
    >>> frame2['debt']=np.arange(5.)
    >>> frame2
           year   state  pop  debt
    one    2000    Ohio  1.5   0.0
    two    2001    Ohio  1.7   1.0
    three  2002    Ohio  3.6   2.0
    four   2001  Nevada  2.4   3.0
    five   2002  Nevada  2.9   4.0

    将列表或数组赋值给某个列时,长度必须跟DataFrame的长度相匹配

    如果赋值的是一个Series,就会精确匹配DataFrame的索引,所有的空位都将被填上缺省值

    >>> val=Series([-1.2,-1.5,-1.7],index=['two','four','five'])
    >>> frame2['debt']=val
    >>> frame2
           year   state  pop  debt
    one    2000    Ohio  1.5   NaN
    two    2001    Ohio  1.7  -1.2
    three  2002    Ohio  3.6   NaN
    four   2001  Nevada  2.4  -1.5
    five   2002  Nevada  2.9  -1.7

    为不存在的列赋值会创建出一个新列

    >>> frame2['eastern']=frame2.state == 'Ohio'
    >>> frame2
           year   state  pop  debt eastern
    one    2000    Ohio  1.5   NaN    True
    two    2001    Ohio  1.7  -1.2    True
    three  2002    Ohio  3.6   NaN    True
    four   2001  Nevada  2.4  -1.5   False
    five   2002  Nevada  2.9  -1.7   False

    关键字del用于删除列

    >>> del frame2['eastern']
    >>> frame2
           year   state  pop  debt
    one    2000    Ohio  1.5   NaN
    two    2001    Ohio  1.7  -1.2
    three  2002    Ohio  3.6   NaN
    four   2001  Nevada  2.4  -1.5
    five   2002  Nevada  2.9  -1.7

    通过索引方式返回的列只是相应数据的视图而已,并不是副本。对返回的Series所做的任何就地修改全部会反映到源DataFrame上

    另一个常见的数据形式是嵌套字典(字典的字典)

    外层字典的键作为列,内层键则作为行索引

    >>> pop={'Nevada':{2001:2.4,2002:2.9},
    ...     'Ohio':{2000:1.5,2001:1.7,2002:3.6}}
    >>> frame3=DataFrame(pop)
    >>> frame3
          Nevada  Ohio
    2000     NaN   1.5
    2001     2.4   1.7
    2002     2.9   3.6

    对结果进行转置

    >>> frame3.T
            2000  2001  2002
    Nevada   NaN   2.4   2.9
    Ohio     1.5   1.7   3.6

    内层字典的键会被合并,排序以形成最终的索引

    >>> DataFrame(pop,index=[2001,2002,2003])
          Nevada  Ohio
    2001     2.4   1.7
    2002     2.9   3.6
    2003     NaN   NaN

    设置DataFrame的index和columns的name属性

    >>> frame3.index.name='year';frame3.columns.name='state'
    >>> frame3
    state  Nevada  Ohio
    year
    2000      NaN   1.5
    2001      2.4   1.7
    2002      2.9   3.6

    跟Series一样,values属性也会以二维ndarray的形式返回DataFrame中的数据

    >>> frame3.values
    array([[ nan,  1.5],
           [ 2.4,  1.7],
           [ 2.9,  3.6]])

    如果DataFrame各列的数据类型不同,则值数组的数据类型就会选用兼容所有列的数据的数据类型

    >>> frame2.values
    array([[2000, 'Ohio', 1.5, nan],
           [2001, 'Ohio', 1.7, -1.2],
           [2002, 'Ohio', 3.6, nan],
           [2001, 'Nevada', 2.4, -1.5],
           [2002, 'Nevada', 2.9, -1.7]], dtype=object)

    索引对象

    pandas数据模型的重要组成部分

    负责管理轴标签和其它元数据。构建Series或DataFrame时,所用到的任何数组或其它序列的标签都会被转换成一个Index

    >>> obj=Series(range(3),index=['a','b','c'])
    >>> index=obj.index
    >>> index
    Index([u'a', u'b', u'c'], dtype='object')

    Index对象是不可修改的(immutable),使index对象在多个数据结构之间安全共享

    重新索引

    >>> obj=Series([4.5,7.2,-5.3,3.6],index=['d','b','a','c'])
    >>> obj
    d    4.5
    b    7.2
    a   -5.3
    c    3.6
    dtype: float64

    调用该Series的reindex将会根据新索引进行重排,索引值不存在引入缺失值

    >>> obj2=obj.reindex(['a','b','c','d','e'])
    >>> obj2
    a   -5.3
    b    7.2
    c    3.6
    d    4.5
    e    NaN
    dtype: float64

    设定默认的缺失值

    >>> obj2=obj.reindex(['a','b','c','d','e'],fill_value=0)
    >>> obj2
    a   -5.3
    b    7.2
    c    3.6
    d    4.5
    e    0.0
    dtype: float64

    对于时间序列这样的有序数据,重新索引时可以需要做一些插值处理

    method选项

    >>> obj3=Series(['blue','purple','yellow'],index=[0,2,4])

    reindex的插值method选项

    ffill或pad;前向填充或搬运值

    >>> obj3.reindex(range(6),method='ffill')
    0      blue
    1      blue
    2    purple
    3    purple
    4    yellow
    5    yellow
    dtype: object

    对于DataFrame,reindex可以修改行索引,列,或两个都修改。如果仅传入一个序列,则会重新索引行

    >>> frame=DataFrame(np.arange(9).reshape((3,3)),index=['a','c','d'],columns=['Ohio','Texas','California'])
    >>> frame
       Ohio  Texas  California
    a     0      1           2
    c     3      4           5
    d     6      7           8
    
    
    >>> frame2=frame.reindex(['a','b','c','d'])
    >>> frame2
       Ohio  Texas  California
    a   0.0    1.0         2.0
    b   NaN    NaN         NaN
    c   3.0    4.0         5.0
    d   6.0    7.0         8.0

    使用columns关键字即可重新索引列

    >>> states=['Texas','Utah','California']
    >>> frame.reindex(columns=states)
       Texas  Utah  California
    a      1   NaN           2
    c      4   NaN           5
    d      7   NaN           8

    可以同时对行和列进行重新索引,而插值则只能按行应用

    >>> frame.reindex(index=['a','b','c','d'],method='ffill',columns=states)
       Texas  Utah  California
    a      1   NaN           2
    b      1   NaN           2
    c      4   NaN           5
    d      7   NaN           8

    利用ix的标签索引功能,重新索引任务可以变得更简洁

    >>> frame.ix[['a','b','c','d'],states]
       Texas  Utah  California
    a    1.0   NaN         2.0
    b    NaN   NaN         NaN
    c    4.0   NaN         5.0
    d    7.0   NaN         8.0

    reindex函数的参数

    index       索引的新序列
    method      插值填充方式
    fill_value  在重新索引的过程中,需要引入缺失值时使用的替代值
    limit       前向或后向填充时的最大填充量
    level       在MultiIndex的指定级别上匹配简单索引,否则选取其子集
    copy        默认为true,无论如何都复制

    丢弃指定轴上的项

    >>> obj=Series(np.arange(5.),index=['a','b','c','d','e'])
    >>> obj
    a    0.0
    b    1.0
    c    2.0
    d    3.0
    e    4.0
    dtype: float64
    
    >>> new_obj=obj.drop('c')
    >>> new_obj
    a    0.0
    b    1.0
    d    3.0
    e    4.0
    dtype: float64

    对于DataFrame,可以删除任意轴上的索引值

    >>> data=DataFrame(np.arange(16).reshape((4,4)),index=['Ohio','Colorado','Utah','New York'],columns=['one','two','three','four'])
    >>> data
              one  two  three  four
    Ohio        0    1      2     3
    Colorado    4    5      6     7
    Utah        8    9     10    11
    New York   12   13     14    15 
    
    
    >>> data.drop(['Colorado','Ohio'])
              one  two  three  four
    Utah        8    9     10    11
    New York   12   13     14    15
    
    
    >>> data.drop('two',axis=1)
              one  three  four
    Ohio        0      2     3
    Colorado    4      6     7
    Utah        8     10    11
    New York   12     14    15
    
    >>> data.drop(['two','four'],axis=1)
              one  three
    Ohio        0      2
    Colorado    4      6
    Utah        8     10
    New York   12     14

    索引,选取和过滤

    类似于Numpy数组的索引,只不过Series的索引值不只是整数

    Series利用标签的切片运算与普通的python切片运算不同,其末端是包含的

    >>> data=DataFrame(np.arange(16).reshape((4,4)),index=['Ohio','Colorado','Utah','New York'],columns=['one','two','three','four'])
    >>> data
              one  two  three  four
    Ohio        0    1      2     3
    Colorado    4    5      6     7
    Utah        8    9     10    11
    New York   12   13     14    15

    DataFrame的切片

    >>> data[:2]
              one  two  three  four
    Ohio        0    1      2     3
    Colorado    4    5      6     7
    
    
    
    >>> data<5
                one    two  three   four
    Ohio       True   True   True   True
    Colorado   True  False  False  False
    Utah      False  False  False  False
    New York  False  False  False  False
    
    >>> data[data<5]=0
    >>> data
              one  two  three  four
    Ohio        0    0      0     0
    Colorado    0    5      6     7
    Utah        8    9     10    11
    New York   12   13     14    15
    
    >>> data.ix['Colorado',['two','three']]
    two      5
    three    6
    Name: Colorado, dtype: int64
    
    >>> data.ix[['Colorado','Utah'],[3,0,1]]
              four  one  two
    Colorado     7    0    5
    Utah        11    8    9
    
    >>> data.ix[2]
    one       8
    two       9
    three    10
    four     11
    Name: Utah, dtype: int64
    
    >>> data.ix[:'Utah','two']
    Ohio        0
    Colorado    5
    Utah        9
    Name: two, dtype: int64

    算术对齐和数据对齐

    自动的数据对齐操作在不重叠的索引处引入NA值

    对于DataFrame,对齐操作会同时发生在行和列上

    使用add方法,传入加数以及一个fill_value参数:obj.add(obj2,fill_value=0)

    DataFrame和Series之间的运算

    >>> arr=np.arange(12.).reshape((3,4))
    >>> arr
    array([[  0.,   1.,   2.,   3.],
           [  4.,   5.,   6.,   7.],
           [  8.,   9.,  10.,  11.]])
    >>> arr[0]
    array([ 0.,  1.,  2.,  3.])
    >>> arr-arr[0]
    array([[ 0.,  0.,  0.,  0.],
           [ 4.,  4.,  4.,  4.],
           [ 8.,  8.,  8.,  8.]])

    这叫做广播(broadcasting)

    >>> frame=DataFrame(np.arange(12.).reshape((4,3)),columns=list('bde'),index=['Utah','Ohio','Texas','Oregon'])
    >>> series=frame.ix[0]
    >>> frame
              b     d     e
    Utah    0.0   1.0   2.0
    Ohio    3.0   4.0   5.0
    Texas   6.0   7.0   8.0
    Oregon  9.0  10.0  11.0
    >>> series
    b    0.0
    d    1.0
    e    2.0
    Name: Utah, dtype: float64

    默认情况下,DataFrame和Series之间的算术运算会将Series的索引匹配到DataFrame的行,然后沿着行一直向下广播

    >>> frame-series
              b    d    e
    Utah    0.0  0.0  0.0
    Ohio    3.0  3.0  3.0
    Texas   6.0  6.0  6.0
    Oregon  9.0  9.0  9.0

    如果某个索引值在DataFrame的列或Series的索引中找不到,则参与运算的两个对象被重新索引以形成并集

    >>> series2=Series(range(3),index=['b','e','f'])
    >>> series2
    b    0
    e    1
    f    2
    dtype: int64
    
    >>> frame+series2
              b   d     e   f
    Utah    0.0 NaN   3.0 NaN
    Ohio    3.0 NaN   6.0 NaN
    Texas   6.0 NaN   9.0 NaN
    Oregon  9.0 NaN  12.0 NaN

    匹配行且在列上广播,则必须使用算术运算方法

    >>> frame.sub(series3,axis=0)
              b    d    e
    Utah   -1.0  0.0  1.0
    Ohio   -1.0  0.0  1.0
    Texas  -1.0  0.0  1.0
    Oregon -1.0  0.0  1.0

    函数应用和映射

    许多最为常见的数组统计功能都被实现成DataFrame的方法

    >>> frame=DataFrame(np.random.randn(4,3),columns=list('bde'),index=['Utah','Ohio','Texas','Oregon'])
    >>> frame
                   b         d         e
    Utah   -1.120701 -0.772813 -1.183221
    Ohio   -0.690566  0.610834  0.382371
    Texas   0.287303 -0.001705 -1.055101
    Oregon  1.149945  1.056177 -0.178909
    
    >>> deff(x):
    ...     return Series([x.min(),x.max()],index=['min','max'])
    ... 
    >>> frame.apply(f)
                b         d         e
    min -1.120701 -0.772813 -1.183221
    max  1.149945  1.056177  0.382371

    frame中各个浮点值的格式化字符串

    >>> format=lambda x:'%.2f' % x
    >>> frame.applymap(format)
                b      d      e
    Utah    -1.12  -0.77  -1.18
    Ohio    -0.69   0.61   0.38
    Texas    0.29  -0.00  -1.06
    Oregon   1.15   1.06  -0.18

    Series有一个用于元素级函数的map方法

    >>> frame['e'].map(format)
    Utah      -1.18
    Ohio       0.38
    Texas     -1.06
    Oregon    -0.18
    Name: e, dtype: object

    排序和排名

    >>> obj=Series(range(4),index=['d','a','b','c'])
    >>> obj.sort_index()
    a    1
    b    2
    c    3
    d    0
    dtype: int64

    对于DataFrame,则可以根据任意一个轴上的索引进行排序

    数据默认是按升序排列的,但也可以降序排列

    >>> frame=DataFrame(np.arange(8).reshape((2,4)),index=['three','one'],columns=['d','a','b','c'])
    >>> frame.sort_index(axis=1)
           a  b  c  d
    three  1  2  3  0
    one    5  6  7  4
    >>> frame.sort_index(axis=1,ascending=False)
           d  c  b  a
    three  0  3  2  1
    one    4  7  6  5

    按值对Series进行排序,可使用其order方法

    >>> obj=Series([4,7,-3,2])
    >>> obj.order()
    
    >>> obj.sort_values()
    2   -3
    3    2
    0    4
    1    7
    dtype: int64

    在排序时,任何缺失值默认都会被放到Series的末尾

    >>> frame=DataFrame({'b':[4,7,-3,2],'a':[0,1,0,1]})
    >>> frame
       a  b
    0  0  4
    1  1  7
    2  0 -3
    3  1  2
    
    >>> frame.sort_values(by='b')
       a  b
    2  0 -3
    3  1  2
    0  0  4
    1  1  7

    排名

    根据某种规则破坏平级关系

    >>> obj=Series([7,-5,7,4,2,0,4])
    >>> obj
    0    7
    1   -5
    2    7
    3    4
    4    2
    5    0
    6    4
    dtype: int64
    >>> obj.rank()
    0    6.5
    1    1.0
    2    6.5
    3    4.5
    4    3.0
    5    2.0
    6    4.5
    dtype: float64

    排名时用于破坏平级关系的method选项

    average 默认,在相等分组中,为各个值分配平均排名
    min     使用整个分组的最小排名
    max     使用整个分组的最大排名
    first   按值在原始数据中的出现顺序分配排名

    带有重复值的轴索引

    许多pandas函数(eg:reindex)都要求标签唯一,但并不是强制性

    索引的is_unique属性

    >>> obj.index.is_unique
    False

    某个索引对应多个值,则返回一个Series

    >>> obj['a']
    a    0
    a    1
    dtype: int64

    对应单值,返回一个标量值

    >>> obj['c']
    4

    汇总和计算描述统计

    sum求和,传入axis=1将会按行进行求和运算

    NA值会自动被排除,除非整个切片(行或列)都是NA

    通过skipna选项可以禁用该功能,df.mean(axis=1,skipna=False)

    describe,用于一次性产生多个汇总统计

    相关系数与协方差

    唯一值,值计数以及成员资格

    可以从一维Series的值中抽取信息

    >>> obj=Series(['c','a','d','a','a','b','b','c','c'])
    >>> uniques=obj.unique()
    >>> uniques
    array(['c', 'a', 'd', 'b'], dtype=object)

    计算一个Series中各值出现的频率

    >>> obj.value_counts()
    c    3
    a    3
    b    2
    d    1
    dtype: int64

    矢量化集合的成员资格,可用于选取Series中或DataFrame列中数据的子集

    >>> mask=obj.isin(['b','c'])
    >>> mask
    0     True
    1    False
    2    False
    3    False
    4    False
    5     True
    6     True
    7     True
    8     True
    dtype: bool
    
    >>> obj[mask]
    0    c
    5    b
    6    b
    7    c
    8    c
    dtype: object

    处理缺失数据

    missing data在大部分数据分析应用中都很常见,pandas的设计目标是让缺失数据的处理任务尽量轻松

    python内置的None值也会被当做NA处理

    滤除缺失数据

    >>> from numpy import nan as NA
    >>> data=Series([1,NA,3.5,NA,7])
    >>> data
    0    1.0
    1    NaN
    2    3.5
    3    NaN
    4    7.0
    dtype: float64
    >>> data.dropna()
    0    1.0
    2    3.5
    4    7.0
    dtype: float64

    通过布尔型索引

    >>> data[data.notnull()]
    0    1.0
    2    3.5
    4    7.0
    dtype: float64

    对于DataFrame,dropna默认丢弃任何含有缺失值的行

    传入how='all'将会丢弃全为NA的那些行

    >>> data=DataFrame([[1.,6.5,3.],[1.,NA,NA],[NA,NA,NA],[NA,6.5,3.]])
    >>> data
         0    1    2
    0  1.0  6.5  3.0
    1  1.0  NaN  NaN
    2  NaN  NaN  NaN
    3  NaN  6.5  3.0
    >>> data.dropna(how='all')
         0    1    2
    0  1.0  6.5  3.0
    1  1.0  NaN  NaN
    3  NaN  6.5  3.0
    
    
    >>> data[4]=NA
    >>> data
         0    1    2   4
    0  1.0  6.5  3.0 NaN
    1  1.0  NaN  NaN NaN
    2  NaN  NaN  NaN NaN
    3  NaN  6.5  3.0 NaN
    
    >>> data.dropna(axis=1,how='all')
         0    1    2
    0  1.0  6.5  3.0
    1  1.0  NaN  NaN
    2  NaN  NaN  NaN
    3  NaN  6.5  3.0

    时间序列数据,只想留下一部分观测数据

    >>> df=DataFrame(np.random.randn(7,3))
    >>> df
              0         1         2
    0  1.367974 -0.556556  0.679336
    1 -0.480919 -1.535185 -0.299710
    2  0.230583  0.140626  0.604209
    3  0.437830 -0.467286 -0.859989
    4 -0.254706 -0.227431 -0.956299
    5  0.966204 -2.010860 -0.010693
    6 -0.673721  1.497827 -0.257273
    
    >>> df.ix[:4,1]=NA
    >>> df.ix[:2,2]
    
    >>> df
              0         1         2
    0  1.367974       NaN       NaN
    1 -0.480919       NaN       NaN
    2  0.230583       NaN       NaN
    3  0.437830       NaN -0.859989
    4 -0.254706       NaN -0.956299
    5  0.966204 -2.010860 -0.010693
    6 -0.673721  1.497827 -0.257273

    一行中至少有3个非NA值将其保留

    >>> df.dropna(thresh=3)
              0         1         2
    5  0.966204 -2.010860 -0.010693
    6 -0.673721  1.497827 -0.257273

    填充缺失数据

    fillna方法是最主要的函数

    >>> df.fillna(0)
              0         1         2
    0  1.367974  0.000000  0.000000
    1 -0.480919  0.000000  0.000000
    2  0.230583  0.000000  0.000000
    3  0.437830  0.000000 -0.859989
    4 -0.254706  0.000000 -0.956299
    5  0.966204 -2.010860 -0.010693
    6 -0.673721  1.497827 -0.257273

    一个字典调用fillna,就可以实现对不同列填充不同的值

    >>> df.fillna({1:0.5,3:-1})
              0         1         2
    0  1.367974  0.500000       NaN
    1 -0.480919  0.500000       NaN
    2  0.230583  0.500000       NaN
    3  0.437830  0.500000 -0.859989
    4 -0.254706  0.500000 -0.956299
    5  0.966204 -2.010860 -0.010693
    6 -0.673721  1.497827 -0.257273

    fillna默认会返回新对象,但也可以对现有对象进行就地修改

    返回被填充对象的引用

    >>> _=df.fillna(0,inplace=True)
    >>> df
              0         1         2
    0  1.367974  0.000000  0.000000
    1 -0.480919  0.000000  0.000000
    2  0.230583  0.000000  0.000000
    3  0.437830  0.000000 -0.859989
    4 -0.254706  0.000000 -0.956299
    5  0.966204 -2.010860 -0.010693
    6 -0.673721  1.497827 -0.257273

    对reindex有效的那些插值方法也可以用fillna

    >>> df=DataFrame(np.random.randn(6,3))
    >>> df.ix[2:,1]=NA;df.ix[4:,2]=NA
    >>> df
              0         1         2
    0  0.647866  0.891312 -0.211922
    1 -1.455856 -0.629213 -1.043685
    2  2.078467       NaN -0.067846
    3 -0.223047       NaN  0.513800
    4  0.306559       NaN       NaN
    5  0.404265       NaN       NaN

    填充最靠近行的数值填充,列行为

    >>> df.fillna(method='ffill')
              0         1         2
    0  0.647866  0.891312 -0.211922
    1 -1.455856 -0.629213 -1.043685
    2  2.078467 -0.629213 -0.067846
    3 -0.223047 -0.629213  0.513800
    4  0.306559 -0.629213  0.513800
    5  0.404265 -0.629213  0.513800
    
    >>> df.fillna(method='ffill',limit=2)
              0         1         2
    0  0.647866  0.891312 -0.211922
    1 -1.455856 -0.629213 -1.043685
    2  2.078467 -0.629213 -0.067846
    3 -0.223047 -0.629213  0.513800
    4  0.306559       NaN  0.513800
    5  0.404265       NaN  0.513800

    层次化索引

    hierachical indexing

    一个轴上拥有多个(两个以上)索引级别

    >>> data=Series(np.random.randn(10),index=[['a','a','a','b','b','b','c','c','d','d'],[1,2,3,1,2,3,1,2,2,3]])
    >>> data
    a  1   -0.521370
       2    0.658209
       3    0.841101
    b  1    0.354237
       2   -0.426983
       3    0.835357
    c  1   -0.246308
       2    0.709859
    d  2   -1.215098
       3    0.400793
    dtype: float64

    这就是带有MultiIndex索引的Series的格式化输出形式

    >>> data.index
    MultiIndex(levels=[[u'a', u'b', u'c', u'd'], [1, 2, 3]],
               labels=[[0, 0, 0, 1, 1, 1, 2, 2, 3, 3], [0, 1, 2, 0, 1, 2, 0, 1, 1, 2]])
    
    
    >>> data['b']
    1    0.354237
    2   -0.426983
    3    0.835357
    dtype: float64

    内层进行选取

    >>> data[:,2]
    a    0.658209
    b   -0.426983
    c    0.709859
    d   -1.215098
    dtype: float64

    层次化索引在数据重塑和基于分组的操作(如透视表生成)扮演重要的角色

    >>> data.unstack()
              1         2         3
    a -0.521370  0.658209  0.841101
    b  0.354237 -0.426983  0.835357
    c -0.246308  0.709859       NaN
    d       NaN -1.215098  0.400793

    unstack的逆运算是stack

    对于DataFrame,每条轴都可以有分层索引

    >>> frame=DataFrame(np.arange(12).reshape((4,3)),index=[['a','a','b','b'],[1,2,1,2]],columns=[['Ohio','Ohio','Colorado'],['Green','Red','Green']])
    >>> frame
         Ohio     Colorado
        Green Red    Green
    a 1     0   1        2
      2     3   4        5
    b 1     6   7        8
      2     9  10       11

    各层都可以有名字(可以是字符串,也可以是别的python对象)

    >>> frame.index.names=['key1','key2']
    >>> frame
               Ohio     Colorado
              Green Red    Green
    key1 key2 
    a    1        0   1        2
         2        3   4        5
    b    1        6   7        8
         2        9  10       11
    
    
    >>> frame.columns.names=['state','color']
    >>> frame
    state      Ohio     Colorado
    color     Green Red    Green
    key1 key2
    a    1        0   1        2
         2        3   4        5
    b    1        6   7        8
         2        9  10       11

    有了分部的列索引,可以轻松选取列分组

    可以单独创建MultiIndex然后复用

    >>> MultiIndex.from_arrays([['Ohio','Ohio','Colorado'],['Green','Red','Green']],names=['state','color'])

    重排分级顺序

    swaplevel接受两个级别编号或名称,并返回一个互换了级别的新对象

    >>> frame.swaplevel('key1','key2')
    state      Ohio     Colorado
    color     Green Red    Green
    key2 key1
    1    a        0   1        2
    2    a        3   4        5
    1    b        6   7        8
    2    b        9  10       11

    sortlevel则根据单个级别中的值对数据进行排序

    >>> frame
    state      Ohio     Colorado
    color     Green Red    Green
    key1 key2 
    a    1        0   1        2
         2        3   4        5
    b    1        6   7        8
         2        9  10       11
    
    >>> frame.sortlevel(1)
    state      Ohio     Colorado
    color     Green Red    Green
    key1 key2
    a    1        0   1        2
    b    1        6   7        8
    a    2        3   4        5
    b    2        9  10       11
    
    >>> frame.swaplevel(0,1).sortlevel(0)
    state      Ohio     Colorado
    color     Green Red    Green
    key2 key1
    1    a        0   1        2
         b        6   7        8
    2    a        3   4        5
         b        9  10       11

    在层次化索引的对象上,如果索引是按字典方式从外向内排序,即调用sortlevel(0)或sort_index()的结果,数据选取操作的性能要好的多

    根据级别汇总统计

    许多对DataFrame和Series的描述和汇总统计都有一个level选项,用于指定在某条轴上求和的级别

    >>> frame.sum(level='key2')
    state  Ohio     Colorado
    color Green Red    Green
    key2
    1         6   8       10
    2        12  14       16
    
    >>> frame.sum(level='color',axis=1)
    color      Green  Red
    key1 key2
    a    1         2    1
         2         8    4
    b    1        14    7
         2        20   10

    使用DataFrame的列

    想要将DataFrame的一个或多个列当做行索引来用,或者将行索引当成DataFrame的列

    >>> frame=DataFrame({'a':range(7),'b':range(7,0,-1),'c':['one','one','one','two','two','two','two'],'d':[0,1,2,0,1,2,3]})
    >>> frame
       a  b    c  d
    0  0  7  one  0
    1  1  6  one  1
    2  2  5  one  2
    3  3  4  two  0
    4  4  3  two  1
    5  5  2  two  2
    6  6  1  two  3

    set_index()函数会将其一个或多个列转换为行索引,并创建一个新的DataFrame

    >>> frame2=frame.set_index(['c','d'])
    >>> frame2
           a  b
    c   d
    one 0  0  7
        1  1  6
        2  2  5
    two 0  3  4
        1  4  3
        2  5  2
        3  6  1

    默认情况下,那些列会从DataFrame中移除,但也可以将其保留下来

    >>> frame.set_index(['c','d'],drop=False)
           a  b    c  d
    c   d
    one 0  0  7  one  0
        1  1  6  one  1
        2  2  5  one  2
    two 0  3  4  two  0
        1  4  3  two  1
        2  5  2  two  2
        3  6  1  two  3

    reset_index的功能相反,层次化索引的级别会被转移到列里面

    >>> frame2.reset_index()
         c  d  a  b
    0  one  0  0  7
    1  one  1  1  6
    2  one  2  2  5
    3  two  0  3  4
    4  two  1  4  3
    5  two  2  5  2
    6  two  3  6  1
  • 相关阅读:
    容斥相关
    动态点分治
    杜教筛
    上下界网络流的理解
    putty securecrt
    借款费用
    在线画图工具
    费用包括(营业成本)和(期间费用)
    信用减值损失
    采用权益法核算的长期股权投资,被投资单位宣告发放现金股利
  • 原文地址:https://www.cnblogs.com/bubu99/p/12215941.html
Copyright © 2020-2023  润新知