基于Python的大数据的分页模型代码

最近在写一个cmdb系统的分页，尽管Django本身有分页的模块儿，但是还是想实现一下自己心中想的分页的一种逻辑

因为，在我们工作当中，当我们的数据量超级大的时候，其实我们每次分页查询都不必将所有的数据查询出来，而是可以按阶段的查询，举个例子

每次查询5页数据，当需要第六页的时候，再次进行加载,为了更加明了，做了举例

首先我们规定好每次查询5页，每页4条数据　　

db_data = list(range(1,100))

final_page = len(db_data)

one_page_num = 5 #每页4条
one_time_num = 5 #每次查5页

1、查询索引的逻辑 设定findIndex查询的索引

    # page = 1  int(1/5) + 1  findIndex = 1
    # page = 2  int(2/5) + 1  findIndex = 1
    # page = 3  int(3/5) + 1  findIndex = 1
    # page = 4  int(4/5) + 1  findIndex = 1
    # page = 5  int(5/5)  findIndex = 1
    # page = 6  int(6/5) + 1  findIndex = 2
    # page = 7  int(7/5) + 1  findIndex = 2
    # page = 8  int(8/5) + 1  findIndex = 2
    # page = 9  int(9/5) + 1  findIndex = 2
    # page = 10  int(10/5)  findIndex = 2
    # page = 11  int(11/5) + 1  findIndex = 3
    # page = 12  int(12/5) + 1  findIndex = 3
    # page = 13  int(13/5) + 1  findIndex = 3
    # page = 14  int(14/5) + 1  findIndex = 3
    # page = 15  int(15/5)  findIndex = 3

所以我们可以写如下代码

if page/one_time_num > int(page/one_time_num):
    findIndex = int(page / one_time_num) + 1
else:
    findIndex = int(page / one_time_num)
#进行第一次查询，查询每次5页，每页4条的数据
select_num = one_time_num * one_page_num #这里决一次定查询20条
select_start = (findIndex - 1) * select_num  # 开始查询的索引
select_down =  findIndex * select_num  # 结束查询的索引
select_data = db_data[select_start: select_down]  #这里查询出了一次20条数据

这个时候是查询出20条数据了，但是，我们想要的具体的数据和页码对不上，所以做了以下的数据

#下面对20条数据进行截取
    #findIndex 1
        # page = 1  1-(0*5)  gnPage = 1
        # page = 2  2-(0*5)  gnPage = 2
        # page = 3  3-(0*5)  gnPage = 3
        # page = 4  4-(0*5)  gnPage = 4
        # page = 5  5-(0*5)  gnPage = 5
    #findIndex 2
        # page = 6  6-(1*5)  gnPage = 1
        # page = 7  7-(1*5)  gnPage = 2
        # page = 8  8-(1*5)  gnPage = 3
        # page = 9  9-(1*5)  gnPage = 4
        # page = 10  10-(1*5)  gnPage = 5
    #findIndex 3
        # page = 11  11-(2*5)  gnPage = 1
        # page = 12  12-(2*5)  gnPage = 2
        # page = 13  13-(2*5)  gnPage = 3
        # page = 14  14-(2*5)  gnPage = 4
        # page = 15  15-(2*5)  gnPage = 5

所以我们可以得到以下代码来获取具体的一页数据

#设定 截取的索引位now_index
now_index = page - (findIndex-1)*one_time_num
#设定单页的截取起始
page_start = (now_index - 1) * one_page_num
page_end = now_index * one_page_num
#开始截取
page_data = select_data[page_start:page_end]

最后进行代码结合

#coding:utf-8

db_data = list(range(1,100))

one_page_num = 5 #每页4条
one_time_num = 5 #每次查5页


while True:
    page = int(input("page >>>"))  #输入页码

    #查询索引的逻辑 设定findIndex查询的索引
    # page = 1  int(1/5) + 1  findIndex = 1
    # page = 2  int(2/5) + 1  findIndex = 1
    # page = 3  int(3/5) + 1  findIndex = 1
    # page = 4  int(4/5) + 1  findIndex = 1
    # page = 5  int(5/5)  findIndex = 1
    # page = 6  int(6/5) + 1  findIndex = 2
    # page = 7  int(7/5) + 1  findIndex = 2
    # page = 8  int(8/5) + 1  findIndex = 2
    # page = 9  int(9/5) + 1  findIndex = 2
    # page = 10  int(10/5)  findIndex = 2
    # page = 11  int(11/5) + 1  findIndex = 3
    # page = 12  int(12/5) + 1  findIndex = 3
    # page = 13  int(13/5) + 1  findIndex = 3
    # page = 14  int(14/5) + 1  findIndex = 3
    # page = 15  int(15/5)  findIndex = 3

    if page/one_time_num > int(page/one_time_num):
        findIndex = int(page / one_time_num) + 1
    else:
        findIndex = int(page / one_time_num)
    #进行第一次查询，查询每次5页，每页4条的数据
    select_num = one_time_num * one_page_num #这里决一次定查询20条
    select_start = (findIndex - 1) * select_num  # 开始查询的索引
    select_down =  findIndex * select_num  # 结束查询的索引

    select_data = db_data[select_start: select_down]  #这里查询出了一次20条数据

    #下面对20条数据进行截取
    #findIndex 1
        # page = 1  1-(0*5)  gnPage = 1
        # page = 2  2-(0*5)  gnPage = 2
        # page = 3  3-(0*5)  gnPage = 3
        # page = 4  4-(0*5)  gnPage = 4
        # page = 5  5-(0*5)  gnPage = 5
    #findIndex 2
        # page = 6  6-(1*5)  gnPage = 1
        # page = 7  7-(1*5)  gnPage = 2
        # page = 8  8-(1*5)  gnPage = 3
        # page = 9  9-(1*5)  gnPage = 4
        # page = 10  10-(1*5)  gnPage = 5
    #findIndex 3
        # page = 11  11-(2*5)  gnPage = 1
        # page = 12  12-(2*5)  gnPage = 2
        # page = 13  13-(2*5)  gnPage = 3
        # page = 14  14-(2*5)  gnPage = 4
        # page = 15  15-(2*5)  gnPage = 5
    #设定 截取的索引位now_index
    now_index = page - (findIndex-1)*one_time_num
    #设定单页的截取起始
    page_start = (now_index - 1) * one_page_num
    page_end = now_index * one_page_num
    #开始截取
    page_data = select_data[page_start:page_end]
    #输出效果
    print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    print("当前是第%s次查询"%findIndex)
    print("当前查询到的所有数据是 
 %s" % select_data)
    print("当前是 %s 页" % page)
    print("当前的数据是 
 %s " % page_data)
    print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

大家试一下吧。