应用场景:处理大量数据(14W条数据)进行批量插入数据库操作,如果14W条数据直接进行插入会导致数据库服务器CPU负载过大、出现慢日志,解决的方法就是对这个字典进行分割,分组去进行插入。
将14W条数据(dict 字典)转为列表(list),列表中的每一组都还是字典,每组150条数据。
python对字典数据进行分组:
# 对字典进行分割 def dict_chunk(self, dicts, size): new_list = [] dict_len = len(dicts) # 获取分组数 while_count = dict_len // size + 1 if dict_len % size != 0 else dict_len / size split_start = 0 split_end = size while (while_count > 0): # 把字典的键放到列表中,然后根据偏移量拆分字典 new_list.append({k: dicts[k] for k in list(dicts.keys())[split_start:split_end]}) split_start += size split_end += size while_count -= 1 return new_list
举个例子:(这里是分割成每组10条数据)
原始字典数据: phones = { 'a_01':'1200x1500', 'a_02':'1280x1480', 'a_03':'1220x1520', 'a_04':'1240x1540', 'a_05':'1240x1540', 'a_06':'1220x1520', 'a_07':'1240x1540', 'a_08':'1200x1500', 'a_09':'1240x1540', 'a_10':'1240x1540', 'a_11':'1280x1480', 'a_12':'1240x1540', 'a_13':'1220x1520', 'a_14':'1200x1500', 'a_15':'1280x1480', 'a_16':'1240x1540', 'a_17':'1200x1500', 'a_18':'1280x1480', 'a_19':'1240x1540', 'a_20':'1280x1480', 'a_21':'1240x1540', 'a_22':'1280x1480', }
# 这里直接调用咱们之前写的那个demo就好了 res = self.dict_chunk(phones,10) print(res)
[{'a_01': '1200x1500', 'a_02': '1280x1480', 'a_03': '1220x1520', 'a_04': '1240x1540', 'a_05': '1240x1540', 'a_06': '1220x1520', 'a_07': '1240x1540', 'a_08': '1200x1500', 'a_09': '1240x1540', 'a_10': '1240x1540'}, {'a_11': '1280x1480', 'a_12': '1240x1540', 'a_13': '1220x1520', 'a_14': '1200x1500', 'a_15': '1280x1480', 'a_16': '1240x1540', 'a_17': '1200x1500', 'a_18': '1280x1480', 'a_19': '1240x1540', 'a_20': '1280x1480'}, {'a_21': '1240x1540', 'a_22': '1280x1480'}]
这样就得到了新的数据