• 2秒统计一亿数据


    题目如下:

    1、生成两个文件:
    1)文件名Person.txt
    字段:
    personid,long,主键
    name,string,内容随机
    记录条数 1千万条
    格式:txt,csv或者二进制,都可以。

    2)文件名Order.txt
    字段:
    orderid,long,主键
    personid,long,外键,内容必须在Person.txt里面,取值随机
    amount,double,金额,数值随机
    记录条数 1亿条,personid的模拟尽量均匀
    格式:txt,csv或者二进制,都可以。

    2、写程序以刚才生成的文件为依据,读取其内容,统计出等同与以下语句的一组数据,并写入文件 sqlresult.txt
    1)给定一个person name ,SELECT o.amount FROM person p,order o WHERE o.personid=p. personid AND p.name=? ORDER BY o.amount limit 1000
    2)SELECT p.name, sum(o.amount) sum, count(*) from person p,order o where o.personid=p. personid group by o.personid order by sum limit 1000 【选做】

    Java要求:-Xmx1024m,即JVM最大内存超过1G
    .NET要求在32位机器上跑。
    开发要求:可以使用任何开源组件,包括单机数据库,但不允许使用SQL语句。

    1、程序运行时间:
        1)本机时间测试:
        按名称查询:50ms左右
        汇总查询:2000ms左右
        2)局域网测试
        按名称查询:13573ms
        汇总查询:287162ms(5分钟左右,建议把文件Copy到本机)
       
    2、文件构成:
        1)存储方式
        二进制,按位存储
        2)文件构成(共5个文件)
        person.nk:                 Person信息(190M),           ID:long(8),Name:String(12)
        person.nk.name:       Person索引文件(190M),    Name、PersonID
        order.nk:                    订单信息(2.23G),                ID:long(8),PersonID:long(8),Amount:double(8)
        order.nk.personid:     订单索引文件(1.48G),        personID,OrderID
        order.nk.amount:       订单索引文件(1.48G),        PersonID,Amount
    3、设计思路:
        1)按照名称查询订单信息说明:
        因为person有1千万条,Order有1亿条,平均每人有10个订单;
        首先根据名称查询PersonID:person.nk.name,根据名称排序,二分法查找;
        然后再根据PersonID查询金额:order.nk.personid,根据PersonID排序,二分法查找订单;
        2)查询汇总信息:
        循环订单,统计金额和数量。
        直接查询order.nk文件,因为文件比较大,会花费大量的时间,所以创建PersonID和金额的索引,按PersonID排序,就可以统计出需要的结果。

    源码:

    test.java

     1 import java.util.ArrayList;
     2 import java.util.List;
     3 
     4 public class test {
     5 
     6     public static void main(String[] args) {
     7 
     8         //测试名称(也可通过 PersonOperate.printPersonData(); 打印所有人员)
     9         // Id:233745 Name:尉全平
    10         // Id:233746 Name:汪哲宁
    11         // Id:233747 Name:宦英凡
    12         // Id:233748 Name:余彪朗
    13         // Id:233749 Name:贾栋信
    14         // Id:233750 Name:秋泽波
    15         // Id:233751 Name:司栋固
    16         // Id:233752 Name:黎苑芬
    17         // Id:233753 Name:吉士天
    18         // Id:233754 Name:轩兰兰
    19         // Id:233755 Name:西冰凤
    20         // Id:233756 Name:冀克新
    21         // Id:233757 Name:梁韵婵
    22         // Id:233758 Name:吉兰琳
    23         // Id:233759 Name:齐艳纨
    24         // Id:233760 Name:马枫聪
    25         // Id:233761 Name:阙菊婉
    26 
    27 
    28         //如有重名,默认取第一个人员,请通过 PersonOperate.getPersonsByName("东楠固") 方法确认
    29         PersonOperate.printOrderByPersonName("支浩毅", 1000, true);
    30 
    31         OrderOperate.printTop1000SumData(1000, true);
    32 
    33 
    34 
    35     }
    36 
    37 }
    View Code

    PersonOperate.java

      1 import java.io.File;
      2 import java.io.IOException;
      3 import java.io.RandomAccessFile;
      4 import java.nio.MappedByteBuffer;
      5 import java.nio.channels.FileChannel;
      6 import java.util.ArrayList;
      7 import java.util.Collections;
      8 import java.util.Comparator;
      9 import java.util.List;
     10 
     11 public class PersonOperate {
     12 
     13     /**
     14      * 单条插入
     15      *
     16      * @param person
     17      */
     18     public static void insert(Person person) {
     19 
     20         File file = new File(CommonFunc.psnPath);
     21         long fileLen = file.length();
     22 
     23         RandomAccessFile raf = null;
     24         try {
     25             raf = new RandomAccessFile(file, "rw");
     26 
     27             long id = person.getPersonId();
     28             byte[] idByte = CommonFunc.getBytes(id);
     29             String name = person.getName();
     30             byte[] nameByte = CommonFunc.getBytes(name, CommonFunc.nameLen);
     31 
     32             byte[] idNameByte = new byte[CommonFunc.psnIdLen + CommonFunc.nameLen];
     33             System.arraycopy(idByte, 0, idNameByte, 0, CommonFunc.psnIdLen);
     34             System.arraycopy(nameByte, 0, idNameByte, CommonFunc.psnIdLen, CommonFunc.nameLen);
     35 
     36             raf.seek(fileLen);
     37             raf.write(idNameByte);
     38 
     39         } catch (Exception ex) {
     40             System.out.println(ex.getMessage());
     41         } finally {
     42             if (null != raf) {
     43                 try {
     44                     raf.close();
     45                 } catch (IOException e) {
     46                     System.out.println(e.getMessage());
     47                 }
     48             }
     49         }
     50     }
     51 
     52     /**
     53      * 批量插入
     54      *
     55      * @param personList
     56      */
     57     public static void batchInsert(Person[] personList) {
     58 
     59         File file = new File(CommonFunc.psnPath);
     60         long fileLen = file.length();
     61 
     62         RandomAccessFile raf = null;
     63         try {
     64             byte[] idNameBytes = new byte[(CommonFunc.psnIdLen + CommonFunc.nameLen) * personList.length];
     65             for (int i = 0; i < personList.length; i++) {
     66                 Person person = personList[i];
     67 
     68                 long id = person.getPersonId();
     69                 byte[] idByte = CommonFunc.getBytes(id);
     70                 String name = person.getName();
     71                 byte[] nameByte = CommonFunc.getBytes(name, CommonFunc.nameLen);
     72 
     73                 System.arraycopy(idByte, 0, idNameBytes, i * (CommonFunc.psnIdLen + CommonFunc.nameLen), CommonFunc.psnIdLen);
     74                 System.arraycopy(nameByte, 0, idNameBytes, i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen, CommonFunc.nameLen);
     75             }
     76 
     77             raf = new RandomAccessFile(file, "rw");
     78             raf.seek(fileLen);
     79             raf.write(idNameBytes);
     80         } catch (Exception ex) {
     81             System.out.println(ex.getMessage());
     82         } finally {
     83             if (null != raf) {
     84                 try {
     85                     raf.close();
     86                 } catch (IOException e) {
     87                     System.out.println(e.getMessage());
     88                 }
     89             }
     90         }
     91     }
     92 
     93     /**
     94      * 根据ID查对象
     95      *
     96      * @param personId
     97      * @return
     98      */
     99     public static Person getPersonById(long personId) {
    100         Person rtn = new Person();
    101 
    102         File file = new File(CommonFunc.psnPath);
    103         //获取位置
    104         long pos = (personId - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen);
    105 
    106         RandomAccessFile raf = null;
    107         try {
    108             raf = new RandomAccessFile(file, "r");
    109 
    110             raf.seek(pos);
    111             rtn.setPersonId(raf.readLong());
    112             pos += CommonFunc.psnIdLen;
    113 
    114             raf.seek(pos);
    115             byte[] nameByte = new byte[CommonFunc.nameLen];
    116             raf.read(nameByte);
    117             rtn.setName(CommonFunc.getString(nameByte));
    118         } catch (Exception ex) {
    119             System.out.println(ex.getMessage());
    120         } finally {
    121             if (null != raf) {
    122                 try {
    123                     raf.close();
    124                 } catch (IOException e) {
    125                     System.out.println(e.getMessage());
    126                 }
    127             }
    128         }
    129 
    130         return rtn;
    131     }
    132 
    133     /**
    134      * 根据名字查询Person对象
    135      *
    136      * @param name
    137      * @return
    138      */
    139     public static List<Person> getPersonsByName(String name) {
    140         List<Person> rtn = new ArrayList<>();
    141 
    142         //获取要查询的ID列表
    143         File idxFile = new File(CommonFunc.psnIdxPath);
    144         long fileLen = idxFile.length();
    145         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    146 
    147         FileChannel idxFileChannel = null;
    148         try {
    149             idxFileChannel = new RandomAccessFile(idxFile, "r").getChannel();
    150             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, idxFile.length());
    151 
    152             byte[] nameBytes = CommonFunc.getBytes(name, 12);
    153             long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameBytes[0], nameBytes[1], nameBytes[2]});
    154 
    155             //二分法查找,确定位置
    156             long findRow = -1;
    157             long minRow = 1;
    158             long maxRow = rowNum;
    159             while (maxRow >= minRow) {
    160                 long midRow = (minRow + maxRow) / 2;
    161                 idxMappedByteBuffer.position((int) (midRow - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    162                 byte[] nameByte = new byte[CommonFunc.nameLen];
    163                 idxMappedByteBuffer.get(nameByte);
    164                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    165 
    166                 if (tempFirstNameNum > firstNameNum) {
    167                     if (maxRow == midRow) {
    168                         break;
    169                     }
    170                     maxRow = midRow;
    171                 } else if (tempFirstNameNum < firstNameNum) {
    172                     if (minRow == midRow) {
    173                         minRow++;
    174                         continue;
    175                     }
    176                     minRow = midRow;
    177                 } else {
    178                     findRow = midRow - 1;
    179                     break;
    180                 }
    181             }
    182 
    183             if (findRow == -1) {
    184                 return rtn;
    185             }
    186 
    187             //前后搜索,确定该数据的最大行和最小行
    188             long minRowIndex = 0;
    189             while ((findRow - minRowIndex) >= 0) {
    190                 idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    191                 byte[] nameByte = new byte[CommonFunc.nameLen];
    192                 idxMappedByteBuffer.get(nameByte);
    193                 String tempName = CommonFunc.getString(nameByte);
    194                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    195 
    196                 if (tempFirstNameNum != firstNameNum) {
    197                     break;
    198                 }
    199 
    200                 if (tempName.equals(name) == true) {
    201                     idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
    202                     long personId = idxMappedByteBuffer.getLong();
    203 
    204                     Person person = new Person();
    205                     person.setPersonId(personId);
    206                     person.setName(tempName);
    207                     rtn.add(person);
    208                 }
    209                 minRowIndex++;
    210             }
    211 
    212             long maxRowIndex = 1;
    213             while ((findRow + maxRowIndex) <= rowNum) {
    214                 idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    215                 byte[] nameByte = new byte[CommonFunc.nameLen];
    216                 idxMappedByteBuffer.get(nameByte);
    217                 String tempName = CommonFunc.getString(nameByte);
    218                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    219 
    220                 if (tempFirstNameNum != firstNameNum) {
    221                     break;
    222                 }
    223 
    224                 if (tempName.equals(name) == true) {
    225                     idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
    226                     long personId = idxMappedByteBuffer.getLong();
    227 
    228                     Person person = new Person();
    229                     person.setPersonId(personId);
    230                     person.setName(tempName);
    231                     rtn.add(person);
    232                 }
    233 
    234                 maxRowIndex++;
    235             }
    236         } catch (Exception ex) {
    237             System.out.println(ex.getMessage());
    238         } finally {
    239             if (null != idxFileChannel) {
    240                 try {
    241                     idxFileChannel.close();
    242                 } catch (IOException e) {
    243                     System.out.println(e.getMessage());
    244                 }
    245             }
    246         }
    247 
    248         return rtn;
    249     }
    250 
    251     public static List<Person> getPersonsByName_bak(String name) {
    252         List<Person> rtn = new ArrayList<>();
    253 
    254         File file = new File(CommonFunc.psnPath);
    255         long fileLen = file.length();
    256         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    257 
    258         RandomAccessFile raf = null;
    259         try {
    260             raf = new RandomAccessFile(file, "r");
    261             for (int i = 0; i < rowNum; i++) {
    262                 Person person = new Person();
    263                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    264                 person.setPersonId(raf.readLong());
    265 
    266                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
    267                 byte[] nameByte = new byte[CommonFunc.nameLen];
    268                 raf.read(nameByte);
    269                 String pName = CommonFunc.getString(nameByte);
    270                 person.setName(pName);
    271 
    272                 if (name.equals(pName) == true) {
    273                     rtn.add(person);
    274                 }
    275             }
    276         } catch (Exception ex) {
    277             System.out.println(ex.getMessage());
    278         } finally {
    279             if (null != raf) {
    280                 try {
    281                     raf.close();
    282                 } catch (IOException e) {
    283                     System.out.println(e.getMessage());
    284                 }
    285             }
    286         }
    287 
    288         return rtn;
    289     }
    290 
    291     public static List<Person> getPersonsByName_bak2(String name) {
    292         List<Person> rtn = new ArrayList<>();
    293 
    294         //获取要查询的ID列表
    295         File idxFile = new File(CommonFunc.psnIdxPath);
    296         long fileLen = idxFile.length();
    297         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    298 
    299         RandomAccessFile idxRaf = null;
    300         try {
    301             idxRaf = new RandomAccessFile(idxFile, "r");
    302 
    303             byte[] nameBytes = CommonFunc.getBytes(name, 12);
    304             long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameBytes[0], nameBytes[1], nameBytes[2]});
    305 
    306             //二分法查找,确定位置
    307             long findRow = -1;
    308             long minRow = 1;
    309             long maxRow = rowNum;
    310             while (maxRow >= minRow) {
    311                 long midRow = (minRow + maxRow) / 2;
    312                 idxRaf.seek((midRow - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    313                 byte[] nameByte = new byte[CommonFunc.nameLen];
    314                 idxRaf.read(nameByte);
    315                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    316 
    317                 if (tempFirstNameNum > firstNameNum) {
    318                     if (maxRow == midRow) {
    319                         break;
    320                     }
    321                     maxRow = midRow;
    322                 } else if (tempFirstNameNum < firstNameNum) {
    323                     if (minRow == midRow) {
    324                         minRow++;
    325                         continue;
    326                     }
    327                     minRow = midRow;
    328                 } else {
    329                     findRow = midRow - 1;
    330                     break;
    331                 }
    332             }
    333 
    334             if (findRow == -1) {
    335                 return rtn;
    336             }
    337 
    338             //前后搜索,确定该数据的最大行和最小行
    339             long minRowIndex = 0;
    340             while ((findRow - minRowIndex) >= 0) {
    341                 idxRaf.seek((findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    342                 byte[] nameByte = new byte[CommonFunc.nameLen];
    343                 idxRaf.read(nameByte);
    344                 String tempName = CommonFunc.getString(nameByte);
    345                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    346 
    347                 if (tempFirstNameNum != firstNameNum) {
    348                     break;
    349                 }
    350 
    351                 if (tempName.equals(name) == true) {
    352                     idxRaf.seek((findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
    353                     byte[] idByte = new byte[CommonFunc.psnIdLen];
    354                     idxRaf.read(idByte);
    355                     long personId = CommonFunc.getLong(idByte);
    356 
    357                     Person person = getPersonById(personId);
    358                     rtn.add(person);
    359                 }
    360                 minRowIndex++;
    361             }
    362 
    363             long maxRowIndex = 1;
    364             while ((findRow + maxRowIndex) <= rowNum) {
    365                 idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    366                 byte[] nameByte = new byte[CommonFunc.nameLen];
    367                 idxRaf.read(nameByte);
    368                 String tempName = CommonFunc.getString(nameByte);
    369                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    370 
    371                 if (tempFirstNameNum != firstNameNum) {
    372                     break;
    373                 }
    374 
    375                 if (tempName.equals(name) == true) {
    376                     idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
    377                     byte[] idByte = new byte[CommonFunc.psnIdLen];
    378                     idxRaf.read(idByte);
    379                     long personId = CommonFunc.getLong(idByte);
    380 
    381                     Person person = getPersonById(personId);
    382                     rtn.add(person);
    383                 }
    384 
    385                 maxRowIndex++;
    386             }
    387         } catch (Exception ex) {
    388             System.out.println(ex.getMessage());
    389         } finally {
    390             if (null != idxRaf) {
    391                 try {
    392                     idxRaf.close();
    393                 } catch (IOException e) {
    394                     System.out.println(e.getMessage());
    395                 }
    396             }
    397         }
    398 
    399         return rtn;
    400     }
    401 
    402     /**
    403      * 创建名称索引
    404      * <p/>
    405      * 每个字符占3个字节,取Int值,占4个字节
    406      * 3个字符,进行相加,然后进行排序
    407      * <p/>
    408      * 1个字节的最大值为 256*256*256 = 16777216
    409      * 3个字节的最大值为 16777216 * 3 = 50331648
    410      * <p/>
    411      * 如:孙(15052185)天(15049897)天(15049897):45151979
    412      */
    413     public static void createfirstNameIndex() {
    414 
    415         long totalStartTime = System.currentTimeMillis();
    416         //1、创建临时文件
    417         System.out.println("开始创建临时文件");
    418         long startTime = System.currentTimeMillis();
    419 
    420         createTempFile();
    421 
    422         long endTime = System.currentTimeMillis();
    423         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
    424         System.out.println("临时文件创建完成");
    425 
    426         //2、整理临时文件
    427         System.out.println("整理临时文件");
    428         startTime = System.currentTimeMillis();
    429 
    430         fillTempFile();
    431 
    432         endTime = System.currentTimeMillis();
    433         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
    434         System.out.println("整理临时文件完成");
    435 
    436         //3、创建索引文件
    437         System.out.println("开始创建索引文件");
    438         startTime = System.currentTimeMillis();
    439 
    440         createIdxFile();
    441 
    442         endTime = System.currentTimeMillis();
    443         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
    444         System.out.println("索引文件创建完成");
    445 
    446         long totalEndTime = System.currentTimeMillis();
    447         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
    448     }
    449 
    450     private static void createTempFile() {
    451         File tempFile = new File(CommonFunc.tempPath);
    452         File file = new File(CommonFunc.psnPath);
    453         long fileLen = file.length();
    454         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    455 
    456         FileChannel tempFileChannel = null;
    457         RandomAccessFile raf = null;
    458         try {
    459             if (tempFile.exists() == true) {
    460                 tempFile.delete();
    461             }
    462             tempFile.createNewFile();
    463 
    464             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
    465             raf = new RandomAccessFile(file, "r");
    466             for (int i = 0; i < rowNum; i++) {
    467                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
    468                 byte[] nameByte = new byte[CommonFunc.nameLen];
    469                 raf.read(nameByte);
    470                 long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    471 
    472                 MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_ONLY, (firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen), CommonFunc.tempLen);
    473                 byte[] longByte = new byte[CommonFunc.tempLen];
    474                 mappedByteBuffer.get(longByte);
    475                 long tempVal = CommonFunc.getLong(longByte);
    476 
    477                 mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, (firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen), CommonFunc.tempLen + CommonFunc.tempLen);
    478                 tempVal++;
    479                 longByte = CommonFunc.getBytes(tempVal);
    480                 byte[] zeroByte = new byte[CommonFunc.tempLen];
    481                 byte[] idNameByte = new byte[CommonFunc.tempLen + CommonFunc.tempLen];
    482                 System.arraycopy(longByte, 0, idNameByte, 0, CommonFunc.tempLen);
    483                 System.arraycopy(zeroByte, 0, idNameByte, CommonFunc.tempLen, CommonFunc.tempLen);
    484                 mappedByteBuffer.put(idNameByte);
    485 
    486                 if (i % 100000 == 0) {
    487                     System.out.println("创建临时文件 " + i);
    488                 }
    489             }
    490         } catch (Exception ex) {
    491             System.out.println(ex.getMessage());
    492         } finally {
    493             if (null != tempFileChannel) {
    494                 try {
    495                     tempFileChannel.close();
    496                 } catch (IOException e) {
    497                     System.out.println(e.getMessage());
    498                 }
    499             }
    500             if (null != raf) {
    501                 try {
    502                     raf.close();
    503                 } catch (IOException e) {
    504                     System.out.println(e.getMessage());
    505                 }
    506             }
    507         }
    508     }
    509 
    510     private static void fillTempFile() {
    511         File tempFile = new File(CommonFunc.tempPath);
    512         long fileLen = tempFile.length();
    513         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
    514 
    515         FileChannel tempFileChannel = null;
    516         try {
    517             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
    518             long preNum = 0;
    519             for (int i = 0; i < rowNum; i++) {
    520                 MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_ONLY, i * (CommonFunc.tempLen + CommonFunc.tempLen), CommonFunc.tempLen);
    521                 byte[] longByte = new byte[8];
    522                 mappedByteBuffer.get(longByte);
    523                 long tempVal = CommonFunc.getLong(longByte);
    524 
    525                 if (tempVal <= 0) {
    526                     continue;
    527                 }
    528 
    529                 mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen, CommonFunc.tempLen);
    530                 longByte = CommonFunc.getBytes(preNum);
    531                 mappedByteBuffer.put(longByte);
    532                 preNum += tempVal;
    533 
    534                 if (i % 100000 == 0) {
    535                     System.out.println("填充临时文件 " + i);
    536                 }
    537             }
    538         } catch (Exception ex) {
    539             System.out.println(ex.getMessage());
    540         } finally {
    541             if (null != tempFileChannel) {
    542                 try {
    543                     tempFileChannel.close();
    544                 } catch (IOException e) {
    545                     System.out.println(e.getMessage());
    546                 }
    547             }
    548         }
    549     }
    550 
    551     private static void createIdxFile() {
    552         File file = new File(CommonFunc.psnPath);
    553         File tempFile = new File(CommonFunc.tempPath);
    554         File idxFile = new File(CommonFunc.psnIdxPath);
    555         long fileLen = file.length();
    556         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    557 
    558         RandomAccessFile raf = null;
    559         RandomAccessFile tempRaf = null;
    560         RandomAccessFile idxRaf = null;
    561         try {
    562             if (idxFile.exists() == true) {
    563                 idxFile.delete();
    564             }
    565             idxFile.createNewFile();
    566 
    567             raf = new RandomAccessFile(file, "r");
    568             tempRaf = new RandomAccessFile(tempFile, "rw");
    569             idxRaf = new RandomAccessFile(idxFile, "rw");
    570 
    571             for (int i = 0; i < rowNum; i++) {
    572                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    573                 byte[] idByte = new byte[CommonFunc.psnIdLen];
    574                 raf.read(idByte);
    575                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
    576                 byte[] nameByte = new byte[CommonFunc.nameLen];
    577                 raf.read(nameByte);
    578                 long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
    579 
    580                 tempRaf.seek((firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
    581                 byte[] longByte = new byte[CommonFunc.tempLen];
    582                 tempRaf.read(longByte);
    583                 long tempVal = CommonFunc.getLong(longByte);
    584                 tempRaf.seek((firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
    585                 longByte = new byte[CommonFunc.tempLen];
    586                 tempRaf.read(longByte);
    587                 long tempPreNum = CommonFunc.getLong(longByte);
    588                 tempRaf.seek((firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
    589                 tempVal--;
    590                 longByte = CommonFunc.getBytes(tempVal);
    591                 tempRaf.write(longByte);
    592 
    593                 //写入
    594                 idxRaf.seek((tempPreNum + tempVal) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    595                 idxRaf.write(nameByte);
    596                 idxRaf.seek((tempPreNum + tempVal) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
    597                 idxRaf.write(idByte);
    598 
    599                 if (i % 100000 == 0) {
    600                     System.out.println("创建索引文件 " + i);
    601                 }
    602             }
    603         } catch (Exception ex) {
    604             System.out.println(ex.getMessage());
    605         } finally {
    606             if (null != raf) {
    607                 try {
    608                     raf.close();
    609                 } catch (IOException e) {
    610                     System.out.println(e.getMessage());
    611                 }
    612             }
    613             if (null != tempRaf) {
    614                 try {
    615                     tempRaf.close();
    616                 } catch (IOException e) {
    617                     System.out.println(e.getMessage());
    618                 }
    619             }
    620             if (null != idxRaf) {
    621                 try {
    622                     idxRaf.close();
    623                 } catch (IOException e) {
    624                     System.out.println(e.getMessage());
    625                 }
    626             }
    627         }
    628     }
    629 
    630     //以下为测试用方法
    631 
    632     //标准数据:1千万条
    633     public static void createData() {
    634 
    635         File file = new File(CommonFunc.psnPath);
    636         if (file.exists() == true) {
    637             file.delete();
    638         }
    639         try {
    640             file.createNewFile();
    641         } catch (IOException e) {
    642             System.out.println(e.getMessage());
    643         }
    644 
    645         long totalStartTime = System.currentTimeMillis();
    646         for (int j = 0; j < 10; j++) {
    647 
    648             long startTime = System.currentTimeMillis();
    649             Person[] personList = new Person[1000000];
    650             for (int i = (j * 1000000); i < (j * 1000000 + 1000000); i++) {
    651 
    652                 Person person = new Person();
    653                 personList[i - (j * 1000000)] = person;
    654 
    655                 person.setPersonId(i + 1);
    656                 person.setName(CommonFunc.getPersonName());
    657             }
    658 
    659             PersonOperate.batchInsert(personList);
    660             long endTime = System.currentTimeMillis();
    661             System.out.println("程序第 " + j + " 次运行时间: " + (endTime - startTime) + "ms");
    662         }
    663         long totalEndTime = System.currentTimeMillis();
    664         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
    665     }
    666 
    667     //测试数据 3条
    668     public static void createTestData() {
    669 
    670         File file = new File(CommonFunc.psnPath);
    671         if (file.exists() == true) {
    672             file.delete();
    673         }
    674         try {
    675             file.createNewFile();
    676         } catch (IOException e) {
    677             System.out.println(e.getMessage());
    678         }
    679 
    680         //批量提交:1KW数据
    681         long totalStartTime = System.currentTimeMillis();
    682         for (int j = 0; j < 1; j++) {
    683 
    684             long startTime = System.currentTimeMillis();
    685             Person[] personList = new Person[3];
    686             for (int i = (j * 3); i < (j * 3 + 3); i++) {
    687 
    688                 Person person = new Person();
    689                 personList[i - (j * 3)] = person;
    690 
    691                 person.setPersonId(i + 1);
    692                 person.setName(CommonFunc.getPersonName());
    693             }
    694 
    695             PersonOperate.batchInsert(personList);
    696             long endTime = System.currentTimeMillis();
    697             System.out.println("程序第 " + j + " 次运行时间: " + (endTime - startTime) + "ms");
    698         }
    699         long totalEndTime = System.currentTimeMillis();
    700         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
    701     }
    702 
    703     public static void printPersonData() {
    704 
    705         List<Person> rtn = new ArrayList<>();
    706 
    707         File file = new File(CommonFunc.psnPath);
    708         long fileLen = file.length();
    709         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    710 
    711         RandomAccessFile raf = null;
    712         try {
    713             raf = new RandomAccessFile(file, "r");
    714             for (int i = 0; i < rowNum; i++) {
    715                 Person person = new Person();
    716                 rtn.add(person);
    717 
    718                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    719                 person.setPersonId(raf.readLong());
    720 
    721                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
    722                 byte[] nameByte = new byte[CommonFunc.nameLen];
    723                 raf.read(nameByte);
    724                 person.setName(CommonFunc.getString(nameByte));
    725 
    726                 System.out.println("Id:" + person.getPersonId() + " Name:" + person.getName());
    727             }
    728         } catch (Exception ex) {
    729             System.out.println(ex.getMessage());
    730         } finally {
    731             if (null != raf) {
    732                 try {
    733                     raf.close();
    734                 } catch (IOException e) {
    735                     System.out.println(e.getMessage());
    736                 }
    737             }
    738         }
    739     }
    740 
    741     public static void printTempData() {
    742 
    743         File tempFile = new File(CommonFunc.tempPath);
    744         long fileLen = tempFile.length();
    745         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
    746 
    747         RandomAccessFile tempRaf = null;
    748         try {
    749             tempRaf = new RandomAccessFile(tempFile, "rw");
    750             for (int i = 0; i < rowNum; i++) {
    751                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen));
    752                 byte[] longByte = new byte[8];
    753                 tempRaf.read(longByte);
    754                 long tempVal = CommonFunc.getLong(longByte);
    755 
    756                 if (tempVal <= 0) {
    757                     continue;
    758                 }
    759 
    760                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
    761                 longByte = new byte[8];
    762                 tempRaf.read(longByte);
    763                 long tempPreNum = CommonFunc.getLong(longByte);
    764 
    765                 System.out.println("PersonNameNum : " + (i + 1) + " ; Num : " + tempVal + " ; PreNum : " + tempPreNum);
    766             }
    767         } catch (Exception ex) {
    768             System.out.println(ex.getMessage());
    769         } finally {
    770             if (null != tempRaf) {
    771                 try {
    772                     tempRaf.close();
    773                 } catch (IOException e) {
    774                     System.out.println(e.getMessage());
    775                 }
    776             }
    777         }
    778     }
    779 
    780     public static void printIdxData() {
    781 
    782         File tempFile = new File(CommonFunc.psnIdxPath);
    783         long fileLen = tempFile.length();
    784         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
    785 
    786         RandomAccessFile tempRaf = null;
    787         try {
    788             tempRaf = new RandomAccessFile(tempFile, "rw");
    789             for (int i = 0; i < rowNum; i++) {
    790 
    791                 tempRaf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
    792                 byte[] nameByte = new byte[CommonFunc.nameLen];
    793                 tempRaf.read(nameByte);
    794                 String name = CommonFunc.getString(nameByte);
    795 
    796                 tempRaf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
    797                 byte[] idByte = new byte[CommonFunc.psnIdLen];
    798                 tempRaf.read(idByte);
    799                 long id = CommonFunc.getLong(idByte);
    800 
    801                 System.out.println("idx : " + i + " ; name : " + name + " ; id : " + id);
    802             }
    803         } catch (Exception ex) {
    804             System.out.println(ex.getMessage());
    805         } finally {
    806             if (null != tempRaf) {
    807                 try {
    808                     tempRaf.close();
    809                 } catch (IOException e) {
    810                     System.out.println(e.getMessage());
    811                 }
    812             }
    813         }
    814     }
    815 
    816     public static void printOrderByPersonName(String name) {
    817 
    818         printOrderByPersonName(name, 0, false);
    819     }
    820 
    821     public static void printOrderByPersonName(String name, int totalNum, final boolean desc) {
    822 
    823         long startTime = System.currentTimeMillis();
    824         List<Person> personList = PersonOperate.getPersonsByName(name);
    825         if (personList.size() <= 0) {
    826             System.out.println("该名称没有对应的人员");
    827             return;
    828         }
    829         //默认取第一个人
    830         Person person = personList.get(0);
    831         List<Order> orderList = OrderOperate.getOrderByPersonId(person.getPersonId());
    832 
    833         //排序
    834         Collections.sort(orderList, new Comparator<Order>() {
    835             @Override
    836             public int compare(Order o1, Order o2) {
    837                 int rtn = 0;
    838                 if (o1.getAmount() < o2.getAmount()) {
    839                     rtn = desc == false ? -1 : 1;
    840                 } else if (o1.getAmount() > o2.getAmount()) {
    841                     rtn = desc == false ? 1 : -1;
    842                 }
    843                 return rtn;
    844             }
    845         });
    846 
    847         List<Order> rtn;
    848         if (totalNum > 0 && orderList.size() > totalNum) {
    849             rtn = orderList.subList(0, totalNum);
    850         } else {
    851             rtn = orderList;
    852         }
    853 
    854         if (rtn.size() <= 0) {
    855             System.out.println("该名称对应的订单为0");
    856         }
    857 
    858         long endTime = System.currentTimeMillis();
    859 
    860         for (Order order : rtn) {
    861             System.out.println("订单ID:" + order.getOrderId() + " 价格:" + order.getAmount());
    862         }
    863         System.out.println("共查出 " + rtn.size() + " 条");
    864         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
    865     }
    866 }
    View Code

    OrderOperate.java

       1 import java.io.File;
       2 import java.io.IOException;
       3 import java.io.RandomAccessFile;
       4 import java.nio.MappedByteBuffer;
       5 import java.nio.channels.FileChannel;
       6 import java.util.ArrayList;
       7 import java.util.List;
       8 
       9 public class OrderOperate {
      10 
      11     /**
      12      * 插入订单
      13      *
      14      * @param order
      15      */
      16     public static void insert(Order order) {
      17 
      18         File file = new File(CommonFunc.orderPath);
      19         long fileLen = file.length();
      20 
      21         RandomAccessFile raf = null;
      22         try {
      23             raf = new RandomAccessFile(file, "rw");
      24 
      25             long id = order.getOrderId();
      26             byte[] idByte = CommonFunc.getBytes(id);
      27             long personId = order.getPersonId();
      28             byte[] personIdByte = CommonFunc.getBytes(personId);
      29             double amount = order.getAmount();
      30             byte[] amountByte = CommonFunc.getBytes(amount);
      31 
      32             byte[] idNameByte = new byte[CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen];
      33             System.arraycopy(idByte, 0, idNameByte, 0, CommonFunc.orderIdLen);
      34             System.arraycopy(personIdByte, 0, idNameByte, CommonFunc.orderIdLen, CommonFunc.personIdLen);
      35             System.arraycopy(amountByte, 0, idNameByte, CommonFunc.orderIdLen + CommonFunc.personIdLen, CommonFunc.amountLen);
      36 
      37             raf.seek(fileLen);
      38             raf.write(idNameByte);
      39         } catch (Exception ex) {
      40             System.out.println(ex.getMessage());
      41         } finally {
      42             if (null != raf) {
      43                 try {
      44                     raf.close();
      45                 } catch (IOException e) {
      46                     System.out.println(e.getMessage());
      47                 }
      48             }
      49         }
      50     }
      51 
      52     /**
      53      * 批量插入订单
      54      *
      55      * @param orders
      56      */
      57     public static void batchInsert(Order[] orders) {
      58 
      59         File file = new File(CommonFunc.orderPath);
      60         long fileLen = file.length();
      61 
      62         RandomAccessFile raf = null;
      63         try {
      64             raf = new RandomAccessFile(file, "rw");
      65 
      66             byte[] idNameBytes = new byte[(CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) * orders.length];
      67             for (int i = 0; i < orders.length; i++) {
      68                 Order order = orders[i];
      69 
      70                 long id = order.getOrderId();
      71                 byte[] idByte = CommonFunc.getBytes(id);
      72                 long personId = order.getPersonId();
      73                 byte[] personIdByte = CommonFunc.getBytes(personId);
      74                 double amount = order.getAmount();
      75                 byte[] amountByte = CommonFunc.getBytes(amount);
      76 
      77                 System.arraycopy(idByte, 0, idNameBytes, i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen), CommonFunc.orderIdLen);
      78                 System.arraycopy(personIdByte, 0, idNameBytes, i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen, CommonFunc.personIdLen);
      79                 System.arraycopy(amountByte, 0, idNameBytes, i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + (CommonFunc.orderIdLen + CommonFunc.personIdLen), CommonFunc.amountLen);
      80             }
      81             raf.seek(fileLen);
      82             raf.write(idNameBytes);
      83 
      84         } catch (Exception ex) {
      85             System.out.println(ex.getMessage());
      86         } finally {
      87             if (null != raf) {
      88                 try {
      89                     raf.close();
      90                 } catch (IOException e) {
      91                     System.out.println(e.getMessage());
      92                 }
      93             }
      94         }
      95     }
      96 
      97     /**
      98      * 根据ID查询Order(二分法查询)
      99      *
     100      * @param orderId
     101      * @return
     102      */
     103     public static Order getOrderById(long orderId) {
     104         Order rtn = new Order();
     105 
     106         File file = new File(CommonFunc.orderPath);
     107         //获取位置
     108         long pos = (orderId - 1) * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
     109 
     110         RandomAccessFile raf = null;
     111         try {
     112             raf = new RandomAccessFile(file, "r");
     113 
     114             raf.seek(pos);
     115             rtn.setOrderId(raf.readLong());
     116             pos += CommonFunc.orderIdLen;
     117 
     118             raf.seek(pos);
     119             rtn.setPersonId(raf.readLong());
     120             pos += CommonFunc.personIdLen;
     121 
     122             raf.seek(pos);
     123             rtn.setAmount(raf.readDouble());
     124         } catch (Exception ex) {
     125             System.out.println(ex.getMessage());
     126         } finally {
     127             if (null != raf) {
     128                 try {
     129                     raf.close();
     130                 } catch (IOException e) {
     131                     System.out.println(e.getMessage());
     132                 }
     133             }
     134         }
     135 
     136         return rtn;
     137     }
     138 
     139     /**
     140      * 根据PersonID查询所有订单
     141      *
     142      * @param personId
     143      * @return
     144      */
     145     public static List<Order> getOrderByPersonId(long personId) {
     146         List<Order> rtn = new ArrayList<>();
     147 
     148         //获取要查询的ID列表
     149         File idxFile = new File(CommonFunc.orderIdxPath);
     150         long fileLen = idxFile.length();
     151         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
     152 
     153         FileChannel idxFileChannel = null;
     154         try {
     155             idxFileChannel = new RandomAccessFile(idxFile, "r").getChannel();
     156             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, idxFile.length());
     157 
     158             //二分法查找,确定位置
     159             long findRow = -1;
     160             long minRow = 1;
     161             long maxRow = rowNum;
     162             while (maxRow > minRow) {
     163                 long midRow = (minRow + maxRow) / 2;
     164                 idxMappedByteBuffer.position((int) (midRow - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
     165                 long tempPersonId = idxMappedByteBuffer.getLong();
     166                 if (tempPersonId > personId) {
     167                     if (maxRow == midRow) {
     168                         break;
     169                     }
     170                     maxRow = midRow;
     171                 } else if (tempPersonId < personId) {
     172                     if (minRow == midRow) {
     173                         minRow++;
     174                         continue;
     175                     }
     176                     minRow = midRow;
     177                 } else {
     178                     findRow = midRow - 1;
     179                     break;
     180                 }
     181             }
     182 
     183             if (findRow == -1) {
     184                 return rtn;
     185             }
     186 
     187             //前后搜索,确定该数据的最大行和最小行
     188             long minRowIndex = 0;
     189             while ((findRow - minRowIndex) >= 0) {
     190                 idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
     191                 long tempPersonId = idxMappedByteBuffer.getLong();
     192                 if (tempPersonId != personId) {
     193                     break;
     194                 }
     195 
     196                 idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
     197                 long orderId = idxMappedByteBuffer.getLong();
     198                 Order order = getOrderById(orderId);
     199                 rtn.add(order);
     200                 minRowIndex++;
     201             }
     202 
     203             long maxRowIndex = 1;
     204             while ((findRow + maxRowIndex) <= rowNum) {
     205                 idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
     206                 long tempPersonId = idxMappedByteBuffer.getLong();
     207                 if (tempPersonId != personId) {
     208                     break;
     209                 }
     210 
     211                 idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
     212                 long orderId = idxMappedByteBuffer.getLong();
     213                 Order order = getOrderById(orderId);
     214                 rtn.add(order);
     215                 maxRowIndex++;
     216             }
     217         } catch (Exception ex) {
     218             System.out.println(ex.getMessage());
     219         } finally {
     220             if (null != idxFileChannel) {
     221                 try {
     222                     idxFileChannel.close();
     223                 } catch (IOException e) {
     224                     System.out.println(e.getMessage());
     225                 }
     226             }
     227         }
     228 
     229         return rtn;
     230     }
     231 
     232     public static List<Order> getOrderByPersonId_bak(long personId) {
     233         List<Order> rtn = new ArrayList<>();
     234 
     235         //获取要查询的ID列表
     236         File idxFile = new File(CommonFunc.orderIdxPath);
     237         long fileLen = idxFile.length();
     238         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
     239 
     240         RandomAccessFile idxRaf = null;
     241         try {
     242             idxRaf = new RandomAccessFile(idxFile, "r");
     243 
     244             //二分法查找,确定位置
     245             long findRow = -1;
     246             long minRow = 1;
     247             long maxRow = rowNum;
     248             while (maxRow > minRow) {
     249                 long midRow = (minRow + maxRow) / 2;
     250                 idxRaf.seek((midRow - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
     251                 byte[] longByte = new byte[8];
     252                 idxRaf.read(longByte);
     253                 long tempPersonId = CommonFunc.getLong(longByte);
     254                 if (tempPersonId > personId) {
     255                     if (maxRow == midRow) {
     256                         break;
     257                     }
     258                     maxRow = midRow;
     259                 } else if (tempPersonId < personId) {
     260                     if (minRow == midRow) {
     261                         minRow++;
     262                         continue;
     263                     }
     264                     minRow = midRow;
     265                 } else {
     266                     findRow = midRow - 1;
     267                     break;
     268                 }
     269             }
     270 
     271             if (findRow == -1) {
     272                 return rtn;
     273             }
     274 
     275             //前后搜索,确定该数据的最大行和最小行
     276             long minRowIndex = 0;
     277             while ((findRow - minRowIndex) >= 0) {
     278                 idxRaf.seek((findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
     279                 byte[] longByte = new byte[8];
     280                 idxRaf.read(longByte);
     281                 long tempPersonId = CommonFunc.getLong(longByte);
     282 
     283                 if (tempPersonId != personId) {
     284                     break;
     285                 }
     286 
     287                 idxRaf.seek((findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
     288                 longByte = new byte[8];
     289                 idxRaf.read(longByte);
     290                 long orderId = CommonFunc.getLong(longByte);
     291 
     292                 Order order = getOrderById(orderId);
     293                 rtn.add(order);
     294 
     295                 minRowIndex++;
     296             }
     297 
     298             long maxRowIndex = 1;
     299             while ((findRow + maxRowIndex) <= rowNum) {
     300                 idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
     301                 byte[] longByte = new byte[8];
     302                 idxRaf.read(longByte);
     303                 long tempPersonId = CommonFunc.getLong(longByte);
     304 
     305                 if (tempPersonId != personId) {
     306                     break;
     307                 }
     308 
     309                 idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
     310                 longByte = new byte[8];
     311                 idxRaf.read(longByte);
     312                 long orderId = CommonFunc.getLong(longByte);
     313 
     314                 Order order = getOrderById(orderId);
     315                 rtn.add(order);
     316 
     317                 maxRowIndex++;
     318             }
     319         } catch (Exception ex) {
     320             System.out.println(ex.getMessage());
     321         } finally {
     322             if (null != idxRaf) {
     323                 try {
     324                     idxRaf.close();
     325                 } catch (IOException e) {
     326                     System.out.println(e.getMessage());
     327                 }
     328             }
     329         }
     330 
     331         return rtn;
     332     }
     333 
     334     /**
     335      * 创建PersonID索引(时间太长,待进一步优化)
     336      */
     337     public static void createPersonIdIndex() {
     338 
     339         long totalStartTime = System.currentTimeMillis();
     340 
     341         // 1)做一个PersonId,数量的文件;
     342         System.out.println("开始创建临时文件");
     343         long startTime = System.currentTimeMillis();
     344 
     345         createTempFile();
     346 
     347         long endTime = System.currentTimeMillis();
     348         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     349         System.out.println("临时文件创建完成");
     350 
     351         // 2)做一个空间;
     352         System.out.println("整理临时文件");
     353         startTime = System.currentTimeMillis();
     354 
     355         fillTempFile();
     356 
     357         endTime = System.currentTimeMillis();
     358         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     359         System.out.println("整理临时文件完成");
     360 
     361         // 3)填充空间;
     362         System.out.println("开始创建索引文件");
     363         startTime = System.currentTimeMillis();
     364 
     365         createIdxFile();
     366 
     367         endTime = System.currentTimeMillis();
     368         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     369         System.out.println("索引文件创建完成");
     370 
     371         long totalEndTime = System.currentTimeMillis();
     372         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
     373 
     374     }
     375 
     376     public static void createPersonIdAmontIndex() {
     377 
     378         long totalStartTime = System.currentTimeMillis();
     379 
     380         // 1)做一个PersonId,数量的文件;
     381         System.out.println("开始创建临时文件");
     382         long startTime = System.currentTimeMillis();
     383 
     384         createTempFile();
     385 
     386         long endTime = System.currentTimeMillis();
     387         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     388         System.out.println("临时文件创建完成");
     389 
     390         // 2)做一个空间;
     391         System.out.println("整理临时文件");
     392         startTime = System.currentTimeMillis();
     393 
     394         fillTempFile();
     395 
     396         endTime = System.currentTimeMillis();
     397         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     398         System.out.println("整理临时文件完成");
     399 
     400         // 3)填充空间;
     401         System.out.println("开始创建索引文件");
     402         startTime = System.currentTimeMillis();
     403 
     404         createIdxAmountFile();
     405 
     406         endTime = System.currentTimeMillis();
     407         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     408         System.out.println("索引文件创建完成");
     409 
     410         long totalEndTime = System.currentTimeMillis();
     411         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
     412 
     413     }
     414 
     415     private static void createTempFile() {
     416         File tempFile = new File(CommonFunc.tempPath);
     417         File file = new File(CommonFunc.orderPath);
     418         long fileLen = file.length();
     419         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
     420 
     421         FileChannel tempFileChannel = null;
     422         RandomAccessFile raf = null;
     423         try {
     424             if (tempFile.exists() == true) {
     425                 tempFile.delete();
     426             }
     427             tempFile.createNewFile();
     428 
     429             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
     430             MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, tempFile.length());
     431             raf = new RandomAccessFile(file, "r");
     432             for (long i = 0; i < rowNum; i++) {
     433                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
     434                 byte[] longByte = new byte[8];
     435                 raf.read(longByte);
     436                 long personId = CommonFunc.getLong(longByte);
     437 
     438                 int currentLen = (int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
     439                 if (mappedByteBuffer.capacity() < currentLen + (CommonFunc.tempLen + CommonFunc.tempLen)) {
     440                     mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, currentLen + (CommonFunc.tempLen + CommonFunc.tempLen));
     441                 }
     442 
     443                 mappedByteBuffer.position(currentLen);
     444                 long tempVal = mappedByteBuffer.getLong();
     445 
     446                 tempVal++;
     447                 mappedByteBuffer.position(currentLen);
     448                 mappedByteBuffer.putLong(tempVal);
     449 
     450                 if (i % 1000000 == 0) {
     451                     System.out.println("创建临时文件 " + i);
     452                 }
     453             }
     454         } catch (Exception ex) {
     455             System.out.println(ex.getMessage());
     456         } finally {
     457             if (null != tempFileChannel) {
     458                 try {
     459                     tempFileChannel.close();
     460                 } catch (IOException e) {
     461                     System.out.println(e.getMessage());
     462                 }
     463             }
     464             if (null != raf) {
     465                 try {
     466                     raf.close();
     467                 } catch (IOException e) {
     468                     System.out.println(e.getMessage());
     469                 }
     470             }
     471         }
     472     }
     473 
     474     private static void fillTempFile() {
     475         File tempFile = new File(CommonFunc.tempPath);
     476         long fileLen = tempFile.length();
     477         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
     478 
     479         FileChannel tempFileChannel = null;
     480         try {
     481             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
     482             MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, fileLen);
     483             long preNum = 0;
     484             for (int i = 0; i < rowNum; i++) {
     485                 mappedByteBuffer.position(i * (CommonFunc.tempLen + CommonFunc.tempLen));
     486                 long tempVal = mappedByteBuffer.getLong();
     487 
     488                 if (tempVal <= 0) {
     489                     continue;
     490                 }
     491 
     492                 mappedByteBuffer.position(i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
     493                 mappedByteBuffer.putLong(preNum);
     494                 preNum += tempVal;
     495 
     496                 if (i % 1000000 == 0) {
     497                     System.out.println("填充临时文件 " + i);
     498                 }
     499             }
     500         } catch (Exception ex) {
     501             System.out.println(ex.getMessage());
     502         } finally {
     503             if (null != tempFileChannel) {
     504                 try {
     505                     tempFileChannel.close();
     506                 } catch (IOException e) {
     507                     System.out.println(e.getMessage());
     508                 }
     509             }
     510         }
     511     }
     512 
     513     private static void createIdxFile() {
     514         File file = new File(CommonFunc.orderPath);
     515         File tempFile = new File(CommonFunc.tempPath);
     516         File idxFile = new File(CommonFunc.orderIdxPath);
     517         long fileLen = file.length();
     518         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
     519 
     520         RandomAccessFile raf = null;
     521         FileChannel tempFileChannel = null;
     522         FileChannel idxFileChannel = null;
     523         try {
     524             if (idxFile.exists() == true) {
     525                 idxFile.delete();
     526             }
     527             idxFile.createNewFile();
     528 
     529             raf = new RandomAccessFile(file, "r");
     530             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
     531             idxFileChannel = new RandomAccessFile(idxFile, "rw").getChannel();
     532             MappedByteBuffer tempMappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, tempFile.length());
     533             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, rowNum * (CommonFunc.personIdLen + CommonFunc.orderIdLen));
     534             for (long i = 0; i < rowNum; i++) {
     535                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen));
     536                 byte[] idByte = new byte[CommonFunc.orderIdLen];
     537                 raf.read(idByte);
     538                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
     539                 byte[] personIdByte = new byte[CommonFunc.personIdLen];
     540                 raf.read(personIdByte);
     541                 long personId = CommonFunc.getLong(personIdByte);
     542 
     543                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
     544                 long tempVal = tempMappedByteBuffer.getLong();
     545                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)) + CommonFunc.tempLen);
     546                 long tempPreNum = tempMappedByteBuffer.getLong();
     547                 tempVal--;
     548                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
     549                 tempMappedByteBuffer.putLong(tempVal);
     550                 //写入
     551                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)));
     552                 idxMappedByteBuffer.put(personIdByte);
     553                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)) + CommonFunc.personIdLen);
     554                 idxMappedByteBuffer.put(idByte);
     555 
     556                 if (i % 1000000 == 0) {
     557                     System.out.println("创建索引文件 " + i);
     558                 }
     559             }
     560         } catch (Exception ex) {
     561             System.out.println(ex.getMessage());
     562         } finally {
     563             if (null != raf) {
     564                 try {
     565                     raf.close();
     566                 } catch (IOException e) {
     567                     System.out.println(e.getMessage());
     568                 }
     569             }
     570             if (null != tempFileChannel) {
     571                 try {
     572                     tempFileChannel.close();
     573                 } catch (IOException e) {
     574                     System.out.println(e.getMessage());
     575                 }
     576             }
     577             if (null != idxFileChannel) {
     578                 try {
     579                     idxFileChannel.close();
     580                 } catch (IOException e) {
     581                     System.out.println(e.getMessage());
     582                 }
     583             }
     584         }
     585     }
     586 
     587     private static void createIdxAmountFile() {
     588         File file = new File(CommonFunc.orderPath);
     589         File tempFile = new File(CommonFunc.tempPath);
     590         File idxFile = new File(CommonFunc.orderIdxAmountPath);
     591         long fileLen = file.length();
     592         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
     593 
     594         RandomAccessFile raf = null;
     595         FileChannel tempFileChannel = null;
     596         FileChannel idxFileChannel = null;
     597         try {
     598             if (idxFile.exists() == true) {
     599                 idxFile.delete();
     600             }
     601             idxFile.createNewFile();
     602 
     603             raf = new RandomAccessFile(file, "r");
     604             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
     605             idxFileChannel = new RandomAccessFile(idxFile, "rw").getChannel();
     606             MappedByteBuffer tempMappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, tempFile.length());
     607             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, rowNum * (CommonFunc.personIdLen + CommonFunc.orderIdLen));
     608             for (long i = 0; i < rowNum; i++) {
     609                 //raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen));
     610                 //byte[] idByte = new byte[CommonFunc.orderIdLen];
     611                 //raf.read(idByte);
     612                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
     613                 byte[] personIdByte = new byte[CommonFunc.personIdLen];
     614                 raf.read(personIdByte);
     615                 long personId = CommonFunc.getLong(personIdByte);
     616                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen + CommonFunc.personIdLen);
     617                 double amount = raf.readDouble();
     618 
     619                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
     620                 long tempVal = tempMappedByteBuffer.getLong();
     621                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)) + CommonFunc.tempLen);
     622                 long tempPreNum = tempMappedByteBuffer.getLong();
     623                 tempVal--;
     624                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
     625                 tempMappedByteBuffer.putLong(tempVal);
     626                 //写入
     627                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)));
     628                 idxMappedByteBuffer.put(personIdByte);
     629                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)) + CommonFunc.personIdLen);
     630                 idxMappedByteBuffer.putDouble(amount);
     631 
     632                 if (i % 1000000 == 0) {
     633                     System.out.println("创建索引文件 " + i);
     634                 }
     635             }
     636         } catch (Exception ex) {
     637             System.out.println(ex.getMessage());
     638         } finally {
     639             if (null != raf) {
     640                 try {
     641                     raf.close();
     642                 } catch (IOException e) {
     643                     System.out.println(e.getMessage());
     644                 }
     645             }
     646             if (null != tempFileChannel) {
     647                 try {
     648                     tempFileChannel.close();
     649                 } catch (IOException e) {
     650                     System.out.println(e.getMessage());
     651                 }
     652             }
     653             if (null != idxFileChannel) {
     654                 try {
     655                     idxFileChannel.close();
     656                 } catch (IOException e) {
     657                     System.out.println(e.getMessage());
     658                 }
     659             }
     660         }
     661     }
     662 
     663     //以下为测试用方法
     664 
     665     //标准数据:一亿条
     666     public static void createData() {
     667 
     668         File file = new File(CommonFunc.orderPath);
     669         if (file.exists() == true) {
     670             file.delete();
     671         }
     672         try {
     673             file.createNewFile();
     674         } catch (IOException e) {
     675             System.out.println(e.getMessage());
     676         }
     677 
     678         long totalStartTime = System.currentTimeMillis();
     679         for (int j = 0; j < 100; j++) {
     680 
     681             long startTime = System.currentTimeMillis();
     682             Order[] orderList = new Order[1000000];
     683             for (int i = (j * 1000000); i < (j * 1000000 + 1000000); i++) {
     684 
     685                 Order order = new Order();
     686                 orderList[i - (j * 1000000)] = order;
     687 
     688                 order.setOrderId(i + 1);
     689                 order.setPersonId(CommonFunc.getRandomPersonId(10000000));
     690                 order.setAmount(CommonFunc.getRandomDouble(10000));
     691             }
     692 
     693             OrderOperate.batchInsert(orderList);
     694             long endTime = System.currentTimeMillis();
     695             System.out.println("程序第 " + j + " 次运行时间: " + (endTime - startTime) + "ms");
     696         }
     697         long totalEndTime = System.currentTimeMillis();
     698         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
     699 
     700     }
     701 
     702     //测试数据:10条
     703     public static void createTestData() {
     704 
     705         File file = new File(CommonFunc.orderPath);
     706         if (file.exists() == true) {
     707             file.delete();
     708         }
     709         try {
     710             file.createNewFile();
     711         } catch (IOException e) {
     712             System.out.println(e.getMessage());
     713         }
     714 
     715         long totalStartTime = System.currentTimeMillis();
     716         for (int j = 0; j < 1; j++) {
     717 
     718             long startTime = System.currentTimeMillis();
     719             Order[] orderList = new Order[10];
     720             for (int i = (j * 10); i < (j * 10 + 10); i++) {
     721 
     722                 Order order = new Order();
     723                 orderList[i - (j * 10)] = order;
     724 
     725                 order.setOrderId(i + 1);
     726                 order.setPersonId(CommonFunc.getRandomPersonId(3));
     727                 order.setAmount(CommonFunc.getRandomDouble(10000));
     728             }
     729 
     730             OrderOperate.batchInsert(orderList);
     731             long endTime = System.currentTimeMillis();
     732             System.out.println("程序第 " + j + " 次运行时间: " + (endTime - startTime) + "ms");
     733         }
     734         long totalEndTime = System.currentTimeMillis();
     735         System.out.println("程序运行总时间: " + (totalEndTime - totalStartTime) + "ms");
     736 
     737     }
     738 
     739     public static void printOrderData() {
     740 
     741         List<Order> rtn = new ArrayList<>();
     742 
     743         File file = new File(CommonFunc.orderPath);
     744         long fileLen = file.length();
     745         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
     746 
     747         RandomAccessFile raf = null;
     748         try {
     749             raf = new RandomAccessFile(file, "r");
     750             for (long i = 0; i < rowNum; i++) {
     751                 Order order = new Order();
     752                 rtn.add(order);
     753 
     754                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen));
     755                 order.setOrderId(raf.readLong());
     756 
     757                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
     758                 byte[] personIdByte = new byte[CommonFunc.personIdLen];
     759                 raf.read(personIdByte);
     760                 order.setPersonId(CommonFunc.getLong(personIdByte));
     761 
     762                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + (CommonFunc.orderIdLen + CommonFunc.personIdLen));
     763                 byte[] amountLenByte = new byte[CommonFunc.amountLen];
     764                 raf.read(amountLenByte);
     765                 order.setAmount(CommonFunc.getDouble(amountLenByte));
     766 
     767                 System.out.println("ID:" + order.getOrderId() + " PersonId:" + order.getPersonId() + " Amount:" + order.getAmount());
     768             }
     769         } catch (Exception ex) {
     770             System.out.println(ex.getMessage());
     771         } finally {
     772             if (null != raf) {
     773                 try {
     774                     raf.close();
     775                 } catch (IOException e) {
     776                     System.out.println(e.getMessage());
     777                 }
     778             }
     779         }
     780     }
     781 
     782     public static void printTempData() {
     783 
     784         File tempFile = new File(CommonFunc.tempPath);
     785         long fileLen = tempFile.length();
     786         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
     787 
     788         RandomAccessFile tempRaf = null;
     789         try {
     790             tempRaf = new RandomAccessFile(tempFile, "rw");
     791             for (int i = 0; i < rowNum; i++) {
     792                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen));
     793                 byte[] longByte = new byte[8];
     794                 tempRaf.read(longByte);
     795                 long tempVal = CommonFunc.getLong(longByte);
     796 
     797                 if (tempVal <= 0) {
     798                     continue;
     799                 }
     800 
     801                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
     802                 longByte = new byte[8];
     803                 tempRaf.read(longByte);
     804                 long tempPreNum = CommonFunc.getLong(longByte);
     805 
     806                 System.out.println("PersonId : " + (i + 1) + " ; Num : " + tempVal + " ; PreNum : " + tempPreNum);
     807             }
     808         } catch (Exception ex) {
     809             System.out.println(ex.getMessage());
     810         } finally {
     811             if (null != tempRaf) {
     812                 try {
     813                     tempRaf.close();
     814                 } catch (IOException e) {
     815                     System.out.println(e.getMessage());
     816                 }
     817             }
     818         }
     819     }
     820 
     821     public static void printIdxData() {
     822 
     823         File tempFile = new File(CommonFunc.orderIdxPath);
     824         long fileLen = tempFile.length();
     825         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen);
     826 
     827         RandomAccessFile tempRaf = null;
     828         try {
     829             tempRaf = new RandomAccessFile(tempFile, "rw");
     830             for (long i = 0; i < rowNum; i++) {
     831 
     832                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen));
     833                 byte[] longByte = new byte[CommonFunc.personIdLen];
     834                 tempRaf.read(longByte);
     835                 long personid = CommonFunc.getLong(longByte);
     836 
     837                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen) + CommonFunc.personIdLen);
     838                 longByte = new byte[CommonFunc.orderIdLen];
     839                 tempRaf.read(longByte);
     840                 long orderid = CommonFunc.getLong(longByte);
     841 
     842                 System.out.println("idx : " + i + " ; personid : " + personid + " ; orderid : " + orderid);
     843             }
     844         } catch (Exception ex) {
     845             System.out.println(ex.getMessage());
     846         } finally {
     847             if (null != tempRaf) {
     848                 try {
     849                     tempRaf.close();
     850                 } catch (IOException e) {
     851                     System.out.println(e.getMessage());
     852                 }
     853             }
     854         }
     855     }
     856 
     857     public static void printIdxAmountData() {
     858 
     859         File tempFile = new File(CommonFunc.orderIdxAmountPath);
     860         long fileLen = tempFile.length();
     861         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen);
     862 
     863         RandomAccessFile tempRaf = null;
     864         try {
     865             tempRaf = new RandomAccessFile(tempFile, "rw");
     866             for (long i = 0; i < rowNum; i++) {
     867 
     868                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen));
     869                 byte[] longByte = new byte[CommonFunc.personIdLen];
     870                 tempRaf.read(longByte);
     871                 long personid = CommonFunc.getLong(longByte);
     872 
     873                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen) + CommonFunc.personIdLen);
     874                 double amount = tempRaf.readDouble();
     875 
     876                 System.out.println("idx : " + i + " ; personid : " + personid + " ; amount : " + amount);
     877             }
     878         } catch (Exception ex) {
     879             System.out.println(ex.getMessage());
     880         } finally {
     881             if (null != tempRaf) {
     882                 try {
     883                     tempRaf.close();
     884                 } catch (IOException e) {
     885                     System.out.println(e.getMessage());
     886                 }
     887             }
     888         }
     889     }
     890 
     891     public static void printTop1000SumData(int totalNum, boolean desc) {
     892 
     893         long startTime = System.currentTimeMillis();
     894 
     895         List<OrderQuery> rtn = getOrderQueryList(totalNum, desc);
     896 
     897         long endTime = System.currentTimeMillis();
     898 
     899         for (OrderQuery orderQuery : rtn) {
     900             System.out.println("姓名:" + orderQuery.getPersonName() + " 数量:" + orderQuery.getOrderCount() + " 总价:" + orderQuery.getOrderSum());
     901         }
     902         System.out.println("共查出 " + rtn.size() + " 条");
     903         System.out.println("程序运行时间: " + (endTime - startTime) + "ms");
     904     }
     905 
     906     private static List<OrderQuery> getOrderQueryList(int totalNum, boolean desc) {
     907 
     908         List<OrderQuery> rtn = new ArrayList<>();
     909 
     910         File idxFile = new File(CommonFunc.orderIdxAmountPath);
     911         long fileLen = idxFile.length();
     912         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
     913         File psnFile = new File(CommonFunc.psnPath);
     914 
     915         FileChannel idxFileChannel = null;
     916         FileChannel psnFileChannel = null;
     917         try {
     918             idxFileChannel = new RandomAccessFile(idxFile, "r").getChannel();
     919             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, idxFile.length());
     920 
     921             psnFileChannel = new RandomAccessFile(psnFile, "r").getChannel();
     922             MappedByteBuffer psnMappedByteBuffer = psnFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, psnFile.length());
     923 
     924             OrderQuery orderQuery = null;
     925             long prePersonId = -1;
     926             for (int i = 0; i < rowNum; i++) {
     927                 long personId = idxMappedByteBuffer.getLong();
     928                 double amount = idxMappedByteBuffer.getDouble();
     929 
     930                 if (prePersonId == personId) {
     931                     orderQuery.setOrderCount(orderQuery.getOrderCount() + 1);
     932                     orderQuery.setOrderSum(orderQuery.getOrderSum() + amount);
     933                 } else {
     934                     //处理完的数据,添加入列表
     935                     if (orderQuery != null) {
     936                         insertOrderQueryList(rtn, orderQuery, totalNum, desc);
     937                     }
     938 
     939                     //获取Person姓名
     940                     int psnNamePos = (int) (personId - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen;
     941                     psnMappedByteBuffer.position(psnNamePos);
     942                     byte[] psnNameByte = new byte[CommonFunc.nameLen];
     943                     psnMappedByteBuffer.get(psnNameByte);
     944                     String personName = CommonFunc.getString(psnNameByte);
     945                     //设置对象
     946                     orderQuery = new OrderQuery();
     947                     orderQuery.setPersonName(personName);
     948                     orderQuery.setOrderCount(1);
     949                     orderQuery.setOrderSum(amount);
     950 
     951                     prePersonId = personId;
     952                 }
     953 
     954                 if (i % 1000000 == 0) {
     955                     System.out.println("处理条数 " + i);
     956                 }
     957             }
     958 
     959             //添加最后一条数据
     960             if (orderQuery != null) {
     961                 insertOrderQueryList(rtn, orderQuery, totalNum, desc);
     962             }
     963 
     964         } catch (Exception ex) {
     965             System.out.println(ex.getMessage());
     966         } finally {
     967             if (null != idxFileChannel) {
     968                 try {
     969                     idxFileChannel.close();
     970                 } catch (IOException e) {
     971                     System.out.println(e.getMessage());
     972                 }
     973             }
     974             if (null != psnFileChannel) {
     975                 try {
     976                     psnFileChannel.close();
     977                 } catch (IOException e) {
     978                     System.out.println(e.getMessage());
     979                 }
     980             }
     981         }
     982 
     983         return rtn;
     984     }
     985 
     986     private static void insertOrderQueryList(List<OrderQuery> list, OrderQuery orderQuery, int listMaxNum, boolean desc) {
     987 
     988         int insertIdx = desc ? getInsertDescIndex(list, orderQuery) : getInsertIndex(list, orderQuery);
     989         if (insertIdx < listMaxNum || listMaxNum <= 0) {
     990             list.add(insertIdx, orderQuery);
     991         }
     992 
     993         if (list.size() > listMaxNum && listMaxNum > 0) {
     994             for (int i = list.size() - 1; i >= listMaxNum; i--) {
     995                 list.remove(i);
     996 
     997             }
     998         }
     999     }
    1000 
    1001     private static int getInsertIndex(List<OrderQuery> list, OrderQuery orderQuery) {
    1002         int rtn = -1;
    1003 
    1004         //处理边缘点
    1005         if (list.size() <= 0 || list.get(0).getOrderSum() > orderQuery.getOrderSum()) {
    1006             return 0;
    1007         }
    1008         if (list.get(list.size() - 1).getOrderSum() <= orderQuery.getOrderSum()) {
    1009             return list.size();
    1010         }
    1011 
    1012         //常规处理
    1013         int minIdx = 0;
    1014         int maxIdx = list.size() - 1;
    1015         while (maxIdx > minIdx) {
    1016             //中间数据处理
    1017             if (minIdx + 1 == maxIdx) {
    1018                 rtn = maxIdx;
    1019                 break;
    1020             }
    1021 
    1022             int midIdx = (minIdx + maxIdx) / 2;
    1023             double currentNum = list.get(midIdx).getOrderSum();
    1024             if (currentNum > orderQuery.getOrderSum()) {
    1025                 maxIdx = midIdx;
    1026             } else if (currentNum < orderQuery.getOrderSum()) {
    1027                 minIdx = midIdx;
    1028             } else {
    1029                 rtn = midIdx + 1;
    1030                 break;
    1031             }
    1032         }
    1033 
    1034         return rtn;
    1035     }
    1036 
    1037     private static int getInsertDescIndex(List<OrderQuery> list, OrderQuery orderQuery) {
    1038         int rtn = -1;
    1039 
    1040         //处理边缘点
    1041         if (list.size() <= 0 || list.get(0).getOrderSum() < orderQuery.getOrderSum()) {
    1042             return 0;
    1043         }
    1044         if (list.get(list.size() - 1).getOrderSum() >= orderQuery.getOrderSum()) {
    1045             return list.size();
    1046         }
    1047 
    1048         //常规处理
    1049         int minIdx = 0;
    1050         int maxIdx = list.size() - 1;
    1051         while (maxIdx > minIdx) {
    1052             //中间数据处理
    1053             if (minIdx + 1 == maxIdx) {
    1054                 rtn = maxIdx;
    1055                 break;
    1056             }
    1057 
    1058             int midIdx = (minIdx + maxIdx) / 2;
    1059             double currentNum = list.get(midIdx).getOrderSum();
    1060             if (currentNum > orderQuery.getOrderSum()) {
    1061                 minIdx = midIdx;
    1062             } else if (currentNum < orderQuery.getOrderSum()) {
    1063                 maxIdx = midIdx;
    1064             } else {
    1065                 rtn = midIdx + 1;
    1066                 break;
    1067             }
    1068         }
    1069 
    1070         return rtn;
    1071     }
    1072 
    1073 }
    View Code

    CommonFunc.java

      1 import java.nio.charset.Charset;
      2 import java.util.Random;
      3 
      4 public class CommonFunc {
      5 
      6     //private static String dbPath = "\\192.168.1.163\Theme\zuoye\order.nk";
      7 
      8 //    public static String psnPath = "D:\test\person.nk";
      9 //    public static String psnIdxPath = "D:\test\person.nk.name";
     10 //    public static String orderPath = "D:\test\order.nk";
     11 //    public static String orderIdxPath = "D:\test\order.nk.personid";
     12 //    public static String orderIdxAmountPath = "D:\test\order.nk.amount";
     13 //    public static String tempPath = "D:\test\temp.nk";
     14 
     15     public static String psnPath = "\\192.168.1.163\Theme\zuoye\person.nk";
     16     public static String psnIdxPath = "\\192.168.1.163\Theme\zuoye\person.nk.name";
     17     public static String orderPath = "\\192.168.1.163\Theme\zuoye\order.nk";
     18     public static String orderIdxPath = "\\192.168.1.163\Theme\zuoye\order.nk.personid";
     19     public static String orderIdxAmountPath = "\\192.168.1.163\Theme\zuoye\order.nk.amount";
     20     public static String tempPath = "\\192.168.1.163\Theme\zuoye\temp.nk";
     21 
     22     public static int psnIdLen = 8;
     23     public static int nameLen = 12;
     24     public static int orderIdLen = 8;
     25     public static int personIdLen = 8;
     26     public static int amountLen = 8;
     27     public static int tempLen = 8;
     28 
     29     private static String firstName = "赵钱孙李周吴郑王冯陈褚卫蒋沈韩杨朱秦尤许何吕施张孔曹严华金魏陶姜戚谢邹喻柏水窦章云苏潘葛奚范彭郎鲁韦昌马苗凤花方俞任袁柳酆鲍史唐费廉岑薛雷贺倪汤滕殷罗毕郝邬安常乐于时傅皮卞齐康伍余元卜顾孟平黄和穆萧尹姚邵湛汪祁毛禹狄米贝明臧计伏成戴谈宋茅庞熊纪舒屈项祝董梁杜阮蓝闵席季麻强贾路娄危江童颜郭梅盛林刁钟徐邱骆高夏蔡田樊胡凌霍虞万支柯咎管卢莫经房裘缪干解应宗宣丁贲邓郁单杭洪包诸左石崔吉钮龚程嵇邢滑裴陆荣翁荀羊於惠甄魏加封芮羿储靳汲邴糜松井段富巫乌焦巴弓牧隗山谷车侯宓蓬全郗班仰秋仲伊宫宁仇栾暴甘钭厉戎祖武符刘姜詹束龙叶幸司韶郜黎蓟薄印宿白怀蒲台从鄂索咸籍赖卓蔺屠蒙池乔阴郁胥能苍双闻莘党翟谭贡劳逄姬申扶堵冉宰郦雍却璩桑桂濮牛寿通边扈燕冀郏浦尚农温别庄晏柴瞿阎充慕连茹习宦艾鱼容向古易慎戈廖庚终暨居衡步都耿满弘匡国文寇广禄阙东殴殳沃利蔚越夔隆师巩厍聂晁勾敖融冷訾辛阚那简饶空曾毋沙乜养鞠须丰巢关蒯相查后江红游竺权逯盖益桓公万俟司马上官欧阳夏侯诸葛闻人东方赫连皇甫尉迟公羊澹台公冶宗政濮阳淳于仲孙太叔申屠公孙乐正轩辕令狐钟离闾丘长孙慕容鲜于宇文司徒司空亓官司寇仉督子车颛孙端木巫马公西漆雕乐正壤驷公良拓拔夹谷宰父谷粱晋楚阎法汝鄢涂钦段干百里东郭南门呼延归海羊舌微生岳帅缑亢况后有琴梁丘左丘东门西门商牟佘佴伯赏南宫墨哈谯笪年爱阳佟第五言福百家姓续";
     30     private static String girl = "秀娟英华慧巧美娜静淑惠珠翠雅芝玉萍红娥玲芬芳燕彩春菊兰凤洁梅琳素云莲真环雪荣爱妹霞香月莺媛艳瑞凡佳嘉琼勤珍贞莉桂娣叶璧璐娅琦晶妍茜秋珊莎锦黛青倩婷姣婉娴瑾颖露瑶怡婵雁蓓纨仪荷丹蓉眉君琴蕊薇菁梦岚苑婕馨瑗琰韵融园艺咏卿聪澜纯毓悦昭冰爽琬茗羽希宁欣飘育滢馥筠柔竹霭凝晓欢霄枫芸菲寒伊亚宜可姬舒影荔枝思丽 ";
     31     private static String boy = "伟刚勇毅俊峰强军平保东文辉力明永健世广志义兴良海山仁波宁贵福生龙元全国胜学祥才发武新利清飞彬富顺信子杰涛昌成康星光天达安岩中茂进林有坚和彪博诚先敬震振壮会思群豪心邦承乐绍功松善厚庆磊民友裕河哲江超浩亮政谦亨奇固之轮翰朗伯宏言若鸣朋斌梁栋维启克伦翔旭鹏泽晨辰士以建家致树炎德行时泰盛雄琛钧冠策腾楠榕风航弘";
     32 
     33     /**
     34      * 随机获取中文名字
     35      *
     36      * @return
     37      */
     38     public static String getPersonName() {
     39 
     40         String nameStr = getRandomNum(2) == 1 ? boy : girl;
     41 
     42         int firstRandom = getRandomNum(firstName.length() - 1);
     43         String first = firstName.substring(firstRandom, firstRandom + 1);
     44         int secondRandom = getRandomNum(nameStr.length() - 1);
     45         String second = nameStr.substring(secondRandom, secondRandom + 1);
     46         int thridRandom = getRandomNum(nameStr.length() - 1);
     47         String third = nameStr.substring(thridRandom, thridRandom + 1);
     48 
     49         return first.concat(second).concat(third);
     50     }
     51 
     52     /**
     53      * 随机获取PersonID
     54      *
     55      * @return
     56      */
     57     public static long getRandomPersonId(int max) {
     58         return getRandomNum(max) + 1;
     59     }
     60 
     61     /**
     62      * 获取随机数,不包含传入的最大数
     63      *
     64      * @param maxNum
     65      * @return
     66      */
     67     public static int getRandomNum(int maxNum) {
     68         return (int) (Math.random() * maxNum);
     69     }
     70 
     71     /**
     72      * 获取Double类型的随机数
     73      *
     74      * @param maxNum
     75      * @return
     76      */
     77     public static double getRandomDouble(int maxNum) {
     78         Random randomno = new Random();
     79         return randomno.nextDouble() * maxNum;
     80     }
     81 
     82     public static byte[] getBytes(long data) {
     83         byte[] bytes = new byte[8];
     84         bytes[7] = (byte) (data & 0xff);
     85         bytes[6] = (byte) ((data >> 8) & 0xff);
     86         bytes[5] = (byte) ((data >> 16) & 0xff);
     87         bytes[4] = (byte) ((data >> 24) & 0xff);
     88         bytes[3] = (byte) ((data >> 32) & 0xff);
     89         bytes[2] = (byte) ((data >> 40) & 0xff);
     90         bytes[1] = (byte) ((data >> 48) & 0xff);
     91         bytes[0] = (byte) ((data >> 56) & 0xff);
     92         return bytes;
     93     }
     94 
     95     public static byte[] getBytes(int data) {
     96         byte[] bytes = new byte[4];
     97         bytes[3] = (byte) (data & 0xff);
     98         bytes[2] = (byte) ((data & 0xff00) >> 8);
     99         bytes[1] = (byte) ((data & 0xff0000) >> 16);
    100         bytes[0] = (byte) ((data & 0xff000000) >> 24);
    101         return bytes;
    102     }
    103 
    104     public static byte[] getBytes(double data) {
    105         long intBits = Double.doubleToLongBits(data);
    106         return getBytes(intBits);
    107     }
    108 
    109     public static byte[] getBytes(String data, int len) {
    110         return getBytes(data, len, "UTF-8");
    111     }
    112 
    113     public static byte[] getBytes(String data, int len, String charsetName) {
    114         byte[] rtn = new byte[len];
    115 
    116         Charset charset = Charset.forName(charsetName);
    117         byte[] strByte = data.getBytes(charset);
    118         //取小的数据
    119         int copyLen = strByte.length < len ? strByte.length : len;
    120         System.arraycopy(strByte, 0, rtn, 0, copyLen);
    121 
    122         return rtn;
    123     }
    124 
    125     public static long getLong(byte[] bytes) {
    126         return (0xffL & (long) bytes[7]) |
    127                 (0xff00L & ((long) bytes[6] << 8)) |
    128                 (0xff0000L & ((long) bytes[5] << 16)) |
    129                 (0xff000000L & ((long) bytes[4] << 24)) |
    130                 (0xff00000000L & ((long) bytes[3] << 32)) |
    131                 (0xff0000000000L & ((long) bytes[2] << 40)) |
    132                 (0xff000000000000L & ((long) bytes[1] << 48)) |
    133                 (0xff00000000000000L & ((long) bytes[0] << 56));
    134     }
    135 
    136     public static int getInt(byte[] bytes) {
    137         return (0xff & bytes[3]) |
    138                 (0xff00 & (bytes[2] << 8)) |
    139                 (0xff0000 & (bytes[1] << 16)) |
    140                 (0xff000000 & (bytes[0] << 24));
    141     }
    142 
    143     public static double getDouble(byte[] bytes) {
    144         long l = getLong(bytes);
    145         return Double.longBitsToDouble(l);
    146     }
    147 
    148     public static String getString(byte[] bytes) {
    149         return getString(bytes, "UTF-8");
    150     }
    151 
    152     public static String getString(byte[] bytes, String charsetName) {
    153         Charset charset = Charset.forName(charsetName);
    154         return new String(bytes, charset).trim();
    155     }
    156 }
    View Code

    OrderQuery.java

     1 public class OrderQuery {
     2 
     3     private String personName;
     4     private int OrderCount;
     5     private double OrderSum;
     6 
     7     public String getPersonName() {
     8         return personName;
     9     }
    10 
    11     public void setPersonName(String personName) {
    12         this.personName = personName;
    13     }
    14 
    15     public int getOrderCount() {
    16         return OrderCount;
    17     }
    18 
    19     public void setOrderCount(int orderCount) {
    20         OrderCount = orderCount;
    21     }
    22 
    23     public double getOrderSum() {
    24         return OrderSum;
    25     }
    26 
    27     public void setOrderSum(double orderSum) {
    28         OrderSum = orderSum;
    29     }
    30 }
    View Code

    Person.java

     1 public class Person {
     2 
     3     private long personId;
     4     private String name;
     5 
     6     public long getPersonId() {
     7         return personId;
     8     }
     9 
    10     public void setPersonId(long personId) {
    11         this.personId = personId;
    12     }
    13 
    14     public String getName() {
    15         return name;
    16     }
    17 
    18     public void setName(String name) {
    19         this.name = name;
    20     }
    21 }
    View Code

    Order.java

     1 public class Order {
     2 
     3     private long orderId;
     4     private long personId;
     5     private double amount;
     6 
     7     public long getOrderId() {
     8         return orderId;
     9     }
    10 
    11     public void setOrderId(long orderId) {
    12         this.orderId = orderId;
    13     }
    14 
    15     public long getPersonId() {
    16         return personId;
    17     }
    18 
    19     public void setPersonId(long personId) {
    20         this.personId = personId;
    21     }
    22 
    23     public double getAmount() {
    24         return amount;
    25     }
    26 
    27     public void setAmount(double amount) {
    28         this.amount = amount;
    29     }
    30 }
    View Code
  • 相关阅读:
    2018.09.08什么是ajax
    2018.09.03怎样让网页自适应所有屏幕宽度
    2018.08.25字符串和二维数组之间的转换
    2018.08.20MySQL常用命令总结(二)
    2018.08.15解决MySQL1290问题
    2018.08.13MySQL常用命令总结(一)
    2018.08.11MySQL无法启动错误码1067的解决方法
    2018.08.10 css中position定位问题
    2018.08.10jQuery导航栏置顶
    2018.08.07css实现图片放大
  • 原文地址:https://www.cnblogs.com/sshoub/p/4865418.html
Copyright © 2020-2023  润新知