Ceph相关

Ceph基础知识和基础架构简介

http://www.xuxiaopang.com/2020/10/09/list/#more大话Ceph

http://www.xuxiaopang.com/2016/11/08/easy-ceph-CRUSH/ 大话Ceph CRUSH算法

在Ceph中一切数据（图片、视频，音频、文本、字符串等）都看成个对象来存储（读取其二进制流存储），而不以其格式来区分它们。

架构：

RADOS：对象存储。底层存储系统，最常用的存储方式

LIBRADOS：各种语言的客户端库

RADOSGW、RBD、CEPH FS：基于RADOS和LIBRADOS进一步实现的 Web API库、块存储、文件系统

可见，提供三种存储方式：对象存储、块存储、文件系统存储

其他概念：

pool（逻辑上，对象存储池）

pg（逻辑上，placement group对象归置组）

osd（物理上，object storage device可理解为一个硬盘，一台主机里可能有多个）

pool、pg是逻辑上的概念，起到namespace的作用，以对数据进行分区，在创建一个pool时指定pg数。

 

数据存储位置计算：

存储一个对象时毋庸置疑需要指定对象名(如 objectKey)，此外也指定了所属的pool，要做的就是确定对象最终应存在哪个OSD上。

计算过程可由服务端完成，但是这样的话任何一个对象存储时都需要服务端计算存储位置，服务端压力会大。一个合格的分布式存储系统肯定应将此计算任务交由客户端来完成。

计算过程主要包括两部分（PS：数据分布原理很简单，官方吹嘘时总爱故弄玄虚。万变不离其宗，数据分布原理与读硕时所做时空数据索引系统的本质上大同小异）：

1、由对象的pool和objectKey确定所属的pg，Ceph用hash实现（pg数改了后重hash的问题？由于创建一个pool时指定了pool数，相当于pg数固定所以不用考虑重hash？）

2、确定一个pg映射到哪个或哪些osd上，对应到多个是为了冗余存储提高可靠性。Ceph采用CRUSH算法，本质上就是一种伪随机选择的过程：

对于一个pg：

a、CRUSH_HASH( PG_ID, OSD_ID, r ) ===> draw ：其和每个osd分别确定一个随机数，称为draw

b、( draw &0xffff ) * osd_weight ===> osd_straw ：各osd的权重（该osd以T为单位的存储容量值）乘各自的draw得到一个值，称为straw

c、pick up high_osd_straw ：选straw最大的osd作为pg应存入的osd

这里第一步中的 r 为一个常数，用于通过调节之来为同一个pg对应到多个osd上，如分别为0、1、2等。

原理图如下：

 利用Ceph实现一个网盘：

package com.marchon.sensestudy.web;

import java.io.BufferedOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.io.InputStreamResource;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.DeleteObjectsResult;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.marchon.sensestudy.common.config.ConfigParam;
import com.marchon.sensestudy.common.utils.CephClientUtils;
import com.marchon.sensestudy.common.utils.ControllerUtils;
import com.marchon.sensestudy.responsewrapper.ApiCustomException;
import com.marchon.sensestudy.responsewrapper.ApiErrorCode;

/**
 * 用户存储空间管理（网盘功能）。存储位于Ceph上，Ceph本身是key-value存储，为模拟文件系统目录，这里key采用层级路径形式：${userId}+分隔符+${absolutePath}。<br>
 * 可在读或写时模拟文件系统层级关系，若 读维护则增删快查改慢、写维护则与之相反。这里在读时维护<br>
 * <br>
 * note：<br>
 * 1. 增、改需要校验文件(夹)名有效性，查、删不需要。文件或目录名不能包含/等；<br>
 * 2. 在Ceph上并没有目录概念，也不存储目录，所有数据都是以key-value存储的，这里key命名时采用层级路径形式如/a/b/1.jpg、/a/b/2、/a/b/3/，展现给用户的文件夹或文件列表通过解析层级路径来获取。<br>
 * 为区分Ceph中一个key代表的是用户的文件还是文件夹，此以最后一层级后是否带"/"来区分：带与不带分别表示用户的文件夹和文件。可见：<br>
 * <li><b>用户在"/a/b/c"下可看到文件夹"d"的充要条件是Ceph中存在以"/a/b/c/d/"为前缀的key(注意d后的"/"不可少)</b></li> <br>
 * <li>用户看到的文件夹"b"可能来源于Ceph中的两种形式：实际创建的文件夹(在层级末尾，即如key="/a/b/")和虚拟文件夹(在层级中间，如key="/a/b/c/"或"/a/b/c.jpg")。两者可能在Ceph中同时存在，如用户创建一个文件夹再往其中传文件时</li>
 * 3. 写(上传文件、创建文件夹、重命名)时要解决重名问题，确保存储的：同级"目录"下文件之间、文件夹之间、文件和文件夹之间都不重名。解决：加数字后缀（传文件或文件夹时）、抛错（重命名时）<br>
 * <br>
 * TODO 涉及到组合操作，如重命名时复制原数据到新数据然后删除原数据，如何确保原子性?
 */
@RestController
public class AccountStorageController {// 操作：查(文件信息、文件下载)、删、增(文件夹、文件)、改、存储空间大小(查、改)
    private static final Logger logger = LoggerFactory.getLogger(AccountStorageController.class);
    private static final String SEPERATOR_FOR_KEY = "/";

    @Autowired
    private ControllerUtils controllerUtils;

    // 1. 以下为网盘CRUD相关
    /** 查。模拟文件系统目录形式展现Ceph上的文件。可通过Ceph的带delimiter的listObjects获取到当前目录下文件和文件夹列表，但此时文件夹没有大小、修改时间等信息，故弃之自实现 */
    @GetMapping("/api/v1/account/storage/entries")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public List<EntryItem> getFileOrFolderList(@RequestParam("parentDirAbsPath") String parentDirAbsPath,
            HttpServletRequest request) {
        parentDirAbsPath = parentDirAbsPath.endsWith(SEPERATOR_FOR_KEY) ? parentDirAbsPath
                : (parentDirAbsPath + SEPERATOR_FOR_KEY);// 确保以分隔符结尾，否则查询结果会有false
        // positive
        String userId = controllerUtils.getUserIdFromJwt(request);
        String objKeyPrefix = getJoinedObjKey(userId, parentDirAbsPath);

        // 查询，获取key以objKeyPrefix为前缀的所有对象信息
        List<S3ObjectSummary> listRes = CephClientUtils
                .listObjects(ConfigParam.cephBucketName_accountStorage, objKeyPrefix).stream()
                .map(objList -> objList.getObjectSummaries()).flatMap(s3ObjList -> s3ObjList.stream())
                .collect(Collectors.toList());
        // System.out.println(listRes);

        // 提取文件夹名或文件名
        Map<String, EntryItem> res = new HashMap<>();
        for (S3ObjectSummary s3ObjSum : listRes) {
            if (s3ObjSum.getKey().equals(objKeyPrefix)) {
                continue;
            }
            // 获取纯文件名或目录名
            String entryName = s3ObjSum.getKey().substring(objKeyPrefix.length());// 去除前缀
            boolean isFolder = entryName.indexOf(SEPERATOR_FOR_KEY) >= 0;// 对于用户创建的文件夹Ceph存储key时以分隔符结尾
            entryName = isFolder ? entryName.substring(0, entryName.indexOf(SEPERATOR_FOR_KEY)) : entryName;// 得到目录名或文件夹名，不包含任何额外路径信息

            // 获取fileType：folder, file, txt, doc, ...
            int lastIndexOfDot = entryName.lastIndexOf(".");
            String fileType = isFolder ? SpecialFileType.FOLDER.typeName
                    : (lastIndexOfDot >= 0 && lastIndexOfDot < entryName.length() - 1
                            ? entryName.substring(lastIndexOfDot + 1)
                            : SpecialFileType.FILE.typeName);
            // 组织层级信息返回
            EntryItem entryItem = null;
            if (!res.containsKey(entryName)) {// 新建entry
                entryItem = new EntryItem(parentDirAbsPath, entryName, s3ObjSum.getSize(), s3ObjSum.getLastModified(),
                        ConfigParam.cephBucketName_accountStorage + "/" + s3ObjSum.getKey(), fileType);
                res.put(entryName, entryItem);
            } else {// 更新entry，只有文件夹会有此操作
                entryItem = res.get(entryName);
                // 更新数据大小
                entryItem.setByteSize(entryItem.getByteSize() + s3ObjSum.getSize());
                // 更新最后修改时间
                if (s3ObjSum.getLastModified().after(entryItem.getLastModify())) {
                    entryItem.setLastModify(s3ObjSum.getLastModified());
                }
            }
        }
        return res.values().stream().sorted(new Comparator<EntryItem>() {// 排序：文件夹靠前、再先按类型排、再按时间倒排
            private String fileType1, fileType2;

            @Override
            public int compare(EntryItem o1, EntryItem o2) {
                fileType1 = o1.getFileType();
                fileType2 = o2.getFileType();

                // 文件夹靠前排
                if (fileType1.equals(SpecialFileType.FOLDER.typeName)
                        && !fileType2.equals(SpecialFileType.FOLDER.typeName)) {
                    return -1;
                } else if (!fileType1.equals(SpecialFileType.FOLDER.typeName)
                        && fileType2.equals(SpecialFileType.FOLDER.typeName)) {
                    return 1;
                }
                // 同为文件夹 或 同非文件夹：按类型排，类型相同再按时间倒排
                if (fileType1.equals(fileType2)) {
                    return -(o1.getLastModify().compareTo(o2.getLastModify()));
                }
                return fileType1.compareTo(fileType2);

            }
        }).collect(Collectors.toList());
    }

    /** 查。下载文件或文件夹。zip压缩使用org.apache.tools.zip库而不是JDK自带zip库，后者继承自前者故常用API几乎一样。 */
    @GetMapping(value = "/api/v1/account/storage/entries/download")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    // @GetMapping(value = WebSecurityConfig.URL_TEST_PUBLIC)
    public ResponseEntity<InputStreamResource> downLoadFiles(@RequestParam("parentDirAbsPath") String parentDirAbsPath,
            @RequestParam("entryNames") List<String> entryNames, HttpServletRequest request,
            HttpServletResponse response) throws IOException {
        // 获取待下载entry的绝对路径
        parentDirAbsPath = parentDirAbsPath.endsWith(SEPERATOR_FOR_KEY) ? parentDirAbsPath
                : parentDirAbsPath + SEPERATOR_FOR_KEY;
        List<String> entryAbsPathes = new ArrayList<>(entryNames.size());
        for (String entryName : entryNames) {
            entryAbsPathes.add(parentDirAbsPath + entryName);
        }

        String userId = controllerUtils.getUserIdFromJwt(request);
        // String userId = "63b3fb3e-a8a7-49bb-8d3b-93517955cf13";
        boolean isDownloadSingleFile = false;

        // 判断是否是单文件下载
        String objKey;
        if (entryAbsPathes.size() == 1) {
            // 检测是否是文件
            String entryAbsPath = entryAbsPathes.get(0);
            objKey = getJoinedObjKey(userId, entryAbsPath);
            if (!entryAbsPath.endsWith(SEPERATOR_FOR_KEY)
                    && CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, objKey)) {// 文件存在
                isDownloadSingleFile = true;
            }
        }

        // 下载
        if (isDownloadSingleFile) {// 单文件，直接下载
            objKey = getJoinedObjKey(userId, entryAbsPathes.get(0));
            if (CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, objKey)) {// 文件存在
                URL url = CephClientUtils.generatePresignedUrl(ConfigParam.cephBucketName_accountStorage, objKey, null);
                // InputStream in = CephClientUtils.getObject(ConfigParam.cephBucketName_accountStorage, objKey)
                // .getObjectContent();// doesn't work, can only get the first byte

                String fileName = objKey.substring(objKey.lastIndexOf(SEPERATOR_FOR_KEY) + 1);
                InputStream in = url.openStream();
                // System.out.println("file " + objKey + " size: " + in.available());
                return ResponseEntity.ok().contentLength(in.available()).contentType(MediaType.APPLICATION_OCTET_STREAM)
                        .header("Content-Disposition", "attachment;filename=" + URLEncoder.encode(fileName, "UTF-8"))
                        .body(new InputStreamResource(in));
            }
        } else if (entryAbsPathes.size() > 0) {// 多文件下载，边压缩边下载
            // 获取指定路径在Ceph中的所有对象key
            Set<String> downloadObjKeys = new HashSet<>();
            for (String entryAbsPath : entryAbsPathes) {
                if (entryAbsPath.trim().equals("")) {
                    continue;
                }
                objKey = getJoinedObjKey(userId, entryAbsPath);
                if (!entryAbsPath.endsWith(SEPERATOR_FOR_KEY)
                        && CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, objKey)) {// 文件存在
                    downloadObjKeys.add(objKey);
                } else {// 当成下载文件夹
                    entryAbsPath = entryAbsPath.endsWith(SEPERATOR_FOR_KEY) ? entryAbsPath
                            : entryAbsPath + SEPERATOR_FOR_KEY;// 确保查询时key以分隔符结尾，防止false
                    // positive
                    objKey = getJoinedObjKey(userId, entryAbsPath);
                    // 列取该文件夹下的所有文件的key
                    List<String> tmpObjKeys = CephClientUtils
                            .listObjects(ConfigParam.cephBucketName_accountStorage, objKey).stream()
                            .map(objList -> objList.getObjectSummaries()).flatMap(s3ObjList -> s3ObjList.stream())
                            .map(e -> e.getKey())//
                            // .filter(key -> !key.endsWith(SEPERATOR_FOR_KEY))// 过滤掉空文件夹
                            .collect(Collectors.toList());
                    downloadObjKeys.addAll(tmpObjKeys);
                }
            }
            // System.out.println(downloadObjKeys);

            // 下载
            response.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);
            response.setHeader("Content-Disposition", "attachment;filename="
                    + URLEncoder.encode(entryNames.size() == 1 ? entryNames.get(0) + ".zip" : "download.zip", "UTF-8"));// 若只有一个文件夹则压缩包直接以文件夹命名

            ZipOutputStream zipos = new ZipOutputStream(new BufferedOutputStream(response.getOutputStream())); // 设置压缩流：直接写入response，实现边压缩边下载
            zipos.setMethod(ZipOutputStream.DEFLATED); // 设置压缩方法
            // zipos.setEncoding("UTF-8");

            logger.info("user {} download {} files", userId, downloadObjKeys.size());
            byte[] buffer = new byte[20 * 1024];
            ZipEntry zipEntry;
            for (String downloadObjKey : downloadObjKeys) {
                // 获取文件流
                URL url = CephClientUtils.generatePresignedUrl(ConfigParam.cephBucketName_accountStorage,
                        downloadObjKey, null);
                InputStream inputStream = url.openStream();
                logger.info("file {} size: {}", downloadObjKey, inputStream.available());

                // 添加ZipEntry
                String fileName = downloadObjKey.substring(userId.length() + 1)// 去除userId前缀，包括分隔符
                        .substring(parentDirAbsPath.length() == 1 ? 0 : parentDirAbsPath.length());// 去除parentDirAbsPath前缀，parentDirAbsPath本身包括分隔符
                zipEntry = new ZipEntry(fileName);
                zipEntry.setComment("generated by marchon");
                zipos.putNextEntry(zipEntry);// 名字带"/"后缀即可添加文件夹如/a/b/会添加文件夹b，/a/b/1.txt也会添加文件夹b。直接打开zip文件可能看不到空文件夹，但解压后即有
                // 传数据
                {
                    int length = 0;
                    while ((length = inputStream.read(buffer)) != -1) {
                        zipos.write(buffer, 0, length);
                    }
                }
                // 关闭流
                inputStream.close();
                zipos.closeEntry();
            }
            // 关闭流
            zipos.close();
        }
        return null;
    }

    /** 删。"*"表示删除当前所在目录下的所有内容 */
    @DeleteMapping("/api/v1/account/storage/entries")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public List<String> deleteFiles(@RequestBody List<EntryItem> entryItems, HttpServletRequest request) {
        String userId = controllerUtils.getUserIdFromJwt(request);

        List<String> deletedEntryNameList = new ArrayList<>();
        for (EntryItem entryItem : entryItems) {
            String parentDirAbsPath = entryItem.getParentDirAbsPath();
            String entryName = entryItem.getEntryName();
            String fileType = entryItem.getFileType();// 用以确定是否是目录

            // 若entryItems元素非空，则每个元素的fileType、entryName必传
            ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM,
                    parentDirAbsPath != null,
                    "Bad Request: Required String parameter 'parentDirAbsPath' is not present");
            ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, entryName != null,
                    "Bad Request: Required String parameter 'entryName' is not present");
            ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, fileType != null,
                    "Bad Request: Required String parameter 'fileType' is not present");

            // 组合获取key
            parentDirAbsPath = parentDirAbsPath.endsWith(SEPERATOR_FOR_KEY) ? parentDirAbsPath
                    : (parentDirAbsPath + SEPERATOR_FOR_KEY);// 确保以分隔符结尾

            // 删除
            String objKeyPrefix;
            if (entryName.trim().equals("*") || fileType.equals(SpecialFileType.FOLDER.typeName)) {// 删除所有 或 删除文件夹
                if (entryName.trim().equals("*")) {// 删除当前目录下的所有文件
                    objKeyPrefix = getJoinedObjKey(userId, parentDirAbsPath);
                } else {
                    entryName = entryName.endsWith(SEPERATOR_FOR_KEY) ? entryName : entryName + SEPERATOR_FOR_KEY;// 若是文件夹则确保以分隔符结尾，避免误删。如若指定文件夹名为code，若不加/则codeExamples文件夹也会被删
                    objKeyPrefix = getJoinedObjKey(userId, parentDirAbsPath + entryName);
                }
                DeleteObjectsResult deletedRes = CephClientUtils
                        .deleteObjects(ConfigParam.cephBucketName_accountStorage, objKeyPrefix);

                List<String> deletedObjPathes = deletedRes.getDeletedObjects().stream().map(ele -> ele.getKey())// 获取key
                        .map(key -> key.substring(userId.length(), key.length()))// 去掉userId前缀和/后缀
                        .collect(Collectors.toList());
                deletedEntryNameList.addAll(deletedObjPathes);
            } else {// 删除文件
                String objKey = getJoinedObjKey(userId, parentDirAbsPath + entryName);
                if (CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, objKey)) {
                    CephClientUtils.deleteObject(ConfigParam.cephBucketName_accountStorage, objKey);
                    deletedEntryNameList.add(objKey.substring(userId.length()));
                }
            }

            // 解决删除后父目录丢失的问题
            {
                // 删除完后对用户所见而言有可能丢了若干层父目录，如对于请求parentDirAbsPath=/a/b、fileType=folder，若b是虚拟文件夹且/a/b/下只有一个entry，则执行删除后，用户就看不到b文件夹了，若a是虚拟文件夹则a用户也看不到了，依之类推。
                // 解决：创建一个parentDirAbsPath key
                objKeyPrefix = getJoinedObjKey(userId, parentDirAbsPath);
                int numOfKeyWithparentDirAbsPathPrefix = CephClientUtils
                        .listObjects(ConfigParam.cephBucketName_accountStorage, objKeyPrefix).stream()
                        .map(objList -> objList.getObjectSummaries()).flatMap(s3ObjList -> s3ObjList.stream())
                        .collect(Collectors.toList()).size();// 以parentDirAbsPath为前缀的对象数
                // System.out.println(numOfKeyWithparentDirAbsPathPrefix);
                if (numOfKeyWithparentDirAbsPathPrefix < 1) {
                    CephClientUtils.writeObject(ConfigParam.cephBucketName_accountStorage, objKeyPrefix, "");
                }
            }
        }

        {// 更新capacity信息
            updateCurrentCapacity(userId);
        }

        return deletedEntryNameList;
    }

    /** 增。创建文件夹。须确保存到Ceph时文件夹名以 / 结尾；已存在该名字的文件或文件夹时加数字后缀 */
    @PostMapping(value = "/api/v1/account/storage/folder", produces = "application/json")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public String createFolder(@RequestBody Map<String, Object> reqMap, HttpServletRequest request) {
        String parentDirAbsPath = (String) reqMap.get("parentDirAbsPath");
        String inputFolderName = (String) reqMap.get("folderName");

        ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, parentDirAbsPath != null,
                "Bad Request: Required String parameter 'parentDirAbsPath' is not present");
        ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, inputFolderName != null,
                "Bad Request: Required String parameter 'inputFolderName' is not present");

        ApiCustomException.assertTrue(ApiErrorCode.ILLEGAL_VALUE, isEntryNameValid(inputFolderName, true),
                "Invalid folder name");

        String userId = controllerUtils.getUserIdFromJwt(request);
        parentDirAbsPath = parentDirAbsPath.endsWith(SEPERATOR_FOR_KEY) ? parentDirAbsPath
                : (parentDirAbsPath + SEPERATOR_FOR_KEY);// 确保以分隔符结尾
        inputFolderName = inputFolderName.endsWith(SEPERATOR_FOR_KEY)
                ? inputFolderName.substring(0, inputFolderName.length() - 1)
                : inputFolderName;// 确保不易/结尾

        // 若当前目录下已存在同名文件夹或文件则加数字后缀
        String savedFolderName = getRenamedEntrynameIfExist(userId, parentDirAbsPath, inputFolderName, true);

        // 创建
        savedFolderName = savedFolderName + SEPERATOR_FOR_KEY;// 存入时确保以分隔符结尾
        String objKey = getJoinedObjKey(userId, parentDirAbsPath + savedFolderName);
        CephClientUtils.writeObject(ConfigParam.cephBucketName_accountStorage, objKey, "");
        logger.info("upload folder to ceph: " + objKey);
        return parentDirAbsPath + savedFolderName;
    }

    /** 增。上传文件。即使文件夹上传获取到的也只是其中的所有文件之。已存在该名字的文件或文件夹时加数字后缀 */
    @PostMapping(value = "/api/v1/account/storage/entries")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public List<String> uploadFiles(@RequestParam("userFile") List<MultipartFile> files,
            @RequestParam("parentDirAbsPath") String parentDirAbsPath, HttpServletRequest request) throws Exception {

        parentDirAbsPath = parentDirAbsPath.endsWith(SEPERATOR_FOR_KEY) ? parentDirAbsPath
                : (parentDirAbsPath + SEPERATOR_FOR_KEY);// 确保以分隔符结尾
        String userId = controllerUtils.getUserIdFromJwt(request);
        List<String> res = new ArrayList<>();
        for (MultipartFile file : files) {
            if (file.isEmpty()) {
                continue;
            }
            ApiCustomException.assertTrue(ApiErrorCode.ILLEGAL_VALUE,
                    isEntryNameValid(file.getOriginalFilename(), false),
                    "Invalid file name '" + file.getOriginalFilename() + "'");// 不能包含/等

            {// TODO 文件过滤
                // if (!file.getOriginalFilename().endsWith(".csv")) {// 类型过滤
                // throw new ApiCustomException(ApiErrorCode.INVALID_FILE_CONTENT, "only support .csv file");
                // }
                // if (file.getSize() > 100) {// 大小过滤
                // throw new ApiCustomException(ApiErrorCode.INVALID_FILE_CONTENT, "file to large");
                // }
            }

            // 若当前目录下已存在同名文件夹或文件则加数字后缀
            String savedEntryName = getRenamedEntrynameIfExist(userId, parentDirAbsPath, file.getOriginalFilename(),
                    false);

            // 存储
            String objKey = getJoinedObjKey(userId, parentDirAbsPath + savedEntryName);

            {// TODO 修改权限，不能全public
                CephClientUtils.writeObject(ConfigParam.cephBucketName_accountStorage, objKey, file,
                        CannedAccessControlList.PublicRead);
                CephClientUtils.generatePublicUrl(ConfigParam.cephBucketName_accountStorage, objKey);
            }
            res.add(parentDirAbsPath + savedEntryName);

            logger.info("upload file to ceph:{}, size:{}", objKey, file.getSize());

        }

        {// 更新capacity信息
            updateCurrentCapacity(userId);
        }
        return res;
    }

    /** 改。新名字的文件夹或文件已存在时抛错 */
    @PutMapping("/api/v1/account/storage/entry")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public List<EntryItem> updateEntryName(@RequestBody Map<String, Object> reqMap, HttpServletRequest request) {
        String parentDirAbsPath = (String) reqMap.get("parentDirAbsPath");
        String fileType = (String) reqMap.get("fileType");
        String oldEntryName = (String) reqMap.get("oldEntryName");
        String newEntryName = (String) reqMap.get("newEntryName");

        // 各参数必传
        ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, parentDirAbsPath != null,
                "Bad Request: Required String parameter 'parentDirAbsPath' is not present");
        ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, fileType != null,
                "Bad Request: Required String parameter 'fileType' is not present");
        ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, oldEntryName != null,
                "Bad Request: Required String parameter 'oldEntryName' is not present");
        ApiCustomException.assertTrue(HttpStatus.BAD_REQUEST, ApiErrorCode.NOT_FOUND_PARAM, newEntryName != null,
                "Bad Request: Required String parameter 'newEntryName' is not present");

        // 确保新旧名不同
        if (oldEntryName.equals(newEntryName)) {
            return getFileOrFolderList(parentDirAbsPath, request);
        }

        String userId = controllerUtils.getUserIdFromJwt(request);
        parentDirAbsPath = parentDirAbsPath.endsWith(SEPERATOR_FOR_KEY) ? parentDirAbsPath
                : (parentDirAbsPath + SEPERATOR_FOR_KEY);// 确保以分隔符结尾
        boolean isFolder = fileType.equals(SpecialFileType.FOLDER.typeName);

        ApiCustomException.assertTrue(ApiErrorCode.ILLEGAL_VALUE, isEntryNameValid(newEntryName, isFolder),
                "Invalid file/folder name.");// 不能包含/等

        if (isFolder) {// 文件夹重命名
            oldEntryName = oldEntryName.endsWith(SEPERATOR_FOR_KEY) ? oldEntryName : oldEntryName + SEPERATOR_FOR_KEY;
            newEntryName = newEntryName.endsWith(SEPERATOR_FOR_KEY) ? newEntryName : newEntryName + SEPERATOR_FOR_KEY;
            String oldObjKeyPrefix = getJoinedObjKey(userId, parentDirAbsPath + oldEntryName);
            String newObjKeyPrefix = getJoinedObjKey(userId, parentDirAbsPath + newEntryName);

            // 确保新文件夹名的文件夹未存在
            if (CephClientUtils.listObjects(ConfigParam.cephBucketName_accountStorage, newObjKeyPrefix)

                    .stream().map(objList -> objList.getObjectSummaries()).flatMap(s3ObjList -> s3ObjList.stream())
                    .collect(Collectors.toList()).size() > 0) {
                throw new ApiCustomException(ApiErrorCode.DUPLICATE_DATA, "the target folder already exists");
            }
            // 确保原文件夹存在
            List<S3ObjectSummary> listRes = CephClientUtils
                    .listObjects(ConfigParam.cephBucketName_accountStorage, oldObjKeyPrefix).stream()
                    .map(objList -> objList.getObjectSummaries()).flatMap(s3ObjList -> s3ObjList.stream())
                    .collect(Collectors.toList());
            if (listRes.size() < 1) {
                throw new ApiCustomException(ApiErrorCode.NOT_EXIST_DATA, "the source folder doesn't exist");
            }

            // 移动
            String oldObjKey, newObjKey;
            for (S3ObjectSummary s3ObjSum : listRes) {
                // 获取重命名后的新key，别直接replace因为有可能原key中有多处被replace掉
                oldObjKey = s3ObjSum.getKey();
                newObjKey = newObjKeyPrefix + oldObjKey.substring(oldObjKeyPrefix.length());// 去掉旧前缀，加上新前缀
                // 转存
                CephClientUtils.copyObject(ConfigParam.cephBucketName_accountStorage, oldObjKey,
                        ConfigParam.cephBucketName_accountStorage, newObjKey);
                CephClientUtils.deleteObject(ConfigParam.cephBucketName_accountStorage, oldObjKey);
            }
        } else {// 文件重命名
            String oldObjKey = getJoinedObjKey(userId, parentDirAbsPath + oldEntryName);
            String newObjKey = getJoinedObjKey(userId, parentDirAbsPath + newEntryName);

            // 确保新文件名的文件未存在
            if (CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, newObjKey)) {
                throw new ApiCustomException(ApiErrorCode.DUPLICATE_DATA, "the target file already exists");
            }
            // 确保原文件存在
            if (!CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, oldObjKey)) {
                throw new ApiCustomException(ApiErrorCode.NOT_EXIST_DATA, "the source file doesn't exist");
            }

            CephClientUtils.copyObject(ConfigParam.cephBucketName_accountStorage, oldObjKey,
                    ConfigParam.cephBucketName_accountStorage, newObjKey);
            CephClientUtils.deleteObject(ConfigParam.cephBucketName_accountStorage, oldObjKey);
        }
        return getFileOrFolderList(parentDirAbsPath, request);
    }

    // 2. 以下为工具方法或工具类
    /** 检查当前目录下是否已经有与输入名同名的文件或文件夹，有则加数字后缀：若是文件夹或不带扩展名的文件则直接在尾部加后缀如 a -> a(1)、若是带扩展名的文件则在扩展名前加后缀如 a.jpg -> a(1).jpg */
    private String getRenamedEntrynameIfExist(String userId, String parentDirAbsPath, String inputEntryName,
            boolean isInputEntryFolder) {

        // 获取当前目录下同前缀的文件或文件夹key列表
        String objKey = getJoinedObjKey(userId, parentDirAbsPath + inputEntryName);
        List<ObjectListing> objectListings = CephClientUtils.listObjects(ConfigParam.cephBucketName_accountStorage,
                objKey, SEPERATOR_FOR_KEY);
        List<String> fileKeysWithCommonprefix = objectListings.stream().map(e -> e.getObjectSummaries())
                .flatMap(e -> e.stream()).map(e -> e.getKey()).collect(Collectors.toList());// 每个元素形如
                                                                                            // xx/a/b/1.jpg、xx/a/b/2
        List<String> folderKeysWithCommonprefix = objectListings.stream().map(e -> e.getCommonPrefixes())
                .flatMap(e -> e.stream())// 每个元素形如xx/a/b/，末尾有"/"
                .map(key -> key.substring(0, key.length() - 1)).collect(Collectors.toList());// 去除额外添加的"/"后缀以与filekey的形式一样

        // 查重
        Set<String> existedKeySet = new HashSet<>(fileKeysWithCommonprefix);
        existedKeySet.addAll(folderKeysWithCommonprefix);
        // System.out.println(existedKeySet);
        if (existedKeySet.contains(objKey)) {// 与同级目录下的现有文件或文件夹重名
            int lastIndexOfDot = inputEntryName.lastIndexOf(".");
            String prefix = inputEntryName;
            String suffix = "";
            // System.out.println(prefix + "," + suffix);

            // 带扩展名的文件由于在扩展名前加后缀，故还需选出去与除扩展名后的前缀同名的文件或文件夹。如已有文件a.jpg、文件夹a(1).jpg，此时再添加文件a.jpg时需把该文件夹也选出
            if (!isInputEntryFolder && 0 <= lastIndexOfDot && lastIndexOfDot < inputEntryName.length() - 1) {
                prefix = inputEntryName.substring(0, lastIndexOfDot);
                suffix = inputEntryName.substring(lastIndexOfDot);
                String tmpObjKey = getJoinedObjKey(userId, parentDirAbsPath + prefix);
                List<ObjectListing> tmpObjListings = CephClientUtils
                        .listObjects(ConfigParam.cephBucketName_accountStorage, tmpObjKey, SEPERATOR_FOR_KEY);
                List<String> tmpFileKeysWithCommonprefix = tmpObjListings.stream().map(e -> e.getObjectSummaries())
                        .flatMap(e -> e.stream()).map(e -> e.getKey()).collect(Collectors.toList());
                List<String> tmpFolderKeysWithCommonprefix = tmpObjListings.stream().map(e -> e.getCommonPrefixes())
                        .flatMap(e -> e.stream()).map(key -> key.substring(0, key.length() - 1))
                        .collect(Collectors.toList());
                existedKeySet.addAll(tmpFileKeysWithCommonprefix);
                existedKeySet.addAll(tmpFolderKeysWithCommonprefix);
                // System.out.println(existedKeySet);
            }
            // 找出最小的未使用的数字后缀
            int size = existedKeySet.size();
            String tmpName;
            for (int i = 1; i <= size; i++) {
                tmpName = prefix + "(" + i + ")" + suffix;
                if (!existedKeySet.contains(getJoinedObjKey(userId, parentDirAbsPath + tmpName))) {
                    inputEntryName = tmpName;
                    break;
                }
            }
        }
        return inputEntryName;
    }

    /** 以一个分隔符依次连接字段。需要确保输入的userId不以分隔符结尾 */
    private String getJoinedObjKey(String userId, String fileNameOrPath) {
        // 确保以一个 分隔符 连接各字段
        userId = userId.endsWith(SEPERATOR_FOR_KEY) ? userId.substring(0, userId.length() - 1) : userId;
        fileNameOrPath = fileNameOrPath.startsWith(SEPERATOR_FOR_KEY) ? fileNameOrPath
                : SEPERATOR_FOR_KEY + fileNameOrPath;
        return userId + fileNameOrPath;
    }

    /** 增、改需要校验名字，查、删不需要 */
    private boolean isEntryNameValid(String entryName, boolean isFolder) {
        // String regEx = "[ _`~!@#$%^&*()+=|{}':;',\[\].<>/?~！@#￥%……&*（）——+|{}【】‘；：”“’。，、？]|
|
|	";
        String regEx = "[<>|*?,]";// <,>,|,*,?, 文件不能包含这些特殊字符，文件夹还不能包含分隔符等字符

        boolean res = entryName.length() > 0 && !Pattern.compile(regEx).matcher(entryName).find();
        if (isFolder) {
            res = res && !entryName.contains(SEPERATOR_FOR_KEY) && !entryName.equals(SpecialDirectory.CURRENT.name)
                    && !entryName.equals(SpecialDirectory.PARENT.name);// .和..分别表示当前目录和上层目录，系统默认，不能取该名。若前后或中间加空格则允许，不视为该二特殊目录
        }
        return res;
    }

    enum SpecialFileType {
        FOLDER("folder"), // 文件夹类型
        FILE("file");// 未知具体格式者指定为此类型
        public String typeName;

        private SpecialFileType(String typeName) {
            this.typeName = typeName;
        }
    }

    enum SpecialDirectory {
        CURRENT("."), // 当前目录
        PARENT("..");// 父目录
        String name;

        private SpecialDirectory(String name) {
            this.name = name;
        }
    }

    // 3. 以下为存储容量相关。容量信息存到Ceph，需要确保不与上传文件或文件夹的key重名。key:userId, value:
    // ${currentCapacity}_${totalCapacity}。
    /** 获取容量，优先从capacityInfo取currentCapacity而不是通过listObjects算得，以减少开销。 */
    @GetMapping("/api/v1/account/storage/capacity")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public Map<String, Long> getCapacityInfo(HttpServletRequest request) {

        String userId = controllerUtils.getUserIdFromJwt(request);
        List<Long> capacityList = getCapacity(userId);

        // 组装结果
        Map<String, Long> res = new HashMap<>();
        res.put("currentCapacityByte", capacityList.get(0));
        res.put("totalCapacityByte", capacityList.get(1));
        return res;
    }

    /** 扩容 */
    @PutMapping("/api/v1/account/storage/capacity")
    @PreAuthorize("hasAnyRole( 'ROLE_STUDENT' , 'ROLE_TEACHER' )")
    public Map<String, Long> updateTotalCapacity(@RequestParam("totalCapacityGB") Double newTotalCapacityGB,
            HttpServletRequest request) {

        String userId = controllerUtils.getUserIdFromJwt(request);
        List<Long> capacityList = getCapacity(userId);

        long currentCapacity = capacityList.get(0);

        long newTotalCapacityBytes = (long) (newTotalCapacityGB * 1024 * 1024 * 1024);
        ApiCustomException.assertTrue(ApiErrorCode.ILLEGAL_VALUE, newTotalCapacityBytes >= currentCapacity,
                "the totalCapacity is less than currentCapacity");

        // 更新
        String objKey = userId;
        String objValue = currentCapacity + "_" + newTotalCapacityBytes;
        CephClientUtils.writeObject(ConfigParam.cephBucketName_accountStorage, objKey, objValue);

        // 组装结果
        Map<String, Long> res = new HashMap<>();
        res.put("currentCapacityByte", currentCapacity);
        res.put("totalCapacityByte", newTotalCapacityBytes);
        return res;
    }

    /** 通过listObjects更新存储的currentCapacity信息，以供查询使用。应该在增或删操作之后调用此 */
    private long updateCurrentCapacity(String userId) {
        long currentCapacity, totalCapacity;

        String objKey = userId;
        if (CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, objKey)) {
            String tmpStr = CephClientUtils.getObjectStr(ConfigParam.cephBucketName_accountStorage, objKey);
            String[] capacityStrs = tmpStr.split("_");
            totalCapacity = Long.parseLong(capacityStrs[1]);
        } else {
            totalCapacity = (long) (ConfigParam.cephAccountStorageDefaultCapacityGB * 1024 * 1024 * 1024);
        }

        currentCapacity = getCurrentCapacityByListingObjects(userId);

        String objValue = currentCapacity + "_" + totalCapacity;
        CephClientUtils.writeObject(ConfigParam.cephBucketName_accountStorage, objKey, objValue);

        return currentCapacity;
    }

    /** 获取Capacity信息，返回两个值：依次为currentCapacity、totalCapacity */
    private List<Long> getCapacity(String userId) {
        long currentCapacity, totalCapacity;

        String objKey = userId;
        if (CephClientUtils.isObjExist(ConfigParam.cephBucketName_accountStorage, objKey)) {
            String tmpStr = CephClientUtils.getObjectStr(ConfigParam.cephBucketName_accountStorage, objKey);
            String[] capacityStrs = tmpStr.split("_");
            currentCapacity = Long.parseLong(capacityStrs[0]);
            totalCapacity = Long.parseLong(capacityStrs[1]);
        } else {
            currentCapacity = getCurrentCapacityByListingObjects(userId);
            totalCapacity = (long) (ConfigParam.cephAccountStorageDefaultCapacityGB * 1024 * 1024 * 1024);
        }

        List<Long> res = new ArrayList<>();
        res.add(currentCapacity);
        res.add(totalCapacity);
        return res;
    }

    /** 获取指定用户的实际已用存储空间，单位byte。通过listObjects实现，比较耗时耗资源，尽量少调用 */
    private long getCurrentCapacityByListingObjects(String userId) {
        long currentCapacity = 0;

        String objKeyPrefix = userId.endsWith(SEPERATOR_FOR_KEY) ? userId : userId + SEPERATOR_FOR_KEY;// 确保以分隔符结尾，以免存储capacity的object也被包含在内
        List<S3ObjectSummary> s3ObjectSummaryList = CephClientUtils
                .listObjects(ConfigParam.cephBucketName_accountStorage, objKeyPrefix).stream()
                .map(e -> e.getObjectSummaries()).flatMap(e -> e.stream()).collect(Collectors.toList());
        for (S3ObjectSummary s3ObjSum : s3ObjectSummaryList) {
            currentCapacity += s3ObjSum.getSize();
        }
        return currentCapacity;
    }
}

/** 文件或文件夹抽象表示为Entry */
class EntryItem {
    private String parentDirAbsPath;
    private String entryName;
    private Long byteSize;
    private Date lastModify;
    private String accessPath;
    private String fileType;// pdf, xls, txt ...

    public EntryItem() {// necessary

    }

    public EntryItem(String parentDirAbsPath, String entryName, Long byteSize, Date lastModify, String accessPath,
            String fileType) {
        this.parentDirAbsPath = parentDirAbsPath;
        this.entryName = entryName;
        this.byteSize = byteSize;
        this.lastModify = lastModify;
        this.accessPath = accessPath;
        this.fileType = fileType;
    }

    public String getParentDirAbsPath() {
        return parentDirAbsPath;
    }

    public void setParentDirAbsPath(String parentDirAbsPath) {
        this.parentDirAbsPath = parentDirAbsPath;
    }

    public String getEntryName() {
        return entryName;
    }

    public void setEntryName(String entryName) {
        this.entryName = entryName;
    }

    public Long getByteSize() {
        return byteSize;
    }

    public void setByteSize(Long byteSize) {
        this.byteSize = byteSize;
    }

    public Date getLastModify() {
        return lastModify;
    }

    public void setLastModify(Date lastModify) {
        this.lastModify = lastModify;
    }

    public String getAccessPath() {
        return accessPath;
    }

    public void setAccessPath(String accessPath) {
        this.accessPath = accessPath;
    }

    public String getFileType() {
        return fileType;
    }

    public void setFileType(String fileType) {
        this.fileType = fileType;
    }
}

package com.marchon.sensestudy.common.utils;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.stream.Collectors;

import org.springframework.web.multipart.MultipartFile;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.CopyObjectResult;
import com.amazonaws.services.s3.model.DeleteObjectsRequest;
import com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion;
import com.amazonaws.services.s3.model.DeleteObjectsResult;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
import com.amazonaws.services.s3.model.S3Object;
import com.marchon.sensestudy.common.config.ConfigParam;

public class CephClientUtils {

    private static AmazonS3 amazonS3Client = null;
    private static Protocol protocol = Protocol.HTTP;
    public static String CEPH_URL_STR = protocol.name() + "://" + ConfigParam.cephHost + ":" + ConfigParam.cephPort;

    static {
        AWSCredentials credentials = new BasicAWSCredentials(ConfigParam.cephAccessKey, ConfigParam.cephSecretKey);
        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setProtocol(protocol);
        amazonS3Client = new AmazonS3Client(credentials, clientConfig);
        amazonS3Client.setEndpoint(ConfigParam.cephHost + ":" + ConfigParam.cephPort);
    }

    /** 批量删除key以指定前缀开头的数据 */
    public static DeleteObjectsResult deleteObjects(String bucketName, String keyPrefix) {
        List<String> keyStrs = amazonS3Client.listObjects(bucketName, keyPrefix).getObjectSummaries().stream()
                .map(e -> e.getKey()).collect(Collectors.toList());
        List<KeyVersion> keys = keyStrs.stream().map(e -> new KeyVersion(e)).collect(Collectors.toList());
        DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(bucketName);
        deleteObjectsRequest.setKeys(keys);
        return amazonS3Client.deleteObjects(deleteObjectsRequest);
    }

    /** 删除精确指定key的数据 */
    public static void deleteObject(String bucketName, String objKey) {
        amazonS3Client.deleteObject(bucketName, objKey);
    }

    /**
     * 获取存储的资源信息列表。通过正确指定prefix和delimiter可以以类文件系统的方式列取资源信息，即若多个文件在当前目录的同一个子文件夹下则它们只以一个子文件夹的形合并显示在当前目录下。这里的合并规则为
     * ”${prefix}w*${first_delimiter}“，即 prefix到delimiter首次出现
     * 间(包括prefix和delimiter自身)的subKey视为共有文件夹。缺点：这里的”文件夹“缺乏最近修改时间、大小等元信息。<br>
     * 典型应用：<br>
     * <li>若delimiter为空串则查得所有以prefix为前缀的key；</li><br>
     * <li>若所存的key是以"/"分隔的如"xx/a/b"、"xx/a/b"等，则当delimiter为"/"是：若prefix以"/"结尾则查得的是当前目录下的所有文件夹或文件、否则查得的是当前目录下以prefix为前缀的所有文件夹或文件</li>
     * 
     * @return 返回的ObjectListing List至少有一个元素
     */
    public static List<ObjectListing> listObjects(String bucketName, String keyPrefix, String delimiter) {
        ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
        listObjectsRequest.setBucketName(bucketName);
        listObjectsRequest.setPrefix(keyPrefix);
        listObjectsRequest.setDelimiter(delimiter);
        // listObjectsRequest.setMaxKeys(2);// 1000 in default
        // listObjectsRequest.setMarker(null);

        // 可能有多个truncate
        List<ObjectListing> objectListings = new ArrayList<>();
        ObjectListing res = amazonS3Client.listObjects(listObjectsRequest);
        objectListings.add(res);
        while (res.isTruncated()) {
            res = amazonS3Client.listNextBatchOfObjects(res);
            objectListings.add(res);
        }
        return objectListings;
    }

    public static List<ObjectListing> listObjects(String bucketName, String keyPrefix) {
        return listObjects(bucketName, keyPrefix, null);
    }

    /** 文件存储 */
    public static PutObjectResult writeObject(String bucketName, String objKey, MultipartFile file,
            CannedAccessControlList cannedAccessControlList) {
        PutObjectResult res;
        if (!amazonS3Client.doesBucketExist(bucketName)) {
            amazonS3Client.createBucket(bucketName);
        }
        ObjectMetadata objectMetadata = new ObjectMetadata();
        objectMetadata.setContentLength(file.getSize()); // must set, otherwise stream contents will be buffered in
                                                            // memory and could result in out of memory errors.
        // objectMetadata.getUserMetadata().put("type", "pdf");//are added in http request header, which cann't only
        // contain iso8859-1 charset
        InputStream inputStream = null;
        try {
            inputStream = file.getInputStream();
        } catch (IOException e1) {
            e1.printStackTrace();
        }
        res = amazonS3Client.putObject(new PutObjectRequest(bucketName, objKey, inputStream, objectMetadata));
        if (null != cannedAccessControlList) {
            amazonS3Client.setObjectAcl(bucketName, objKey, cannedAccessControlList);
        }
        try {
            inputStream.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return res;
    }

    /** 文件获取 */
    public static S3Object getObject(String bucketName, String objKey) {
        return amazonS3Client.getObject(bucketName, objKey);
    }

    public static PutObjectResult writeObject(String bucketName, String objKey, MultipartFile file) {
        return writeObject(bucketName, objKey, file, null);
    }

    /** 生成直接访问的永久URL。要求被访问资源的权限为public才能访问到 */
    public static URL generatePublicUrl(String bucketName, String objKey) {
        return amazonS3Client.getUrl(bucketName, objKey);
    }

    /** 生成直接访问的临时URL，失效时间默认15min */
    // url formate returned: scheme://host[:port]/bucketName/objKey?{Query}
    public static URL generatePresignedUrl(String bucketName, String key, Date expiration) {
        return amazonS3Client.generatePresignedUrl(bucketName, key, expiration);// 失效时间点必设，默认为15min，最多只能7天
    }

    /**
     * 5 default metadata are set, for example: <br>
     * {Accept-Ranges=bytes, Content-Length=5, Content-Type=text/plain, ETag=5d41402abc4b2a76b9719d911017c592,
     * Last-Modified=Sun Nov 04 15:35:17 CST 2018}
     */
    public static ObjectMetadata getObjectMetaData(String bucketName, String objKey) {
        return amazonS3Client.getObjectMetadata(bucketName, objKey);
    }

    public static boolean isObjExist(String bucketName, String objKey) {
        return amazonS3Client.doesObjectExist(bucketName, objKey);
    }

    public static CopyObjectResult copyObject(String sourceBucketName, String sourceKey, String destinationBucketName,
            String destinationKey) {
        return amazonS3Client.copyObject(sourceBucketName, sourceKey, destinationBucketName, destinationKey);
    }

    /** 文本存储，会被s3 sdk以UTF-8格式编码成字节流存储 */
    public static PutObjectResult writeObject(String bucketName, String objKey, String objContent) {
        if (null == objContent) {// objContent为null时下面putObject会出错
            objContent = "";
        }
        if (!amazonS3Client.doesBucketExist(bucketName)) {
            amazonS3Client.createBucket(bucketName);
        }
        return amazonS3Client.putObject(bucketName, objKey, objContent);// String will be encoded to bytes with UTF-8
                                                                        // encoding.
    }

    /** 文本获取 */
    public static String getObjectStr(String bucketName, String objKey) {
        if (!amazonS3Client.doesObjectExist(bucketName, objKey)) {// 判断是否存在，不存在时下面直接获取会报错
            return null;
        }

        S3Object s3Object = amazonS3Client.getObject(bucketName, objKey);
        BufferedReader bufferedReader = null;
        try {
            bufferedReader = new BufferedReader(new InputStreamReader(s3Object.getObjectContent(), "UTF-8"));// 存入时被UTF-8编码了，故对应之
        } catch (UnsupportedEncodingException e1) {
            e1.printStackTrace();
        }

        String res = null;
        try {
            res = bufferedReader.readLine();
        } catch (IOException e) {
            e.printStackTrace();
        }
        try {
            bufferedReader.close();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return res;
    }

    private static byte[] getObjectBytes(String bucketName, String objKey) throws IOException {
        S3Object s3Object = amazonS3Client.getObject(bucketName, objKey);
        InputStream in = s3Object.getObjectContent();

        byte[] bytes = new byte[(int) s3Object.getObjectMetadata().getInstanceLength()];
        in.read(bytes);

        in.close();
        return bytes;
    }
}