FastDFS分布式文件系统环境搭建和代码测试

FastDFS分布式文件系统环境搭建和代码测试

待配置的机器：

192.168.44.10 192.168.44.11 192.168.44.12 192.168.44.13

注意：关闭linux防火墙，或者开放FastDFS所需的一些列端口

一、安装必要的工具和依赖

1、工具

yum install lrzsz wget vim unzip net-tools -y

2、依赖

yum install gcc perl openssl openssl-devel pcre pcre-devel zlib zlib-devel libevent libevent-devel -y

二、安装FastDFS

1、安装包下载地址

码云地址：https://gitee.com/fastdfs100
下载FastDFS、libfastcommon、fastdfs-nginx-module(可选)、fastdfs-client-java(可选)

使用rz -y 上传文件，使用sz下载文件，这个工具在lrzsz安装，有这个工具后，用拖拽也可以。

2、安装libfastcommon

libfastcommon 库是 FastDFS 文件系统运行需要的公共 C 语言函数库

解压：tar -zxvf fastdfs100-libfastcommon-V1.0.51.tar.gz

如果是zip包使用：unzip fastdfs100-libfastcommon-V1.0.51.zip

进入目录，编译：./make.sh

编译没报错可以安装：./make.sh install

3、安装FastDFS

解压：tar -zxvf fastdfs100-fastdfs-V6.07.tar.gz
或 unzip fastdfs100-fastdfs-V6.07.zip

进入目录，编译：./make.sh

编译没报错可以安装：./make.sh install

所有编译出来的文件存放在/usr/bin****目录下

所有配置文件存放在/etc/fdfs****目录下

通过ls /usr/bin/fdfs* 可以查看所有fastDFS的命令

三、配置FastDFS

1、将/etc/fdfs下的配置文件样例全部改名，去掉sample后缀

 mv client.conf.sample client.conf
 mv storage.conf.sample storage.conf
 mv storage_ids.conf.sample storage_ids.conf
 mv tracker.conf.sample tracker.conf

2、将安装目录conf下的http.conf和拷贝到/etc/fdfs下

cp http.conf /etc/fdfs/
cp mime.types /etc/fdfs/

3、分组

group1：192.168.44.10 192.168.44.11

group2： 192.168.44.12 192.168.44.13

tracker server: 192.168.44.10 192.168.44.12

storage server: 全部机器

group1和group2的数据在各自的组内自动复制，做冗余备份。

group1和group2之间数据不重复，只为提高并发和容量。

4、创建日志目录和数据目录

mkdir -p /opt/fastdfs/tracker #tracker的日志目录
mkdir -p /opt/fastdfs/storage #storage的日志目录
mkdir -p /opt/fastdfs/storage/files #真正存放文件的目录

5、修改tracker.conf

192.168.44.10

base_path = /opt/fastdfs/tracker
store_group = group1

192.168.44.12

base_path = /opt/fastdfs/tracker
store_group = group2

store_group参数与store_lookup参数相关

注意：FastDFS默认是带有负载均衡策略的可以在tracker的2台机器中修改tracker.conf文件
store_lookup=1

0 随机存放策略
1 指定组
2 选择磁盘空间的优先存放 默认值

修改后重启服务
fdfs_trackerd /etc/fdfs/tracker.conf restart

6、修改storage.conf

# group1或group2，看分组配置
group_name = group1 
base_path = /opt/fastdfs/storage
store_path0 = /opt/fastdfs/storage/files
tracker_server = 192.168.44.10:22122
tracker_server = 192.168.44.12:22122

7、启动tracker server

192.168.44.10和192.168.44.12

fdfs_trackerd /etc/fdfs/tracker.conf start

ps -ef | grep fdfs

root       1753      1  0 11:30 ?        00:00:00 fdfs_trackerd /etc/fdfs/tracker.conf

8、启动storage server

fdfs_storaged /etc/fdfs/storage.conf start

需要通过ps -ef | grep fdfs检查是否启动。

/opt/fastdfs/storage/files/data 这个目录下是数据存放位置，data目录是FastDFS自动创建，这个目录下有00~FF个目录

9、查看storage注册情况

fdfs_monitor /etc/fdfs/storage.conf

10、重启FastDFS

重启tracker

fdfs_trackerd /etc/fdfs/tracker.conf restart

重启storage

fdfs_storaged /etc/fdfs/storage.conf restart

11、关闭FastDFS

fdfs_trackerd /etc/fdfs/tracker.conf stop
fdfs_storaged /etc/fdfs/storage.conf stop

不建议在生产环境使用 kill -9 强制关闭，因为可能会导致文件信息不同步问题

四、环境测试

使用FastDFS自带的测试工具

1、修改client.conf配置文件，主要修改两个配置：

base_path=/opt/fastdfs/client
tracker_server=192.168.44.10:22122

2、测试

执行上传命令

fdfs_test /etc/fdfs/client.conf upload /root/test.txt

删除测试

fdfs_delete_file /etc/fdfs/client.conf group1/要删除的文件路径

注意：

没有搭建集群默认只有一个组group1

后缀名包含-m的为属性文件(meta)

五、代码测试

使用Springboot进行代码测试

https://gitee.com/asker124143222/my-boot.git

1、依赖

        <dependency>
            <groupId>com.github.tobato</groupId>
            <artifactId>fastdfs-client</artifactId>
            <version>1.27.2</version>
        </dependency>

2、service

property外置前缀upload到application.yml里

@ConfigurationProperties(prefix = "upload")
@Data
public class UploadProperties {
    private String baseUrl;
    private List<String> allowTypes;
}

UploadService

package com.home.system.service;

import com.github.tobato.fastdfs.domain.fdfs.FileInfo;
import com.github.tobato.fastdfs.domain.fdfs.StorePath;
import com.github.tobato.fastdfs.service.FastFileStorageClient;
import com.home.system.config.UploadProperties;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import javax.annotation.Resource;
import javax.imageio.ImageIO;
import javax.servlet.http.HttpServletResponse;
import java.awt.image.BufferedImage;
import java.io.Closeable;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashSet;

/**
 * @Author: xu.dm
 * @Date: 2021/6/16 14:44
 * @Version: 1.0
 * @Description: TODO
 **/
@Slf4j
@EnableConfigurationProperties(UploadProperties.class)
@Service
public class UploadService {

    @Resource
    private FastFileStorageClient storageClient;
    @Resource
    private UploadProperties properties;

    public String uploadImage(MultipartFile file) {
        // 1、校验文件类型
        String contentType = file.getContentType();
        if (!properties.getAllowTypes().contains(contentType)) {
            throw new RuntimeException("文件类型不支持");
        }
        // 2、校验文件内容
        try {
            BufferedImage image = ImageIO.read(file.getInputStream());
            if (image == null || image.getWidth() == 0 || image.getHeight() == 0) {
                throw new RuntimeException("上传文件有问题");
            }
        } catch (IOException e) {
            log.error("校验文件内容失败....{}", e);
            throw new RuntimeException("校验文件内容失败" + e.getMessage());
        }
        try {
            // 3、上传到FastDFS
            // 3.1、获取扩展名
            String extension = StringUtils.substringAfterLast(file.getOriginalFilename(), ".");
            // 3.2、上传
            StorePath storePath = storageClient.uploadFile(file.getInputStream(), file.getSize(), extension, new HashSet<>());
            System.out.println(storePath.getGroup());
            System.out.println(storePath.getPath());
            System.out.println(storePath);
            // 返回路径
            return properties.getBaseUrl() + storePath.getFullPath();
        } catch (IOException e) {
            log.error("【文件上传】上传文件失败！...." + e);

            throw new RuntimeException("【文件上传】上传文件失败！" + e.getMessage());

        }
    }

    public String uploadFile(MultipartFile file) {
        try {
            // 3、上传到FastDFS
            // 3.1、获取扩展名
            String extension = StringUtils.substringAfterLast(file.getOriginalFilename(), ".");
            // 3.2、上传
            StorePath storePath = storageClient.uploadFile(file.getInputStream(), file.getSize(), extension, new HashSet<>());
            System.out.println(storePath.getGroup());
            System.out.println(storePath.getPath());
            System.out.println(storePath);
            // 返回路径
            return properties.getBaseUrl() + storePath.getFullPath();
        } catch (IOException e) {
            log.error("【文件上传】上传文件失败！...." + e);

            throw new RuntimeException("【文件上传】上传文件失败！" + e.getMessage());

        }
    }

    public void downloadFile(HttpServletResponse response, String groupName, String path) {
        FileInfo fileInfo = null;
        try {
            fileInfo = storageClient.queryFileInfo(groupName, path);
        }catch (Exception e){
            throw new RuntimeException("文件不存在");
        }



        if(fileInfo == null) {
            throw new RuntimeException("文件不存在");
        }

        InputStream inputStream = storageClient.downloadFile(groupName, path, ins -> ins);
        String filename = path.replace("/", ".");
        response.setContentType("application/octet-stream;charset=UTF-8");
        response.setHeader("Content-Disposition", "attachment; filename="+filename);
        try {
            streamCopy(inputStream,response.getOutputStream());
            response.flushBuffer();
        } catch (IOException e) {
            closeQuietly(inputStream);
            e.printStackTrace();
            throw new RuntimeException("下载文件失败！");
        }
    }

    public void deleteFile(String groupName, String path) {
        storageClient.deleteFile(groupName, path);
    }

    private void streamCopy(InputStream inp, OutputStream out) throws IOException {
        byte[] buff = new byte[4096];
        int count;
        while ((count = inp.read(buff)) != -1) {
            if (count > 0) {
                out.write(buff, 0, count);
            }
        }
    }

    public static void closeQuietly( final Closeable closeable ) {
        // no need to log a NullPointerException here
        if(closeable == null) {
            return;
        }

        try {
            closeable.close();
        } catch ( Exception e ) {
            log.error( "Unable to close resource: " + e,
                    e );
        }
    }
}

3、controller

package com.home.system.controller;

import com.home.system.service.UploadService;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.multipart.MultipartFile;

import javax.annotation.Resource;
import javax.servlet.http.HttpServletResponse;
import java.util.HashMap;
import java.util.Map;

/**
 * @Author: xu.dm
 * @Date: 2021/6/16 14:50
 * @Version: 1.0
 * @Description: TODO
 **/
@Controller
public class UploadController {

    @Resource
    private UploadService uploadService;


    @GetMapping("/upload")
    public String upload() {
        return "upload";
    }

    @GetMapping("/fileDelete")
    public String fileDelete() {
        return "fileDelete";
    }

    @GetMapping("/fileDownload")
    public String fileDownload() {
        return "download";
    }


    @PostMapping("file/download")
    public void doDownload(HttpServletResponse response, String groupName, String path){
        uploadService.downloadFile(response,  groupName, path);
    }


    @PostMapping("file/delete")
    @ResponseBody
    public String doDelete(String groupName, String path){
        uploadService.deleteFile(groupName, path);
        return "删除成功";
    }

    @RequestMapping("upload/doUpload")
    @ResponseBody
    public Map<String,Object> doUpload(MultipartFile file){
        System.out.println(file.getOriginalFilename());
        Map<String, Object> upload =new HashMap<>();
        String path = this.uploadService.uploadImage(file);
        upload.put("path",path);
        return upload;
    }

    @RequestMapping("upload/doFileUpload")
    @ResponseBody
    public Map<String,Object> doFileUpload(MultipartFile file){
        System.out.println(file.getOriginalFilename());
        Map<String, Object> upload =new HashMap<>();
        String path = this.uploadService.uploadFile(file);
        upload.put("path",path);
        return upload;
    }
}

4、application.yml文件配置

server:
  port: 8080
  tomcat:
    accept-count: 3
    max-connections: 6
    threads:
      max: 10
    basedir: ./target/

logging:
  file:
    # 一旦日志文件大于设定值，这个日志文件就会被加上日期归档，原文清空重新开始记录日志
    name: ./target/myBoot.log
  logback:
    rollingpolicy:
      max-file-size: 20MB
      max-history: 30

spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
  servlet:
    multipart:
      max-request-size: 500MB
      max-file-size: 500MB

fdfs:
  connect-timeout: 60000
  so-timeout: 120000
  tracker-list:
    - 192.168.44.10:22122
    - 192.168.44.12:22122


upload:
  base-url: http://192.168.1.105/ #换成你的ip
  allow-types:
    - image/jpeg
    - image/png
    - image/bmp