FastDFS 介绍
-
简介
FastDFS是一款类Google FS的开源分布式文件系统，它用纯C语言实现，支持Linux、FreeBSD、AIX等UNIX系统。它只能通过 专有API对文件进行存取访问，不支持POSIX接口方式，不能mount使用。准确地讲，Google FS以及FastDFS、mogileFS、 HDFS、TFS等类Google FS都不是系统级的分布式文件系统，而是应用级的分布式文件存储服务。
FastDFS 是一个开源的高性能分布式文件系统（DFS）。 它的主要功能包括：文件存储，文件同步和文件访问，以及高容量和负载平衡。主要解决了海量数据存储问题，特别适合以中小文件（建议范围：4KB < file_size <500MB）为载体的在线服务。
-
原理导图
FastDFS架构包括 Tracker server和Storage server。客户端请求Tracker server进行文件上传、下载，通过Trackerserver调度最终由Storage server完成文件上传和下载。FastDFS 系统有三个角色：跟踪服务器(Tracker Server)、存储服务器(Storage Server)和客户端 (Client)。
Tracker Server：跟踪服务器，主要做调度工作，起到均衡的作用；负责管理所有的 storage server和 group，每个 storage 在启动后会连接 Tracker，告知自己所属 group 等信息，并保持周期性心跳。通过Trackerserver在文件上传时可以根据一些策略找到Storageserver提供文件上传服务。
Storage Server：存储服务器，主要提供容量和备份服务；以 group 为单位，每个 group 内可以有多台 storage server，数据互为备份。Storage server没有实现自己的文件系统而是利用操作系统 的文件系统来管理文件。
Client：客户端，上传下载数据的服务器，也就是我们自己的项目所部署在的服务器。
-
FastDFS 安装
1、 安装FastDFS Linux依赖包yum install libevent libevent-devel gcc* -y2、 安装FastDFS的官方依赖包
wget -O libfastcommon-1.0.39.tar.gz https://codeload.github.com/happyfish100/libfastcommon/tar.gz/V1.0.39 #解压依赖包 tar -zxvf libfastcommon-1.0.39.tar.gz #编译安装 cd libfastcommon-1.0.39 ./make.sh ./make.sh install3、 配置tracker
Tracker的配置文件在/etc/fdfs/下，首先为了稳定考虑对配置文件进行备份cd /etc/fdfs/ cp tracker.conf.sample tracker.conf vim tracker.conf#重点修改下面的几个点 1、 确保disable是打开的 disable=false 2、 查看提供服务的端口 port=22122 3、 指定生产tracker数据和日志的目录 base_path=/fastdfs/tracker 4、 修改默认的http端口（目的为了防止冲突） http.server_port=9080 5、 创建tracker_base的目录 mkdir -p /fastdfs/tracker 4、 启动tracker #关闭防火墙 systemctl stop firewalld#启动命令 systemctl start fdfs_trackerd #查看状态 systemctl status fdfs_trackerd ``` ```bash #如下： fdfs_trackerd.service - LSB: FastDFS tracker server Loaded: loaded (/etc/rc.d/init.d/fdfs_trackerd; bad; vendor preset: disabled) Active: active (running) since Mon 2020-06-22 01:53:40 EDT; 47min ago Docs: man:systemd-sysv-generator(8) Process: 1017 ExecStart=/etc/rc.d/init.d/fdfs_trackerd start (code=exited, status=0/SUCCESS) CGroup: /system.slice/fdfs_trackerd.service └─1034 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf Jun 22 01:53:40 localhost.localdomain systemd[1]: Starting LSB: FastDFS tracker server... Jun 22 01:53:40 localhost.localdomain fdfs_trackerd[1017]: Starting FastDFS tracker server: Jun 22 01:53:40 localhost.localdomain systemd[1]: Started LSB: FastDFS tracker server. #查看端口 netstat -tulnp|grep fdfs #如下： tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 1034/fdfs_trackerd #设置tracker开机自启动 chkconfig fdfs_trackerd on #关闭tracker /etc/init.d/fdfs_trackerd stop service fdfs_trackerd stop systemctl stop fdfs_trackerd5、 配置storage
storage的配置文件目录同样在/etc/fdfs/#配置storage cd /etc/fdfs/ cp storage.conf.sample storage.conf vim storage.conf 确保日志当中的以下配置 disable=false group_name=group1 port=23000 heart_beat_interval=30 base_path=/fastdfs/storage/base store_path_count=1 store_path0=/fastdfs/storage subdir_count_per_path=256 tracker_server=192.168.194.158:22122 sync_start_time=00:00 sysc_end_time=23:59 http.server_port=9888 #创建对应的配置目录 makedir -p /fastdfs/storage/base6、 启动storage
#启动storage /etc/init.d/fdfs_storaged start service fdfs_storaged start systemctl start fdfs_storaged #查看状态 systemctl status fdfs_storaged #如下 ● fdfs_storaged.service - LSB: FastDFS storage server Loaded: loaded (/etc/rc.d/init.d/fdfs_storaged; bad; vendor preset: disabled) Active: active (running) since Mon 2020-06-22 01:53:40 EDT; 1h 15min ago Docs: man:systemd-sysv-generator(8) Process: 1018 ExecStart=/etc/rc.d/init.d/fdfs_storaged start (code=exited, status=0/SUCCESS) CGroup: /system.slice/fdfs_storaged.service └─1031 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf Jun 22 01:53:40 localhost.localdomain systemd[1]: Starting LSB: FastDFS storage server... Jun 22 01:53:40 localhost.localdomain fdfs_storaged[1018]: Starting FastDFS storage server: Jun 22 01:53:40 localhost.localdomain systemd[1]: Started LSB: FastDFS storage server. #查看端口 netstat -unltp|grep fdfs tcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN 1031/fdfs_storaged #查看tracker和storage是否通信 fdfs_monitor /etc/fdfs/storage.conf #如下 …… server_count=1, server_index=0 tracker server is 192.168.194.158:22122 group count: 1 Group 1: group name = group1 disk total space = 17394 MB disk free space = 15770 MB trunk free space = 0 MB storage server count = 1 active server count = 1 storage server port = 23000 storage HTTP port = 9888 store path count = 1 …… #配置storage开机启动 chkconfig fdfs_storaged on #关闭storage /etc/init.d/fdfs_storaged stop service fdfs_storaged stop systemctl stop fdfs_storaged 7、 配置client client的配置文件同样是存放在/etc/fdfs/下 cd /etc/fdfs/ cp client.conf.sample client.conf vim client.conf #配置如下 base_path=/fastdfs/client tracker_server=192.168.194.158:221228、 测试上传
cd /opt/ fdfs_upload_file /etc/fdfs/client.conf bjh.jpg 效果如下 group1/M00/00/00/wKjCnl7wXLSAT0pPAA8GO9kg11w732.jpg Django_FastDFS使用 Django fastDFS有两个模块可以选择 py3fdfs fdfs_client-py 其中在windows下，python3环境当中使用fdfs_client-py有大概率遇到兼容问题，所以采用py3fdfs文件。 pip install py3fdfs 文件上传 from fdfs_client.client import Fdfs_client, get_tracker_conf tracker_path = get_tracker_conf('./client.conf') client = Fdfs_client(tracker_path) ret = client.upload_by_filename('./by.jpg') print(ret) #效果如下： {'Group name': b'group1', 'Remote file_id': b'group1/M00/00/00/wKjCnl7wcgCAS13GAAA6-XV4BMg226.jpg', 'Status': 'Upload successed.', 'Local file name': './by.jpg', 'Uploaded size': '14.74KB', 'Storage IP': b'192.168.194.158'} 与django进行结合 from django.shortcuts import render from fdfs_client.client import Fdfs_client, get_tracker_conf def index(request): if request.method == "POST": file = request.FILES.get("tfile") tracker_path = get_tracker_conf('./client.conf') client = Fdfs_client(tracker_path) ret = client.upload_by_buffer(file.read(),file_ext_name=file.name) print(ret) return render(request,"index.html")文件下载
文件修改和文件下载文件删除都需要有文件的remote_file_id，这个id在文件上传的时候回返回#保存文件返回的数据 {'Group name': b'group1', 'Remote file_id': b'group1/M00/00/00/wKjCnl7wcgCAS13GAAA6-XV4BMg226.jpg', 'Status': 'Upload successed.', 'Local file name': './by.jpg', 'Uploaded size': '14.74KB', 'Storage IP': b'192.168.194.158'} from fdfs_client.client import Fdfs_client, get_tracker_conf tracker_path = get_tracker_conf('./client.conf') client = Fdfs_client(tracker_path) ret = client.download_to_buffer(remote_name) print(ret) 与django进行结合 #数据建模 from django.db import models class FileManager(models.Model): file_name = models.CharField(max_length = 128) remote_name = models.CharField(max_length = 128) upload_size = models.CharField(max_length = 32) file_type = models.CharField(max_length = 32) upload_time = models.DateTimeField() change_time = models.DateTimeField() group_name = models.CharField(max_length = 32) storage_ip = models.CharField(max_length = 64) #视图文件文件上传 import os import datetime from django.shortcuts import render from django.http import JsonResponse from fdfs_client.client import Fdfs_client, get_tracker_conf from FileManager.models import FileManager def upload(request): result = {"code": "","data": [],"state": ""} if request.method == "POST": file = request.FILES.get("file") file_name = file.name client = os.path.join( os.path.dirname(__file__),"client.conf" ) tracker_path = get_tracker_conf(client) try: client = Fdfs_client(tracker_path) ret = client.upload_by_buffer(file.read(), file_ext_name = file_name) except Exception as e: result["code"] = 500 result["state"] = str(e) result["data"].append(str(e)) else: f = FileManager() f.file_name = file_name f.remote_name = ret.get("Remote file_id").decode() f.upload_size = ret.get("Uploaded size") f.file_type = "图片" f.upload_time = datetime.datetime.now() f.change_time = datetime.datetime.now() f.group_name = ret.get("Group name").decode() f.storage_ip = ret.get("Storage IP").decode() f.save() result["code"] = 200 result["state"] = "上传成功" result["data"].append({ "file_name": file_name, "file_size": f.upload_size, "file_id": f.id }) else: result["code"] = 403 result["state"] = "请求失败" result["data"].append("请求失败") return JsonResponse(result) #文件下载 def download(request): result = {"code": "", "data": [], "state": ""} id_list = request.GET.getlist("file") for id in id_list: file = FileManager.objects.filter(id=id).first() if file: client