如何解决分布式日志exceptionless的写入瓶颈

我们都知道在分布式日志当中，exceptionless客户端是把日志写到

Elasticsearch数据库，就像我们把数据写入到关系数据库一样；

既然是写入，那么在短时间大数据量的情况下，写入就会涉及到效率的问题；

首先我们看下 exceptionless源码

 public class InMemoryObjectStorage : IObjectStorage {
        private readonly Dictionary<string, Tuple<ObjectInfo, object>> _storage = new Dictionary<string, Tuple<ObjectInfo, object>>(StringComparer.OrdinalIgnoreCase);
        private readonly object _lock = new object();

        public InMemoryObjectStorage() : this(1000) {}

        public InMemoryObjectStorage(int maxObjects) {
            MaxObjects = maxObjects;
        }

        public long MaxObjects { get; set; }

        public int Count {
            get { return _storage.Count; }
        }

我们写入到日志的方法  ，首先会把日志写到内存当中；但是为了防止日志过大，导致服务器的内存被过度消耗的问题 ，这边是限制了 日志的条数的 1000；

当有新的日志进来时，如果是超过1000，那么旧的日志是会被抛弃的；

  public bool SaveObject<T>(string path, T value) where T : class {
            if (String.IsNullOrWhiteSpace(path))
                throw new ArgumentNullException("path");

            lock (_lock) {
                _storage[path] = Tuple.Create(new ObjectInfo {
                    Created = DateTime.Now,
                    Modified = DateTime.Now,
                    Path = path
                }, (object)value);

                if (_storage.Count > MaxObjects)
                    _storage.Remove(_storage.OrderByDescending(kvp => kvp.Value.Item1.Created).First().Key);
            }

            return true;
        }

这显然不是 我们想要的，我们需要的是完整的日志；

 public SubmissionResponse PostEvents(IEnumerable<Event> events, ExceptionlessConfiguration config, IJsonSerializer serializer) {
            if (!config.IsValid)
                return new SubmissionResponse(500, message: "Invalid client configuration settings");

            string data = serializer.Serialize(events);
            string url = String.Format("{0}/events", GetServiceEndPoint(config));

            HttpResponseMessage response;
            try {
                HttpContent content = new StringContent(data, Encoding.UTF8, "application/json");

                // don't compress data smaller than 4kb
                if (data.Length > 1024 * 4)
                    content = new GzipContent(content);

                _client.Value.AddAuthorizationHeader(config.ApiKey);
                response = _client.Value.PostAsync(url, content).ConfigureAwait(false).GetAwaiter().GetResult();
            } catch (Exception ex) {
                return new SubmissionResponse(500, exception: ex);
            }

            int settingsVersion;
            if (Int32.TryParse(GetSettingsVersionHeader(response.Headers), out settingsVersion))
                SettingsManager.CheckVersion(settingsVersion, config);

            return new SubmissionResponse((int)response.StatusCode, GetResponseMessage(response));
        }

所以我们需要在PostEvents 这个方法做文章；

目前我们的做法是：

 

就是把日志直接写到消息队列，然后做一个消费日志的订阅队列的服务，从队列读取数据，在写到Elasticsearch；