• Cassandra Issue with Tombstone


    1. Cassandra is quicker than postgre and have lower change to lose data. Cassandra doesn't have foreign keys, locking mechanism and etcs, so that it's quicker on writes.

    2. Everything in cassandra is a write. Insert/update/delete is also write.

    3. Setting a column to null/ deleting a column will create a tomestone; Deleting a row/primary key/partitio will create a single row tomestone

    4. Could adjust tombstone_warn_threshold and tombstone_failure_threshold in cassandra.yaml.

    5. Could adjust gc_grace_seconds when creating table 

    6. Hitting tombstone limit only happens per query.

    Related Attributes

    Delete will create tombstones

    • tombstone_warn_threshold: 1000 (default), could be found in cassandra.yaml

    • tombstone_failure_threshold: 100000 (default), could be found in cassandra.yaml

    • tombstone_compaction_interval: table attribute

    • min_compaction_threshold: table attribute    #Compaction will only be eligible after min_compaction_threshold SSTables exist, by default it’s 4.

    • gc_grace_seconds: table attribute

    • snapshot_before_compaction: false

     

    Check table attributes here http://docs.datastax.com/en/cassandra/2.1/cassandra/reference/referenceTableAttributes.html


    Could consider using DateTieredCompactionStrategy instead of the default SizeTieredCompactionStrategy.

    Cassandra MBean

    Use Jconsole to remotely connect to:

    hostname:7199

    e.g. localhost:7199


    Check/change the TombstoneFailureThreshold attribute inside StorageService MBean.

    Force a flush and compaction

    sudo nodetool -h localhost -p 7199 -u OC_APP_RAINBOWDBA -pw a3c224d4b89192d2ea3ea943dd7e9648 flush rainbowdba undeliveredmessage


    sudo nodetool -h localhost -p 7199 -u OC_APP_RAINBOWDBA -pw a3c224d4b89192d2ea3ea943dd7e9648 compact rainbowdba undeliveredmessage


    Deleted rows will only disappear when gc_grace_seconds time passed and a flush and compaction has been forced

    Truncating Table

    Truncating a table is an immediate operation and won’t leave any tomestones.


    Don’t insert Null into columns

    Inserting a null value to the column will leave a cell tomestone. Deleting a partition/row will also create a single row tombstone.


    Deleting a partition will create a partition tomestone and override the existing cell tomestones. This only happens in memory table not on the disk. Not sure whether creating a partition tomestone will cause a compaction of the cell tomestones on disk.


    Using TTL

    insert into undeliveredmessage("id", "message","type") values('1','message','RAVEN') using ttl 5;


    This query will result in 3 tomestone cells and one row tombstone.

    Cassandra partition size limitation

    In Cassandra, the maximum number of cells (rows x columns) in a single partition is 2 billion.


    Additionally, a single column value may not be larger than 2GB. Partitions greater than 100Mb can cause significant pressure on the heap.

    Performance Test

    Test script TestCassandraPerformance.java could be found in

    Cassandra version: 2.2.3, cqlsh 5.0.1

    1. TombstoneFailureThreshold = 500

    Seems persist 102000 rows and then delete them won’t hit the limit of the tomestone.


    2. TombstoneFailureThreshold = 1

    insert into undeliveredmessage("id","message","sent","type") values('3', 'message3', True, null);


    and then select * from undeliveredmessage is fine


    2. TombstoneFailureThreshold = 1

    insert into undeliveredmessage("id","message","sent","type") values('3', 'message3', null, null);


    and then select * from undeliveredmessage will hit the tomestone limit



    deleted rows number

    existing rows number

    locally recovery time

    vector 2

    recovery time

     

    150_000 * 9

    Operation Timed Out

    Operation Timed Out

     

    150_000 * 5

    9_072 ms

    10_152 ms

     

    150_000 * 3

    5_679 ms

    7_957 ms

     

    150_000 * 2

    3_025 ms

    5_218 ms

     

    150_000

    1_326 ms

    1_879 ms

    150_000

     

    158 ms

    333 ms

    150_000 *2

     

    562 ms

    1_963 ms

    150_000 *3

     

    2_223 ms

    3_833 ms

    150_000 *5

     

    3_476 ms

    9_726 ms

    150_000 *10

     

    Operation Timed Out

    Operation Timed Out

    150_000

    150_000

    1_321 ms

    3_735 ms

    150_000 *2

    150_000

    1_893 ms

    4_939 ms



    Note that we will hit timeout issue when having 150_000 *10 deleted rows in the table.

    Hitting tombstone limit


    For Dash you should see

    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)))


    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)


    For query in command line, you should see something like:

    Traceback (most recent call last):

     File "/usr/bin/cqlsh.py", line 1172, in perform_simple_statement

       rows = future.result(self.session.default_timeout)

     File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 3347, in result

       raise self._final_exception

    ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

  • 相关阅读:
    android音乐播放器开发 SweetMusicPlayer 摇一摇换歌
    kernel logo到开机动画之间闪现黑屏(android 5.X)
    [DLX反复覆盖] hdu 3656 Fire station
    使用清华源 tensorflow 安装
    类别不平衡问题之SMOTE算法(Python imblearn极简实现)
    在Ubuntu 18.04上安装OpenCV 4(C ++和Python)
    OpenCV
    git clone速度太慢的解决办法
    OpenCV 3.4.2 环境搭建(适用于Ubuntu 一键安装)
    ubuntu16.04安装opencv3.4.1教程
  • 原文地址:https://www.cnblogs.com/codingforum/p/6275832.html
Copyright © 2020-2023  润新知