This discussion is archived
9 Replies Latest reply: Jun 6, 2012 7:39 AM by Charles Lamb RSS

KVStrore depoyment related questions

940750 Newbie
Currently Being Moderated
I have deployed a 4 node KVStore with replication facor of 2 and partition to be 2, I have following doubts:

1) Is Number of Replication Group is determined by Replication factor or partition or both?
2) Which among a Replication group is master, as I never assigned it, is it dynamic and is there a provision to control it?
2) if I define partition to be 4, will my data table will be splitted in 4 equal parts on the basis md5 value of keys and can we determine size of partition as per our need?
3) Can we define whch replication group will store which partition.
4) To which machine of KVstore my client should connect?
  • 1. Re: KVStrore depoyment related questions
    Charles Lamb Pro
    Currently Being Moderated
    937747 wrote:
    I have deployed a 4 node KVStore with replication facor of 2 and partition to be 2, I have following doubts:

    1) Is Number of Replication Group is determined by Replication factor or partition or both?
    It is determined by the number of storage nodes and the replication factor.

    You should configure your store to have a much larger number of partitions than 2. In general, it should be about 10x the number of Replication Groups in your system. So 200 would be a better number.
    2) Which among a Replication group is master, as I never assigned it, is it dynamic and is there a provision to control it?
    It is dynamic.
    2) if I define partition to be 4, will my data table will be splitted in 4 equal parts on the basis md5 value of keys and can we determine size of partition as per our need?
    The size of a partition is dynamic depending on what is stored in it. You can not control which partition a k/v pair will be stored in.
    3) Can we define whch replication group will store which partition.
    No.
    4) To which machine of KVstore my client should connect?
    Any of the nodes. Upon initial connection, the client will read the topology of the store and then connect to the actual node containing the k/v pair for future requests.

    Charles Lamb
  • 2. Re: KVStrore depoyment related questions
    940750 Newbie
    Currently Being Moderated
    "You should configure your store to have a much larger number of partitions than 2. In general, it should be about 10x the number of Replication Groups in your system. So 200 would be a better number."

    can u explain reason for same

    Thanks
    Rishabh Agrawal
  • 3. Re: KVStrore depoyment related questions
    Charles Lamb Pro
    Currently Being Moderated
    937747 wrote:
    "You should configure your store to have a much larger number of partitions than 2. In general, it should be about 10x the number of Replication Groups in your system. So 200 would be a better number."

    can u explain reason for same
    Partitions are the unit of granularity for (future) operations like partition migration when we add the ability to increase the number of nodes in the system. If you don't have a small enough unit of granularity, we won't be able to split things up to redistribute them when new nodes are added.

    Charles Lamb
  • 4. Re: KVStrore depoyment related questions
    940750 Newbie
    Currently Being Moderated
    Thanks a lot Chalrles that was really helpful
  • 5. Re: KVStrore depoyment related questions
    Charles Lamb Pro
    Currently Being Moderated
    By the way, using a rep factor of 2 is not ideal. If for no other reason, if you lose a node in a Rep Group (shard) you will not be able to commit any write transactions.

    Charles Lamb
  • 6. Re: KVStrore depoyment related questions
    940750 Newbie
    Currently Being Moderated
    What if I loose only secondary one and not primary one. Will still be there be a writing issue, if yes, then why???


    Thanks and Regards
    Rishabh Agrawal
  • 7. Re: KVStrore depoyment related questions
    Charles Lamb Pro
    Currently Being Moderated
    937747 wrote:
    What if I loose only secondary one and not primary one. Will still be there be a writing issue, if yes, then why???
    It depends on the Durability Replica Ack Policy that you specify. Assuming you use simple_majority (the recommended setting and the default), then it doesn't matter which one you lose -- there will not be a majority. If you use something less than simple_majority (no_ack), then you can still commit transactions. But that is not recommended.

    Charles Lamb
  • 8. Re: KVStrore depoyment related questions
    940750 Newbie
    Currently Being Moderated
    Where exaclty we can define such policy.
  • 9. Re: KVStrore depoyment related questions
    Charles Lamb Pro
    Currently Being Moderated
    You can specify a Durability parameter to any write operation. You can also set the default Durability policy for the KVStore object.

    But do not set the Replica Ack policy less than simple_majority without understanding the implications of recovery and transaction rollback.

    Charles Lamb

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points