Multi-Master, referrals, and chaining

[Ryan Steele]

Hey folks,

As it stands, I've got an environment with six slapd servers (2.4.18) in an N-Way Multi-Master configuration. At any
given moment, only one of these six nodes is receiving client requests (reads, writes, etc.). We use Pacemaker to
provide some basic high-availability services, such that if the current 'master' (a term I'll use in this thread to
refer to the one node receiving all reads/writes from clients) croaks, a shared IP/hostname resource is migrated to the
next available server.

I would like to modify this setup such that ALL of these six nodes can receive client requests, but in the interest of
preserving data consistency, I'd like to refer writes to just one of them (that one being the current 'master'), such
that the rest receive the update(s) via syncrepl. From my own research and reading, it seems as though I'll need a
combination of referrals (specifically, the 'updateref' directive) and chaining, with the latter being introduced so
that the job of chasing the referrals is handled server-side instead of being offloaded to the clients. I have two
questions about this setup, though:

1. In order to prevent an endless loop of referrals, I'm assuming that the terminal node (the 'master') should NOT have
an updateref option that points to the 'master' shared IP/hostname (i.e., itself), like the rest of the nodes do. Is
this correct, or will slapd realize that the referral target is itself, and answer authoritatively?

2. If the assumption in #1 is correct, what is the most advisable way to handle referrals in the situation where the
identity of the terminal node is subject to change? My thought was to trigger an ldapmodify operation which effectively
removes the olcUpdateRef option during the failover, but I'm open to better ideas if they exist.

Thanks!

-Ryan

[Ryan Steele]

I actually realized that my logic was slightly faulty - since each of the six masters is capable of performing the write
operation, the referral will never be chased. Given that, is it impossible to have an N-Way MultiMaster configuration
in which writes are possible on any node, but the writes at any given time are performed on only one of them to ensure
consistency? That is, short of doing what I'm doing now (sending ALL traffic to only one master), which doesn't scale
very well.

Post by Ryan Steele
Hey folks,
As it stands, I've got an environment with six slapd servers (2.4.18) in an N-Way Multi-Master configuration. At any
given moment, only one of these six nodes is receiving client requests (reads, writes, etc.). We use Pacemaker to
provide some basic high-availability services, such that if the current 'master' (a term I'll use in this thread to
refer to the one node receiving all reads/writes from clients) croaks, a shared IP/hostname resource is migrated to the
next available server.
I would like to modify this setup such that ALL of these six nodes can receive client requests, but in the interest of
preserving data consistency, I'd like to refer writes to just one of them (that one being the current 'master'), such
that the rest receive the update(s) via syncrepl. From my own research and reading, it seems as though I'll need a
combination of referrals (specifically, the 'updateref' directive) and chaining, with the latter being introduced so
that the job of chasing the referrals is handled server-side instead of being offloaded to the clients. I have two
1. In order to prevent an endless loop of referrals, I'm assuming that the terminal node (the 'master') should NOT have
an updateref option that points to the 'master' shared IP/hostname (i.e., itself), like the rest of the nodes do. Is
this correct, or will slapd realize that the referral target is itself, and answer authoritatively?
2. If the assumption in #1 is correct, what is the most advisable way to handle referrals in the situation where the
identity of the terminal node is subject to change? My thought was to trigger an ldapmodify operation which effectively
removes the olcUpdateRef option during the failover, but I'm open to better ideas if they exist.
Thanks!
-Ryan

[Aaron Richton]

Post by Ryan Steele
I actually realized that my logic was slightly faulty - since each of the six masters is capable of performing the write
operation, the referral will never be chased. Given that, is it impossible to have an N-Way MultiMaster configuration
in which writes are possible on any node, but the writes at any given time are performed on only one of them to ensure
consistency? That is, short of doing what I'm doing now (sending ALL traffic to only one master), which doesn't scale
very well.

Have you considered mirrormode (possibly combined with some back-ldap on
the consumers so that you can present as only one always-able-to-write
IP)?

With N-way multimaster you lose write performance as you add additional
servers, as the write needs to be committed across an increasing N. More
commits means more complexity means less performance. mirrormode keeps
N=1, for good write performance, while still allowing for redundancy.

[Clowser, Jeff]

I'd probably set it up a little differently. Something like this (very
general/generic):

2 of your 6 servers as suppliers/masters, accepting writes. If you are
using a load balancer, set it up so that you have a vip that always goes
to one of these servers (the "primary" supplier), and directs traffic to
the other ("secondary" supplier) only when the primary in unavailable.
This gives you redundancy for high availability, while minimizing risk
of replication conflicts.

4 of your 6 servers as consumers. If you are using a load balancer,
create a vip for your "read-only" ldap cluster, such that traffic is
balanced across these servers. Referrals on these should go to the
supplier vip (not directly to either "real" supplier). If your client
apps chase referrals, you are fine. If not, you may need to set up
chaining to pass writes to the supplier vip.

To scale read access, add more servers to the consumer cluster (scale
horizontally). To scale your suppliers, get bigger boxes (Scale
vertically - distributing writes across more supplier servers doesn't
help with scaling, since replication kicks in and they all see all the
write load anyway, but details of that have been discussed ad nauseum
here).

Caveats:
If you are using SSL/TLS, there are some complications with naming and
making certs match up.
If you don't use load balancers, you'll have to do some config/dns/etc
tricks, maybe some manual or scripted failover, etc.


If you are wed to your current setup - you'll probably need to do
something like:
- Make only one server the supplier
- Make the rest consumers
- Monitor the supplier, and write a script that makes all the
appropriate config changes to let you quickly promote a consumer to
supplier and redirect the remaining consumers, but this can get tricky,
I'm thinking.

-----Original Message-----
org

penLDAP.org] On Behalf Of Ryan Steele

Sent: Wednesday, March 24, 2010 12:38 PM
Subject: Re: Multi-Master, referrals, and chaining
I actually realized that my logic was slightly faulty - since
each of the six masters is capable of performing the write
operation, the referral will never be chased. Given that, is
it impossible to have an N-Way MultiMaster configuration
in which writes are possible on any node, but the writes at
any given time are performed on only one of them to ensure
consistency? That is, short of doing what I'm doing now
(sending ALL traffic to only one master), which doesn't scale
very well.

Post by Ryan Steele
Hey folks,
As it stands, I've got an environment with six slapd

servers (2.4.18) in an N-Way Multi-Master configuration. At any

Post by Ryan Steele
given moment, only one of these six nodes is receiving

client requests (reads, writes, etc.). We use Pacemaker to

Post by Ryan Steele
provide some basic high-availability services, such that if

the current 'master' (a term I'll use in this thread to

Post by Ryan Steele
refer to the one node receiving all reads/writes from

clients) croaks, a shared IP/hostname resource is migrated to the

Post by Ryan Steele
next available server.
I would like to modify this setup such that ALL of these

six nodes can receive client requests, but in the interest of

Post by Ryan Steele
preserving data consistency, I'd like to refer writes to

just one of them (that one being the current 'master'), such

Post by Ryan Steele
that the rest receive the update(s) via syncrepl. From my

own research and reading, it seems as though I'll need a

Post by Ryan Steele
combination of referrals (specifically, the 'updateref'

directive) and chaining, with the latter being introduced so

Post by Ryan Steele
that the job of chasing the referrals is handled

server-side instead of being offloaded to the clients. I have two

Post by Ryan Steele
1. In order to prevent an endless loop of referrals, I'm

assuming that the terminal node (the 'master') should NOT have

Post by Ryan Steele
an updateref option that points to the 'master' shared

IP/hostname (i.e., itself), like the rest of the nodes do. Is

Post by Ryan Steele
this correct, or will slapd realize that the referral

target is itself, and answer authoritatively?

Post by Ryan Steele
2. If the assumption in #1 is correct, what is the most

advisable way to handle referrals in the situation where the

Post by Ryan Steele
identity of the terminal node is subject to change? My

thought was to trigger an ldapmodify operation which effectively

Post by Ryan Steele
removes the olcUpdateRef option during the failover, but

I'm open to better ideas if they exist.

Post by Ryan Steele
Thanks!
-Ryan

[Ryan Steele]

Hey Darryl,

I decided to use a mixture of MMR and slaves. Specifically, I'll have two masters doing MMR with a proxy (well, a
floating IP/hostname that achieves the same thing, anyways) in front to direct traffic to the 'active' master, with the
high-availability services being provided by Pacemaker, which will swap resources if the active master dies or becomes
unresponsive). Additionally, I'll now have four slaves, each of which will happily perform reads and use an updateref
to send writes to the active master. Along with that, I'll use chaining to have the slaves automatically chase the
referrals for the clients, so it seems transparent from their end. With this model, I can still guarantee the
consistency of my data, distribute whatever reads don't hit the local caches more evenly among the cluster for better
performance, and still have a redundant set of masters to preserve continuity. The nice thing is that altering my
current six-node MMR setup to fit this new architecture is pretty simple, requiring only a few changes to the newly
demoted slaves (such as disabling MirrorMode and simplifying some of the ACLs). Thanks for the suggestion, though!

Ryan,
In order to do what you're suggesting, you'd probably need an intermediate layer ... a proxy server that routes write requests to a specific server. I know Sun's LDAP Proxy that ships with DSEE 6.x can do that and I'm sure there are others as well. I suppose you could also do this with MirrorMode replication rather than the multi-master model. See http://www.openldap.org/doc/admin24/replication.html#MirrorMode%20replication
--Darryl
-----Original Message-----
Behalf Of Ryan Steele
Sent: Wednesday, March 24, 2010 12:38 PM
Subject: Re: Multi-Master, referrals, and chaining
I actually realized that my logic was slightly faulty - since each of the six masters is capable of performing the write
operation, the referral will never be chased. Given that, is it impossible to have an N-Way MultiMaster configuration
in which writes are possible on any node, but the writes at any given time are performed on only one of them to ensure
consistency? That is, short of doing what I'm doing now (sending ALL traffic to only one master), which doesn't scale
very well.

Post by Ryan Steele
Hey folks,
As it stands, I've got an environment with six slapd servers (2.4.18) in an N-Way Multi-Master configuration. At any
given moment, only one of these six nodes is receiving client requests (reads, writes, etc.). We use Pacemaker to
provide some basic high-availability services, such that if the current 'master' (a term I'll use in this thread to
refer to the one node receiving all reads/writes from clients) croaks, a shared IP/hostname resource is migrated to the
next available server.
I would like to modify this setup such that ALL of these six nodes can receive client requests, but in the interest of
preserving data consistency, I'd like to refer writes to just one of them (that one being the current 'master'), such
that the rest receive the update(s) via syncrepl. From my own research and reading, it seems as though I'll need a
combination of referrals (specifically, the 'updateref' directive) and chaining, with the latter being introduced so
that the job of chasing the referrals is handled server-side instead of being offloaded to the clients. I have two
1. In order to prevent an endless loop of referrals, I'm assuming that the terminal node (the 'master') should NOT have
an updateref option that points to the 'master' shared IP/hostname (i.e., itself), like the rest of the nodes do. Is
this correct, or will slapd realize that the referral target is itself, and answer authoritatively?
2. If the assumption in #1 is correct, what is the most advisable way to handle referrals in the situation where the
identity of the terminal node is subject to change? My thought was to trigger an ldapmodify operation which effectively
removes the olcUpdateRef option during the failover, but I'm open to better ideas if they exist.
Thanks!
-Ryan

The information contained in this message and any attachment may be
proprietary, confidential, and privileged or subject to the work
product doctrine and thus protected from disclosure. If the reader
of this message is not the intended recipient, or an employee or
agent responsible for delivering this message to the intended
recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited.
If you have received this communication in error, please notify me
immediately by replying to this message and deleting it and all
copies and backups thereof. Thank you.