Managing Replication and Referrals

eplication and referrals are both important mechanisms for extending your directory service beyond a single server configuration. This chapter describes how you can use both replication and referrals for your directory service. This chapter contains the following sections:

• "Replication"

• "Configuring a Server for Replication"

• "Creating Replication Agreements"

• "Managing Supplier-Initiated Agreements"

• "Managing Consumer-Initiated Agreements"

• "Initializing Consumers"

• "Supplier-Initiated Replication Algorithm"

• "Supplier-Initiated Replication Algorithm"

• "Consumer-Initiated Replication Algorithm"

• "Machine data"

• "Managing Referrals"

Replication

Note that all directory data is mastered in one and only one directory location. The server that masters directory data is called the supplier server. The supplier server can then propagate changes made to its directory data to other servers which are known as consumer servers.

There are two basic forms of replication available to you: supplier-initiated replication and consumer-initiated replication. Supplier-initiated replication allows you to configure a supplier server to push data to consumer server. Consumer-initiated replication allows you to configure consumer servers to pull directory data from supplier servers.

Directory servers use replication agreements to define replication. A replication agreement identifies the directory objects that are candidates for replication, the times during which replication can occur, the server to which the replicated data is to be pushed, or the server from which replicated data is to be pulled.

This chapter provides detailed information on how to setup replication agreements for two basic forms of replication: supplier-initiated replication and consumer-initiated replication. For more overview information on what replication is and how you might use it, see the Netscape Directory Server Deployment Guide.

Configuring a Server for Replication

Configuring Servers for Supplier-Initiated Replication

1. On the consumer server, configure a DN for the supplier to use to bind to the consumer and a corresponding password (if you are using normal authentication) or a certificate subject DN (if you are using SSL).

2. On the supplier server, configure a change log directory and then restart the supplier server.

3. Create a supplier-initiated replication agreement.

Configuring Servers for Consumer-Initiated Replication

1. On the supplier server, configure a change log directory and then restart the supplier server.

2. On the supplier server, set up consumer access to the change log directory. This includes creating a DN and password for the consumer to use to bind to the supplier, as well as setting the access control instructions to allow the consumer to search and read the change log.

3. Create a consumer-initiated replication agreement.

Before you can create replication agreements, you have to configure your server to either be a supplier or a consumer (or both). To do this, you must configure basic information about the server.

Configuring the Supplier DN

Supplier servers use the supplier DN to bind to this server. Entries supplied to this server from another server can only be updated if the LDAP client binds as the supplier DN; all other update operations for the supplied data will be referred to the supplier server that masters the directory data. It is therefore important that the only LDAP clients that bind to this server using the supplier DN are supplier servers.

You can configure the server to accept normal bind operations from the supplier, or you can configure the server to accept certificate-based authentication. The following sections describe how to configure these two methods of authentication.

Configuring the Server to Accept Normal Authentication

After changing the values in these fields, you must apply your changes and then restart your server.

Configuring the Server to Accept Certificate-Based Authentication

For replication purposes, you must tell your consumer server what certificate(s) correspond to the supplier DN that you have defined for the consumer server. To do this, identify the supplier DN for the local server in the Supplier DN field just as you would for the normal authentication case. Leave the password fields blank, and in the Supplier SSL Clients box, enter the subject DN of the certificate that the supplier server will use to bind to this server. If you have more than one server supplying entries to this consumer, enter a subject DN for each supplier server. Each DN should be placed on a separate line in the dialog box.

You can find the subject DN of the certificate used by a supplier server by going to the supplier server's manager and looking at the bottom of the Server Preferences | Server On/Off form.

Related Topics

• "Managing Supplier-Initiated Agreements"

• "Supplier-Initiated Replication Algorithm"

• "Using Certificate-Based Authentication"

• "Activating SSL"

To configure a change log, you must identify the physical location where the change log will be stored and a directory suffix to use for the change log directory.

• In Changelog DB Directory, enter a fully qualified path name to the directory that will contain the change log. This directory must exist before identifying it to this parameter. Also, this directory must be located on a local disk.

• In Changelog Suffix, enter a DN to be used as this directory's suffix. Typically, this suffix is: cn=changelog

• In Max Changelog Records, enter the maximum number of records the change log can contain. If you leave this field blank, there is no maximum size for the change log.

• In Max Changelog Age, enter the maximum age of an entry in the change log. If you leave this field blank, entries will never be removed from the change log due to their age.

• "Managing Supplier-Initiated Agreements"

• "Supplier-Initiated Replication Algorithm"

• "Managing Consumer-Initiated Agreements"

• "Consumer-Initiated Replication Algorithm"

1. Create a directory entry that can be used by your consumer servers to read your change log. This directory entry does not have to be created in your change log directory tree; it can instead be a normal entry in your primary directory tree.

2. Provide an ACI statement at the root level of your change log tree that grants the user from step 1 read, search, and compare access to the entire change log tree. In addition, this ACI should grant full read, search, and compare privileges for the tree or subtree that the consumer server will retrieve from the supplier server.

For security reasons, you are recommended to not configure anonymous access for your change log directory tree. Also, you should grant only read, search, and compare access to the DN that your consumers will bind to your supplier with; do not provide any form of write or delete access to this tree.

Finally, for logging and auditing purposes, you may want to configure a different directory entry to be used by each consumer server for this purpose. This will allow you to track which consumer is binding to your server and when.

Related Topics

• Chapter 5, "Managing Access Control"

• "Viewing the Access Log"

• "Consumer-Initiated Replication Algorithm"

This form has two distinct areas:

• Supplier Initiated Agreements identify the agreements that this server is using to supply directory entries to consumer servers. From this area you can manage supplier-initiated replication.

• Consumer Initiated Agreements identify the agreements that this server is using to pull directory data from supplier servers. From this area, you can manage consumer-initiated replication.

• "Supplier-Initiated Replication Algorithm"

• "Consumer-Initiated Replication Algorithm"

• Add Consumer brings up the Replication Editor.

• Initialize Consumer causes the consumer corresponding to the highlighted replication agreement to be initialized or reinitialized. The consumer server must be running in order for it to be initialized.

  Use this button with caution since this button will cause your server to use online consumer creation. For more information, see "Initializing Consumers".

• Edit Agreement brings up the Replication Editor for the highlighted replication agreement.

• Delete Agreement causes the highlighted replication agreement to be deleted.

1. On the consumer server:
   • Configure a supplier DN and password.

   • Make sure that a suffix configured on the supplier server matches the suffix of the entries that will be supplied to the consumer.

   • If you are going to replicate just a subtree, make sure the consumer server has the subtree's parent entry. That is, if you are replicating the subtree represented by ou=people, st=Minnesota, o=airius.com, make sure that the entry for st=Minnesota, o=airius.com exists on the consumer server.

2. On the supplier server:
   • Configure a change log

   • Use Replica Config | Configure Replication Agreement to create a replication agreement.

   • Initialize the consumer. For details, see "Initializing Consumers".

Adding and Editing a Replication Agreement

The Replication Editor now appears.

In the Nickname field at the top of the editor, enter a unique name representative of this agreement.

You now have three tabs that you use to add a new consumer: Destination, Content, and Schedule. A fourth tab, Status, allows you to track replication activity.

The Destination Tab

• Enter the server name and the port on which the consumer server is running.

• Click Encrypted if you want to use SSL communications between the supplier and the consumer server. Note that you must have first configured your servers to use SSL.

• If you are not using SSL, or if you are using SSL with simple authentication, enter the consumer server's supplier DN in the Bind As field the consumer server's password in the Password field.

• If you are using SSL Client Authentication (which is available only if you are using SSL), click SSL Client Authentication. The Bind As and Password fields are now unavailable because the server will use its security certificate to authenticate to the consumer server.

  In order for you to select this option, you must first do the following:

  • Configure SSL for both your supplier and consumer servers (see "Activating SSL").

  • Configure your consumer server to recognize your supplier server's certificate as the supplier DN (see "Configuring the Supplier DN").

• Choose whether you want to replicate the entire tree beneath your directory suffix, or just some subtree within your directory tree. If you want to replicate a subtree, enter the subtree's DN (for example: ou=people, o=airius.com), or click Choose to browse and select the desired subtree.

  Note that if you are going to replicate a subtree, you must make sure the appropriate parent entry is available on the consumer server. For example, if you are replicating the ou=people, o=airius.com subtree, then you must first make sure the consumer server contains the o=airius.com entry

• "Configuring a Server for Replication"

Select "Always keep directories in sync" if you do not want the replication window to ever close.

The Status Tab

Managing Consumer-Initiated Agreements

• Add Supplier brings up the Replication Editor.

• Initialize Consumer causes this consumer to initialize or reinitialize itself from the supplier represented by the highlighted agreement. The supplier server must be running in order for your consumer to be initialized.

  Use this button with caution since this button will cause your server to use online consumer creation. For more information, see "Initializing Consumers".

• Edit Agreement brings up the Replication Editor for the highlighted replication agreement.

• Delete Agreement causes the highlighted replication agreement to be deleted.

1. On the supplier server:
   • Configure a change log.

   • Create the directory entry and access control rights necessary for the consumer server to read the change log directory.

2. On the consumer server:
   • Make sure that a suffix configured on the supplier server matches the suffix of the entries that will be supplied to the consumer.

   • If you are going to replicate just a subtree, make sure the consumer server has the subtree's parent entry. That is, if you are replicating the subtree represented by ou=people, st=Minnesota, o=airius.com, make sure that the entry for st=Minnesota, o=airius.com exists on the consumer server.

   • Use Replica Config | Configure Replication Agreement to create a replication agreement.

   • Initialize the consumer. For details, see "Initializing Consumers".

Adding and Editing a Replication Agreement

The Replication Editor now appears.

In the Nickname field at the top of the editor, enter a name or short description that is representative of this agreement.

You now have three tabs that you use to add a new consumer: Source, Content, and Schedule. A fourth tab, Status, allows you to track replication activity.

The Source Tab

• Enter the server name and the port on which the supplier server is running.

• Click Encrypted if you want to use SSL communications between the supplier and the consumer server. Note that you must have first configured your servers to use SSL.

• If you are not using SSL, enter the bind DN and password that the consumer should bind to the supplier with in the Bind As and Password fields. This bind DN and password must have read privileges for the supplier's change log directory.

• If you are using SSL Client Authentication (which is available only if you are using SSL) then click SSL Client Authentication. The Bind As and Password fields are now unavailable because the server will use its security certificate to authenticate to the supplier server.

  In order for you to select this option, you must first do the following:

  • Configure SSL for both your supplier and consumer servers (see "Activating SSL".)

  • Configure your supplier server to map your consumer server's certificate to a DN that has read privileges for the supplier's change log directory. For more information, see the "Mapping Client Certificates to LDAP" section in Managing Netscape Servers.

• Choose whether you want to replicate the supplier server's entire tree beneath its directory suffix, or just some subtree. If you want to replicate a subtree, enter the subtree's DN (for example: ou=people, o=airius.com), or click Choose to browse and select the desired subtree. Note that the DN browser will only show you those directory entries and branch points that the consumer server has read and search access for. This access is determined by the ACIs set on the supplier for the DN you configured in the Source tab.

  If you are going to replicate a subtree, you must make sure the appropriate parent entry is available on the consumer server. For example, if you are replicating the ou=people, o=airius.com subtree, then you must first make sure the consumer server contains the o=airius.com entry.

• "Configuring a Server for Replication"

Select "Always keep directories in sync" if you do not want the replication window to ever close.

The Status Tab

Initializing Consumers

There are two ways that a consumer can be initialized:

• Online Consumer Creation -- This method is the easiest to perform but is prohibitively time consuming for databases that are larger than 5,000 - 10,000 entries in size.

• Manual Consumer Creation -- This is the more difficult but most efficient method.

When to Initialize a Consumer

Once the tree has been physically placed on the consumer, the supplier server can begin begin replaying update operations to the consumer server. In addition, any attempts to modify data on the consumer that is owned by the supplier are referred to the supplier server.

Under normal operations, the consumer should not ever have to be initialized again. However, there are several major events that can require a reinitialization of the consumer server:

• The supplier's database version number does not match the version number stored on the consumer for the replicated entries. This will happen if the supplier's database is either reloaded or recovered from a backup.

  The database version number is a unique identifier that allows the supplier and the consumer to know that the database has not been reloaded since the last synchronization.

• The change log on the supplier server is damaged to the extent that the supplier cannot determine what changes to replay to the consumer. This can happen if the supplier's change log becomes corrupted (such as might happen in the event of a disk failure) or if the change log is trimmed before the trimmed changes can be replayed to the consumer server.

  This situation is most likely to arise if the consumer's database is restored from a backup and the supplier's change log was truncated sometime after that backup was taken.

  Change logs are trimmed based on the Maximum Changelog Age and Maximum Changelog Size parameters. For more information, see "Configuring the Change Log".

• The supplier server is continually repairing inconsistencies on the consumer server, or if you find that the supplier is unable to repair data inconsistencies on the consumer server, then you should reinitialize the consumer server. In almost all cases, the supplier can successfully repair inconsistencies on a consumer server. Further, even if it cannot repair data inconsistencies the supplier will continue to replicate to the consumer server. When the supplier server has detected a data inconsistency on a supplier server, the supplier issues the following message to the error log:

		Inconsistency detected while replaying change <n>, 
		entry <DN>, to replica <host>:<port>/<DN>

The process that you use to initialize or reinitialize a consumer differs depending on the type of consumer creation that you are using. See "Online Consumer Creation" (next) or "Manual Consumer Creation" for more information.

Note

Online consumer creation works by moving data from the supplier to the consumer server over LDAP. That is, the replicated information is placed on the consumer server using LDAP add operations. This is exactly the mechanism used by the supplier server to synchronize a consumer, and it requires very little action on your part to succeed.

Before using online consumer creation, consider the performance implications of this method of consumer initialization. On a reasonably fast single processor (such as an Intel Pentium II or a Sun Sparc Ultra 1), you can expect online consumer creation to proceed at the following rates:

• For fresh initializations (that is, if the consumer server's tree is empty), the supplier can add 18,000 - 36,000 entries per hour. The actual rate will depend on characteristics of your server such as the size of your entries, the amount of indexing your consumer is performing, the speed of your disk, the amount of RAM available to the consumer server, and the speed of your networks.

• For reinitializations, the supplier can add from 9,000 to 18,000 entries per hour. The performance drops by half because the online consumer creation process deletes all previously replicated entries from the consumer server before the consumer is initialized with a fresh set of data.

In addition, you should always use online consumer creation whenever your consumer server contains a directory tree that is mastered by more than one server. In this case, the manual creation process offers no performance advantage over online consumer creation.

How to Use Online Consumer Creation

1. Create a replication agreement in Replication | Configure Replication Agreements.

   For details on creating replication agreements, see "Creating a Supplier- Initiated Agreement" or "Creating a Consumer-Initiated Agreement".

2. Highlight the appropriate replication agreement.

3. Click the Initialize Consumer button, and then click Initialize in the confirmation box.

Online consumer creation begins immediately. You can find out if online consumer creation is finished by clicking the status tab. If online consumer creation is in progress, the status shows that a replica is being initialized. To update this window, you must click the Update button. When online consumer creation is finished, the status is changed to reflect this.

Note that you can configure your server to automatically reinitialize a consumer. To do this, place the following line in the slapd.conf file of either the supplier server (for supplier-initiated replication) or the consumer server (for consumer-initiated replication):

	orcauto on

Manual Consumer Creation

You should use the manual process whenever you find that the online process is inappropriate due to performance concerns. Also, you should never use the manual process if your consumer server contains directory data that is mastered by more than one directory server. That is, use the manual process only if your entire consumer server's database is supplied to it from a single supplier server. For more information, see "How to Perform Manual Consumer Creation".

For information on consumer initialization, see "Initializing Consumers". For information on the online consumer creation process, see "Online Consumer Creation".

How to Perform Manual Consumer Creation

1. Create a replication agreement as described in "Creating a Supplier-Initiated Agreement" or "Creating a Consumer-Initiated Agreement".

2. On the supplier server, convert the local directory tree that you want to replicate to LDIF (see below).

3. Import the LDIF file to the consumer server (see below).

To convert the local tree to LDIF, do the following on the supplier server:

1. Put your supplier's database in read-only mode to ensure the consistency of the LDIF file you are going to create.

   For information on how to put your database into read-only mode, see "Placing Your Database in Read-Only Mode". Note that this process requires a server restart.

2. From the Server Manager, go to Replication | Manual Consumer Initialization.

3. In the Copy Subtree field, enter the distinguished name of the subtree to be converted to LDIF.

4. In the To LDIF File field, enter the name of the LDIF file to be created. The file you specify will be placed in the NSHOME\slapd-<serverId>\ldif directory; you cannot specify a full file path in this field.

5. Click OK.

6. Take your database out of read-only mode. Again, this process requires a server restart.

To import the LDIF file to the consumer server:

1. Using FTP or some other mechanism, transfer the LDIF statements created by the supplier server to your consumer server. Remember that LDIF files are ASCII files, so if you are transferring the file between systems with different end-of-line (EOL) characters (such as between Windows NT and Unix), be sure to perform an ASCII transfer so that the EOL characters are converted appropriately.

2. Create your consumer server's database from the LDIF file by using either the Database Management | Import form, or the ns-slapd ldif2ldbm command-line utility.

   For more information, see "Importing LDIF Using the Server Manager", or "Importing LDIF from the Command Line".

It is important to note that you should never use this manual consumer creation process if your supplier server is mastering only part of the tree contained on the consumer server. This is because this process erases the consumer server's database which wipes out the rest of the consumer's tree.

If your consumer server contains data that is also mastered either by itself or by some other supplier server, then use the online consumer creation process (for details, see "Online Consumer Creation"). While it is possible to manually import this LDIF file, you must do so using ldapmodify which offers no performance improvement over online consumer creation because both mechanisms are adding entries over LDAP.

If for some reason you decide that you must manually initialize a consumer that contains data mastered by some other server than your supplier server, make sure you do the following:

• Make sure you create on the consumer server any entries that are parents of the replicated subtree before adding the replicated data. That is, if you are replicating l=Minneapolis, ou=Global, o=airius.com, make sure that you have created ou=Global, o=airius.com and o=airius.com on your consumer server before running the add operation.

• If you are reinitializing a consumer server, make sure you delete all of the contents of the replicated tree before running the add operation. If you do not delete the currently existing replicated tree, the server will fail the add operations, stating that the entries already exist.

• When you are adding or deleting replicated entries, bind to your consumer server using the supplier DN configured for that server. If you use any other DN (including the root DN), the consumer server will simply refer the modify operation to the supplier server.

1. Based on a schedule that you set, the supplier server determines that it is time to synchronize a consumer. The directory server identifies those subtrees that are replicated and the servers to which it is supplying those trees by using directory entries contained in the Machine data tree. Each consumer server is identified by a separate machine data entry. Part of each such entry is an identification of the root point of the replicated tree and a schedule indicating when the consumer should be updated.

2. The supplier server binds to the consumer server using the supplier DN and password that you provide. The supplier must use this special DN for the bind or all updates to the replicated tree will be referred back to the supplier server.

   You use the Replica Config | Configure Replication Agreements form to configure the supplier DN for a consumer server.

3. Each replicated tree contained on a consumer server should include a copiedFrom attribute that identifies the supplier of the subtree. This attribute is maintained by the replication subsystem. The supplier server examines the copiedFrom attribute in the replicated subtree's root point to ensure that no other supplier is identified by this attribute.

   If no such attribute exists for the subtree, the supplier server:

   1. Writes the attribute to the replicated root point along with the appropriate attribute value.

   2. Resets the consumer version identification to zero.

   3. Aborts and immediately retries the synchronization.

   If the copiedFrom attribute identifies some server (server B) other than the current supplier (server A), the supplier (server A) aborts synchronization and returns an error. This prevents any one replicated entry on the consumer from having multiple suppliers.

   If the copiedFrom attribute is set correctly, the supplier compares version identification on the replicated tree against the supplier's change log to determine whether the replicated tree needs to be updated. The change log is a log of all the changes made to the supplier's entries. Among other things, this log contains version identification used for synchronization purposes.

4. If no changes are required, the supplier terminates synchronization normally. Otherwise, the supplier updates and/or deletes all appropriate entries on the consumer as indicated by the change log. The supplier also records the last update number applied to the replica and sets the copiedFrom attribute at the top of the replicated tree on the consumer server. This identifies the tree as being a replica and, more importantly, identifies the supplier server as the master of the information in that tree.

5. The supplier exits synchronization normally.

Consumer-Initiated Replication Algorithm

1. Based on a schedule that you set (which is stored in the consumer server's machine data tree), the consumer server determines that it wants to be updated.

2. The consumer server obtains from its own machine data tree the location of each of its own directory trees that are supplied to it from another server. The consumer then examines the root point of each of these trees to make sure that a copiedFrom attribute is set on the root point.

   If no copiedFrom attribute exists on the tree, the consumer server adds one so that all write operations are appropriately referred to the supplier server.

3. The consumer server binds to the supplier server and searches the supplier's change log directory to see if the last update that the consumer has recorded in its directory is still contained in the supplier's directory. If it does not, then the consumer has no way of knowing what other changes may have occurred on the supplier since the last time that the consumer was updated. In this situation, the consumer simply reinitializes itself from the supplier server.

4. If the consumer's last update is still contained in the supplier's change log, then the consumer determines if any changes occurred on the supplier server that must be made to the consumer server's directory. If no changes are required, the consumer terminates synchronization normally. Otherwise, the consumer updates and/or deletes all appropriate entries in its directory tree as indicated by the change log. The consumer sets the appropriate version identification at the same time.

5. The consumer exits synchronization normally.

Machine data

A second tree is also created when you install your directory server. This tree is used to contain machine data. The machine data tree contains two top-level entries that identify the local server. The first entry uses the NetscapeMachineData object class to identify the domain components of the machine on which the server is installed. The second entry uses the LDAPServer object class to identify the port on which the LDAP server is listening.

The machine data tree also contains zero or more entries that identify consumer or supplier servers. On the suppler server, the machine data tree contains an entry for each consumer server to which the server replicates data. These consumer server entries use object class LDAPReplica. On the consumer servers, the machine data tree contains an entry for each supplier server. The suppler server entries use object class
cirReplicaSource.

See Appendix A, "Object Classes" for information on the NetscapeMachineData, LDAPServer, LDAPReplica, cirReplicaSource object classes.

The suffix for the machine data directory tree is

dc=<serverID>, dc=<domain>, dc=<domain_type>

For example, if your directory server is running on directory.airius.com, then the machine data suffix is

dc=directory, dc=airius, dc=com

Managing Referrals

• The client is attempting to modify an entry that is not mastered on the local server. That is, the entry is supplied to the local server by some other directory server. In this case, the consumer server returns a referral that indicates which server mastered the entry. The client can then follow the referral back to the supplier server and attempt the modification operation there.

• If the client requests a directory entry that cannot reside on the local server, then a referral is returned based on the value of the slapd.conf Referral parameter. The directory server determines whether this kind of a referral should be returned by comparing the DN of the requested directory object against the directory suffixes supported by the local server. If the DN and the supported suffixes do not match, the directory server will return a referral.

  You can manage this default referral mechanism from the Server Preferences | LDAP form. You should add an LDAP URL to this form. For example:

  ldap://directory.airius.com:389/o=airius.com

• If the client searches an entry, or tries to modify an entry, that contains a smart referral, then a referral is returned based on the LDAP URL contained on the smart referral.

Creating and Changing Smart Referrals

You create and manage smart referrals by using ldapmodify (for information on ldapmodify, see "Modifying Entries Using ldapmodify").

To create a smart referral, create the relevant directory entry with the Referral object class. This object class allows a single attribute: ref. The ref attribute is expected to contain an LDAP URL.

For example, to return a smart referral for the entry uid=bjensen, ou=people, o=airius.com, create the entry as follows:

	dn: uid=bjensen, ou=people, o=airius.com
	objectclass: top
	objectclass: referral
	ref: ldap://directory.europe.airius.com/cn=babs jensen, o=people, 
	 l=europe, o=airius.com

For more information on smart referrals, see Chapter 8 of the Netscape Directory Server Deployment Guide. For information on LDAP URLs, see Appendix C, "LDAP URLs."