Consumer bdb log explosion

[M***@humboldt.edu]

We are running slapd 2.3.43 (Using bdb db4-4.3.29-10.el5) with one provider and two consumers and syncrpl (kerberos/SASL based).
Occasionally when heavy changes to the provider are made, the consumer bdb logs go crazy and fill up the bdb volume.
When this occurs we see approximately 1 10MB file created per second until the 14GB of free space in the volume has been exhausted, often in around an hour.
These packages are from the Buchan Milne repo.

Any ideas would be greatly appreciated.

Thanks!

(I have ldap-hot-db-backup running hourly)

DB Volume size = 14GB free.
1 10MB file per second

DB_CONFIG
set_lk_max_lockers 8000
set_lk_max_objects 8000

# one 0.25 GB cache
set_cachesize 0 268435456 1

# Data Directory
#set_data_dir db

# Transaction Log settings
set_lg_regionmax 262144
set_lg_bsize 2097152
#set_lg_dir logs

[Quanah Gibson-Mount]

Post by M***@humboldt.edu
We are running slapd 2.3.43 (Using bdb db4-4.3.29-10.el5) with one
provider and two consumers and syncrpl (kerberos/SASL based).
Occasionally when heavy changes to the provider are made, the consumer
bdb logs go crazy and fill up the bdb volume.
When this occurs we see approximately 1 10MB file created per second
until the 14GB of free space in the volume has been exhausted, often in
around an hour.
These packages are from the Buchan Milne repo.
Any ideas would be greatly appreciated.

I would suggest you add the auto remove flag, so that unused logs are
deleted.

--Quanah


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Aaron Richton]

Post by M***@humboldt.edu
We are running slapd 2.3.43 (Using bdb db4-4.3.29-10.el5) with one provider and two consumers and syncrpl (kerberos/SASL based).
Occasionally when heavy changes to the provider are made, the consumer bdb logs go crazy and fill up the bdb volume.

Remove unnecessary log files prior to volume fill, hopefully in
conjunction with a sane backup procedure (I don't know what
ldap-hot-backup is, but hopefully it eventually calls slapcat(8) and that
output gets put somewhere safe). I assume "10MB file per second" refers to
transaction log file size, which sounds like sufficient granularity to
remove in a timely fashion in most installations.

http://www.openldap.org/faq/data/cache/738.html

[M***@humboldt.edu]

Thank you Aaron and Quanah,
At Quanah's recommendation, I added the DB_CONFIG flag to auto-remove inactive files.
Essentially the ldap-hot-db-backup runs the db_archieve utility hourly!
The archive utility is supposed to backup transaction logs to a different file-system and delete the inactive files.
Hopefully this config change will reduce the excessive number of transaction logs and solve our problems.

The database is backed up every night using slapcat, and I can also recover from the backup directory.

I guess what I don't understand is why the transaction logs occasionally build up so fast. I don't have these problems on the provider (master).
We've had these problems off and on since we went to syncrpl last summer.

Thanks for your comments.
Mark
----- Original Message -----
From: "Aaron Richton"
Subject: Re: Consumer bdb log explosion

Post by M***@humboldt.edu
We are running slapd 2.3.43 (Using bdb db4-4.3.29-10.el5) with one provider and two consumers and syncrpl (kerberos/SASL based).
Occasionally when heavy changes to the provider are made, the consumer bdb logs go crazy and fill up the bdb volume.

Remove unnecessary log files prior to volume fill, hopefully in
conjunction with a sane backup procedure (I don't know what
ldap-hot-backup is, but hopefully it eventually calls slapcat(8) and that
output gets put somewhere safe). I assume "10MB file per second" refers to
transaction log file size, which sounds like sufficient granularity to
remove in a timely fashion in most installations.

http://www.openldap.org/faq/data/cache/738.html