pls help: performance tuning

[ST Wong (ITSC)]

Hi all,

I'm using openldap 2.4.21 + BerkeleyDB 4.8.26 on a machine with 4 cores. The system occassionally does accept connection and get error:

May 9 03:03:31 holiday slapd[2382]: connection_input: conn=645254 deferring operation: too many executing

I currently set thread in slapd.conf to 32 (default x 2). While I think shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase no. of threads, at the cost of read performance. Besides, after increasing thread to over 32, say 100, I got warning of "threads larger than twice the default".

Would like to know the recommended way to tune the system in our case. Would anyone pls help?
Sorry for the newbie question.

Thanks a lot.
/ST Wong (st-***@cuhk.edu.hk)

[Aaron Richton]

Post by ST Wong (ITSC)
I currently set thread in slapd.conf to 32 (default x 2). While I think
shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase no. of
threads, at the cost of read performance. Besides, after increasing
thread to over 32, say 100, I got warning of "threads larger than twice
the default".
Would like to know the recommended way to tune the system in our case.
Would anyone pls help?

"Recommended way to tune the system" is quite the slippery slope, but
here goes nothing...

It sounds like you're focusing primarily on parallelism, which certainly
has its merits. But I might sooner look at latency per operation. For
example, with loglevel stats, you should have visiblity of client
operations that are taking too long (i.e. delay until RESULT lines).
Perhaps an application started searching a field that isn't properly
indexed, for example.

Also, you don't mention results from looking for obvious bottlenecks
(spindles, bus, cpu, etc.). For example, if you're waiting on hard drives,
adding more threads is unlikely to help and in many cases would hurt. The
likely solution there, short of a hardware upgrade, is adding more
replicas (which also has benefits for redundancy/HA concerns, in addition
to alleviating hardware limitations).

Finally, you mention that you're using BerkeleyDB. You should post the
size of your working set and your DB_CONFIG parameters. A 4-core system
sounds fairly modern, it generally shouldn't be going to disk...if
DB_CONFIG is not tuned accordingly, that would drastically affect
performance. You can get some ideas for DB_CONFIG tuning in the
FAQ-O-Matic and list archives.

[Howard Chu]

Post by Aaron Richton

Post by ST Wong (ITSC)
I currently set thread in slapd.conf to 32 (default x 2). While I think
shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase no. of
threads, at the cost of read performance. Besides, after increasing
thread to over 32, say 100, I got warning of "threads larger than twice
the default".
Would like to know the recommended way to tune the system in our case.
Would anyone pls help?

"Recommended way to tune the system" is quite the slippery slope, but
here goes nothing...

Note that the "too many executing" message refers to the number of operations
executing on a single connection. The first thing to do is find out why this
single connection is so busy. Well behaved clients would not be spitting out
hundreds of operations over a single connection all at once.

Post by Aaron Richton
It sounds like you're focusing primarily on parallelism, which certainly
has its merits. But I might sooner look at latency per operation. For
example, with loglevel stats, you should have visiblity of client
operations that are taking too long (i.e. delay until RESULT lines).
Perhaps an application started searching a field that isn't properly
indexed, for example.
Also, you don't mention results from looking for obvious bottlenecks
(spindles, bus, cpu, etc.). For example, if you're waiting on hard drives,
adding more threads is unlikely to help and in many cases would hurt. The
likely solution there, short of a hardware upgrade, is adding more
replicas (which also has benefits for redundancy/HA concerns, in addition
to alleviating hardware limitations).
Finally, you mention that you're using BerkeleyDB. You should post the
size of your working set and your DB_CONFIG parameters. A 4-core system
sounds fairly modern, it generally shouldn't be going to disk...if
DB_CONFIG is not tuned accordingly, that would drastically affect
performance. You can get some ideas for DB_CONFIG tuning in the
FAQ-O-Matic and list archives.

[ST Wong (ITSC)]

Hi all,

Thanks for your help.

Regarding the "too many executing" source, it is an anti-spam/anti-virus
gateway blackbox that performs recipient checking using ldap query.
I'm afraid it's difficult to change her behaviour to split out hundreds
of operations over multiple connections. I'm check the product manual
of this blackbox, but probably won't be able to change it. Besides,
there are some other deferring operations logged from the same source:

11:May 10 03:09:38 fiesta slapd[31600]: connection_input: conn=633362
deferring operation: pending operations
11:May 10 03:04:56 fiesta slapd[31600]: connection_input: conn=712193
deferring operation: binding

The slapd doesn't respond or respond slowly when the problem occurs.
The servers each has 4 GB RAM.
Here comes some setting in slapd.conf and DB_CONFIG:

slapd.conf:
sizeLimit 50
timeLimit 600
cacheSize 1000000
checkPoint 1024 10
IDLcacheSize 3000
threads 32
sockbuf_max_incoming 4194303


DB_CONFIG:
set_cachesize 2 0 1
set_lg_bsize 20971520
set_lg_max 83886080
set_lg_regionmax 2621440
set_lk_max_locks 10000
set_lk_max_objects 5000


Would like to know if there is any problem with the setting.
Thanks again.
/ST Wong

-----Original Message-----
From: openldap-software-bounces+st-wong=***@OpenLDAP.org
[mailto:openldap-software-bounces+st-wong=***@OpenLDAP.org] On
Behalf Of Howard Chu
Sent: Wednesday, May 12, 2010 2:25 AM
To: Aaron Richton
Cc: ST Wong (ITSC); openldap-***@openldap.org
Subject: Re: pls help: performance tuning

Post by Aaron Richton

Post by ST Wong (ITSC)
I currently set thread in slapd.conf to 32 (default x 2). While I
think shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase
no. of threads, at the cost of read performance. Besides, after
increasing thread to over 32, say 100, I got warning of "threads
larger than twice the default".
Would like to know the recommended way to tune the system in our case.
Would anyone pls help?

"Recommended way to tune the system" is quite the slippery slope, but
here goes nothing...

Note that the "too many executing" message refers to the number of
operations executing on a single connection. The first thing to do is
find out why this single connection is so busy. Well behaved clients
would not be spitting out hundreds of operations over a single
connection all at once.

Post by Aaron Richton
It sounds like you're focusing primarily on parallelism, which
certainly has its merits. But I might sooner look at latency per
operation. For example, with loglevel stats, you should have visiblity
of client operations that are taking too long (i.e. delay until RESULT

lines).

Post by Aaron Richton
Perhaps an application started searching a field that isn't properly
indexed, for example.
Also, you don't mention results from looking for obvious bottlenecks
(spindles, bus, cpu, etc.). For example, if you're waiting on hard
drives, adding more threads is unlikely to help and in many cases
would hurt. The likely solution there, short of a hardware upgrade, is
adding more replicas (which also has benefits for redundancy/HA
concerns, in addition to alleviating hardware limitations).
Finally, you mention that you're using BerkeleyDB. You should post the
size of your working set and your DB_CONFIG parameters. A 4-core
system sounds fairly modern, it generally shouldn't be going to
disk...if DB_CONFIG is not tuned accordingly, that would drastically
affect performance. You can get some ideas for DB_CONFIG tuning in the
FAQ-O-Matic and list archives.