too many open files

[Hugo Monteiro]

Hello list,

Although the subject suggests a fairly know issue, i cannot seem to
understand the cause.

Log file presents

Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.allow: Too many open files
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.deny: Too many open files

, but

proxyldap1:~# cat /proc/sys/fs/file-max
100000
proxyldap1:~# cat /proc/sys/fs/file-nr
1088 0 100000
proxyldap1:~# lsof -p `pidof slapd`| wc -l
460
proxyldap1:~# su -c "ulimit -n" openldap
8192

slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm,
using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.

Have i overlooked any aspect?

Thank you in advance,

Hugo Monteiro.
--
fct.unl.pt:~# cat .signature

Hugo Monteiro
Email : ***@fct.unl.pt
Telefone : [phone] Ext.15307
Web : http://hmonteiro.net

Divisão de Informática
Faculdade de Ciências e Tecnologia da
Universidade Nova de Lisboa
Quinta da Torre 2829-516 Caparica Portugal
Telefone: [phone] Fax: [phone]
www.fct.unl.pt ***@fct.unl.pt

fct.unl.pt:~# _

[Quanah Gibson-Mount]

--On Thursday, April 22, 2010 1:10 PM +0100 Hugo Monteiro

Post by Hugo Monteiro
Hello list,
Although the subject suggests a fairly know issue, i cannot seem to
understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.allow: Too many open files
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.deny: Too many open files
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm,
using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.

<http://www.openldap.org/lists/openldap-technical/200806/msg00084.html>

Generally caused by linking against tcp wrappers, although I thought 64-bit
OS shouldn't hit that issue.

--Quanah



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Hugo Monteiro]

Post by Quanah Gibson-Mount
--On Thursday, April 22, 2010 1:10 PM +0100 Hugo Monteiro

Post by Hugo Monteiro
Hello list,
Although the subject suggests a fairly know issue, i cannot seem to
understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.allow: Too many open files
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.deny: Too many open files
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm,
using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.

<http://www.openldap.org/lists/openldap-technical/200806/msg00084.html>
Generally caused by linking against tcp wrappers, although I thought
64-bit OS shouldn't hit that issue.
--Quanah

Hello Quanah,

I also saw that thread, and in fact the debian packages are built with
--enable-wrappers, but i also assumed it wouldn't be an issue on a 64bit
machine.
I'll recompile without tcp wrapper support and see how it goes.

Regards,

Hugo Monteiro.
--
fct.unl.pt:~# cat .signature

Hugo Monteiro
Email : ***@fct.unl.pt
Telefone : [phone] Ext.15307
Web : http://hmonteiro.net

Divisão de Informática
Faculdade de Ciências e Tecnologia da
Universidade Nova de Lisboa
Quinta da Torre 2829-516 Caparica Portugal
Telefone: [phone] Fax: [phone]
www.fct.unl.pt ***@fct.unl.pt

fct.unl.pt:~# _

[Buchan Milne]

Post by Hugo Monteiro
Hello list,
Although the subject suggests a fairly know issue, i cannot seem to
understand the cause.
Log file presents
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.allow: Too many open files
Apr 22 12:57:36 proxyldap1 slapd[1511]: warning: cannot open
/etc/hosts.deny: Too many open files
, but
proxyldap1:~# cat /proc/sys/fs/file-max
100000
proxyldap1:~# cat /proc/sys/fs/file-nr
1088 0 100000
proxyldap1:~# lsof -p `pidof slapd`| wc -l
460

And 'grep files /proc/`pidof slapd`/limits' ?

Post by Hugo Monteiro
proxyldap1:~# su -c "ulimit -n" openldap
8192
slapd is version 2.4.17-2 (Debian Unstable) and is running on a xen vm,
using 64bit kernel 2.6.26-2-xen-amd64 in a Debian Lenny system.
Have i overlooked any aspect?

The preceding lines, showing the ACCEPT lines with the fd number, prior to the
"Too many open files" error would have been useful.

I recently tested (in the "max open files" thread from March), and found that
with tcp_wrappers, I could get more than 4000 connections (with ulimit -n set
to 4096).

You may want to review some Debian bug reports, specifically this one, which
also has a script you can use to test with:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=378261


Regards,
Buchan

[Hugo Monteiro]

Post by Buchan Milne
And 'grep files /proc/`pidof slapd`/limits' ?

Bingo.

proxyldap1:~# grep files /proc/`pidof slapd`/limits
Max open files 1024 1024 files


Now this i don't undestand. Howcome is this value set to 1024 while i
defined 8192 for nofile in /etc/security/limits ?

So now i added ulimit -n 4096 to /etc/default/slapd and i restarted the
server.
I now get a proper value in /proc/`pidof slapd`/limits.

Now it's wait and see, but i'm feeling confident.

Thank you for your help.

Hugo Monteiro.
--
fct.unl.pt:~# cat .signature

Hugo Monteiro
Email : ***@fct.unl.pt
Telefone : [phone] Ext.15307
Web : http://hmonteiro.net

Divisão de Informática
Faculdade de Ciências e Tecnologia da
Universidade Nova de Lisboa
Quinta da Torre 2829-516 Caparica Portugal
Telefone: [phone] Fax: [phone]
www.fct.unl.pt ***@fct.unl.pt

fct.unl.pt:~# _

[Klemens Kittan]

Hello,

don't want to discourage you, but my slapd (2.4.11 on Debian Lenny)
still has problems with more than 1024 files.

I have
-- ulimit -n 4096 in /etc/default/slapd
-- checked the output of /proc/`pidof slapd`/limits
-- downloaded the source package from Debian (version 2.4.11),
did a grep for FD_SETSIZE and found that it is set to 4096
-- built a new package from these sources with configure options:
--disable-tcpwrapper

In a loop from a python script, I then connect, bind, search the ldap
server and keep all the connections open, like so:

...
def ldap_connection_func():
global n
l = ldap.initialize("ldaps://ldap...")
l.simple_bind_s(username, password)
l.search_s('dc=haiti,...', ldap.SCOPE_SUBTREE, 'objectclass=*')
n.append(l)

for x in range(0,1100):
ldap_connection_func()
...

I see the number of open files of the slapd process rise to about 1080,
then all connections opened from the script are closed while slapd keeps
running.

We now will try to set the idletimeout parameter to some sensible value

0. Hopefully, this will not lead to trouble with syncrepl.

Klemens

Post by Buchan Milne
And 'grep files /proc/`pidof slapd`/limits' ?

Bingo.
proxyldap1:~# grep files /proc/`pidof slapd`/limits
Max open files 1024 1024 files
Now this i don't undestand. Howcome is this value set to 1024 while i
defined 8192 for nofile in /etc/security/limits ?
So now i added ulimit -n 4096 to /etc/default/slapd and i restarted the
server.
I now get a proper value in /proc/`pidof slapd`/limits.
Now it's wait and see, but i'm feeling confident.
Thank you for your help.
Hugo Monteiro.

--
Klemens Kittan
Systemadministrator

Uni-Potsdam, Inst. f. Informatik
August-Bebel-Str. 89
14482 Potsdam

Tel. : [phone]
Fax. : [phone]
eMail : ***@cs.uni-potsdam.de

gpg --recv-keys --keyserver wwwkeys.de.pgp.net 6EA09333

[Buchan Milne]

Post by Klemens Kittan
Hello,
don't want to discourage you, but my slapd (2.4.11 on Debian Lenny)
still has problems with more than 1024 files.
I have
-- ulimit -n 4096 in /etc/default/slapd
-- checked the output of /proc/`pidof slapd`/limits
-- downloaded the source package from Debian (version 2.4.11),
did a grep for FD_SETSIZE and found that it is set to 4096
--disable-tcpwrapper
In a loop from a python script, I then connect, bind, search the ldap
...
global n
l = ldap.initialize("ldaps://ldap...")
l.simple_bind_s(username, password)
l.search_s('dc=haiti,...', ldap.SCOPE_SUBTREE, 'objectclass=*')
n.append(l)
ldap_connection_func()
...

What was the ulimit -n in the shell from which you ran this python script? If
you are using a single process to generate the load to the server, this
process obviously also needs to be able to open sufficient files.

Post by Klemens Kittan
I see the number of open files of the slapd process rise to about 1080,
then all connections opened from the script are closed while slapd keeps
running.

It is funny that slapd opened more than 1024 files then, as it should either
have stopped at less than 1024 fd's (as it has other files open), or the script
should have reached 1100 connections, or your script is running with a file
descriptor limit.

Post by Klemens Kittan
We now will try to set the idletimeout parameter to some sensible value

It looks like your slapd file descriptor limit is now taking effect, you just
need to test better, and/or interpret the results better ....

Regards,
Buchan

[Klemens Kittan]

Hello,

Post by Buchan Milne

Post by Klemens Kittan
Hello,
don't want to discourage you, but my slapd (2.4.11 on Debian Lenny)
still has problems with more than 1024 files.
I have
-- ulimit -n 4096 in /etc/default/slapd
-- checked the output of /proc/`pidof slapd`/limits
-- downloaded the source package from Debian (version 2.4.11),
did a grep for FD_SETSIZE and found that it is set to 4096
--disable-tcpwrapper
In a loop from a python script, I then connect, bind, search the ldap
...
global n
l = ldap.initialize("ldaps://ldap...")
l.simple_bind_s(username, password)
l.search_s('dc=haiti,...', ldap.SCOPE_SUBTREE, 'objectclass=*')
n.append(l)
ldap_connection_func()
...

What was the ulimit -n in the shell from which you ran this python script? If
you are using a single process to generate the load to the server, this
process obviously also needs to be able to open sufficient files.

Ah, I see.
I just have adjusted the ulimit size on the server side (to 4096). I
never thought about the client's side, where I started the script. Here,
the "ulimit -n" is 1024. Indeed, if the client tries to open less than
1024 files, everything is fine. Even if I start the script 3 times (each
in a separate shell) everything runs as expected. The clients and the
server do not have a problem, in detail, the server opened more than
1024 files at the same time successfully.

Post by Buchan Milne

Post by Klemens Kittan
I see the number of open files of the slapd process rise to about 1080,
then all connections opened from the script are closed while slapd keeps
running.

It is funny that slapd opened more than 1024 files then, as it should either
have stopped at less than 1024 fd's (as it has other files open), or the script
should have reached 1100 connections, or your script is running with a file
descriptor limit.

Post by Klemens Kittan
We now will try to set the idletimeout parameter to some sensible value

It looks like your slapd file descriptor limit is now taking effect, you just
need to test better, and/or interpret the results better ....

Yes, you are right.
I should set ulimit in the /etc/default/slapd and it will work.

Post by Buchan Milne
Regards,
Buchan

Regards,
Klemens
--
Klemens Kittan
Systemadministrator

Uni-Potsdam, Inst. f. Informatik
August-Bebel-Str. 89
14482 Potsdam

Tel. : [phone]
Fax. : [phone]
eMail : ***@cs.uni-potsdam.de

gpg --recv-keys --keyserver wwwkeys.de.pgp.net 6EA09333