Subject: Shared Library vs Static Library Linking

From: Mark Nickel <[email-masked]>
Date: Wed, 14 Jul 1999 09:09:44 -0500

To all,

We are going to be rolling out an IMAP/Netscape based email solution at
my company.  After trolling through deja.com looking for IMAP
performance tuning hints, I came across a discussion about building the
c-client library (and consequently the IMAP daemon and related tools) as
a Shared Library rather than a Static Library.

My question: is building the c-client libray as Shared and linking
applications to it as such going to give me a benefit over Static
linking.

Here are our estimates of user utilization:
o  4 servers with less than 100 users per server.
o  Each user will most likely start their imap client software
(Netscape), log into
    the server, and leave the client software running for the entire
day.

I have successfully built the c-client library as Shared and linked all
relevant applications to it.  The executibles are small and everything
seems to work just great.

Architecture:
o  4 Dell PowerEdge 1300's
    o  64mb ram
    o  4gb SCSI harddrive for the OS and apps
    o  9gb SCSI harddrive for the filesystem mounted under /home

o  clients:  Netscape 4.6
o  IMAP 4.6 - beta
o  mbx - high performance mail box format

Thanks for your insight!

Mark Nickel
(Ok, so I'm .sig-less)

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 14 Jul 1999 13:10:52 -0700 (Pacific Daylight Time)

> applications to it as such going to give me a benefit over Static

The only benefit will be if you run more than one c-client application.
If the only application you run is imapd, then I doubt that you'll get any
benefit since all the imapd processes will share their code pages.  In
fact, you may have (extremely) slight additional overhead.

Basically, the memory savings is (n-1)*(size of c-client) where n is the
number of different c-client applications running simultaneously.  For
example, if c-client is 500KB, and you have 50 imapds and 20 Pines, then
you'll save 500KB.

This presumes a decent implementation of UNIX that shares code pages.  If
code pages aren't shared, then the savings would be much more.  I wouldn't
have much respect for a UNIX system that has shared libraries but not
shared code pages though... ;-)

---

From: "Alex Shvedov" <[email-masked]>
Date: Wed, 14 Jul 1999 16:25:49 -0400

> The only benefit will be if you run more than one c-client
application.

Sorry if I sound foolish, but still: as we know, c-client is not
thread-safe; how can it be that some shared (really shared, with
shared code pages) library is NOT thread safe?
I would say that it's impossible.
Am I missing something?

---

From: Mark Crispin <[email-masked]>
Date: Wed, 14 Jul 1999 13:53:10 -0700 (Pacific Daylight Time)

> I would say that it's impossible.

The issue of "thread safe" has to do with global variables; in a
multi-threaded application global variables are shared.  In many cases,
this is not intended behavior (most of today's generation of programmers
have never heard of shared writeable data or how such a thing could
possibly be useful).

This is not the case with a shared library, which shares only the
non-writeable code pages.

Strictly speaking, c-client *is* thread-safe -- the globals in c-client
are *intended* to be shared (yes!).  However, certain of the C library
routines called by c-client are not thread-safe in most versions of C
library, which in turn renders c-client non thread-safe.

For example, strtok() in most C libraries stores its state in a global
variable.  A thread-aware version of strtok() would maintain a separate
state per thread (and do this for all similar functions).  Unfortunately,
most C libraries are written by unimaginative programmers and require that
you use an alternative version of strtok() (with a name like strtok_r())
which takes an extra argument pointing to an automatic that holds the
state.

---

From: "Alex Shvedov" <[email-masked]>
Date: Wed, 14 Jul 1999 17:19:16 -0400

> The issue of "thread safe" has to do with global variables;
A little more than that.

> This is not the case with a shared library, which shares only
the non-writeable code pages.
Correct, but -

> Strictly speaking, c-client *is* thread-safe -- the globals in
c-client are *intended* to be shared (yes!).
I already posted my own empirical results and I know that
c-client is not thread-safe.

Text-book "strtok" case is only part of the c-client problems;
namely, gethostbyname is to be replaced by gethostbyname_r, if
any, due to internal statics (not only globals make life
non-reenterable).

Making c-client thread-safe is not a VERY big enterprise;
actually, I just serialized it by mutexes in 3 places and 3-5
threads I used for my tests started working together fine. This,
of course, is a dumb solution; real one can be built through
POSIX pthread_setspecific(...) calls.

I worked under AIX 4.2, but problems I found now are not
OS-related at all.

---

From: Mark Nickel <[email-masked]>
Date: Thu, 15 Jul 1999 17:31:23 -0500

Ooops...  A little snafu here...

Our operating system of choice will be Redhat 6.0 with my custom built
version of imap-4.6-Beta to use the c-client as a shared library.

<Off-Topic Digression>
   We may be scrapping the whole "home grown" method of delivering a
messaging solution because of obstacles with integrating sendmail and LDAP.
As a result, we may go with CommuniGate Pro from Stalker software or another
one I saw on freshmeat.net.

This solution will still give us problems with unified Directory Services as
CommuniGate Pro is great for organizations that have one server for all user
accounts that are local to that host.  At least from what I'm able to
discover, there's no real maildrop capability if the user account is not on
the server where CommuniGate is running.

I hate to give up on the IMAP+sendmail+LDAP but I'm starting to feel the
time crunch to get off of cc:Mail....
</Off-Topic Digression>

Thanks again for your insight's

Mark Nickel

---

From: "Alex Shvedov" <[email-masked]>
Date: Fri, 16 Jul 1999 12:52:44 -0400

>Could you mail me your patches or souces of the thread-safe
c-client lib?

I thought that I did that already... I'll have to find that stuff
once again, so wait a little, OK?

---

From: Mark Nickel <[email-masked]>
Date: Wed, 15 Sep 1999 09:18:52 -0500

A while back I wrote to this list inquiring about building the c-client
library shared vs. static.

Well, I think that I've got some interesting results from that little
experiment.  Nothing, though, that Mark Crispin didn't really elude to
in replies to my previous messages.

Here goes:

My Platform:
Dell PowerEdge 1300, PII 333mHz w/ 192mb ram
4GB HD for OS and 9GB HD for /home
imapd-4.6-Beta
we are using Netscape Mail Client 4.61 (the assumption is that most
email users are going to leave their mail client running throughout the
day, not just check mail when they want to but letting the client do the
checking every 10 minutes)
we estimate that approximately 80 - 100 people will eventually be on
this box.

c-client was built as a shared library.  imapd and its related tools
were also built linking to the shared c-client.

We are testing this setup with the 5 of us in the IS department.

Here is the results of a "free" command:

                      total       used            free     shared
buffers     cached
Mem:        192892   133192       59700      25016      82264      26236

-/+ buffers/cache:      24692     168200
Swap:       128484       1112     127372

(At the end of this message I'm including output of my "ps -aux | sort"
command so you have an idea of what is running on this box.)

Something just seems wrong with that memory utilization.  I'm I spec'ing
these machines too light on physical memory?  Should I scrap the shared
library thing and build as static?  Will I see better memory
utilization?

My gut instinct is that I'm underestimating the physical memory that
will be required for 80-100 people leaving imapd running.  Each imapd
process seems to use average 1.1mb of ram.  (The amout used seems to
vary based on what the users are doing within netscape)  So 100 users @
1.1mb each is about 110mb of ram...  The 192mb I've got should be enough
to handle that number of users, but as the "free" command shows, it
seems that 192mb isn't enough.

I would really appreciate any insights.

Thanks!

Mark

[output of ps -aux | sort ]
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
bin        245  0.0  0.1  1088  288 ?        S    Sep13   0:00 portmap
daemon     317  0.0  0.1  1112  284 ?        S    Sep13   0:00
/usr/sbin/atd
klippert 11218  0.0  0.8  2576 1656 ?        S    07:53   0:00 imapd
mbrochtr 11229  0.0  0.6  2280 1348 ?        S    07:59   0:00 imapd
mnickel  11476  0.0  0.6  2220 1236 ?        S    08:48   0:00 imapd
mnickel  11553  0.0  0.6  2260 1328 ?        S    09:00   0:00 imapd
mnickel  11581  0.0  0.4  1712  952 pts/0    S    09:08   0:00 -bash
mnickel  11607  0.0  0.4  2644  956 pts/0    R    09:12   0:00 ps -aux
mnickel  11608  0.0  0.4  1712  952 pts/0    R    09:12   0:00 -bash
nobody     526  0.0  0.4  2556  916 ?        S    Sep13   0:00 httpd
nobody     527  0.0  0.5  2556  968 ?        S    Sep13   0:00 httpd
nobody     528  0.0  0.5  2556  972 ?        S    Sep13   0:00 httpd
nobody     529  0.0  0.5  2556  972 ?        S    Sep13   0:00 httpd
nobody     530  0.0  0.5  2556  972 ?        S    Sep13   0:00 httpd
root         1  0.0  0.2  1096  476 ?        S    Sep13   0:07 init [3]
root         2  0.0  0.0     0    0 ?        SW   Sep13   0:00 [kflushd]

root         3  0.0  0.0     0    0 ?        SW   Sep13   0:00 [kpiod]
root         4  0.0  0.0     0    0 ?        SW   Sep13   0:00 [kswapd]
root         5  0.0  0.0     0    0 ?        SW<  Sep13   0:00
[mdrecoveryd]
root       292  0.0  0.2  1264  540 ?        S    Sep13   0:08 syslogd
-m 0
root       303  0.0  0.3  1360  648 ?        S    Sep13   0:00 klogd
root       331  0.0  0.2  1284  532 ?        S    Sep13   0:00 crond
root       349  0.0  0.2  1236  512 ?        S    Sep13   0:00 inetd
root       363  0.0  0.4  2264  888 ?        S    Sep13   0:09 named
root       377  0.0  0.2  1284  516 ?        S    Sep13   0:00 lpd
root       395  0.0  0.2  1128  488 ?        S    Sep13   0:00 rpc.statd

root       406  0.0  0.0  1072  124 ?        S    Sep13   0:00
rpc.rquotad
root       417  0.0  0.0  1132  132 ?        S    Sep13   0:00
rpc.mountd
root       432  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       433  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       434  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       435  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       436  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       437  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       438  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       439  0.0  0.0     0    0 ?        SW   Sep13   0:00 [nfsd]
root       440  0.0  0.0     0    0 ?        SW   Sep13   0:00 [lockd]
root       441  0.0  0.0     0    0 ?        SW   Sep13   0:00 [rpciod]
root       469  0.0  0.8  3752 1616 ?        S    Sep13   0:00 slapd
root       470  0.0  0.8  3752 1616 ?        S    Sep13   0:02 slapd
root       471  0.0  0.8  3752 1616 ?        S    Sep13   0:06 slapd
root       492  0.0  0.3  1992  744 ?        S    Sep13   0:03 sendmail:
accepti
root       507  0.0  0.1  1124  384 ?        S    Sep13   0:00 gpm -t
imps2
root       521  0.0  0.5  2384 1072 ?        S    Sep13   0:02 httpd
root       556  0.0  0.2  2208  524 ?        S    Sep13   0:00 smbd -D
root       567  0.0  0.3  1856  668 ?        S    Sep13   0:02 nmbd -D
root       639  0.0  0.5  2136 1008 tty1     S    Sep13   0:00 login --
root
root       640  0.0  0.1  1060  384 tty2     S    Sep13   0:00
/sbin/mingetty tt
root       641  0.0  0.1  1060  384 tty3     S    Sep13   0:00
/sbin/mingetty tt
root       642  0.0  0.1  1060  384 tty4     S    Sep13   0:00
/sbin/mingetty tt
root       643  0.0  0.1  1060  384 tty5     S    Sep13   0:00
/sbin/mingetty tt
root       644  0.0  0.1  1060  384 tty6     S    Sep13   0:00
/sbin/mingetty tt
root       646  0.0  0.0  1052  116 ?        S    Sep13   0:02 update
(bdflush)
root     11235  0.0  0.5  1740  980 tty1     S    08:07   0:00 -bash
root     11579  0.0  0.4  1672  860 ?        S    09:08   0:00
in.telnetd
root     11580  0.0  0.5  2156 1060 pts/0    S    09:08   0:00 login --
mnickel
swolf    11385  0.0  0.7  2320 1440 ?        S    08:23   0:00 imapd
xfs        541  0.0  0.4  1988  908 ?        S    Sep13   0:00 xfs