Re: [SAGE] simple database problem

[Jon maddog Hall <maddog@li.org>]

Andrew,

First of all I would try Greg Rose's suggestion, which is basically another
way of doing what I suggested.....extracting the keys and sorting them into
disk address order.

If that does not work, and seeing that you do not want to make too much
"investment" into it, you might try simply dedicating a system to the database
to cut down on the I/O buffer and disk contention that you see.

andrew@research.att.com said:
> 	we have plenty of memory (4GB). the database is modest in size (20MB
> 	of data; 160MB on disk). mlock won't work because it is not a server
> 	based program; rather, each time the access program runs, it opens
> 	the gdbm database and does its business.

Another way of treating this might be to create a RAM-based file system to
"shadow" your disk-based one and do all of your reading from that?  On booting
you can load the database off disk into the ram-based filesystem.  All writes
and updates would go to both filesystems.

andrew@research.att.com said:
> i reiterate that we do not see similiar slowdowns with any other OS we use;
> just linux.

You have had the same load, including all of the other applications, on the
same hardware with the same file layout with a different OS?

You talked about lack of buffer cache.  I can not speak for all of the
different operating systems but Digital Unix had a shared VM and buffer I/O
pool.  As the needs for real, physical memory went up from the VM subsystem,
the I/O buffer pool got smaller.  As the need for real memory went down, the
I/O buffer could grow.  All of this was within limits set by the administrator
in installing it, but even the default settings allowed for a great set of
range, and of course with larger real memory sizes the flexibility got
even larger.

It is possible that the other operating systems either had this type of
mechanism or had their I/O buffer pool configured to a larger size so that
the database did not thrash against itself.

Since you have the sources for the Linux kernel, you could probably try
bumping up the size of the I/O buffer and see if that helps.

md
-- 
Jon "maddog" Hall
Executive Director           Linux International(R)
email: maddog@li.org         80 Amherst St. 
Voice: +1.603.672.4557       Amherst, N.H. 03031-3032 U.S.A.
WWW: http://www.li.org

Board Member: Uniforum Association, USENIX Association

(R)Linux is a registered trademark of Linus Torvalds in several countries.
(R)Linux International is a registered trademark in the USA used pursuant
   to a license from Linux Mark Institute, authorized licensor of Linus
   Torvalds, owner of the Linux trademark on a worldwide basis
(R)UNIX is a registered trademark of The Open Group in the USA and other
   countries.