Memory mapped files for IDL

Intro - Specs - Compiling - Usage - Sparse Files - Shared Memory - Bugs - Download - Release History

Introduction

Shortly after beginning to use IDL, I became annoyed with a couple features of IDL. First, when working with many large images, I would often run out of virtual memory, despite having 127 MB available. Second, the ASSOC feature, which associates a file with an IDL array, does not work as I had hoped. Rather than easily allowing access to any element of any array contained in a file, it requires that elements be copied into temporary arrays, and then written back to the file array. Eventually I got tired of it both of these problems and decided to do something about it. I sat down and wrote VARRAY. It has the advantages of solving both problems, and a couple I never thought of.

VARRAY: Specifications

VARRAY is in external system routine in IDL. It is written in C and has been tested on IDL 4.X and 5.X under SunOS 4.1.3, Solaris 2.X, and RedHat Linux 6.1. It should, however, work with minor modifications under most UNIX systems that support the mmap() and ftruncate() functions. Making it work under Win32 is more of an effort, but should be possible. I have no idea whether there is a mmap() equivalent under VMS.

Compiling VARRAY

VARRAY is shipped with a Makefile that may require some modifications depending upon the system used. Macros are provided for SunOS systems with GCC and ACC compilers (GCC is recommended), Solaris, and Linux systems. Once the Makefile has been modified, VARRAY is compiled with the command:

make varray.so

VARRAY: Usage

The shared object file "varray.so" needs to be linked into the running IDL process using the LINKIMAGE routine. I place the following command in my IDL_STARTUP routine:

LINKIMAGE,'VARRAY','~/idl/mmap/varray.so',1,'varray',min_args=1,max_args=10,/keywords

Once the shared object is loaded, the VARRAY function becomes available. The syntax of the VARRAY function is:

array=VARRAY([filename],element,[dim1,dim2,dim3,...dim8],[/writable],[/status])

where

"filename" is the name of the file you wish to associate with the array. If the filename is omitted, a writable temporary file with a unique name is created in the /tmp directory.
"element" is a variable type of each element of the array. Currently only numeric scalar types are allowed.
"dim1..dim8" are the dimensions of the array. If the file is writable and the dimensions are larger than the current file size, the file is ftruncate()d to the appropriate size.
The "/writable" keyword specified that the file is writable and that the changes to array elements are shared and written to disk.
The "/status" keyword causes VARRAY to print the number of open files to the standard output. Currently VARRAY supports 32 simultaneously open files.

For example, the command

A=VARRAY("test.dat",byte(0),512,512,/writable)

opens a file named "test.dat", creating it if it doesn't exist, and assigns it to a 512 by 512 byte array. The elements of this array can be accessed and written to. For example:

A(*,*)=byte(255*randomu(seed,512,512))

will write random values into the file. When the variable is deleted (i.e DELVAR,A) or reallocated (i.e. A=SOMETHING) all changes will be updated on disk.

Sparse Files

VARRAY supports sparse files. In a sparse file, only those portions of the file that contain non-zero data are written to disk. Try the following in IDL:

a=fltarr(8192,8192)

Chances are, you just saw the message (unless you had 256 MB free):

% Unable to allocate memory: to make array.
  Not enough memory
% Execution halted at:  $MAIN$

Now, with VARRAY loaded try the following:

a=varray("test.dat",float(0),8192,8192,/writable)
help,a

You should see...

A               FLOAT     = Array(8192, 8192)

Going to UNIX and doing "ls -l", we see that the file is 268435456 bytes long and takes up 24k of disk space. Now convince yourself that the array is real by doing

a(4096,4096)=!pi
print,a(4096,4096)

You'd better see 3.14159. Checking the file size again you'll see that it's still 268435456 bytes long, but now it takes up 40k of disk space. Check that things are repeatable by deleting the variable, and reloading it.

delvar,a
a=varray("test.dat",float(0),8192,8192,/writable)
print,a(4096,4096)

You should still see 3.14159. If you want to sit around for a long time you can even "print,total(a)".

Shared Memory

A happy circumstance of VARRAY is that it allows memory to be shared between IDL processes. If you map a file with the /writable keyword, the changes will be shared with any other process that maps the file. As an example, start two idl processes and link "varray.so" to them. In each, enter:

a=varray("shm.dat",fix(0),100,/writable)

now, in one enter "a(0)=1" then in the other, enter "print,a(0)." Presto, interprocess communication. Of course there's no protection for simultaneous access, so for each variable I would recommend that one process read and the other write.

Bugs and Stuff-To-Do

There's always another bug or feature. Here are a few you should note:

Arrays that are not mapped as writable use up swap space, as the system ensures that enough swap space is available to support changes to the array values. Thus, in order to save swap space, files must be mapped "/writable."
In the above shared memory example, if the array in the reader process is not mapped "/writable" and is written to, the array looses it's mapping to the file, and the interprocess connection disappears.
A planned "OFFSET" keyword, which specifies an offset in the file for the mapping to begin, has not yet been implemented. Therefore, only headerless files are supported for now.

Download VARRAY

This software is Copyright 1998 UC Regents, All Rights Reserved. Permission to modify and distribute this program is granted, provided the copyright notice in the source code remains unchanged. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The authors and copyright holders are not liable for any damages resulting from the use or misuse of this software.

Download Source Code

Download SPARC SunOS 4.1.3 Shared Object

Download SPARC Solaris 2.6 Shared Object

Download RedHat Linux 6.1 (glibc 2.1) Shared Object

Release History

Version 1.00 First release
Version 1.01 Fixed several function definitions
Version 1.02 Background syncronization
Version 1.03 Bug Fix, removed third parameter from the munmap() call. Added Solaris section to Makefile.
Version 1.03b Added some support for Digital Unix, improved Unix 97 compatibility.
Version 1.04 Bug Fix, stupid error in line 202.

Acknowledgements

My thanks to Angelos Vourlidas at the Naval Research Lab, for providing the Solaris binary and for pointing out the bug in the munmap() call.

Thanks to Nick Bower for providing the RedHat binary and reminding me to fix the printf bug.

You are visitor

Counter provided by LE FastCounter

Intro - Specs - Compiling - Usage - Sparse Files - Shared Memory - Bugs - Download - Release History

korpela@ssl.berkeley.edu