Back to Eric's Home Page

Memory mapped files for IDL

Intro - Specs - Compiling - Usage - Sparse Files - Shared Memeory - Bugs - Download


Shortly after beginning to use IDL, I became annoyed with a couple features of IDL. First, when working with many large images, I would often run out of virtual memory, despite having 127 MB available. Second, the ASSOC feature, which associates a file with an IDL array, does not work as I had hoped. Rather than easily allowing access to any element of any array contained in a file, it requires that elements be copied into temporary arrays, and then written back to the file array. Eventually I got tired of it both of these problems and decided to do something about it. I sat down and wrote VARRAY. It has the advantages of solving both problems, and a couple I never thought of.

VARRAY: Specifications

VARRAY is in external system routine in IDL. It is written in C and has been tested on IDL 4.0.1 under SunOS 4.1.3. It should, however, work with minor modifications under most UNIX systems that support the mmap() and ftruncate() functions. Making it work under Win32 is more of an effort, but should be possible. I have no idea whether there is a mmap() equivalent under VMS.

Compiling VARRAY

VARRAY is shipped with a Makefile that may require some modifications depending upon the system used. Macros are provided for SunOS systems with GCC and ACC compilers (GCC is recommended) and Linux systems. (The code has never been compiled or tested on a Linux system. Modifications are likely to be required.) Once the Makefile has been modified, VARRAY is compiled with the command:


The shared object file "" needs to be linked into the running IDL process using the LINKIMAGE rotine. I place the following command in my IDL_STARTUP routine:

Once the shared object is loaded, the VARRAY function becomes available. The syntax of the VARRAY function is:


For example, the command

opens a file named "test.dat", creating it if it doesn't exist, and assigns it to a 512 by 512 byte array. The elements of this array can be accessed and written to. For example:
will write random values into the file. When the variable is deleted (i.e DELVAR,A) or reallocated (i.e. A=SOMETHING) all changes will be updated on disk.

Sparse Files

VARRAY supports sparse files. In a sparse file, only those portions of the file that contain non-zero data are written to disk. Try the following in IDL:
Chances are, you just saw the message (unless you had 256 MB free):
% Unable to allocate memory: to make array.
  Not enough memory
% Execution halted at:  $MAIN$                 
Now, with VARRAY loaded try the following:
You should see...
A               FLOAT     = Array(8192, 8192)
Going to UNIX and doing "ls -l", we see that the file is 268435456 bytes long and takes up 24k of disk space. Now convince yourself that the array is real by doing
You'd better see 3.14159. Checking the file size again you'll see that it's still 268435456 bytes long, but now it takes up 40k of disk space. Check that things are repeatable by deleting the variable, and reloading it.
You should still see 3.14159. If you want to sit around for a long time you can even "print,total(a)".

Shared Memory

A happy circumstance of VARRAY is that it allows memory to be shared between IDL processes. If you map a file with the /writable keyword, the changes will be shared with any other process that maps the file. As an example, start two idl processes and link "" to them. In each, enter:
now, in one enter "a(0)=1" then in the other, enter "print,a(0)." Presto, interprocess communication. Of course there's no protection for simultaneous access, so for each variable I would recommend that one process read and the other write.

Bugs and Stuff-To-Do

There's always another bug or feature. Here are a few you should note:

Download VARRAY

This software is Copyright 1998 UC Regents, All Rights Reserved. Permission to modify and distribute this file is granted, provided the copyright notice in the source code remains intact. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The authors and copyright holders are not liable for any damages resulting from the use or misuse of this software.

Download Source Code

Download SPARC SunOS 4.1.3 Shared Object

You are visitor
Counter provided by LE FastCounter

Intro - Specs - Compiling - Usage - Sparse Files - Shared Memeory - Bugs - Download

Copyright © 1998 Eric Korpela