64bit RMark

[danikaschube]

I am running RMark in Windows 7 on a 64bit machine but am running of memory at the process.data step of my CJS analysis. I have successfully run my script on several largish data sets but have a few very large datasets that have a lot of groups. I have removed all extraneous data, etc to reduce the amount of background memory being used but still hit the upper limit of memory available and get the error "Error: cannot allocate vector of size 1.7 G" at the process.data step.

I was wondering if a 64bit version of RMark was in the works so I could run my scripts in the 64bit version of R. I can work with the data in Mark without the RMark interface, but I really like the easy of use that RMark provides.

Thanks in advance for any help.

Cheers,
Danielle

[bacollier]

Danielle,
I think Jeff is out of town, so I will attempt to comment (but I expect others to chime in as well). What you are seeing is a issue that you often get when dealing with really large datasets in R. I don't think that a 64 bit version of RMark will help you out much (see Evan's comments on other threads regarding the gain in speed/power when you transfer from 32-64 bit). I think you can change memory.limit in R (not at machine with R on it, but I think its memory.limit), but I don't think this is a RMark problem, or a 32 vs. 64bit problem, but a general R issue as this topic comes up quite often on the R-help list. So, I doubt that the fix you requested is the fix you need. But, I may be wrong (anyone else have a thought?)

Bret

[cooch]

Danielle,
I think Jeff is out of town, so I will attempt to comment (but I expect others to chime in as well). What you are seeing is a issue that you often get when dealing with really large datasets in R. I don't think that a 64 bit version of RMark will help you out much (see Evan's comments on other threads regarding the gain in speed/power when you transfer from 32-64 bit). I think you can change memory.limit in R (not at machine with R on it, but I think its memory.limit), but I don't think this is a RMark problem, or a 32 vs. 64bit problem, but a general R issue as this topic comes up quite often on the R-help list. So, I doubt that the fix you requested is the fix you need. But, I may be wrong (anyone else have a thought?)

Bret

From the R FAQ:

2.9 There seems to be a limit on the memory it uses!

Indeed there is. It is set by the command-line flag --max-mem-size (see How do I install R for Windows?) and defaults to the smaller of the amount of physical RAM in the machine and 0.5GB less than the limit of user virtual memory for a process (most often 2GB on a 32-bit build). It can be set to any amount between 32Mb and 4095Mb (on a 32-bit build).

Use ?Memory and ?memory.size for information about memory usage. The limit can be raised by calling memory.limit within a running R session.

The 32-bit executables Rgui.exe and Rterm.exe support up to 3GB of user address space per process under suitably enabled versions of 32-bit Windows (see http://www.microsoft.com/whdc/system/pl ... AEmem.mspx, http://msdn.microsoft.com/en-us/library ... 5%29.aspx; this is not enabled by default). On such systems, the default for --max-mem-size is the smaller of the amount of RAM and 2.5GB. On all but the earliest 64-bit versions of Windows the user address space for a 32-bit process is 4GB, and there the default for --max-mem-size is the smaller of the amount of RAM and 3.5GB. For a 64-bit build, the default is the amount of RAM.

[jlaake]

Danielle-

I will be putting together a 64 bit version later this summer. What is large in terms of number of groups, occasions, and number of capture histories? I've worked with some very large data sets and have never run short of memory yet. You may want to start with a completely empty .Rdata workspace. If they are large you'll also want to use the external=TRUE argument so it doesn't save the mark object in the workspace.

regards --jeff

[cooch]

Danielle-

I will be putting together a 64 bit version later this summer. ..

Might as well hold off until there is a 64-bit mark.exe. While I've compiled one for Linux (although the source code is out of date), a Windows version doesn't exist.

[danikaschube]

Thanks everyone for the replys.

I had tried all the memory expansion tricks listed in the manuals/help files but I am still short. I think my problem is too many groups rather than too many individuals. The problem data set is American Robin which is caught at a lot of banding stations across the continent, but there are never large numbers of individuals caught at each station.

I will await a 64bit Windows versions of MARK.exe and RMark while I work with this data in regular MARK.

Cheers,
Danielle

[jlaake]

I've not advertised it much but there is an alternative in RMark for CJS and JS models that does not use MARK. If you want to look at it, look at the help for the ?cjs and ?crm and related functions. It will not have the problens with large numbers of groups because the groups are not used to create PIMS in this alternate approach, but I've not tried it with large data sets so you may still run into memory problems. I've been working on the 64 bit approach in between meetings and will post as soon as I have something,

--jeff