Using Process Memory Matrix script for understanding Oracle process memory usage
This tool currently works only on Solaris. I will write support for (newer) Linux versions soon and possibly also for HP-UX.
When working on a problem I wrote a script which helps to present the output of Solaris pmap in a better way. If you don't know what pmap is, it's a tool available on Solaris, Linux, HP-UX (and AIX where it's called procmap) which displays you the breakdown of each processes address space - virtual memory mappings. This is way better than just relying on ps or top command SIZE and RSS columns.
My script gives a better overview of how much memory an Oracle process is really using. Historically this has been problematic due various differences of memory accounting of shared memory segments (Oracle SGA) and the large amount of data returned from pmap command.
My script procmm.sh (Process Memory Matrix) will simply run pmap on the processes specified and aggregate the output into a matrix.
Example output is here:
On the Horizontal axis you see the various memory sizes pmap reports (like VIRTUAL size and SWAP_RSVD - swap space reservation) and on vertical "SEGMENT_TYPE" axis you'll see for what (which mapping) in that process address these memory figures are shown (for example, Oracle binary, heap memory (think malloc()), libraries and process private memory allocations which you'll see as "anon").
I don't do any of my own computation magic here, I just show output from couple pmap commands in a better aggregated and understandable manner. So if you want to read official documentation about these figures then just run man pmap.
I will explain the columns shortly here too:
In the above example I examined only a single process. Below I pass all processes of an instance as a parameter and procmm walks through them. This is not a cheap and fast process, so shouldn't run this frequently.
The -t option below means "Total", the script doesn't show individual PID memory breakdown but sum of all PIDs passed to it.
The ANON figure reports roughly 50404 kB as the Oracle instance processes actual memory usage (I should be more precise and say memory allocation).
However the total 50404 kB also includes 13180 kB of ANON memory allocated by various libraries (in addition to Oracle libraries also multiple OS libraries which are not under Oracle's control). Also, total 3336 kB of private (writable) ANON memory has been allocated "in" oracle binary. This is because the BSS section in the binary which holds various static global variables (static function-local variables often go to stack, but global variables used over object modules go to BSS section).
Every new Oracle process reuses the shared Oracle binary pages (only one copy of Oracle binary is in memory), but when the process tries to write to the variables section, then the OS copies the shared page into a new physical page and maps the new page to process address space as a writable page. That's called copy on write.
To Be Continued...