No memory although more than 1GB free

I have an application, which makes heavily use of stl and memory
consumption.
After a while the application is not be able to allocate memory from the
system anymore. The following functions fail for claiming 0.5MB:
HeapAlloc with current default heap
creating a new heap with CreateHeap
using new operator

I use XP pro with 2GB physical RAM and fixed 1GB page file.
Task Manager provide the following information at this point for Commit
Charge in KBytes:
Total: 1631776
Limit: 2967716

Process Viewer shows for this application:
Working Set: 289036 KB
Heap Usage: 1421896 KB

What's strange is, I am still be able to start another program, which
allocates in a loop chunks of 1MB RAM. This is possible for approximately
further 1.2 GB until the limit 2967716 KB is reached.

Why is it not possible for my application to allocate more memory while
another process is able to do so?

Regards,
Klaus

"Klaus Bonadt" wrote in message
...
What's strange is, I am still be able to start another program, which
allocates in a loop chunks of 1MB RAM. This is possible for
approximately
further 1.2 GB until the limit 2967716 KB is reached.

Why is it not possible for my application to allocate more memory
while
another process is able to do so?

It looks like your application runs out of virtual address space, and/or
virtual memory gets fragmented. You see, you may have plenty of physical
memory, but no free virtual address window to map it to.
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken

Sounds like you are using a old application. What is it you are trying to
run..?



"Klaus Bonadt" wrote in message
...
I have an application, which makes heavily use of stl and memory
consumption.
After a while the application is not be able to allocate memory from the
system anymore. The following functions fail for claiming 0.5MB:
HeapAlloc with current default heap
creating a new heap with CreateHeap
using new operator

I use XP pro with 2GB physical RAM and fixed 1GB page file.
Task Manager provide the following information at this point for Commit
Charge in KBytes:
Total: 1631776
Limit: 2967716

Process Viewer shows for this application:
Working Set: 289036 KB
Heap Usage: 1421896 KB

What's strange is, I am still be able to start another program, which
allocates in a loop chunks of 1MB RAM. This is possible for approximately
further 1.2 GB until the limit 2967716 KB is reached.

Why is it not possible for my application to allocate more memory while
another process is able to do so?

Regards,
Klaus

It seems that you could have that case. You could try to walk the virtual
address space and see what's going on. Check out VirtualQueryEx().

Nick

"Igor Tandetnik" wrote in message
...
"Klaus Bonadt" wrote in message
...
What's strange is, I am still be able to start another program, which
allocates in a loop chunks of 1MB RAM. This is possible for
approximately
further 1.2 GB until the limit 2967716 KB is reached.

Why is it not possible for my application to allocate more memory
while
another process is able to do so?

It looks like your application runs out of virtual address space, and/or
virtual memory gets fragmented. You see, you may have plenty of physical
memory, but no free virtual address window to map it to.
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken

Why is it not possible for my application to allocate more memory
while
another process is able to do so?

It looks like your application runs out of virtual address space, and/or
virtual memory gets fragmented. You see, you may have plenty of physical
memory, but no free virtual address window to map it to.

What do you mean with "virtual address space"? If it means "memory", even
defragmented memory, any other application should not get memory in huge
chunks anymore?
Does it mean something like "page handles"? Maybe this number is restricted?

Thanks for clarifying.
Regards,
Klaus

It seems that you could have that case. You could try to walk the virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

2.
I will collect sets of memory regions with different attributes. However, I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Regards,
Klaus

"Klaus Bonadt" wrote in message
...
Why is it not possible for my application to allocate more memory
while
another process is able to do so?

It looks like your application runs out of virtual address space,
and/or
virtual memory gets fragmented. You see, you may have plenty of
physical
memory, but no free virtual address window to map it to.

What do you mean with "virtual address space"? If it means "memory",
even
defragmented memory, any other application should not get memory in
huge
chunks anymore?

I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer.
Upper 2GB are reserved for the system (note - this does not mean the
system uses up 2GB of RAM, it just means that you cannot allocate memory
with an address in the upper 2GB). The bottom 2GB are partially occupied
by your executable code and any DLLs it may load. The rest is available
for allocation.

Now, with very large amounts of RAM, it may so happen that the process
runs out of addresses before it runs out of physical memory (in fact, if
you have more than 2GB of RAM you simply cannot address it all as a flat
space). That's the primary motivation for moving to 64bit processors.

Even if you have enough address space, it may be fragmented. That is,
there are many small stretches of unused address space, but none large
enough to accomodate your allocation request.

Does it mean something like "page handles"? Maybe this number is
restricted?

I don't know what you are talking about.
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken

"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

You can start it at 0 and then increment it based on the info returned in
lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION).

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Yes, but this will give you more info than a simple sum. To understand what
you're being given again see MEMORY_BASIC_INFORMATION.

BTW, what failture code to you get from the allocation functions that you
call?

HTH,
Nick

"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

You can start it at 0 and then increment it based on the info returned in
lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION).

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Yes, but this will give you more info than a simple sum. To understand what
you're being given again see MEMORY_BASIC_INFORMATION.

BTW, what failture code to you get from the allocation functions that you
call?

HTH,
Nick

One other tool useful for a meaningful dump of the process address space
is the '!address' command in cdb/ntsd/windbg.
If will also try to give the usage of the allocated regions, their size,
flags, etc, etc.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Regards,
Klaus

One other tool useful for a meaningful dump of the process address space
is the '!address' command in cdb/ntsd/windbg.
If will also try to give the usage of the allocated regions, their size,
flags, etc, etc.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Regards,
Klaus

"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

You can start it at 0 and then increment it based on the info returned in
lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION).

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Yes, but this will give you more info than a simple sum. To understand what
you're being given again see MEMORY_BASIC_INFORMATION.

BTW, what failture code to you get from the allocation functions that you
call?

HTH,
Nick

I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer.
Upper 2GB are reserved for the system (note - this does not mean the
system uses up 2GB of RAM, it just means that you cannot allocate memory
with an address in the upper 2GB). The bottom 2GB are partially occupied
by your executable code and any DLLs it may load. The rest is available
for allocation.

Thus, each process is able to allocate 2GB at maximum. Indeed, I can almost
allocate 2GB with my test program, allocating chunks of 1MB.
However, when my crucial application runs out of memory, Task Manager tells
that there is only 1,6GB in use (see my first mail). Furthermore, starting
my test program at this point in time, this program is able to allocate
further 1,2 GB until my whole virtual memory is allocated. Thus, all other
processes (system and my application) share 3GB - 1,2GB = 1,8GB. One more
indication that my crucial application was still not be able to allocate
nearly 2GB.

My question is, why could my application allocate only 1,4GB (this is what
Process Viewer (Dev Studio6 tools) says) although it should be able to
allocate 2GB.

Now, with very large amounts of RAM, it may so happen that the process
runs out of addresses before it runs out of physical memory (in fact, if
you have more than 2GB of RAM you simply cannot address it all as a flat
space). That's the primary motivation for moving to 64bit processors.

I have the AMD 64 processor, but I need a special 64-bit XP system, which is
not yet available for AMD, correct?

Even if you have enough address space, it may be fragmented. That is,
there are many small stretches of unused address space, but none large
enough to accomodate your allocation request.

What are PVIEW and Task Manager showing, the sum of heap sizes (including
unused space due to fragmentation) or the sum of actual allocated memory
(HeapAlloc) which was not freed afterwards?
I guess the first case, which means the sum of memory which is reserved for
the process even the process is not able to allocate due to fragmentation?
But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB
unfragmented memory available, otherwise my test program could not allocate
1,2GB until the whole virtual memory is occupied.

Does it mean something like "page handles"? Maybe this number is
restricted?
Maybe there is another limiting resource, for instance the amount of allocs.
I just wrote another test program to clarify this. The test program
allocates only 2Bytes with HeapAlloc() in a loop. It was able to allocate
133,164,202 * 2Bytes = 266,328,404 Bytes, which is 254MB. However, process
viewer shows for this process a heap usage of 2080848KB, which is almost
2GB.
It seems that for every HeapAlloc there are 8 Bytes additional costs in
terms of memory, but the number of allocations
seem to be not limited.

Regards,
Klaus

"Klaus Bonadt" wrote in message
...
I have the AMD 64 processor, but I need a special 64-bit XP system, which
is
not yet available for AMD, correct?

You can download a preview version for free from the MS site.

But anyway, as I have mentioned above there must be at least 2 - 1,8 =
0,2GB
unfragmented memory available, otherwise my test program could not
allocate
1,2GB until the whole virtual memory is occupied.

Each process has a separate virtual address space so fragmentation in one
would not affect another.

It seems that for every HeapAlloc there are 8 Bytes additional costs in
terms of memory, but the number of allocations
seem to be not limited.

Of course. The OS needs to do some bookkeeping for each allocation.

Nick

I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer.
Upper 2GB are reserved for the system (note - this does not mean the
system uses up 2GB of RAM, it just means that you cannot allocate memory
with an address in the upper 2GB). The bottom 2GB are partially occupied
by your executable code and any DLLs it may load. The rest is available
for allocation.

Thus, each process is able to allocate 2GB at maximum. Indeed, I can almost
allocate 2GB with my test program, allocating chunks of 1MB.
However, when my crucial application runs out of memory, Task Manager tells
that there is only 1,6GB in use (see my first mail). Furthermore, starting
my test program at this point in time, this program is able to allocate
further 1,2 GB until my whole virtual memory is allocated. Thus, all other
processes (system and my application) share 3GB - 1,2GB = 1,8GB. One more
indication that my crucial application was still not be able to allocate
nearly 2GB.

My question is, why could my application allocate only 1,4GB (this is what
Process Viewer (Dev Studio6 tools) says) although it should be able to
allocate 2GB.

Now, with very large amounts of RAM, it may so happen that the process
runs out of addresses before it runs out of physical memory (in fact, if
you have more than 2GB of RAM you simply cannot address it all as a flat
space). That's the primary motivation for moving to 64bit processors.

I have the AMD 64 processor, but I need a special 64-bit XP system, which is
not yet available for AMD, correct?

Even if you have enough address space, it may be fragmented. That is,
there are many small stretches of unused address space, but none large
enough to accomodate your allocation request.

What are PVIEW and Task Manager showing, the sum of heap sizes (including
unused space due to fragmentation) or the sum of actual allocated memory
(HeapAlloc) which was not freed afterwards?
I guess the first case, which means the sum of memory which is reserved for
the process even the process is not able to allocate due to fragmentation?
But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB
unfragmented memory available, otherwise my test program could not allocate
1,2GB until the whole virtual memory is occupied.

Does it mean something like "page handles"? Maybe this number is
restricted?
Maybe there is another limiting resource, for instance the amount of allocs.
I just wrote another test program to clarify this. The test program
allocates only 2Bytes with HeapAlloc() in a loop. It was able to allocate
133,164,202 * 2Bytes = 266,328,404 Bytes, which is 254MB. However, process
viewer shows for this process a heap usage of 2080848KB, which is almost
2GB.
It seems that for every HeapAlloc there are 8 Bytes additional costs in
terms of memory, but the number of allocations
seem to be not limited.

Regards,
Klaus