If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
No memory although more than 1GB free
I have an application, which makes heavily use of stl and memory
consumption. After a while the application is not be able to allocate memory from the system anymore. The following functions fail for claiming 0.5MB: HeapAlloc with current default heap creating a new heap with CreateHeap using new operator I use XP pro with 2GB physical RAM and fixed 1GB page file. Task Manager provide the following information at this point for Commit Charge in KBytes: Total: 1631776 Limit: 2967716 Process Viewer shows for this application: Working Set: 289036 KB Heap Usage: 1421896 KB What's strange is, I am still be able to start another program, which allocates in a loop chunks of 1MB RAM. This is possible for approximately further 1.2 GB until the limit 2967716 KB is reached. Why is it not possible for my application to allocate more memory while another process is able to do so? Regards, Klaus |
Ads |
#2
|
|||
|
|||
No memory although more than 1GB free
"Klaus Bonadt" wrote in message
... What's strange is, I am still be able to start another program, which allocates in a loop chunks of 1MB RAM. This is possible for approximately further 1.2 GB until the limit 2967716 KB is reached. Why is it not possible for my application to allocate more memory while another process is able to do so? It looks like your application runs out of virtual address space, and/or virtual memory gets fragmented. You see, you may have plenty of physical memory, but no free virtual address window to map it to. -- With best wishes, Igor Tandetnik "For every complex problem, there is a solution that is simple, neat, and wrong." H.L. Mencken |
#3
|
|||
|
|||
No memory although more than 1GB free
Sounds like you are using a old application. What is it you are trying to
run..? "Klaus Bonadt" wrote in message ... I have an application, which makes heavily use of stl and memory consumption. After a while the application is not be able to allocate memory from the system anymore. The following functions fail for claiming 0.5MB: HeapAlloc with current default heap creating a new heap with CreateHeap using new operator I use XP pro with 2GB physical RAM and fixed 1GB page file. Task Manager provide the following information at this point for Commit Charge in KBytes: Total: 1631776 Limit: 2967716 Process Viewer shows for this application: Working Set: 289036 KB Heap Usage: 1421896 KB What's strange is, I am still be able to start another program, which allocates in a loop chunks of 1MB RAM. This is possible for approximately further 1.2 GB until the limit 2967716 KB is reached. Why is it not possible for my application to allocate more memory while another process is able to do so? Regards, Klaus |
#4
|
|||
|
|||
No memory although more than 1GB free
It seems that you could have that case. You could try to walk the virtual
address space and see what's going on. Check out VirtualQueryEx(). Nick "Igor Tandetnik" wrote in message ... "Klaus Bonadt" wrote in message ... What's strange is, I am still be able to start another program, which allocates in a loop chunks of 1MB RAM. This is possible for approximately further 1.2 GB until the limit 2967716 KB is reached. Why is it not possible for my application to allocate more memory while another process is able to do so? It looks like your application runs out of virtual address space, and/or virtual memory gets fragmented. You see, you may have plenty of physical memory, but no free virtual address window to map it to. -- With best wishes, Igor Tandetnik "For every complex problem, there is a solution that is simple, neat, and wrong." H.L. Mencken |
#5
|
|||
|
|||
No memory although more than 1GB free
Why is it not possible for my application to allocate more memory
while another process is able to do so? It looks like your application runs out of virtual address space, and/or virtual memory gets fragmented. You see, you may have plenty of physical memory, but no free virtual address window to map it to. What do you mean with "virtual address space"? If it means "memory", even defragmented memory, any other application should not get memory in huge chunks anymore? Does it mean something like "page handles"? Maybe this number is restricted? Thanks for clarifying. Regards, Klaus |
#6
|
|||
|
|||
No memory although more than 1GB free
It seems that you could have that case. You could try to walk the virtual
address space and see what's going on. Check out VirtualQueryEx(). DWORD VirtualQuery( LPCVOID lpAddress, // address of region PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer SIZE_T dwLength // size of buffer ); Two questions: 1. How to set lpAddress initially? 2. I will collect sets of memory regions with different attributes. However, I do not know how to interpret these sets. Furthermore, I guess I will end up with the same amount of memory, which "Process viewer" is presenting. Regards, Klaus |
#7
|
|||
|
|||
No memory although more than 1GB free
"Klaus Bonadt" wrote in message
... Why is it not possible for my application to allocate more memory while another process is able to do so? It looks like your application runs out of virtual address space, and/or virtual memory gets fragmented. You see, you may have plenty of physical memory, but no free virtual address window to map it to. What do you mean with "virtual address space"? If it means "memory", even defragmented memory, any other application should not get memory in huge chunks anymore? I mean what I said - address space. Each process is allocated a flat 4GB address space - that's as much as you can address with a 32-bit pointer. Upper 2GB are reserved for the system (note - this does not mean the system uses up 2GB of RAM, it just means that you cannot allocate memory with an address in the upper 2GB). The bottom 2GB are partially occupied by your executable code and any DLLs it may load. The rest is available for allocation. Now, with very large amounts of RAM, it may so happen that the process runs out of addresses before it runs out of physical memory (in fact, if you have more than 2GB of RAM you simply cannot address it all as a flat space). That's the primary motivation for moving to 64bit processors. Even if you have enough address space, it may be fragmented. That is, there are many small stretches of unused address space, but none large enough to accomodate your allocation request. Does it mean something like "page handles"? Maybe this number is restricted? I don't know what you are talking about. -- With best wishes, Igor Tandetnik "For every complex problem, there is a solution that is simple, neat, and wrong." H.L. Mencken |
#8
|
|||
|
|||
No memory although more than 1GB free
"Klaus Bonadt" wrote in message
... It seems that you could have that case. You could try to walk the virtual address space and see what's going on. Check out VirtualQueryEx(). DWORD VirtualQuery( LPCVOID lpAddress, // address of region PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer SIZE_T dwLength // size of buffer ); Two questions: 1. How to set lpAddress initially? You can start it at 0 and then increment it based on the info returned in lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION). 2. I will collect sets of memory regions with different attributes. However, I do not know how to interpret these sets. Furthermore, I guess I will end up with the same amount of memory, which "Process viewer" is presenting. Yes, but this will give you more info than a simple sum. To understand what you're being given again see MEMORY_BASIC_INFORMATION. BTW, what failture code to you get from the allocation functions that you call? HTH, Nick |
#9
|
|||
|
|||
No memory although more than 1GB free
"Klaus Bonadt" wrote in message
... It seems that you could have that case. You could try to walk the virtual address space and see what's going on. Check out VirtualQueryEx(). DWORD VirtualQuery( LPCVOID lpAddress, // address of region PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer SIZE_T dwLength // size of buffer ); Two questions: 1. How to set lpAddress initially? You can start it at 0 and then increment it based on the info returned in lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION). 2. I will collect sets of memory regions with different attributes. However, I do not know how to interpret these sets. Furthermore, I guess I will end up with the same amount of memory, which "Process viewer" is presenting. Yes, but this will give you more info than a simple sum. To understand what you're being given again see MEMORY_BASIC_INFORMATION. BTW, what failture code to you get from the allocation functions that you call? HTH, Nick |
#10
|
|||
|
|||
No memory although more than 1GB free
One other tool useful for a meaningful dump of the process address space
is the '!address' command in cdb/ntsd/windbg. If will also try to give the usage of the allocated regions, their size, flags, etc, etc. -- This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm "Klaus Bonadt" wrote in message ... It seems that you could have that case. You could try to walk the virtual address space and see what's going on. Check out VirtualQueryEx(). DWORD VirtualQuery( LPCVOID lpAddress, // address of region PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer SIZE_T dwLength // size of buffer ); Two questions: 1. How to set lpAddress initially? 2. I will collect sets of memory regions with different attributes. However, I do not know how to interpret these sets. Furthermore, I guess I will end up with the same amount of memory, which "Process viewer" is presenting. Regards, Klaus |
#11
|
|||
|
|||
No memory although more than 1GB free
One other tool useful for a meaningful dump of the process address space
is the '!address' command in cdb/ntsd/windbg. If will also try to give the usage of the allocated regions, their size, flags, etc, etc. -- This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm "Klaus Bonadt" wrote in message ... It seems that you could have that case. You could try to walk the virtual address space and see what's going on. Check out VirtualQueryEx(). DWORD VirtualQuery( LPCVOID lpAddress, // address of region PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer SIZE_T dwLength // size of buffer ); Two questions: 1. How to set lpAddress initially? 2. I will collect sets of memory regions with different attributes. However, I do not know how to interpret these sets. Furthermore, I guess I will end up with the same amount of memory, which "Process viewer" is presenting. Regards, Klaus |
#12
|
|||
|
|||
No memory although more than 1GB free
"Klaus Bonadt" wrote in message
... It seems that you could have that case. You could try to walk the virtual address space and see what's going on. Check out VirtualQueryEx(). DWORD VirtualQuery( LPCVOID lpAddress, // address of region PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer SIZE_T dwLength // size of buffer ); Two questions: 1. How to set lpAddress initially? You can start it at 0 and then increment it based on the info returned in lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION). 2. I will collect sets of memory regions with different attributes. However, I do not know how to interpret these sets. Furthermore, I guess I will end up with the same amount of memory, which "Process viewer" is presenting. Yes, but this will give you more info than a simple sum. To understand what you're being given again see MEMORY_BASIC_INFORMATION. BTW, what failture code to you get from the allocation functions that you call? HTH, Nick |
#13
|
|||
|
|||
No memory although more than 1GB free
I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer. Upper 2GB are reserved for the system (note - this does not mean the system uses up 2GB of RAM, it just means that you cannot allocate memory with an address in the upper 2GB). The bottom 2GB are partially occupied by your executable code and any DLLs it may load. The rest is available for allocation. Thus, each process is able to allocate 2GB at maximum. Indeed, I can almost allocate 2GB with my test program, allocating chunks of 1MB. However, when my crucial application runs out of memory, Task Manager tells that there is only 1,6GB in use (see my first mail). Furthermore, starting my test program at this point in time, this program is able to allocate further 1,2 GB until my whole virtual memory is allocated. Thus, all other processes (system and my application) share 3GB - 1,2GB = 1,8GB. One more indication that my crucial application was still not be able to allocate nearly 2GB. My question is, why could my application allocate only 1,4GB (this is what Process Viewer (Dev Studio6 tools) says) although it should be able to allocate 2GB. Now, with very large amounts of RAM, it may so happen that the process runs out of addresses before it runs out of physical memory (in fact, if you have more than 2GB of RAM you simply cannot address it all as a flat space). That's the primary motivation for moving to 64bit processors. I have the AMD 64 processor, but I need a special 64-bit XP system, which is not yet available for AMD, correct? Even if you have enough address space, it may be fragmented. That is, there are many small stretches of unused address space, but none large enough to accomodate your allocation request. What are PVIEW and Task Manager showing, the sum of heap sizes (including unused space due to fragmentation) or the sum of actual allocated memory (HeapAlloc) which was not freed afterwards? I guess the first case, which means the sum of memory which is reserved for the process even the process is not able to allocate due to fragmentation? But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB unfragmented memory available, otherwise my test program could not allocate 1,2GB until the whole virtual memory is occupied. Does it mean something like "page handles"? Maybe this number is restricted? Maybe there is another limiting resource, for instance the amount of allocs. I just wrote another test program to clarify this. The test program allocates only 2Bytes with HeapAlloc() in a loop. It was able to allocate 133,164,202 * 2Bytes = 266,328,404 Bytes, which is 254MB. However, process viewer shows for this process a heap usage of 2080848KB, which is almost 2GB. It seems that for every HeapAlloc there are 8 Bytes additional costs in terms of memory, but the number of allocations seem to be not limited. Regards, Klaus |
#14
|
|||
|
|||
No memory although more than 1GB free
"Klaus Bonadt" wrote in message
... I have the AMD 64 processor, but I need a special 64-bit XP system, which is not yet available for AMD, correct? You can download a preview version for free from the MS site. But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB unfragmented memory available, otherwise my test program could not allocate 1,2GB until the whole virtual memory is occupied. Each process has a separate virtual address space so fragmentation in one would not affect another. It seems that for every HeapAlloc there are 8 Bytes additional costs in terms of memory, but the number of allocations seem to be not limited. Of course. The OS needs to do some bookkeeping for each allocation. Nick |
#15
|
|||
|
|||
No memory although more than 1GB free
I mean what I said - address space. Each process is allocated a flat 4GB
address space - that's as much as you can address with a 32-bit pointer. Upper 2GB are reserved for the system (note - this does not mean the system uses up 2GB of RAM, it just means that you cannot allocate memory with an address in the upper 2GB). The bottom 2GB are partially occupied by your executable code and any DLLs it may load. The rest is available for allocation. Thus, each process is able to allocate 2GB at maximum. Indeed, I can almost allocate 2GB with my test program, allocating chunks of 1MB. However, when my crucial application runs out of memory, Task Manager tells that there is only 1,6GB in use (see my first mail). Furthermore, starting my test program at this point in time, this program is able to allocate further 1,2 GB until my whole virtual memory is allocated. Thus, all other processes (system and my application) share 3GB - 1,2GB = 1,8GB. One more indication that my crucial application was still not be able to allocate nearly 2GB. My question is, why could my application allocate only 1,4GB (this is what Process Viewer (Dev Studio6 tools) says) although it should be able to allocate 2GB. Now, with very large amounts of RAM, it may so happen that the process runs out of addresses before it runs out of physical memory (in fact, if you have more than 2GB of RAM you simply cannot address it all as a flat space). That's the primary motivation for moving to 64bit processors. I have the AMD 64 processor, but I need a special 64-bit XP system, which is not yet available for AMD, correct? Even if you have enough address space, it may be fragmented. That is, there are many small stretches of unused address space, but none large enough to accomodate your allocation request. What are PVIEW and Task Manager showing, the sum of heap sizes (including unused space due to fragmentation) or the sum of actual allocated memory (HeapAlloc) which was not freed afterwards? I guess the first case, which means the sum of memory which is reserved for the process even the process is not able to allocate due to fragmentation? But anyway, as I have mentioned above there must be at least 2 - 1,8 = 0,2GB unfragmented memory available, otherwise my test program could not allocate 1,2GB until the whole virtual memory is occupied. Does it mean something like "page handles"? Maybe this number is restricted? Maybe there is another limiting resource, for instance the amount of allocs. I just wrote another test program to clarify this. The test program allocates only 2Bytes with HeapAlloc() in a loop. It was able to allocate 133,164,202 * 2Bytes = 266,328,404 Bytes, which is 254MB. However, process viewer shows for this process a heap usage of 2080848KB, which is almost 2GB. It seems that for every HeapAlloc there are 8 Bytes additional costs in terms of memory, but the number of allocations seem to be not limited. Regards, Klaus |
Thread Tools | |
Display Modes | |
|
|