No memory although more than 1GB free

"André Pönitz" wrote in message
...
In microsoft.public.vc.stl Igor Tandetnik wrote:
That's why I think there's fragmentation at work. Imagine the
pathological scenario: your whole memory is occupied by 1K allocated
chunk followed by 1023K free chunk, and so on. 1024 such pairs will eat
up 1GB of RAM in a way that only 1MB is actually used, but you cannot
allocate another 1MB chunk. It's an artificial example of course, it's
highly unlikely to occur in practice, but it demonstrates the idea of
fragmentation nicely.

Does that mean the allocator does _not_ try to collect chunks of 'almost
the same size' in one place and different sizes in another one? [I am
talking about virtual adresses here]

It could but there isn't a perfect solution. Most of the time one won't run
into this problem. If one does then they are also better positioned to fix
it as they know more about how things are and will be allocated than the OS.
MS tries to alleviate this by providing:

http://msdn.microsoft.com/library/de...ation_heap.asp

Nick

By the way, with Windows virtual memory management, you cannot fragment
physical memory at all, you can only fragment virtual address space. The
way it works, all physical memory is broken into pages 4KB (on some
systems 8KB) large. So is virtual memory. When a region of virual memory
is allocated, each page of virtual memory is backed by a page of
physical memory. Now, virtual addresses within this region need to be
consequtive (because your program expects the address arithmetic to
work), but physical RAM pages don't need to be. The system just picks
free RAM pages lying around and maps them to virtual pages. Moreover, if
the system runs out of RAM, it picks a physical page (mapped to some
virtual page in some process A), saves its contents to disk and reuses
it for another virtual page possibly in a different process B. If
process A later needs to refer to that virtual page, some other physical
page may be assigned to it and contents read back from disk.

Thanks a lot, Igor, I got it!

"Klaus Bonadt" wrote in message
...
But anyway, as I have mentioned above there must be at least 2 -
1,8 =
0,2GB
unfragmented memory available, otherwise my test program could not
allocate
1,2GB until the whole virtual memory is occupied.

Each process has a separate virtual address space so fragmentation
in one
would not affect another.

In my current situation, my application runs out of memory due to a
memory
claim of 0.5MB RAM.
Let us assume this is due to fragmented memory, i.e. there is no
contiguous
range with more than 0.5MB RAM in the lower 2 GB of virtual address
space.
The application stops with popping a message box.
Now I start another application, which allocates in a loop chunks of
memory,
each chunk is 1MB. The application does this up to 1.2 GB RAM.

Which part of "each process has a separate virtual address space" do you
have difficulty understanding? A success or failure of a memory
allocation in one process tells you nothing about the state of the
virtual address space in another. It's like saying: "I have machine A
where memory allocation fails. Then I run a test program on machine B,
and it can successfully allocate plenty of memory. So enough memory
should be avaiable on machine A, right?"
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken

Not a problem. We're all here to hopefully learn something. Try re-reading
Igor's posting again. He explains quite well what goes on with
virtual/real
memory and address spaces.

I wrote my answer to you before reading Igor's reply. Anyway, thanks a lot.

Regards,
Klaus

"André Pönitz" wrote in message
...
In microsoft.public.vc.stl Igor Tandetnik wrote:
That's why I think there's fragmentation at work. Imagine the
pathological scenario: your whole memory is occupied by 1K allocated
chunk followed by 1023K free chunk, and so on. 1024 such pairs will
eat
up 1GB of RAM in a way that only 1MB is actually used, but you
cannot
allocate another 1MB chunk. It's an artificial example of course,
it's
highly unlikely to occur in practice, but it demonstrates the idea
of
fragmentation nicely.

Does that mean the allocator does _not_ try to collect chunks of
'almost
the same size' in one place and different sizes in another one? [I am
talking about virtual adresses here]

It may or it may not, I don't know. Whatever strategy a particular
memory allocator employs, it is always possible to construct a
pathological sequence of allocations and deallocations that leads to
fragmentation with this allocator. In my example, I assumed a naive
allocator so that a pathological case would be easy to explain and
understand.
--
With best wishes,
Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken

Does that mean the allocator does _not_ try to collect chunks of 'almost
the same size' in one place and different sizes in another one? [I am
talking about virtual adresses here]

At least it would be worthwile in order to reduce fragmentation to think
about creating different heaps, each responding for allocating chunks with
similar size?

If I create two growable heaps, how the system does position these heaps in
the virtual address space? If the second heap starts at the end of the first
heap, the first heap is actually not growable?
On the other hand, the space between heaps will also cause fragmentation.
Any ideas for balancing?

Regards,
Klaus

Xref: kermit microsoft.public.vc.language:144412 microsoft.public.vc.stl:18875 microsoft.public.windowsxp.perform_maintain:155644

Which part of "each process has a separate virtual address space" do you
have difficulty understanding? A success or failure of a memory
allocation in one process tells you nothing about the state of the
virtual address space in another. It's like saying: "I have machine A
where memory allocation fails. Then I run a test program on machine B,
and it can successfully allocate plenty of memory. So enough memory
should be avaiable on machine A, right?"

I was not aware of your reply, when I wrote this. I got it.
However, could you take a look to my answer regarding André?
After understanding the concept, I am now interesting in figuring out how to
avoid fragmentation.

Regards,
Klaus

"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

You can start it at 0 and then increment it based on the info returned in
lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION).

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Yes, but this will give you more info than a simple sum. To understand what
you're being given again see MEMORY_BASIC_INFORMATION.

BTW, what failture code to you get from the allocation functions that you
call?

HTH,
Nick

"André Pönitz" wrote in message
...
In microsoft.public.vc.stl Igor Tandetnik wrote:
That's why I think there's fragmentation at work. Imagine the
pathological scenario: your whole memory is occupied by 1K allocated
chunk followed by 1023K free chunk, and so on. 1024 such pairs will eat
up 1GB of RAM in a way that only 1MB is actually used, but you cannot
allocate another 1MB chunk. It's an artificial example of course, it's
highly unlikely to occur in practice, but it demonstrates the idea of
fragmentation nicely.

Does that mean the allocator does _not_ try to collect chunks of 'almost
the same size' in one place and different sizes in another one? [I am
talking about virtual adresses here]

It could but there isn't a perfect solution. Most of the time one won't run
into this problem. If one does then they are also better positioned to fix
it as they know more about how things are and will be allocated than the OS.
MS tries to alleviate this by providing:

http://msdn.microsoft.com/library/de...ation_heap.asp

Nick

Not a problem. We're all here to hopefully learn something. Try re-reading
Igor's posting again. He explains quite well what goes on with
virtual/real
memory and address spaces.

I wrote my answer to you before reading Igor's reply. Anyway, thanks a lot.

Regards,
Klaus

"Klaus Bonadt" wrote in message
...
It seems that you could have that case. You could try to walk the
virtual
address space and see what's going on. Check out VirtualQueryEx().

DWORD VirtualQuery(
LPCVOID lpAddress, // address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // information buffer
SIZE_T dwLength // size of buffer
);

Two questions:
1.
How to set lpAddress initially?

You can start it at 0 and then increment it based on the info returned in
lpBuffer (i.e. look up MEMORY_BASIC_INFORMATION).

2.
I will collect sets of memory regions with different attributes. However,
I
do not know how to interpret these sets.
Furthermore, I guess I will end up with the same amount of memory, which
"Process viewer" is presenting.

Yes, but this will give you more info than a simple sum. To understand what
you're being given again see MEMORY_BASIC_INFORMATION.

BTW, what failture code to you get from the allocation functions that you
call?

HTH,
Nick

Thanks a lot, Igor, I got it!

Which container types of the STL are you using Klaus?
And which version of VC++ are you using?

Stephen Howe

Xref: kermit microsoft.public.vc.language:144422 microsoft.public.vc.stl:18899 microsoft.public.windowsxp.perform_maintain:155681

Which container types of the STL are you using Klaus?
And which version of VC++ are you using?

In first place I use map and vector with VC++ version 6 with some header
replacements.

Regards,
Klaus

Thanks a lot, Igor, I got it!

Which container types of the STL are you using Klaus?
And which version of VC++ are you using?

Stephen Howe

Xref: kermit microsoft.public.vc.language:144422 microsoft.public.vc.stl:18899 microsoft.public.windowsxp.perform_maintain:155681

Which container types of the STL are you using Klaus?
And which version of VC++ are you using?

In first place I use map and vector with VC++ version 6 with some header
replacements.

Regards,
Klaus