controlling malloc allocation size

[allynm]

Hello everyone,

Is there a technique for controlling the actual allocation size from the heap using malloc()? I find that if I allocate a block of nominal size n bytes by calling malloc(n), the sys actually allocates much more, some of it "used", much more of it "not used". If I want to trim the not used bytes there is a command that will do this post facto. The smallest sized allocation then becomes just a single page (4096 bytes). In many cases this would be more than ample to handle a dynamic buffer and not generate a segmentation fault.

It would be nice to be able to control the behavior of malloc() so that it allocates no more than the number of bytes requested in the parameter. Is there a way to do this?

Thanks,
Mark Allyn

[dazman19]

Of my somewhat limited c/c++ experience, My understanding is that malloc(n) returns a pointer to the heap memory it allocated, or a null pointer if it fails. This returned pointer doesnt have a type, its actually of void type so you need to cast it then initialize.
int x = 50;
char * buffer;
buffer = (char*) malloc (x);

It might be nice to make your own wrapper function for malloc so you get back types you desire.

If malloc is using more memory than x then i would assume this managed by the system itself but is not of any concern to your application.

[allynm]

Hi Jaunty,

I take your point that the system is managing the unused bytes and there is no particular reason to concern oneself with what is done with them. However, if for some reason you write into the buffer more than it was designed (allocated) to hold, there is no way to know you have done so because as long as you don't exceed the total size (used + unused) you will note generate a seg fault.

Regards,
Mark

[dwhitney67]

Hi Jaunty,

I take your point that the system is managing the unused bytes and there is no particular reason to concern oneself with what is done with them. However, if for some reason you write into the buffer more than it was designed (allocated) to hold, there is no way to know you have done so because as long as you don't exceed the total size (used + unused) you will note generate a seg fault.

Regards,
Mark

Unfortunately, the action of overwriting a buffer does not generate a Segmentation Violation. If you happen to overwrite a buffer and trample on some data that your program expects to use at a later time, then this would probably lead to a Segmentation Violation. This is why troubleshooting these types of problems can be tricky.

Here's a crude example that demonstrates how a program continues running:

#include <stdlib.h>
#include <string.h>
#include <stdio.h>

int main()
{
    char* ptr1 = malloc(10);
    char* ptr2 = malloc(10);

    printf("ptr1 is pointing to: %p\n", ptr1);
    printf("ptr2 is pointing to: %p\n", ptr2);

    strcpy(ptr2, "Hello World\n");

    memset(ptr1, 'J', ptr2-ptr1 + 1);    // overwrite buffer with 'J' chars

    printf("data at ptr2 is: %s\n", ptr2);

    // These are commented out because a SegFault will occur once called.
    //free(ptr1);
    //free(ptr2);

    return 0;
}

Btw, with newer compilers, there is no need to cast the return value of malloc() as dazman19 indicated.

[allynm]

Hi Dazman19,

I have no idea why I gave you the name "Jaunty". Sorry to call you by a ubuntu version name....

Mark

[allynm]

Hi dwhitney67,

Thanks for the very nice demo. I haven't run it yet, but I will. With a debugger as well.

Perhaps you can clarify a point that I find confusing about malloc(). When called it not only allocates a set of bytes and designates them "used". It also generates addtional bytes that are "unused", and as my original post indicted, there is no apparent way I have yet found to conrol the number of "unused" (but apprently, "allocated") bytes. This can be shown easily by a calling mallino() and examining the mallinfo struct it returns.

My original concern was the status of these "unused" bytes. If I write into them (and I most certainly can), then is it possible for me to overwrite already written-to bytes. If so, then in what sense are they "unused". If not, then why are they "allocated unused"? In other words, do I care if I write into the unused blocks out of negligence, or whatever?

Thanks for your help.

Mark

[trent.josephsen]

From memory, don't quote me on the details, but this is the general principle:

Requesting new memory from the operating system is expensive. malloc() does not request new memory from the OS every time you call it because that would incur an unacceptable performance penalty for many applications. Instead, it requests a large chunk of memory all at once and manages it internally. As a result, when you call free() on a pointer, that memory is not released to the OS immediately, but just goes into a list of free blocks that can be returned by a subsequent call to malloc(). All this is done by your process, not by the OS.

The memory you see as "unused" is not unused in the sense that other processes may use it -- it's reserved for your process. It's unused because your process isn't using it. But the "used" blocks aren't necessarily contiguous and don't have to appear in any particular order. You may write to such "unused" memory without causing a segfault, but malloc() won't mark it as "used", so a future call to malloc() could return a pointer that points into the middle of the formerly-unused memory. Then you would be in trouble.

Hope this is clear as mud.

[dazman19]

Hi Dazman19,

I have no idea why I gave you the name "Jaunty". Sorry to call you by a ubuntu version name....

Mark

No Worries

dwhitney67 yeh I have not used C/C++ for a while but its good to know the newer versions of c compilers are easier to use.

[allynm]

Good morning, trent.josephson,

After a call to malloc, a new membory break is established at the end of the "used" and "unused" allocated blocks, as I expect you know. The only way I know to generate a seg fault is to attempt to write beyond the new break. As long as you don't exceed the break you will not generate a seg fault.

But, here's the deal. If I allocate room for 10 ints using:

int * ptr=malloc(10*sizeof(int))

I will obtain 40 bytes of "used" heap and almost 135000 bytes of "unused, but allocated" heap. I seem to have no ex ante ability to limit the amount of "unused" memory. This seems wierd to me. Why all this unused allocation? What purpose does it serve?

Thanks for jumping into this trhead.

Mark

[dwhitney67]

But, here's the deal. If I allocate room for 10 ints using:

int * ptr=malloc(10*sizeof(int))

I will obtain 40 bytes of "used" heap and almost 135000 bytes of "unused, but allocated" heap.

How did you get the number 135000?