Why proper error handling should ALWAYS be done

[tyoc]

For example (not the best example, but anyway), if a program that as its main functionality keeps a respirator going, but uses PulseAudio to beep when human attention is needed, this program would crash on OOM.

Indeed a very bad example, you normally would not use a device that monitor something important and run apps that can take and use all memory in a second in that device.

Yes, there are plenty of actual real world applications where this is an issue. Not many such applications are available for Linux. I wonder why? Unfortunately as long as the developers of core OS functionality refuse to even acknowledge that there are such applications, these applications are unlikely to ever be written for Linux.

I don't get what you say about "real world applications"... I have in front of me a real world application... I'm using it, also you are using it right now (I don't see how you and I can be using something that is not REAL)... I can correct you in what I think you are trying to say "real-time OS applications", there is RTOS, if you want that, you should look at the possibilities.

Unfortunately as long as the developers of core OS functionality refuse to even acknowledge that there are such applications, these applications are unlikely to ever be written for Linux.

Which type of apps, I still cant see what type of apps... if it is not RTOS apps.

where malloc() returning 0 is a very real possibility

You mean that your apps stress the resources to even finally cause an OOM, and that posibility is really high that in each run you fall in an OOM?, say me please what type of applications you write and in what field are you working, Im curious about the field of the app.

malloc returning zero is not a "possibility" is a fact, other thing is: if that fact will happen or not... and that depend on a lot of factors, from system to system, user to user and so on... that is the actual possibility, not malloc returning zero.

If you write "real world applications" where that possibility is high, thus simple, try to low the possibility to happen (one way or the other, software, hardware, users, focus the usage of the machine give it a specific task, etc... etc)... if possible, if not, nice that I don't need to write such an app.

a. How much memory the user of your library may need
This is easy to predict because you can simply decide not to support users of your library who are the type that foolishly and haphazardly abuse system resources.

And how you do that?, put limits?, restrictions of how many objects you can run?, write a monitor that knows the state of usage of your lib for certain thread, process or group of them?, why malloc will need to limit me the memory that I want to get? (it is something variable, if malloc have a limit say 50% of total memory, and if there is a system with very specific task running, but 1 of them need 70% of mem, how that limitation will impact it .

The question is simple. "How much memory the user of your library may need", is not the business of the library, his business is be the library and only of his own resources not decide before the restrictions of usage, not all the possible applications that can use it will have the same behavior and needs, because not a single restriction work for all there is no "one ring to rule them all".

Now, such a program would definitely not use PulseAudio, but to me it is not good library design to force applications to crash without a chance to die gracefully when it would be fully possible to let them.

Don't know if in this thread or in other related (all of them look the same to me) I have said that will be nice if some one can find an app that is a client of PulseAudio and ask them if that has been the case, I also have said that would be nice to skim in the mails or bug report list of that client of app using it, for see if some of the problems they have is loosing data because that behavior.

I also have said that from what I know from servers (Pulse Audio is a server "A sound server is basically a proxy for your sound applications"... and perhaps other things), thus in that context and without see the code, I think that exit is his choice of exit the server, the worker thread or process, whatever he choice to do, I also have said well if you use pa_malloc intead of malloc or your wrapper in your own application, thus your problem , but from the little that I fond in examples of play music with PA haven't seen to this moment usage of pa_malloc in those examples, only what I can judge normal in a server/client communication that is connection and interaction.


I also will like to know the place where pa_malloc is documented like API, ahvent found it to the momment.

This is what I try: http://www.google.com.mx/search?q=pa...pulseaudio.org



-----
Also the argument about audio for a blind people is valid, but the autohr thinking that almost all people that use a PC are not blind and can stop listening data... or stop recording (see that is the client who choose how to handle/store temporarily/final that data and how to process it finally, I hardly think that they need all the audio data in more than needed eg. 1Gb of audio data to be in memory at the same time for do an effect that will have a duration of 30 seconds...).

Being for the blind more valid, simple, knowing how OOM cast the apps with his rules, get up the priority of the audio server, isn¡t? at the end, priority also work for give more importance... I guess/suppose is the reason for be there, use it.

[j_g]

If you ever write a real world application which attempts to allocate enough memory to cause a 0 return value, then you should step-aside and let somebody more qualified write the program.

But Linux over-commits (by default). So you don't know how much memory is enough to cause a 0 return value. All you know for sure is that a 0 return value is possible, and if you "handle" it by calling exit()... well, perhaps say goodbye to your enduser's work. I strongly suspect an enduser would not look favorably upon the "qualifications" of a programmer who does this.

this "exit()" part of these library wrappers is just a comment -- it never gets executed.

How have you determined this to be true? We've already seen that malloc can return 0. Are you claiming that the following code doesn't call exit() if malloc returns 0?

ptr = malloc(SomeAmtOfBytes);
if (!ptr) exit();

I'm not sure I'd be confident in the qualifications of a programmer who answers "yes" to the above question.

If it ever does, then your program has more serious issues to contend with {for instance... its author}.

What specific "issues"?... (other than the sole, unsubstantiated ad hominem that the issue is the software author -- with such ad hominem arguments being a staple in the Ubuntu Programming forum with the apparent tacit approval of at least one moderator, whereas threads containing actual detailed points to back up assertations are to be locked and discouraged).

[pmasiar]

I think reasonable compromise can be established.

We all know that doing proper thing and handle all errors properly is a lot of work, can be easily 30-50-80% of the code. So for simple non-critical app like audio in most cases: whn out of memory, audio app can just die and be done with it, so more important apps can live on. If this is documented behavior, all is kosher. If it does not suit you, please be my guest take my code and add 70% of proper error handling exactly as your app needs. It is open sourse after all.

Some people like to solve right problems any way they can (a.k.a "hack"), other people like to solve problems right way or not at all. In most cases I prefer "good enough" solution over no solution. In some cases, I seek perfection, but audio is not it.

"Perfect" should not be opponent of "good enough".
Or, as "Zen of Python" says: "Practicality beats purity"

IMHO, YMMV, I am NOT a kernel hacker. You can continue discussion, which IMHO look very close to Ambrose Bierce's definition:

Discussion
n. A method of confirming others in their errors

[tyoc]

well, perhaps say goodbye to your enduser's work. I strongly suspect an enduser would not look favorably upon the "qualifications" of a programmer who does this.

Like I said, I have not read the code of PA, but reading the introduction, it say is a server, a proxy... and watching some of the samples, I not see a pa_malloc or pa_free in those samples (yes they are lite) that mean you dont need them there (at less for lite apps).

Most probably without look at code he have choice to close the PA server, NOT the client (that is all that I can argue about why he choice to do that)... do that perhaps is not acceptable from PointOfView of a shared library, but I guess is correct taking that is a "simple" server to serve to multiple clients doing sound data (IO).

Also note that like from a socket, when you read the data you not maintain all the data (eg if you gonna receive 10Gb of a file, you don't have all the file until end for store it), it is a problem of the client only communicate back and fort interacting with the server primarily (open, close , send, receive), secondarily but more specific to a certain client (using the tools to comunicate with the server) is how to handle data and how much process at one time. also how to store or retrieve it, is a problem of the app that is using/acting like the client, the sole purpose of a server is to serve, and the client to be client of the data. what is serving? sound data input (from a mic and related.. to little input if you ask me, I dont see how it can get for example a transference in 1 sec of a input signal that is about 300Mb) and output. I mean well if the client choice to store all the temporal data in memory and then it crash or the server close the connection because it is closed, I dont see why the server should take care of more than communicating to the client things about the content. How a client handle that... from my POV is the problem of the client programm. In other words, why would Apache care about how FF or IE display the content, or how to serve a PHP, ASP, xxDinamicPage?? it sole porpoise is to serve, the clients are the ones that display it in parts and the dynamic is work of other components (not the server), store it in a temp file or in memory, that is not the work of the server.

The above I think it apply to the case, also I repeat haven't checked the code, and don't know from which parts that function that raise this thread is called or if that is called from the client code or not, and how is the server loop, internals etc, etc etc.

[Tuna-Fish]

where malloc() returning 0 is a very real possibility

If you had that much experience, and/or had the decency of spending the whole 2 minutes testing it, you'd know that in "real world applications" malloc doesn't return 0.

Behold, proof:
Code:

#include <stdio.h>
#include <stdlib.h>
#define SIZE 20
#define AMOUNT 1073741824 // 1024*1024*1024
int main()
{
    int i;
    int*a[SIZE];
    for (i=0;i<SIZE;i++){
        a[i] = malloc(AMOUNT);
    }
    printf("pointers:\n");
    for (i=0;i<SIZE;i++){
        printf("%x\n",a[i]);
        free(a[i]);
    }
    return 0;
}

Result:

pointers:
ff8b2010
3f8b3010
7f8b4010
bf8b5010
ff8b6010
3f8b7010
7f8b8010
bf8b9010
ff8ba010
3f8bb010
7f8bc010
bf8bd010
ff8be010
3f8bf010
7f8c0010
bf8c1010
ff8c2010
3f8c3010
7f8c4010
bf8c5010

My machine has 1 GB ram and 2 GB swap.

I just allocated 20GB.
Malloc returns 0 when the amout it is asked to allocate is more than the biggest chunk of contiguous address space on the machine. Not a big issue on a 64 bit machine...

If you need to be sure you don't run out of mem, use calloc. it nulls the pages when it allocates, and if i repeated the above with it, it would return 0 on the 3rd allocation. I don't because I don't want to wait the time it would take to null 2gigs of swap.

[j_g]

Malloc succeeds when there may be no memory left when the process later uses the memory. The malloc/free paradigm is broken.

You got it.

Here's how to improve it (as I noted in a different thread that a particular moderator chose to lock, making it hard to discuss such pertinent, important programming topics here, and instead readers get to wade through yet more posts from people who simply rely upon ad hominem arguments up until they can connive a moderator to stifle discussion that some advocacy case wants stifled for the sake of... purely advocacy. I mean when someone(s) answers a detailed post about programming with little more than "you're a troll", and convinces a moderator to lock the thread, that says it all. The following discussion would have gone into THAT thread since it really should be there. But what can you do when a moderator is not making logical discussion possible?).

What Linux needs is an equivalent to Win32's VirtualAlloc so that the app can literally ask the operating system if there is actually the ability to make particular pages available right now, and if not, the OS can tell the app "No, there isn't, so don't read/write to/from the memory". How does the OS tell the app whether the requested page is actually available or not? Much like malloc(), VirtualAlloc returns non-zero if so, or 0 if not. In other words, there are separate functions to reserve memory, versus committing memory, ie, actually ask for some virtual memory (whether over-committed or not) to be physically made available right now. (Actually, VirtualAlloc serves both purposes. But you can think of it as having two, distinct functions). The OS should expect the app to be sensibly written and properly handle being told "no". (And the app should be sensibly written. It shouldn't do something like "Well I'm just going to loop around this call to VirtualAlloc, constantly asking you to commit").

The OS shouldn't wait until the app actually accesses memory before it does the commit. It should first give the app an opportunity to say "I really do intend to access this memory now, so make sure it is backed up by either RAM or the swap file. Don't just write me out an IOU that you can't cash". Plus, by an app calling this new commit function, then the OS can assume that the app is written not to do things that made Linux resort to over-committing to begin with (ie, forking to run code that shouldn't need copies of its global resources, etc. See my other thread, which is now locked so you can't discuss these things there anymore, for more details).

Of course, especially on a multi-tasking system, there is always the possibility that some "unexpected event" could happen that causes the system to have to resort to freeing up RAM (even though an app hasn't yet offered to give back RAM). But if it's a desktop system with a user sitting at it, hopefully what the OS does is inform the user that he needs to free up RAM, and he should go do that now. That's why Apple took the BSD code and made exactly that happen. Apple said "We make a desktop OS. We bill this as being particularly user friendly. We'll be damned if we handle low memory conditions by terminating software without giving the user some input in the recovery. We could otherwise be ____ing him over").

If the problem needs to be fixed then inside each library is not the place to do it.

That's exactly where we need to start doing it (right after the malloc/free stuff gets fixed). Shared libs are where it impacts the majority of apps running on a system. If the shared libs are fixed, what Linux apps I've written are fairly good to go. (I check for 0 malloc return, and gracefully recover).

But of course, it eventually needs to be fixed in all code.

Each library needs it's own set of platform specific and possibly even kernel version specific code just to allocate memory.

Well, since distros typically update all the software apps at the same time that they update the kernel (and have an update manager that further oversees the entire system), the transition may not be as catastrophic an event as it may seem. Hopefully a distro coordinates things as well as it can (assuming that word is allowed to get out to developers).

But yes, backward compatibility can be a bitch. But dealing with it, while also allowing new features that impact usability and security, is part of the software dev process. Sometimes, you do it in a transparent way (and I think my suggestion is really quite transparent -- old apps continue to "work" but don't yield the same usability and stability as new apps), and sometimes you do it in a less transparent way (ie, Win16 to Win32 migration).

All it takes is one process that doesn't do this to trigger the OOM killer which will potentially kill a "well behaved" process.

Yes, but the less software that does inadequate error checking you have, the less you'll see the OOM Killer, and if the word gets out to enough programmers, the better it will be. (Not via here of course, because any attempts to get the message out here are being "locked" by a moderator, without any reason given other than "we're studying something because some advocates complained". Well, at least it's good that a moderator wouldn't resort to the extremely unprofessional behavior of locking a thread from all further posts, then subsequently and admittedly unlocking the very same thread to specifically allow a certain, non-moderator to post what amounted to another ad hominem argument, and then immediately relocking the thread. I mean, that would be totally inexcusably poor behavior).

I write lots and lots and lots of shared libs, and mine are definitely all ready to go (insofar as checking malloc for a 0 return, and gracefully handling it are concerned). I frankly would have little problem updating my code to also account for a new function to handle committing mem. It's something that I have to do on other platforms anyway, so my code already accounts for it. If other programmers did the same right from the start, they'd be well along the way too. (I'm told that Gnome has been retooling code to really improve error handling issues such as this. They even have a function that attempts to check if mem can be committed, to the extent that it can be implemented outside of the kernel, and they use that).

[Wybiral]

If you had that much experience, and/or had the decency of spending the whole 2 minutes testing it, you'd know that in "real world applications" malloc doesn't return 0.

Behold, proof:
Code:

I ran your exact program and got:

pointers:
77e42008
37e41008
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Behold... Proof.

What version of Ubuntu are you using?

[j_g]

doing proper thing and handle all errors properly is a lot of work

There's an old saying "You can have it quick. Or you can have it good.".

whn out of memory, audio app can just die and be done with it, so more important apps can live on.

If you want to write an MP3 player that does that, fine. (But I'd still prefer one that doesn't). But if you're writing a waveform editing program (ie, Audacity) or software sequencer (ie, RoseGarden), or an audio API used by numerous apps, or a distro that is supposed to be for media work (ie, Ubuntu Studio), then don't do it. Just... don't.

please be my guest take my code and add 70% of proper error handling exactly as your app needs. It is open sourse after all.

Well yeah, I kind of wish that the Pulse Audio author had instead expended his time adding features/redesigns to ALSA's code base (which has already been accepted as the audio API that apps should use) rather than come up with yet another audio API. I don't have the same qualms about ALSA that I do about Pulse Audio. But that's how it goes. What I can do is get the message out. That's even easier than fixing code that eschews the previous open source standard, and maybe shouldn't have been written anyway (if one follows your above advice).

In some cases, I seek perfection, but audio is not it.

Then ALSA already exists, is the standard, and it's not perfect. There you go.

[j_g]

I ran your exact program and got:
Behold... Proof.

Yep.

Wy, this may have to do with the differences between the size of your swap file and total RAM, and his. You're probably running under a more constrained system, so you really are hitting the "low memory handling" of Linux whereas some of these other guys may have systems where their test cases don't trigger the behavior (so they assume it doesn't exist). I haven't looked at the algorithm Linux uses for figuring out how much it will over-commit. It may be that this is relative to how much swap space and RAM you have. I have not heard of a way of asking Linux what this amount would be. And even if you could, it still doesn't change the fact that malloc can return 0. (Can we please put this rumor to bed?)

I think it would be more interesting if these people turned their swap space off (ie, swapoff). They may be surprised at the difference. But then, this underscores how perilous it is to not only assume what constitutes "asking for more memory than you should", but also that malloc will never return a 0.

Wait a minute. Wy, you don't live in the twilight zone, do you? Because people are telling me that what you observed can't happen.

[Tuna-Fish]

I ran your exact program and got:

pointers:
77e42008
37e41008
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Behold... Proof.

What version of Ubuntu are you using?

64bit.

As said, malloc returns 0 if and only if there is not enough adress space available on the system. 32bit system has 2gb of address space available for user.