Page 1 of 5 123 ... LastLast
Results 1 to 10 of 44

Thread: Memory leaks

  1. #1
    Join Date
    Jul 2008
    Beans
    95

    Exclamation Memory leaks

    Hi everyone,

    I'm using pthreads for a multithreaded program. here is a sample code:

    Code:
    while(1):
    {
    Accepter = accept( ListenSocket, NULL, NULL );
    cout << "Accepted" << endl;
    pthread_t t;
    int err=0;
    err=pthread_create(&t,NULL,ProcessThread,(void*)Accepter);
    }
    
    void * ProcessThread(void * arg)
    {
    int Accepter = (int)arg;
    /*WORK*/
    shutdown(Accepter,SHUT_RDWR);
    close(Accepter);
    }
    Questions:

    1) There is a problem in this code. Memory used by this program is ALWAYS INCREASING by accepting new connections!!! It seems that after thread function termination the pthread doesn't release the memory!?! Why? Should I use detach, exit or cleanup? detach first and then exit? where should I call them ?(end of thread function). Any other issues?

    2) Is there any problem with my "Accepter" variable?!!

    3) I have many connections but processing them is mostly less than a sec. Can I thread pool? If yes, how can I do that with pthread?

    Thanks a lot b4
    Last edited by hosseinyounesi; September 27th, 2010 at 07:23 AM. Reason: more speciefic

  2. #2
    Join Date
    Apr 2009
    Location
    Germany
    Beans
    2,134
    Distro
    Ubuntu Development Release

    Re: Memory leaks

    1. you have to detach the thread or join them again at some point.
    Otherwise you will leak memory.
    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
    2. The error checking on the Accepter variable is missing.
    you should use an type which has the same with as an pointer.
    In C99 this would be intptr_t from <inttypes.h>
    pre C99 long mostly does it.

    3. yes you can thread pool, and it may be faster, but as you will have to code it yourself for use a library.
    pthread does not have this built in.
    Last edited by MadCow108; September 27th, 2010 at 09:18 AM.

  3. #3
    Join Date
    Jul 2008
    Beans
    95

    Re: Memory leaks

    Thanks MadCow108.

    Another question:
    What's wrong with following command:
    Code:
    pthread_mutex_t mutex[MAX_THREADS];
    for(int i=0;i<MAX_THREADS;i++)
    {
          mutex[i]=PTHREAD_MUTEX_INITIALIZER;
    }
    after compile the following warning is shown:
    extended initializer lists only available with -std=c++0x or -std=gnu++0x

  4. #4
    Join Date
    Apr 2009
    Location
    Germany
    Beans
    2,134
    Distro
    Ubuntu Development Release

    Re: Memory leaks

    g++ is to dumb to figure out this is all compile time defined.
    so it "initializes" the array in the first line and fails when you want to initialize it again in the loop.

    use pthread_mutex_init instead or (less recommended) initialize it by hand in the first line (ar[] = {INIT, INIT, INIT}).
    Last edited by MadCow108; September 27th, 2010 at 12:47 PM.

  5. #5
    Join Date
    Jul 2008
    Beans
    95

    Re: Memory leaks

    Thanks again,

    1) But my main problem (memory leakage) is not solved
    I wrote detach and exit at end of each thread, but memory is increasing although slower! What's wrong? It appears that although I'm detaching thread or freeing or deleting my memory, OS is not freeing that! I wrote a thread pool too with 100 threads, so threads never die. I assumed that If i use thread pool, required memory will be allocated in program start and no memory increase will be done. But still the same!!!!!!!!!

    (Better to mention that I'm using XMLParser too.)

    2) It seems that I should use fork instead of pthread?! What are issues? forking overhead is more than pthread? how? I have more than 500 clients that each will connect every 30 secs.


    Please help, thanks
    Last edited by hosseinyounesi; September 27th, 2010 at 05:49 PM.

  6. #6
    Join Date
    Jun 2007
    Location
    Maryland, US
    Beans
    6,220
    Distro
    Kubuntu

    Re: Memory leaks

    Quote Originally Posted by hosseinyounesi View Post
    Thanks again,

    1) But my main problem (memory leakage) is not solved
    I wrote detach and exit at end of each thread, but memory is increasing although slower! What's wrong? It appears that although I'm detaching thread or freeing or deleting my memory, OS is not freeing that! I wrote a thread pool too with 100 threads, so threads never die. I assumed that If i use thread pool, required memory will be allocated in program start and no memory increase will be done. But still the same!!!!!!!!!

    (Better to mention that I'm using XMLParser too.)

    2) It seems that I should use fork instead of pthread?! What are issues? forking overhead is more than pthread? how? I have more than 500 clients that each will connect every 30 secs.


    Please help, thanks
    Are you allocating anything in this section of the code:
    Code:
    ...
    /*WORK*/
    ...
    Why would your clients connect so often? Why not just stay connected?

    If you are using a thread-pool, then it would be interesting to see how you have implemented it. I noticed from the code in your opening post that you are using C++.

  7. #7
    Join Date
    Jul 2008
    Beans
    95

    Re: Memory leaks

    Are you allocating anything in this section of the code:
    Code:
    ...
    /*WORK*/
    ...
    Yes i'm allocating but be sure that i free or delete it too!

    Why would your clients connect so often? Why not just stay connected?
    Stay connected for at least 30 secs?!!!

    If you are using a thread-pool, then it would be interesting to see how you have implemented it. I noticed from the code in your opening post that you are using C++.
    Here it is:
    Code:
    mutex = (pthread_mutex_t*) malloc( MAX_THREADS*sizeof(pthread_mutex_t) );
    thrd = (pthread_t*) malloc( MAX_THREADS*sizeof(pthread_t) );
    ti = (ThreadInfo*) malloc( MAX_THREADS*sizeof(ThreadInfo) );
    //initialize
    for(int i=0;i<MAX_THREADS;i++)
    {
    	pthread_mutex_init(&mutex[i],NULL);
    }
    //block all threads
    for(int i=0;i<MAX_THREADS;i++)
    {
            pthread_mutex_lock(&mutex[i]);
    }
    //pool creation
    for(int i=0;i<MAX_THREADS;i++)
    {
    	pthread_create(&thrd[i],NULL,ProcessThread,(void*)i);
    	//all threads are available
    	availableThreads.push_back(i);
    }
    
    while(true)
    {
    	Accepter = accept( ListenSocket, NULL, NULL );
    	cout << "Accepted" << endl;
    	if( Accepter==-1 ) {
    		sprintf(logstr,"%d",errno);
    		LOG_ERROR(logstr,"couldn't accept");
    		continue;
    	}
    	if(availableThreads.size()!=0)
    	{
    		pthread_mutex_lock(&data_mutex);
    		int index=availableThreads.front();
    		availableThreads.pop_front();
    		ti[index].Accepter=Accepter;
    		pthread_mutex_unlock(&data_mutex);
    		pthread_mutex_unlock(&mutex[index]);
    	}
    }
    
    void * ProcessThread(void * arg)
    {
            pthread_mutex_lock(&mutex[i]);
    
    	pthread_mutex_lock(&data_mutex);
    	Accepter=ti[i].Accepter;
    	availableThreads.remove(i);
    	pthread_mutex_unlock(&data_mutex);
    
            SEND & RECV
            PARSING PACKET
            FREE MEMORIES
    
            pthread_detach(pthread_self())
    
            pthread_mutex_lock(&data_mutex);
    	availableThreads.push_back(i);
    	pthread_mutex_unlock(&data_mutex);
    
    }
    Memory size just increases slowly?!!! and after a while I got "Virtual memory exhausted."!
    detach will release the memory? should i use setstack? I'm really confused with pthread and linux

    Waiting for your answers, thanks a lot
    Last edited by hosseinyounesi; September 27th, 2010 at 08:02 PM.

  8. #8
    Join Date
    Sep 2010
    Location
    Linolnshire UK
    Beans
    11

    Re: Memory leaks

    i dont mean to sound completely stupid, but how do you get memory leakage? what sort of memory leakage, like RAM? or storage memory?
    lehmans terms will be fine as i dont really know anything about programming, this title just took my interest

  9. #9
    Join Date
    Jun 2007
    Location
    Maryland, US
    Beans
    6,220
    Distro
    Kubuntu

    Re: Memory leaks

    Quote Originally Posted by hosseinyounesi View Post
    Memory size just increases slowly?!!! detach will release the memory? should i use setstack? I'm really confused with pthread and linux

    Waiting for your answers, thanks a lot
    My experience with POSIX threads is also limited, but I do have some ideas.

    First of all, do not destroy the thread(s) after they have been invoked. Stall them, but don't destroy them... for this is costly in terms of CPU cycles.

    All of your threads do the same thing using one common data parameter... a socket ID.

    I know this may seem like a step into left field, but take a moment to think about what you are attempting to accomplish. From what I see, you have a parent thread with many child-threads. It would be helpful if these child-threads were made aware of their parent, and the (yet to be defined) functions this parent offers.

    Your parent-thread should provide the means for your child-threads to say "pick me! pick me!" when the child is done processing a client. Thus the child-thread needs the mean to notify the parent that it is done processing a client.

    Consider this (and believe me, I've never done exactly like this before, but something similar):
    Code:
    class Thread
    {
    public:
       void start()
       {
          pthread_create(&tid, 0, helper, this);
       }
    
       inline pthread_t getTID() const { return tid; }
    
    protected:
       Thread();
    
       virtual void run() = 0;
    
    private:
       static void* helper(void* arg)
       {
          Thread* t = reinterpret_cast<Thread*>(arg);
    
          t->run();
    
          return 0;
       }
    
       pthread_t tid;
    };
    
    
    class ParentThread : public Thread
    {
    public:
       ParentThread(int numChildThreads, int portNumber);
    
       void childIsDone(int childID);
    
    private:
       void run();
    
       std::vector<ChildThreads> childThreadPool;
       std::queue<int>           availableChildThreadIDs;
       int                       listenSocket;
    };
    
    class ChildThread : public Thread
    {
    public:
       ChildThread(ParentThread& parent, int childID, pthread_mutex_t& mutex);
    
       void handleClient(int clientSocket);
    
    private:
       void run();
    
       ParentThread&    parent;
       int              childID;
       pthread_mutex_t& mutex;
       int              clientSocket;
       bool             remainIdle;
    };
    Here, the Parent Thread creates N child threads, which by default should be sitting idle, which means that even though they are running, they are not processing anything.

    When a child is needed, the Parent Thread obtains an available ID from the queue (something similar to what you are already doing). If successful in getting a child, the parent informs the child of the client socket identifier, and from that point the child does something constructive.

    When the child is done, it notifies it's parent, via the parent's childIsDone() method that it is done... and in essence, wants to be re-inserted into the queue. It then goes back into an idle mode.

    With this design, I'm not sure if the Child Thread really needs a mutex, unless it is sharing access to data objects shared by other Child Threads or the Parent. Access to the Parent's queue should definitely be mutex protected, but you require only one mutex for this, not N mutexes.
    Last edited by dwhitney67; September 27th, 2010 at 08:33 PM.

  10. #10
    Join Date
    Jul 2008
    Beans
    95

    Re: Memory leaks

    Thanks you dwhitney67,

    But I don't think that your way will help me about memory increasing problem. Have you used fork before? Is that a good idea to use fork instead of thread?

    Thanks

Page 1 of 5 123 ... LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •