Page 1 of 3 123 LastLast
Results 1 to 10 of 26

Thread: How to monitor background apps

  1. #1
    Join Date
    Nov 2020
    Beans
    18

    How to monitor background apps

    I am developing an application software running on Ubuntu 20.04. It is made up of a few software modules, all running in background. Named pipes are used to pass data/info among these modules, so they are decoupled. Some of these modules use Ubuntu built-in daemons such as syslogd, snmpd, and others.

    What I want to do is to develop and add a supervisor module to the application. This module is intended to monitor all other modules and also active daemons and somehow auto restart them if they are down. At the moment I am thinking of using named pipe to each of the modules for alive heartbeat/ping approach. I assume I can use process names for monitoring daemons.

    My goal is to look for ways to improve overall software/system operational integrity.

    I am hoping to get some advice or engage with other members on the topic that may be of benefits to others as well.

    Thanks.

  2. #2
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,326

    Re: How to monitor background apps

    If you save the PID when you launch a process, then you can check it with ps.

  3. #3
    Join Date
    Nov 2020
    Beans
    18

    Re: How to monitor background apps

    Thanks for your input. I will implement that PID check. This leads to my next question below.

    Is there a way to make a daemon resilient; i.e. when it dies it can get restarted automatically, maybe through a .service configuration file for systemd?. That would simplify things a lot. If not, maybe I will add a new daemon with its sole purpose of monitoring the supervisor only and that both can check and restart each other.

  4. #4
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,326

    Re: How to monitor background apps

    There is/was a supervisory daemon called inetd that could be used to start and restart services.

    However, if it is only one service, then you can use a bash script with a while loop and a sleep statement, to restart a daemon if it would exit - after a 1 second delay. It is important to wait for the dust to settle before launching the daemon again, otherwise it can fail immediately and the resulting fast spin can bog the whole machine down.

  5. #5
    Join Date
    Nov 2020
    Beans
    18

    Re: How to monitor background apps

    I had a quick read and inetd does not seem to fit, more for socket services. The 1 second delay is a good idea and I will keep that in mind instead of instant restart.

    Effectively there is only one service (supervisor as I call it) that I want to make sure it's always up. I plan to use the while loop/sleep approach and basically every 30 seconds, it wakes up, checks status of supervisor, restarts it if needed, and goes back to sleep. Instead of a bash script as you suggested, I plan to have it run as a service and have supervisor (using the same while loop/sleep concept) to monitor/restart it as well. As mentioned earlier, these two monitor and restart each other to ensure both are up. I am not sure which is better/more reliable this approach vs bash script and cron. It would be great if Linux has some 'solid' service that once you have registered your services with it, it will make sure they are always up. Is there such a thing?.

  6. #6
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,326

    Re: How to monitor background apps

    I don't know whether this is still current and never used it myself (a Bash loop is good enough for me):
    https://www.digitalocean.com/communi...and-debian-vps

  7. #7
    Join Date
    Nov 2020
    Beans
    18

    Re: How to monitor background apps

    I had a look and yes it is available in Ubuntu server 20.04 LTS as I just installed it. It fits exactly what we are discussing here with your suggested bash script. I will add my other foreground programs for it to manage. This is perfect and thanks for your help.

    My application has to be resilient for 24/7 unattended use, pushing a reset button on the box can be a venture and expensive... Thanks again!

  8. #8
    Join Date
    Dec 2014
    Beans
    1,741

    Re: How to monitor background apps

    Quote Originally Posted by tc2020 View Post
    Is there a way to make a daemon resilient; i.e. when it dies it can get restarted automatically, maybe through a .service configuration file for systemd?. That would simplify things a lot. If not, maybe I will add a new daemon with its sole purpose of monitoring the supervisor only and that both can check and restart each other.
    You might want to read 'man 5 systemd.unit' (general format of a unit file), 'man 5 systemd.service' (format of a unit of type service; pay special attention to the keywords 'Restart' and 'WatchdogSec') and 'man 5 systemd.exec' (execution environment for processes started by systemd).

    Holger

  9. #9
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: How to monitor background apps

    Quote Originally Posted by tc2020 View Post
    Thanks for your input. I will implement that PID check. This leads to my next question below.

    Is there a way to make a daemon resilient; i.e. when it dies it can get restarted automatically, maybe through a .service configuration file for systemd?. That would simplify things a lot. If not, maybe I will add a new daemon with its sole purpose of monitoring the supervisor only and that both can check and restart each other.
    systemd can be setup to restart daemons, if you like. Probably want to limit the restart attempts to prevent infinite looping.

    Most of the stuff described was solved decades ago. Look into daemon-tools for examples. There are other existing solutions too. I haven't needed anything more than careful logging to syslog. Monitoring and alarming are common on unix servers.
    https://www.tecmint.com/best-linux-l...agement-tools/
    Zabbix, SysUsage, Nagios, OpenNMS, and similar tools can watch log files, send alarms, and create graphs.

    named-pipes are fine, but not as flexible as sockets.
    Last edited by TheFu; March 5th, 2021 at 09:38 PM.

  10. #10
    Join Date
    Nov 2020
    Beans
    18

    Re: How to monitor background apps

    I have tried both supervisor (suggested by HermanAB) and systemd (suggested by Holger and TheFu). Both seem to do what I was looking for using my simple setup with RPi3 running Ubuntu 20.04 server LTS and a logger bash script.

    Setup for supervisor,

    Code:
    
    $ sudo apt-get install supervisor    
    $ service supervisor restart   
    $ sudo nano /usr/local/bin/testscript.sh 
      
      #/!/bin/bash
      while :
    
      do
         logger $(date)
         sleep 30
      done
    
    $ sudo chmod +x /usr/local/bin/testscript.sh 
    $ sudo nano /etc/supervisor/conf.d/testscript.conf 
    
      [program:testscript] 
      command=sudo /usr/local/bin/testscript.sh   
      autostart=true   
      autorestart=true       
    
    $ sudo supervisorctl reread                                    
    $ sudo supervisorctl update                         
    

    Setup for systemd,

    Code:
    $ sudo nano /etc/systemd/system/testscript.service
    
      [Unit]
      Description= Time Logger testscript
    
      [Service]
      Type=simple
      ExecStart= sudo /usr/local/bin/testscript.sh
      Restart=always
     
      [Install]
      WantedBy=multi-user.target
    
    $ sudo systemctl restart testscript.service
    In both cases, testscript gets spawned immediately after a kill pid command.

    TheFu, you mentioned Nagios, OpenNMS, etc... what are some of the well accepted/commonly used 'open/free' NMS platforms by IT?. I would like to test some.

    Thanks for all the inputs.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •