mysql Ver 14.14 Distrib 5.7.27 stops suddenly on ubuntu 18.04.3 LTS

[marchello_lippi2]

Hi all,
I encoutered issue that mysql Ver 14.14 Distrib 5.7.27 stops suddenly on ubuntu 18.04.3 LTS.
Will add logs here.

Is it any known issue already?

Upd: Syslog is below. Reboot helped me, but this is just a workaround. Please advise.

Nov 12 05:01:03 m3a systemd[1]: Starting MySQL Community Server...
Nov 12 05:01:03 m3a kernel: [316377.797698] audit: type=1400 audit(1573527663.497:21): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=817 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 12 05:01:03 m3a kernel: [316377.866590] audit: type=1400 audit(1573527663.565:22): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=819 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Nov 12 05:01:03 m3a mysqld[819]: Initialization of mysqld failed: 0
Nov 12 05:01:03 m3a systemd[1]: mysql.service: Control process exited, code=exited status=1
Nov 12 05:01:03 m3a systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 12 05:01:03 m3a systemd[1]: Failed to start MySQL Community Server.
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Nov 12 05:01:04 m3a kernel: [316378.321822] audit: type=1400 audit(1573527664.021:23): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=834 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
Nov 12 05:01:04 m3a systemd[1]: Stopped MySQL Community Server.
Nov 12 05:01:04 m3a systemd[1]: Starting MySQL Community Server...
Nov 12 05:01:04 m3a kernel: [316378.342296] audit: type=1400 audit(1573527664.041:24): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=836 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Nov 12 05:01:04 m3a mysqld[836]: Initialization of mysqld failed: 0
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Control process exited, code=exited status=1
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 12 05:01:04 m3a systemd[1]: Failed to start MySQL Community Server.
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Scheduled restart job, restart counter is at 2.
Nov 12 05:01:04 m3a kernel: [316378.823720] audit: type=1400 audit(1573527664.525:25): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=847 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 12 05:01:04 m3a systemd[1]: Stopped MySQL Community Server.
Nov 12 05:01:04 m3a systemd[1]: Starting MySQL Community Server...
Nov 12 05:01:04 m3a kernel: [316378.845412] audit: type=1400 audit(1573527664.545:26): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=849 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Nov 12 05:01:04 m3a mysqld[849]: Initialization of mysqld failed: 0
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Control process exited, code=exited status=1
Nov 12 05:01:04 m3a systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 12 05:01:04 m3a systemd[1]: Failed to start MySQL Community Server.
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Scheduled restart job, restart counter is at 3.
Nov 12 05:01:05 m3a systemd[1]: Stopped MySQL Community Server.
Nov 12 05:01:05 m3a systemd[1]: Starting MySQL Community Server...
Nov 12 05:01:05 m3a kernel: [316379.324082] audit: type=1400 audit(1573527665.025:27): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=861 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 12 05:01:05 m3a kernel: [316379.344849] audit: type=1400 audit(1573527665.045:28): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=863 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Nov 12 05:01:05 m3a mysqld[863]: Initialization of mysqld failed: 0
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Control process exited, code=exited status=1
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 12 05:01:05 m3a systemd[1]: Failed to start MySQL Community Server.
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Nov 12 05:01:05 m3a systemd[1]: Stopped MySQL Community Server.
Nov 12 05:01:05 m3a systemd[1]: Starting MySQL Community Server...
Nov 12 05:01:05 m3a kernel: [316379.824323] audit: type=1400 audit(1573527665.525:29): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=874 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 12 05:01:05 m3a kernel: [316379.845477] audit: type=1400 audit(1573527665.545:30): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=876 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Nov 12 05:01:05 m3a mysqld[876]: Initialization of mysqld failed: 0
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Control process exited, code=exited status=1
Nov 12 05:01:05 m3a systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 12 05:01:05 m3a systemd[1]: Failed to start MySQL Community Server.
Nov 12 05:01:06 m3a systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Nov 12 05:01:06 m3a systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Nov 12 05:01:06 m3a systemd[1]: Stopped MySQL Community Server.
Nov 12 05:01:06 m3a systemd[1]: mysql.service: Start request repeated too quickly.
Nov 12 05:01:06 m3a systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 12 05:01:06 m3a systemd[1]: Failed to start MySQL Community Server.

[SeijiSensei]

First thing I'd check is the permissions on the directory /var/lib/mysql. The directory and its contents should all be owned by user mysql and group mysql.

[marchello_lippi2]

First thing I'd check is the permissions on the directory /var/lib/mysql. The directory and its contents should all be owned by user mysql and group mysql.

Thanks, applied

sudo chown -R mysql:mysql /var/lib/mysql

Will see in couple of days if it will be enough.
Though found another post reg. this issue:
https://support.plesk.com/hc/en-us/a...peration-open-
(just FYI).

[SeijiSensei]

Did the server start after you changed the permissions? Or do you still get

Nov 12 05:01:05 m3a mysqld[876]: Initialization of mysqld failed:

There's also an AppArmor problem.

Nov 12 05:01:05 m3a kernel: [316379.824323] audit: type=1400 audit(1573527665.525:29): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=874 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Are you using a version from the repositories or one from some other source? If the latter, it may not have the required AppArmor markings. Looks like you can disable AppArmor monitoring of an application by following this method: https://help.ubuntu.com/community/Ap...le_one_profile

[marchello_lippi2]

Did the server start after you changed the permissions?

No, it started after I did

sudo reboot

Are you using a version from the repositories

Yes, it's from the repositories.
It happens occasionally after

sudo /etc/init.d/mysql stop; sudo /etc/init.d/mysql start

(but not every time). Thus I will observate it and will update here if it still fails to start in couple of days (and will try to apply AppArmor fix etc).
Thanks!

[marchello_lippi2]

Ok, it happened again today.
I added lines below into /etc/apparmor.d/usr.sbin.mysqld and now journalctl -xe doesn't say anything about apparmor. But mysql still couldn't start until I rebooted machine...

capability dac_read_search,
/proc/*/status r,
@{PROC}/@{pid}/status r,
/sys/devices/system/node/ r,
/sys/devices/system/node/node*/meminfo r,
/sys/devices/system/node/*/* r,
/sys/devices/system/node/* r,

[marchello_lippi2]

Disabled apparmor profile for mysql

$ sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

Restarted mysql (successfully). Will see in couple of days if it helped.

[marchello_lippi2]

So far so good, though waiting more to confirm.

[marchello_lippi2]

Happened again. But this time I just can't trace it.

Just ran

sudo /etc/init.d/mysql start

and it started successfully without machine reboot.

[marchello_lippi2]

Added below crontab record under root:

9 * * * * bash -c "sleep 10; if pgrep mysql; then exit; else /etc/init.d/mysql start; fi" > /dev/null

Will see in couple of days if it helped.