Есть сервер в котором запущены контейнеры backend приложений, всего 2 сервера. Но по какой-то причине на сервере backen-01 начинают падать контейнеры в примерно в 02:30 каждый день.
контейнеры запускаются в docker-compose
для контейнеров стоит restart: always
но сами подняться они не могут, но если выполнить команду docker compose --profile PROFILENAME restart
то все поднимается и работает без проблем до 2 ночи
вывод команды journalctl --since “2024-11-02 02:29:00” --until “2024-11-02 03:35:00”
-- Logs begin at Sat 2024-11-02 02:53:08 MSK, end at Sat 2024-11-02 08:18:04 MSK. --
Nov 02 02:53:08 srv-be01 kernel: IPv6: ADDRCONF(NETDEV_UP): vethb79093f: link is not ready
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 21(veth9fee187) entered disabled state
Nov 02 02:53:08 srv-be01 kernel: device veth9fee187 left promiscuous mode
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 21(veth9fee187) entered disabled state
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 19(veth1e50db0) entered disabled state
Nov 02 02:53:08 srv-be01 kernel: device veth1e50db0 left promiscuous mode
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 19(veth1e50db0) entered disabled state
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.846011706+03:00" level=info msg="shim disconnected" id=e4ff2828166fc91bf4518547aaa4d99b66d09dc3e9ef842ac9979ee88ae4c0fc
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.846056917+03:00" level=warning msg="cleaning up after shim disconnected" id=e4ff2828166fc91bf4518547aaa4d99b66d09dc3e9ef842ac9979ee88ae4c>
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.846066656+03:00" level=info msg="cleaning up dead shim"
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.856975345+03:00" level=warning msg="cleanup warnings time=\"2024-11-02T02:53:08+03:00\" level=info msg=\"starting signal loop\" namespace>
Nov 02 02:53:08 srv-be01 rsyslogd[1314]: imjournal: journal files changed, reloading... [v8.2102.0-15.el8 try https://www.rsyslog.com/e/0 ]
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.846937221+03:00" level=info msg="ignoring event" container=e4ff2828166fc91bf4518547aaa4d99b66d09dc3e9ef842ac9979ee88ae4c0fc module=libcontai>
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.868843545+03:00" level=warning msg="error locating sandbox id d4a56b2b2929dfd954f47cb31ff8042d6d8a4e1ca102cfe4ec964330caa5adc4: sandbox d4a5>
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.875878004+03:00" level=warning msg="failed to delete container from containerd" container=e4ff2828166fc91bf4518547aaa4d99b66d09dc3e9ef842ac9>
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.875942148+03:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=e4ff2828166fc91bf4518547aaa4d99b66d09dc>
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.876697866+03:00" level=error msg="e4ff2828166fc91bf4518547aaa4d99b66d09dc3e9ef842ac9979ee88ae4c0fc cleanup: failed to delete container from >
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.876785886+03:00" level=warning msg="error locating sandbox id d4a56b2b2929dfd954f47cb31ff8042d6d8a4e1ca102cfe4ec964330caa5adc4: sandbox d4a5>
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 20(veth938206e) entered disabled state
Nov 02 02:53:08 srv-be01 kernel: device veth938206e left promiscuous mode
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 20(veth938206e) entered disabled state
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.930621130+03:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd>
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.931014409+03:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containe>
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.931101607+03:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrp>
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.932507049+03:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/cbe8f66a8a6>
Nov 02 02:53:08 srv-be01 systemd-udevd[201218]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 02 02:53:08 srv-be01 systemd-udevd[201218]: Could not generate persistent MAC address for vethc9635ea: No such file or directory
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 7(vethe3c6687) entered blocking state
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 7(vethe3c6687) entered disabled state
Nov 02 02:53:08 srv-be01 kernel: device vethe3c6687 entered promiscuous modesrv-be01
Nov 02 02:53:08 srv-be01 kernel: IPv6: ADDRCONF(NETDEV_UP): vethe3c6687: link is not ready
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 7(vethe3c6687) entered blocking state
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 7(vethe3c6687) entered forwarding state
Nov 02 02:53:08 srv-be01 NetworkManager[1140]: <info> [1730505188.9409] manager: (vethc9635ea): new Veth device (/org/freedesktop/NetworkManager/Devices/4014596)
Nov 02 02:53:08 srv-be01 systemd-udevd[201233]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 02 02:53:08 srv-be01 systemd-udevd[201233]: Could not generate persistent MAC address for vethe3c6687: No such file or directory
Nov 02 02:53:08 srv-be01 NetworkManager[1140]: <info> [1730505188.9429] manager: (vethe3c6687): new Veth device (/org/freedesktop/NetworkManager/Devices/4014597)
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 8(veth55c9386) entered blocking state
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 8(veth55c9386) entered disabled state
Nov 02 02:53:08 srv-be01 kernel: device veth55c9386 entered promiscuous mode
Nov 02 02:53:08 srv-be01 kernel: IPv6: ADDRCONF(NETDEV_UP): veth55c9386: link is not ready
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 8(veth55c9386) entered blocking state
Nov 02 02:53:08 srv-be01 kernel: br-9253fa3d84e5: port 8(veth55c9386) entered forwarding state
Nov 02 02:53:08 srv-be01 NetworkManager[1140]: <info> [1730505188.9515] manager: (veth345142a): new Veth device (/org/freedesktop/NetworkManager/Devices/4014598)
Nov 02 02:53:08 srv-be01 NetworkManager[1140]: <info> [1730505188.9524] manager: (veth55c9386): new Veth device (/org/freedesktop/NetworkManager/Devices/4014599)
Nov 02 02:53:08 srv-be01 dockerd[1991]: time="2024-11-02T02:53:08.987008740+03:00" level=info msg="ignoring event" container=efe7c6de1bd4ca850968c762b477dcd295765c7cd08e420491c008c2b39e48dd module=libcontai>
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.987667998+03:00" level=info msg="shim disconnected" id=efe7c6de1bd4ca850968c762b477dcd295765c7cd08e420491c008c2b39e48dd
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.987743795+03:00" level=warning msg="cleaning up after shim disconnected" id=efe7c6de1bd4ca850968c762b477dcd295765c7cd08e420491c008c2b39e4>
Nov 02 02:53:08 srv-be01 containerd[1182]: time="2024-11-02T02:53:08.987757275+03:00" level=info msg="cleaning up dead shim"
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.009980829+03:00" level=warning msg="cleanup warnings time=\"2024-11-02T02:53:08+03:00\" level=info msg=\"starting signal loop\" namespace>
Nov 02 02:53:09 srv-be01 dockerd[1991]: time="2024-11-02T02:53:09.025139696+03:00" level=warning msg="error locating sandbox id b62e27c7429776ea180deca9dddd1dc2c290f94abdfe4bad5aeb6809dff86f38: sandbox b62e>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.030559900+03:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.030704350+03:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containe>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.030718022+03:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrp>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.031741873+03:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/0ea3e66dd7b>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.041077583+03:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.041175992+03:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containe>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.041186245+03:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrp>
Nov 02 02:53:09 srv-be01 containerd[1182]: time="2024-11-02T02:53:09.041794540+03:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/2cc204aa11e>
Nov 02 02:53:09 srv-be01 dockerd[1991]: time="2024-11-02T02:53:09.042408954+03:00" level=warning msg="failed to delete container from containerd" container=efe7c6de1bd4ca850968c762b477dcd295765c7cd08e420491>
Nov 02 02:53:09 srv-be01 dockerd[1991]: time="2024-11-02T02:53:09.042456446+03:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=efe7c6de1bd4ca850968c762b477dcd295765c7>
Nov 02 02:53:09 srv-be01 dockerd[1991]: time="2024-11-02T02:53:09.045112254+03:00" level=error msg="efe7c6de1bd4ca850968c762b477dcd295765c7cd08e420491c008c2b39e48dd cleanup: failed to delete container from >
Nov 02 02:53:09 srv-be01 dockerd[1991]: time="2024-11-02T02:53:09.045212306+03:00" level=warning msg="error locating sandbox id b62e27c7429776ea180deca9dddd1dc2c290f94abdfe4bad5aeb6809dff86f38: sandbox b62e>
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-b62e27c74297.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-b9389b4117b0.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-c8422bd714fb.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-d4a56b2b2929.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-f0681c25cb75.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-baa7c72521ce.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-e71f6ed6df84.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-59ff2a93fb08.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-40dc507b5b92.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: run-docker-netns-98bf70948849.mount: Succeeded.
Nov 02 02:53:09 srv-be01 systemd[1]: var-lib-docker-overlay2-1a666f0289ced5be0a8d19422d644c58e1818776cc06dd549a7e2e67ccbde6a7-merged.mount: Succeeded.
Вывод tail /var/log/messages
Nov 2 08:21:43 srv-be01 kernel: br-9253fa3d84e5: port 5(vethbcd55b1) entered blocking state
Nov 2 08:21:43 srv-be01 kernel: br-9253fa3d84e5: port 5(vethbcd55b1) entered disabled state
Nov 2 08:21:43 srv-be01 kernel: device vethbcd55b1 entered promiscuous mode
Nov 2 08:21:43 srv-be01 kernel: IPv6: ADDRCONF(NETDEV_UP): vethbcd55b1: link is not ready
Nov 2 08:21:43 srv-be01 kernel: br-9253fa3d84e5: port 5(vethbcd55b1) entered blocking state
Nov 2 08:21:43 srv-be01 kernel: br-9253fa3d84e5: port 5(vethbcd55b1) entered forwarding state
Nov 2 08:21:43 srv-be01 NetworkManager[1140]: <info> [1730524903.7661] manager: (veth5c6b008): new Veth device (/org/freedesktop/NetworkManager/Devices/4086444)
Nov 2 08:21:43 srv-be01 systemd-udevd[3362105]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 2 08:21:43 srv-be01 systemd-udevd[3362105]: Could not generate persistent MAC address for veth5c6b008: No such file or directory
Nov 2 08:21:43 srv-be01 NetworkManager[1140]: <info> [1730524903.7673] manager: (vethbcd55b1): new Veth device (/org/freedesktop/NetworkManager/Devices/4086445)
Подскажите пожалуйста куда копать?