设备环境:威联通 TS-464c(N5095/8G/M2 x2/ HDD x4),内网环境,fnos 1.1.20
BUG现象:我使用docker compose编排容器,一共6个compose,10个容器,全部都用了unless-stopped的重启策略。发现自前几个版本以来(印象中1.1.11,或者更旧),每次重启之后,总会有一些docker的容器没有启动,显示“容器错误退出(255)”。docker界面中查看容器日志没有发现有异常报错。
出现频率:每次必然出现,当具体哪些容器出错每次不一定相同。
联系方式:群645-玖伍贰柒
日志文件:我找到docker 中可疑的日志:
journalctl -u docker
Feb 13 23:47:01 fn464c systemd[1]: Starting docker.service - trim docker service...
Feb 13 23:47:01 fn464c wait_resolv_nameserver.sh[1585]: [INFO] Checking /etc/resolv.conf for nameserver entry...
Feb 13 23:47:01 fn464c wait_resolv_nameserver.sh[1585]: [OK] Found nameserver in /etc/resolv.conf
Feb 13 23:47:01 fn464c systemd[1]: Started docker.service - trim docker service.
Feb 13 23:47:02 fn464c dockerd[1743]: time="2026-02-13T23:47:02.806224538+08:00" level=info msg="Starting up"
Feb 13 23:47:02 fn464c dockerd[1743]: time="2026-02-13T23:47:02.817989826+08:00" level=info msg="OTEL tracing is not configured, using no-op tracer provider"
Feb 13 23:47:02 fn464c dockerd[1743]: time="2026-02-13T23:47:02.818931486+08:00" level=info msg="CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory" dir=/etc/cdi
Feb 13 23:47:02 fn464c dockerd[1743]: time="2026-02-13T23:47:02.818974221+08:00" level=info msg="CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory" dir=/var/run/cdi
Feb 13 23:47:03 fn464c dockerd[1743]: time="2026-02-13T23:47:03.038265876+08:00" level=info msg="Creating a containerd client" address=/run/containerd/containerd.sock timeout=1m0s
Feb 13 23:47:03 fn464c dockerd[1743]: time="2026-02-13T23:47:03.336660043+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Feb 13 23:47:03 fn464c dockerd[1743]: time="2026-02-13T23:47:03.421941940+08:00" level=info msg="Loading containers: start."
Feb 13 23:47:04 fn464c dockerd[1743]: time="2026-02-13T23:47:04.937963127+08:00" level=info msg="Removing stale sandbox" cid=fd10a5919d72 isRestore=false sid=2fd11cfac589
Feb 13 23:47:04 fn464c dockerd[1743]: time="2026-02-13T23:47:04.959914908+08:00" level=warning msg="Failed deleting service host entries to the running container: open : no such file or directory"
Feb 13 23:47:04 fn464c dockerd[1743]: time="2026-02-13T23:47:04.959993294+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint f322f5b9e085e3819da160b3a2b37bff860031adfccbe6d6e41b9aa97adfcc76 afd3a2d94d39f8b30ae572c8c81a20421a87f5fafb25f84a2c88dfc5fb3ffdb2], retrying...."
Feb 13 23:47:04 fn464c dockerd[1743]: time="2026-02-13T23:47:04.995743444+08:00" level=info msg="Removing stale sandbox" cid=aded9dde53c7 isRestore=false sid=686bf210b6ac
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.009943770+08:00" level=warning msg="Failed deleting service host entries to the running container: open : no such file or directory"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.010256317+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint f322f5b9e085e3819da160b3a2b37bff860031adfccbe6d6e41b9aa97adfcc76 f7b8e198fb57d2a9999d1712f186052ac13295bc981b58c58c3426eeab318049], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.076068161+08:00" level=info msg="Removing stale sandbox" cid=c772cf53b576 isRestore=false sid=8273e0310270
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.079973838+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint eb40921fbdb721ccb86da003ae39a2d757adcc62253d83b8bce5d6dd60be1e0b 3b0d33cea5ccf0017a4e2c48908b40cb6e9eef62f9cb297655df969926160e21], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.090869722+08:00" level=info msg="Removing stale sandbox" cid=e215e4cb4136 isRestore=false sid=85df263fc561
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.176015612+08:00" level=warning msg="Failed deleting service host entries to the running container: open : no such file or directory"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.176096414+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint f322f5b9e085e3819da160b3a2b37bff860031adfccbe6d6e41b9aa97adfcc76 444ecbae5a1f7cbdde98d6700fb02aa48d863d28a1c58c2b46e1ae5b365e474b], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.217063289+08:00" level=info msg="Removing stale sandbox" cid=b6f137a98775 isRestore=false sid=87d1d1414b59
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.294009379+08:00" level=warning msg="Failed deleting service host entries to the running container: open : no such file or directory"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.294091699+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint b54f2904b36bbfebafb5cead82c78fcf00cd58db47e5f388253fb126a5b39181 b2614864ecfed5c3ae46afcba8aeb3468869246dcf71eeeb7eed70811904a22f], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.348193200+08:00" level=info msg="Removing stale sandbox" cid=13683e7f0a30 isRestore=false sid=73fcd4e7906d
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.352047269+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint eb40921fbdb721ccb86da003ae39a2d757adcc62253d83b8bce5d6dd60be1e0b 77d1c58a9d1f1afb35c6498b692c6c6823627f75c1044b885b59237850ea1b53], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.361281969+08:00" level=info msg="Removing stale sandbox" cid=3fb18cc8ebad isRestore=false sid=9be661838c30
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.366086716+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint eb40921fbdb721ccb86da003ae39a2d757adcc62253d83b8bce5d6dd60be1e0b ad423ba8e597c1b6d91d8aee3a95dedc556c684dfc2b44cd2e013272e88dd705], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.375341682+08:00" level=info msg="Removing stale sandbox" cid=43c46233ff98 isRestore=false sid=a5c72d00aac0
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.444035165+08:00" level=warning msg="Failed deleting service host entries to the running container: open : no such file or directory"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.444118816+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 667933d30766dd7042fe51944c329c1c2c404a768f57f0abaa5e304a15bc7438 146fc2463a117b7ec1a314b94df4b5f9483a878ea0cd28a7a057b955be7a97d0], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.485181195+08:00" level=info msg="Removing stale sandbox" cid=52c9134326da isRestore=false sid=b0037ecebf84
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.598056340+08:00" level=warning msg="Failed deleting service host entries to the running container: open : no such file or directory"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.598135554+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 9c1ea2ccc845aa0a47f211f2bfb5a8ed90978ae7a182e07841c92e7675a36e70 80db06c0d7778ee39261f7fc2582e9da1784885437d589699a5806ed34bd8ec3], retrying...."
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916509703+08:00" level=warning msg="error locating sandbox id 73fcd4e7906dbacc1f75be61e5d4d9ea73ec299498a1343277dbf12dd170df51: sandbox 73fcd4e7906dbacc1f75be61e5d4d9ea73ec299498a1343277dbf12dd170df51 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916564556+08:00" level=warning msg="error locating sandbox id 686bf210b6ac5baa8b799484840a2a78a409013a90efe8553271f348db185ca5: sandbox 686bf210b6ac5baa8b799484840a2a78a409013a90efe8553271f348db185ca5 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916581765+08:00" level=warning msg="error locating sandbox id a5c72d00aac05baf459ce4ac66fa45c3bd4d053919ec33120a534f060333f165: sandbox a5c72d00aac05baf459ce4ac66fa45c3bd4d053919ec33120a534f060333f165 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916598673+08:00" level=warning msg="error locating sandbox id 85df263fc56126879d49282cbc1f147ad5d4a20c5520aa233f867258dc86c1ad: sandbox 85df263fc56126879d49282cbc1f147ad5d4a20c5520aa233f867258dc86c1ad not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916616772+08:00" level=warning msg="error locating sandbox id 2fd11cfac5892e97a37ff7f14514888b17ce979991e24a420c65497fc7a13a77: sandbox 2fd11cfac5892e97a37ff7f14514888b17ce979991e24a420c65497fc7a13a77 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916632287+08:00" level=warning msg="error locating sandbox id 8273e031027082c56b8cd9a9c4aa2c359cb6aa4d7f6d66f86aa117a24fb9a587: sandbox 8273e031027082c56b8cd9a9c4aa2c359cb6aa4d7f6d66f86aa117a24fb9a587 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916648519+08:00" level=warning msg="error locating sandbox id 9be661838c309399ddccdbea6a78234bc9bcc3e6c19570f8ac458ec59f2b36f7: sandbox 9be661838c309399ddccdbea6a78234bc9bcc3e6c19570f8ac458ec59f2b36f7 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916666668+08:00" level=warning msg="error locating sandbox id 87d1d1414b598f541544e8e297d596b955449ddaf69e4c9519a65e535cb9e487: sandbox 87d1d1414b598f541544e8e297d596b955449ddaf69e4c9519a65e535cb9e487 not found"
Feb 13 23:47:05 fn464c dockerd[1743]: time="2026-02-13T23:47:05.916682740+08:00" level=warning msg="error locating sandbox id b0037ecebf8462473a43721c730c68a9bcbb37dae1c8c0349305809920ecce54: sandbox b0037ecebf8462473a43721c730c68a9bcbb37dae1c8c0349305809920ecce54 not found"
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.184986405+08:00" level=info msg="Loading containers: done."
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.241965972+08:00" level=info msg="Docker daemon" commit=89c5e8f containerd-snapshotter=false storage-driver=overlay2 version=28.5.2
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.242084417+08:00" level=info msg="Initializing buildkit"
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.319764608+08:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:192.168.2.11:42844" dns-server="udp:192.168.1.1:53" error="dns: overflowing header size" question=";postgres.HOST.\tIN\t A"
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.726146540+08:00" level=info msg="Completed buildkit initialization"
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.741772455+08:00" level=info msg="Daemon has completed initialization"
Feb 13 23:47:06 fn464c dockerd[1743]: time="2026-02-13T23:47:06.742121243+08:00" level=info msg="API listen on /var/run/docker.sock"
Feb 13 23:49:23 fn464c dockerd[1743]: time="2026-02-13T23:49:23.091280302+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 13 23:51:38 fn464c dockerd[1743]: time="2026-02-13T23:51:38.259932321+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 13 23:53:53 fn464c dockerd[1743]: time="2026-02-13T23:53:53.429678567+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 13 23:56:08 fn464c dockerd[1743]: time="2026-02-13T23:56:08.595016688+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 13 23:58:23 fn464c dockerd[1743]: time="2026-02-13T23:58:23.763922374+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 14 00:00:38 fn464c dockerd[1743]: time="2026-02-14T00:00:38.931209282+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 14 00:02:54 fn464c dockerd[1743]: time="2026-02-14T00:02:54.099763456+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 14 00:02:54 fn464c dockerd[1743]: time="2026-02-14T00:02:54.300404230+08:00" level=error msg="collecting stats for container /sub2api: NotFound: container \"e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093\" in namespace \"moby\": not found"
Feb 14 00:05:09 fn464c dockerd[1743]: time="2026-02-14T00:05:09.267095364+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 14 00:07:24 fn464c dockerd[1743]: time="2026-02-14T00:07:24.435551922+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 14 00:09:39 fn464c dockerd[1743]: time="2026-02-14T00:09:39.603268507+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 14 00:11:54 fn464c dockerd[1743]: time="2026-02-14T00:11:54.771093596+08:00" level=info msg="ignoring event" container=e215e4cb4136361e135b7db63a28d1266cdd238d1fdf9026613499159b070093 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
对比了docker ps -a的输出,没有成功启动的container id,和日志中“Removing stale sandbox”一行里面的cid能对得上。