bnevets27 Posted June 12, 2017 Share Posted June 12, 2017 Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:38:32 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:42:55 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:43:16 Excelsior shfs/user: err: shfs_create: open: /mnt/cache/system/docker/appdata/plexpy/plexpy.db-shm (28) No space left on device Jun 12 15:43:16 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:43:16 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:43:45 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:43:45 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:43:45 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Jun 12 15:43:45 Excelsior shfs/user: err: shfs_write: write: (28) No space left on device Filesystem 1K-blocks Used Available Use% Mounted on rootfs 20562224 817328 19744896 4% / tmpfs 20637576 588 20636988 1% /run devtmpfs 20562240 0 20562240 0% /dev cgroup_root 20637576 0 20637576 0% /sys/fs/cgroup tmpfs 131072 24904 106168 20% /var/log /dev/sdb1 4013568 620864 3392704 16% /boot /dev/md3 3905078064 3280289896 624788168 85% /mnt/disk3 /dev/md4 3905078064 3606664900 298413164 93% /mnt/disk4 /dev/md9 3905078064 3673819200 231258864 95% /mnt/disk9 /dev/md10 3905110812 2882537336 1022573476 74% /mnt/disk10 /dev/md11 3905110812 1981380952 1923729860 51% /mnt/disk11 /dev/md12 3905110812 3108918476 796192336 80% /mnt/disk12 /dev/md13 3905110812 137634948 3767475864 4% /mnt/disk13 /dev/md14 3905110812 1803370240 2101740572 47% /mnt/disk14 /dev/md15 3905078064 3924448 3901153616 1% /mnt/disk15 /dev/md16 3905078064 3471582932 433495132 89% /mnt/disk16 /dev/md22 2928835740 2849459684 79376056 98% /mnt/disk22 /dev/md23 2928835740 2460082128 468753612 84% /mnt/disk23 /dev/md24 2928835740 259663840 2669171900 9% /mnt/disk24 /dev/sdk1 488386552 281144908 207175988 58% /mnt/cache shfs 47837451600 29519328980 18318122620 62% /mnt/user0 shfs 48325838152 29800473888 18525298608 62% /mnt/user /dev/loop0 41943040 17939664 22928304 44% /var/lib/docker /dev/loop1 1048576 17296 925776 2% /etc/libvirt Complaining about no space left on cache but its clearly only 58% used. Whats going on? Quote Link to comment
bnevets27 Posted June 12, 2017 Author Share Posted June 12, 2017 (edited) can I obtain it via cmd line? nvm found out how Edited June 12, 2017 by bnevets27 Quote Link to comment
bnevets27 Posted June 12, 2017 Author Share Posted June 12, 2017 (edited) attached Edited June 15, 2017 by bnevets27 Quote Link to comment
Squid Posted June 13, 2017 Share Posted June 13, 2017 Its probably this corruption on your cache Jun 12 00:57:25 Excelsior emhttp: shcmd (210): mkdir -p /mnt/cache Jun 12 00:57:25 Excelsior emhttp: shcmd (211): set -o pipefail ; mount -t btrfs -o noatime,nodiratime -U 86791a7c-d6e5-4a33-a9c8-e3669a1c89d1 /mnt/cache |& logger Jun 12 00:57:25 Excelsior kernel: BTRFS info (device sdl1): disk space caching is enabled Jun 12 00:57:25 Excelsior kernel: BTRFS info (device sdl1): has skinny extents Jun 12 00:57:25 Excelsior kernel: BTRFS info (device sdl1): bdev /dev/sdk1 errs: wr 29136313, rd 29882604, flush 1579276, corrupt 1616562, gen 24415 Jun 12 00:57:25 Excelsior kernel: BTRFS info (device sdl1): detected SSD devices, enabling SSD mode Jun 12 00:57:25 Excelsior kernel: BTRFS info (device sdl1): checking UUID tree that's causing this problem (and as a result, the docker.img file is also complaining) Jun 12 00:57:36 Excelsior kernel: BTRFS: device fsid 6b55a11a-d534-4432-aeb9-5589cd54b47a devid 1 transid 1046013 /dev/loop0 Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): disk space caching is enabled Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): has skinny extents Jun 12 00:57:36 Excelsior kernel: BTRFS warning (device loop0): loop0 checksum verify failed on 945078272 wanted AFFEE1BA found BAB79187 level 0 Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): read error corrected: ino 1 off 945078272 (dev /dev/loop0 sector 1862240) Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): read error corrected: ino 1 off 945082368 (dev /dev/loop0 sector 1862248) Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): read error corrected: ino 1 off 945086464 (dev /dev/loop0 sector 1862256) Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): read error corrected: ino 1 off 945090560 (dev /dev/loop0 sector 1862264) Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): bdev /dev/loop0 errs: wr 182, rd 0, flush 0, corrupt 0, gen 0 Jun 12 00:57:36 Excelsior kernel: BTRFS info (device loop0): The free space cache file (29360128) is invalid. skip it @johnnie.black however is the resident expert on btrfs and its recovery from an issue such as this. Quote Link to comment
bnevets27 Posted June 13, 2017 Author Share Posted June 13, 2017 I thought the cache drive getting full the other day (is what I guess happened) is what caused the corruption. Though I have no idea how it got full but it is possible. I did run a scrub after the initial hard shutdown/"full cache" Isn't the log complaining that my appdata folder is full? I didn't see the log complaining about the docker.img itself. If this is just a straight BTRFS issue then this will be the final straw to get rid of it. I thought a raid 1 cache pool would protect me from headache not cause them. Of the 4 machines I've had with various people all BTRFS formated cache drives have had multiple issues. All on single drives so converting to xfs is a easy solution for them. Sent from my SM-N900W8 using Tapatalk Quote Link to comment
itimpi Posted June 13, 2017 Share Posted June 13, 2017 2 hours ago, bnevets27 said: Isn't the log complaining that my appdata folder is full? I didn't see the log complaining about the docker.img itself. The loop0 device that the log mentions is where the docker image is mounted. Quote Link to comment
JorgeB Posted June 13, 2017 Share Posted June 13, 2017 3 hours ago, bnevets27 said: If this is just a straight BTRFS issue then this will be the final straw to get rid of it. I thought a raid 1 cache pool would protect me from headache not cause them. No, all these errors read/write are caused by a hardware issue, check cables on this SSD: Jun 12 00:57:25 Excelsior kernel: BTRFS info (device sdl1): bdev /dev/sdk1 errs: wr 29136313, rd 29882604, flush 1579276, corrupt 1616562, gen 24415 As for the not enough space errors, see this thread, it's the same problem: 1 Quote Link to comment
bnevets27 Posted June 15, 2017 Author Share Posted June 15, 2017 Thanks jonnie.black. Not exactly sure all what happened, I don't think I had size = sized used but can't remember. Ran balance and scrub and everything looks to be normal. was remove so couldn't look at the cable but the log looks clean for now. Next issue is my docker.img went nuts recently. Made some adjustments to what I think was the cause. But whats going on with my docker.img? Is it at 90% usage or 23%? Label: none uuid: 6b55a11a-d534-4432-aeb9-5589cd54b47a Total devices 1 FS bytes used 9.78GiB devid 1 size 50.00GiB used 41.41GiB path /dev/loop0 Filesystem 1K-blocks Used Available Use% Mounted on /dev/loop0 52428800 11628636 39225460 23% /var/lib/docker Quote Link to comment
JorgeB Posted June 15, 2017 Share Posted June 15, 2017 41.41GiB is the allocated space, 9.78GiB is the actually used space at the moment, but at some point in the past it was about 40GiB used and that's a lot for the docker image, something is not correctly mapped, see the docker FAQ. Quote Link to comment
bnevets27 Posted June 15, 2017 Author Share Posted June 15, 2017 Yeah I changed some paths and likely messed one up the other day. Yesterday I went from 30% usage to 99% in about 2hrs. As far as I know I corrected that and did a bit of cleaning which I guess is how I got it back to 23%.Without nuking the docker.img image I assume the allocated space won't ever come down. Having the allocated space 90% of the amount of space assigned to docker I assume isn't a problem? Just for further understanding. I assume this is why you can't shrink the docker.img. So docker will started to allocate more space to itself when it needs it. Kind of like expanding a partition? Later if data is removed then the allocated space never comes back down but you do end up freeing up space from the allocated space (partition). The size that's set in the docker settings is just a limit to how much docker can expand its allocated space?Sent from my SM-N900W8 using Tapatalk Quote Link to comment
JorgeB Posted June 15, 2017 Share Posted June 15, 2017 Allocated space is usually not a problem, it can be if it's very close to the max space, but running a balance will reclaim most of it. Quote Link to comment
bnevets27 Posted June 15, 2017 Author Share Posted June 15, 2017 Interesting. Since there is no gui button in docker for that I assume it has to be run on cmd line. I wonder why there isn't a gui button for balance in the docker config but there is for the cache drive. Also would it make any sense to run balance on a cron job? In the case of docker.img it shouldn't really be changing much so I guess it can be run manually if the user has cleaned up the img. In the case of the cache drive, with constant adding and removing of files off the cache drive it seems to make sense to run a balance on it periodically, possibly a scrub too? Sent from my SM-N900W8 using Tapatalk Quote Link to comment
JorgeB Posted June 15, 2017 Share Posted June 15, 2017 2 hours ago, bnevets27 said: I wonder why there isn't a gui button for balance in the docker config but there is for the cache drive. Because it's not really needed. 2 hours ago, bnevets27 said: Also would it make any sense to run balance on a cron job? If the cache gets completed filled and emptied on a regular base it may make sense to avoid your initial problem, although later kernels are less prone to this. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.