Cleaning out Docker image

[Popple2000]

Will do as soon as I get a chance, thanks !

[kamhighway]

@TexasDave,  I never get the message that disk utilization has returned to normal. 

[johnsanc]

Just adding that you all are not alone. I also have a continuously growing docker.img file and cannot seem to figure out the cause. I can reboot to get it to return to normal, but it quickly grows to capacity in a matter of days.

[kamhighway]

Update:  It's been about 4 days now since I changed my Plex settings to perform transcoding in RAM and the amount of space utilized in my docker.img has changed only slightly.  This result is consistent with my theory that allowing Plex to transcode inside its container was the problem.

 

I don't think the problem is limited to Plex.  Any container that runs an app that uses lots of tmp space is likely to exhibit the same problem. Maybe it should be standard practice to map every docker's /tmp directory to a share on the cache drive or to ram.

 

Now that I'm fairly sure the Plex docker was my problem, here's how we could collaborate to be sure.

 

If you are using a Plex docker, please post your answers to the following questions:

 

1)  Which Plex Docker?  Needo, Linux.io, BinHex, etc.

2)  Did you follow JonP's instructions to set Plex to transcode either in RAM, Cache, or a Share?

3)  Do you know for sure that someone watches content that Plex has to transcode?

4)  Does your utilization of docker.img increase in big jumps (hundreds of MB/day)?

 

 

 

[PhAzE]

Im getting this same problem but im not using plex as a docker. I haven't had time to inspect yet but i did notice sonarr had a queue of shows to process but couldn't until the docker was restarted, that or sab. Ill verify but i also suspect sab may be unpacking rar files in the tmp folder during the unrar process and crashing if the docker image fills.

[kamhighway]

Im getting this same problem but im not using plex as a docker. I haven't had time to inspect yet but i did notice sonarr had a queue of shows to process but couldn't until the docker was restarted, that or sab. Ill verify but i also suspect sab may be unpacking rar files in the tmp folder during the unrar process and crashing if the docker image fills.

 

The problem is most evident when a dockerized app uses /tmp for big files such as transcoding in Plex. Unraring in sab may be another good example. In the sab docker, try mapping /tmp to /mnt/cache/appdata/sabnzbd/tmp. I am considering doing this for all my dockers so they can all use /tmp without utilizing docker.img.

 

I'd be curious to know if this works for you as I am not 100% sure I've diagnosed the primary issue correctly.

[PhAzE]

I had a look ag the tmp folder for sonarr sab and ubooquity and they are all pretty clean. Nothing stands out as bloat yet my docker is filling up still. Needs more investigating.

[kamhighway]

I had a look ag the tmp folder for sonarr sab and ubooquity and they are all pretty clean. Nothing stands out as bloat yet my docker is filling up still. Needs more investigating.

 

The /tmp folder in the dockers may be empty now since the apps do a pretty good job of cleaning up after themselves. I suspect that what is happening is that utilization of docker.img goes up when a dockerized app is using /tmp. However, after the app cleans up /tmp, utilization of docker.img does not go down.

 

This would explain why the size of the containers does not change eventhough docker.img utilization increases. If I am right, then mapping every docker's /tmp to the cache drive would be an easy solution.

[PhAzE]

Ok like the container size grows but doesn't shrink once expanded.

 

Sounds like we need a way to shrink containers via cron or something.

[kamhighway]

Ok like the container size grows but doesn't shrink once expanded.

 

Sounds like we need a way to shrink containers via cron or something.

 

The container does shrink, but it does not seem to free up the space in docker.img. We either need to figure out how to shrink docker.img when a docker is done using space in /tmp. Or, we need to move /tmp outside of docker.img by mapping it to the cache drive.

[BRiT]

Or there needs to be an option to not use docker.img but have it stored in an actual subdirectory in the cache drive.

[ikosa]

If anybody is interested I found some tools to monitor and track docker containers.

 

Prometheus, container_exporter:

http://rancher.com/docker-monitoring-continued-prometheus-and-sysdig/

 

And Prometheus can also get metrics from cadvisor.

https://labs.ctl.io/monitoring-docker-services-with-prometheus/

 

I can't play enough to achieve my goal. Interestingly can't find a metric about container sizes...

 

[shooga]

I have this issue too and have all of my temp folders for sabnzbd and plex pointing outside docker to my cache. I've already expanded by docker image once and would like to avoid doing that repeatedly.

 

Need to find a solution here.

[PhAzE]

So after updating all of my dockers to map /tmp out to /tmp (in ram) the docker is still growing only slightly slower now. I think that may have been only a partial solution. I can see some apps are indeed leaving files in /tmp (like Calibre, and Sonarr).  I believe some of these apps are also growing log files in /var/log in each container, but i'm not sure if it's enough to bloat 10GB of docker.img space.

[PhAzE]

I just wanted to point out, if you need to clear some space in the image and happen to have dangling images, it's best to remove them.  First, run:

 

docker images -f "dangling=true"

 

and if you see anything listed then you have unused images that can be cleared out with the following code:

 

docker rmi $(sudo docker images -f "dangling=true" -q)

 

For me, that got my image back into acceptable utilization levels. Enough to clear the unRaid alert for over utilization.  If docker updates are in fact leaving images behind, then I suppose you could 'cron' this for a weekly run to clear out leftover images. That is assuming that your containers themselves are not growing. In my case, I found that after mapping /tmp to /tmp for all the containers, the images still grew, but likely due to application auto updates which were in fact leaving empty image files after the updates. 

[kamhighway]

@PhAze:  Thank you for your post.  I can believe that there is more than one thing that can cause docker utilization to ramp up.  In my case, setting Plex to transcode from the cache drive seems to have solved most of the problem.  The img is still growing, albeit at a much slower rate.

 

I did not know of the possibility of a "dangling image" so I'll look for that the next time I see a big jump in utilization of docker.img. 

 

It seems to me that we are part of a very small group that is experiencing this problem.  Makes me wonder what I'm doing differently from most other users.

[BRiT]

You're likely using dockers that auto-update. Each time the main program updates in your container the image usage will grow.

 

I do not use auto-updating programs and my docker image usage has been constant since the initial setup.

[shooga]

I took a look and didn't have any dangling images, so that didn't help.

 

Is there a way to view the image sizes? cAdvisor seems to only report the size of the default image because if I add up the numbers that it shows the total is way short of what shows in my docker settings page.

[shooga]

An update for anyone monitoring this thread or searching in the future:

 

I used the command "docker exec -it <container name> bash" to drop into the container and then this, "find / -xdev -type f -size +100M" to find files over 100MB. This helped me find that I had a 1.34GB error log in three of my containers! (eroz's AirVideoServer, needo's PlexMediaServer, and needo's PlexWatch). Deleting that file is an easy temporary fix, but I still need to figure out a long term solution.

 

Any idea what would be causing this error:

 

Nov  1 08:10:18 [server name] sshd[435]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.

Nov  1 08:10:18 [server name] sshd[435]: fatal: Cannot bind any address.

 

I'm guessing it's because ssh is already open on the server. How could I fix this?

[PhAzE]

i didn't have any large logs in any of my dockers. It appears my utilization is still growing, even with TMP mapped to native TMP drive, and dangling images gone. Docker containers do not appear to be growing in size, so that leaves me to believe that it is an issue with something outside the containers. Perhaps docker has it's own log file outside the containers but still in the image, or maybe there are history entries growing the image usage. 

[hoosierdaddy]

I'm also getting this Docker Utilization alert.  Mine is actually going up to 98% sometimes before then dropping down to normal.  The only Docker I'm using is needo/Plex.  During this time, if I'm using PLEX, the movie will stop playing like the file has an error.  I can then normally restart it and it will play.  Frustrating.  I need to pay attending to exactly what files it is happening on.  I setup to have the transcoding done on my cache drive.  This seems to help but I still have the problem...  Not installing any other dockers until I get this figured out.

[Earache]

I'm also getting this and I have no idea how to fix it.  Is there a solution?

 

linuxserver/couchpotato    latest              29d0943b24c0        2 weeks ago         502 MB
linuxserver/quassel-core   latest              0a9f2f07a76f        3 weeks ago         292.6 MB
linuxserver/plexpy         latest              bbda2e6f2f71        3 weeks ago         314.8 MB
linuxserver/plex           latest              3cc54bb12b37        3 weeks ago         583.4 MB
google/cadvisor            latest              c0e56eeefd42        5 weeks ago         22.77 MB
nativedude/plex-requests   latest              94ea7dbfd58d        3 months ago        569.1 MB
<none>                     <none>              14a06126bb6c        3 months ago        282.9 MB
needo/nzbdrone             latest              132b9d473a16        3 months ago        440.4 MB
gfjardim/nzbget            latest              2accc56105d5        3 months ago        529.7 MB

 

I also don't know what <none> is. 

[PhAzE]

I'm also getting this and I have no idea how to fix it.  Is there a solution?

 

linuxserver/couchpotato    latest              29d0943b24c0        2 weeks ago         502 MB
linuxserver/quassel-core   latest              0a9f2f07a76f        3 weeks ago         292.6 MB
linuxserver/plexpy         latest              bbda2e6f2f71        3 weeks ago         314.8 MB
linuxserver/plex           latest              3cc54bb12b37        3 weeks ago         583.4 MB
google/cadvisor            latest              c0e56eeefd42        5 weeks ago         22.77 MB
nativedude/plex-requests   latest              94ea7dbfd58d        3 months ago        569.1 MB
<none>                     <none>              14a06126bb6c        3 months ago        282.9 MB
needo/nzbdrone             latest              132b9d473a16        3 months ago        440.4 MB
gfjardim/nzbget            latest              2accc56105d5        3 months ago        529.7 MB

 

I also don't know what <none> is. 

 

NONE is a dangling (left over but unused) container. Run this to remove it:

 

docker rmi $(sudo docker images -f "dangling=true" -q)

 

------------------------------------------------------------------------

 

My utilization i still going up, and when i compated my container sizes from 3 weeks ago to now, no change, but I've gone up 30% utilization (which is 3gb out of my 10gb.  No temp or log files remain in the containers, and I just can't figure out what is making it grow.

[ljm42]

I'm also getting this Docker Utilization alert.  Mine is actually going up to 98% sometimes before then dropping down to normal.  The only Docker I'm using is needo/Plex.  During this time, if I'm using PLEX, the movie will stop playing like the file has an error.  I can then normally restart it and it will play.  Frustrating.  I need to pay attending to exactly what files it is happening on.  I setup to have the transcoding done on my cache drive.  This seems to help but I still have the problem...  Not installing any other dockers until I get this figured out.

 

It really sounds like the transcoding is still being done in the docker image.  There are two steps to moving it outside of the docker image:

1) Add a volume mapping for /transcode to /tmp (or a location on your cache drive)

2) In Plex, go to Settings -> Server -> Transcoder.  Hit Show Advanced. In the "Transcoder temporary directory" put "/transcode"

That should solve it for you.

 

Edit - I expanded on this idea and added it to the Docker FAQ

 

If Plex is your only Docker, and transcoding really is happening outside of the docker image, and you are still getting to 98%, then your docker image is probably just too small.  Mine is 20gb.

[B_Sinn3d]

It just got worse...I previously set image from 20GB to 30GB, my image exploded last night and completely filled up 100%.  I restarted my docker and now I have 5 orphaned images...grrr.

 

Has LT recognized this as an issue yet?  I can't keep expanding my image as a workaround.