Plex: Guide to Moving Transcoding to RAM


Recommended Posts

Hey everyone, just thought I'd put this up here after reading a syslog by another forum member and realizing a repeating pattern I've seen here where folks decide to let Plex create temporary files for transcoding on an array or cache device instead of in RAM.

 

Why should I move transcoding into RAM?  What do I gain?

In short, transcoding is both CPU and IO intensive.  Many write operations occur to the storage medium used for transcoding, and when using an SSD specifically, this can cause unnecessary wear and tear that would lead to SSD burnouts happening more quickly than is necessary.  By moving transcoding to RAM, you alleviate the burden from your non-volatile storage devices.  RAM isn't subject to "burn out" from usage like an SSD would be, and transcoding doesn't need nearly as much space in memory to perform as some would think.

 

How much RAM do I need for this?

A single stream of video content transcoded to 12mbps on my test system took up 430MB on the root ram filesystem.  The quality of the source content shouldn't matter, only the bitrate to which you are transcoding.  In addition, there are other settings you can tweak to transcoding that would impact this number including how many second of transcoding should occur in advance of being played.  Bottom line:  If you have 4GB or less of total RAM on your system, you may have to tweak settings based on how many different streams you intend on transcoding simultaneously.  If you have 8GB or more, you are probably in the safe zone, but obviously the more RAM you use in general, the less space will be available for transcoding.

 

How do I do this

There are two tweaks to be made in order to move your transcoding into RAM.  One is to the Docker Container you are running and the other is a setting from within the Plex web client itself.

 

Step 1:  Changing your Plex Container Properties

From within the webGui, click on "Docker" and click on the name of the PlexMediaServer container.  From here, add a new volume mapping:

 

/transcode to /tmp

 

Click "Apply" and the container will be started with the new mapping.

 

Step 2:  Changing the Plex Media Server to use the new transcode directory

Connect to the Plex web interface from a browser (e.g. http://tower:32400/web).  From there, click the wrench in the top right corner of the interface to get to settings.  Now click the "Server" tab at the top of this page.  On the left, you should see a setting called "Transcoder."  Clicking on that and then clicking the "Show Advanced" button will reveal the magical setting that let's you redirect the transcoding directory.  Type "/transcode" in there and click apply and you're all set.  You can tweak some of the other settings if desired to see if that improves your media streaming experience.

 

Thanks for reading and enjoy!

  • Like 12
  • Thanks 2
  • Upvote 2
Link to comment

hmmm an interesting think piece. I can certainly see the benefits in some cases, but I would disagree that SSD wear is of any real concern. By your own stats you're only looking at writing 430mb to the SSD. I don't know how long that video is, so I started from your 12mbps value (a reasonable, but even high one) and turned that into 129,600 MB/day or 127GB/day if you were to run a single stream for 24 hours non-stop.

 

Per this article http://techreport.com/review/27062/the-ssd-endurance-experiment-only-two-remain-after-1-5pb it is reasonable to expect ~ 1PB of writes before seeing the start of errors (some drives a bit less, some drives MUCC more). But based on 1PB = 104,8576GB that means we can push a 127GB stream, non-stop, for 8256 days, or 22.6 years.

 

I mean sure, there are other write operations going on our SSD's but I think it is fair to say even if we doubled, or tripled our daily SSD write throughput we're still safe from hitting the endurance limits.

 

My real concern with transcoding to RAM is that pushing three or four simultaneous transcoded streams might stretch the memory limits of some people, though it is unlikely for most.

 

I will add that even in the Plex forums there are discussions about dealing with /tmp (iirc?) writing to ram instead of disc and being a problem. This is just the first link I found in a google search of "plex /tmp ram" https://forums.plex.tv/index.php/topic/119669-issues-with-tmp-on-ram/

Link to comment

hmmm an interesting think piece. I can certainly see the benefits in some cases, but I would disagree that SSD wear is of any real concern. By your own stats you're only looking at writing 430mb to the SSD. I don't know how long that video is, so I started from your 12mbps value (a reasonable, but even high one) and turned that into 129,600 MB/day or 127GB/day if you were to run a single stream for 24 hours non-stop.

 

Per this article http://techreport.com/review/27062/the-ssd-endurance-experiment-only-two-remain-after-1-5pb it is reasonable to expect ~ 1PB of writes before seeing the start of errors (some drives a bit less, some drives MUCC more). But based on 1PB = 104,8576GB that means we can push a 127GB stream, non-stop, for 8256 days, or 22.6 years.

 

I mean sure, there are other write operations going on our SSD's but I think it is fair to say even if we doubled, or tripled our daily SSD write throughput we're still safe from hitting the endurance limits.

 

My real concern with transcoding to RAM is that pushing three or four simultaneous transcoded streams might stretch the memory limits of some people, though it is unlikely for most.

 

I will add that even in the Plex forums there are discussions about dealing with /tmp (iirc?) writing to ram instead of disc and being a problem. This is just the first link I found in a google search of "plex /tmp ram" https://forums.plex.tv/index.php/topic/119669-issues-with-tmp-on-ram/

 

Good write up.  I'll concede that this may or may not be a big deal, but no one can dispute the fact that cells on flash memory do suffer from endurance wear and will eventually burn out whereas RAM does not.

 

As far as people having issues, I haven't researched anything on Plex forums or otherwise regarding that.  All I can say is that I personally have yet to have an issue from doing it this way and I've been doing this pretty much since Docker was implemented in beta 6.  If someone has undesirable effects on their system, of course they should change it back.

 

I just wanted to put this out there because I do it this way, don't have issues, and can see the logic in doing this especially for those that don't have an SSD in their system.

 

Edit:

 

Regarding RAM consumption for multiple streams, I think you're just restating what I mentioned in the OP.  If you don't have enough RAM, don't do this, or try tweaking settings.  I don't know how many folks out there ever have 4 concurrent streams in their home.  Guess it depends on the size of your family.  For my household of 3, it's pretty much no more than 1-2 streams at a time.  I also have 16-32GB of RAM in every system I own, so I'm "RAM rich" and don't worry about the # of streams.

Link to comment

Oh, and you should probably read through that link you posted.  I don't think that was a statement against using /tmp in general, but just one guys issue.  Also, the recommendation in there about "making it as large as your largest piece of media" is just nonsense.  The transcoder doesn't need that much space as it's default setting is to only transcode 60 second of content at a time.

Link to comment

Oh, and you should probably read through that link you posted.  I don't think that was a statement against using /tmp in general, but just one guys issue.  Also, the recommendation in there about "making it as large as your largest piece of media" is just nonsense.  The transcoder doesn't need that much space as it's default setting is to only transcode 60 second of content at a time.

 

Yeah that was just one link from that search query. Others, like yours, had success though they generally admitted to not finding much benefit (usually related to stream initiation speed which was their goal).

 

Its not a horrible idea, and you are right flash does eventually wear out. I just don't know that it is enough to matter or to drive change absent other benefits. Change for change sake and all that [shrug].

 

For sure, I'm glad to see you posting the method, regardless of use case, because it means Limetech is "on it" with regards to plex :)

 

archdraft: if you look at the link / search term I posted you'll see that it does indeed apply as an alternative method for general linux install.  You'll also see discussions about using ramfs vs tmpfs (iirc) and the possibility of rollover into swap if you run out of ram etc etc. That means you need to have swap of course; I do not but I also am not planning to change transcoding to ram right now as can probably tell.

 

StevenD: can you characterize the memory usage with those three streams? Keep in mind you need to let them run to completion to get the full story. Plex does not discard .ts files until the stream is stopped. So usage will grow and grow until that point.

Link to comment

I suppose this still applies if I have Plex running on an Ubuntu VM?

Yes, but in that instance, you have a little more work to do.

 

option a:  use virtfs and mount /tmp to a path somewhere in the VM.  Then use that as your transcode directory following the guide I outline in step 2 in the original post.  Not sure how well this method would work exactly.

 

Option b: manually create a virtual disk under /tmp and mount that to the VM, then specify a path to that mount for transcoding.  Downside with this is you'd have to recreate the vdisk on every reboot (or script it in through the go file).

 

Option C:  get on the docker bangwagon and ditch that old VM to the curb!

Link to comment

I mean sure, there are other write operations going on our SSD's but I think it is fair to say even if we doubled, or tripled our daily SSD write throughput we're still safe from hitting the endurance limits.

 

Thank you. There's so much FUD about SSD wear that I see all sorts of goofy workarounds in lots of places. Your write up was quite succinct.

Link to comment

I have 8GB of RAM. I would never do more than 4 simultaneous streams. Am I safe enough to just map the transcode directory to RAM and get going? I had instances with very old unRAID where my server has crashed many times and this was attributed by some members to suspected unusually high RAM usage by Plex, hence the skepticism.

Link to comment

I have 8GB of RAM. I would never do more than 4 simultaneous streams. Am I safe enough to just map the transcode directory to RAM and get going? I had instances with very old unRAID where my server has crashed many times and this was attributed by some members to suspected unusually high RAM usage by Plex, hence the skepticism.

I would think so but what other apps are you running and are you running VMs at all?

Link to comment

I mean sure, there are other write operations going on our SSD's but I think it is fair to say even if we doubled, or tripled our daily SSD write throughput we're still safe from hitting the endurance limits.

 

Thank you. There's so much FUD about SSD wear that I see all sorts of goofy workarounds in lots of places. Your write up was quite succinct.

 

I think the biggest driver of that is confirmation bias.  People forgetting that stuff fails early. They hear about, or experience, an SSD failure and it validates their feelings that SSDs must be treated with kid gloves. But an HDD fails and they call it what it is; poor QC, poor shipping, dumb luck whatever. Sure there were some models known for having problems [cough] OCZ [cough] but even those were over blown anecdotes versus actual data. They were also usually failures due to poor firmware and not wear of the actual chips. In any case, it doesn't hurt to have options so I can't argue with that.

Link to comment

I mean sure, there are other write operations going on our SSD's but I think it is fair to say even if we doubled, or tripled our daily SSD write throughput we're still safe from hitting the endurance limits.

 

Thank you. There's so much FUD about SSD wear that I see all sorts of goofy workarounds in lots of places. Your write up was quite succinct.

 

I think the biggest driver of that is confirmation bias.  People forgetting that stuff fails early. They hear about, or experience, an SSD failure and it validates their feelings that SSDs must be treated with kid gloves. But an HDD fails and they call it what it is; poor QC, poor shipping, dumb luck whatever. Sure there were some models known for having problems [cough] OCZ [cough] but even those were over blown anecdotes versus actual data. They were also usually failures due to poor firmware and not wear of the actual chips. In any case, it doesn't hurt to have options so I can't argue with that.

If you don't have an SSD or want to limit IO contention to the device used for transcoding, this tweak could help.

 

Even on an SSD, if you have a bunch of VMs or containers all running on the same device and if they are doing lots of IO work, this could impact performance in those instances. While you'd have to be doing some pretty intense stuff, it is possible.

 

And at the end of the day, flash based memory will always eventually suffer burnouts. Over years and years of use, between both Plex transcoding and front ending writes to the array, this could add up.  So am I trying to say that by moving transcoding to ram you "triple ssd life"?  No.  I'm simply stating that RAM does not ever burn out from writes...ever.  So if you are RAM rich, why not implement this tweak? I don't see anything to lose here unless you just don't have enough memory.

Link to comment

If you don't have an SSD or want to limit IO contention to the device used for transcoding, this tweak could help.

 

Even on an SSD, if you have a bunch of VMs or containers all running on the same device and if they are doing lots of IO work, this could impact performance in those instances. While you'd have to be doing some pretty intense stuff, it is possible.

 

Now that's a reason I can get behind. I personally don't have that concern, but I can surely see others might.

 

And at the end of the day, flash based memory will always eventually suffer burnouts. Over years and years of use, between both Plex transcoding and front ending writes to the array, this could add up.  So am I trying to say that by moving transcoding to ram you "triple ssd life"?  No.  I'm simply stating that RAM does not ever burn out from writes...ever.  So if you are RAM rich, why not implement this tweak? I don't see anything to lose here unless you just don't have enough memory.

 

Of course. As with all things it is a question of degrees and use case. I can see the benefits, I just don't know that I'd agree they are very big. The same can be said for the down side. Most people probably won't run into problems with enough ram / few enough streams. Of course "enough" ram is relative depending on how many of those other functions, which might have i/o contention, they are running and how ram intensive they are.

 

I'm really not trying to be completely contrary. I think it is great that you've presented the option and there are surely valid use cases. My nit pick is only that SSD endurance statistically isn't really one of them. Yes they wear out, but they do it at a rate so as to not matter when you are looking at even a high use case giving 20 years of endurance. Cut it in half, or even a quarter and you're still looking at a practical useful life beyond how long most of us will keep these drives before upgrading. 

 

Ok I don't want to beat a dead horse so I'll bow out until I have something else useful to add to the conversation.  Again, thank for the attention to Plex, no matter what I DO appreciate that fact.

Link to comment

Triple SSD life!!! That's the best news ever! Kidding everyone - don't hurt me :)

 

And now for the real reason I'm posting. If I test option A above for my VM, then I am using the ram left for unRAID correct? And not the ram allocated to the ubuntu VM?

 

P.S. Option C is no fun so I think I'll just continue to use my VM's and stick my head in the sand... Thank you very much!

Link to comment

I love Plex. It was the first app I had Eric dockerize when we were testing before beta 6. It is the most simple and intuitive to use media player solution on the marker IMHO. I love XBMC and have heard great things about some of the others, but I just can't get over the completeness of vision that Plex has consistently shown me since my time using it.

  • Like 1
Link to comment

This is a great guide Jonp. I already had the /tmp path within my Plex docker but I wasn't aware of the tweak on the plex transcode page. I've set this now but this does this mean my Plex was never using RAM for transcoding?

 

It may be placebo but I remember when I changed the transcode path to /tmp on my Plex docker that my server fans seemed to be a lot quieter vs using my SSD cache drive.

Link to comment

I get to the point of setting the transcode path and set it to /transcode with volume mapped as /transcode mapped to /tmp and when I go to save the settings I get "Your changes could not be saved." This is on needo version 0.9.11.7

 

could this be due to me not having a plus or pro license preventing me from mapping a volume to /tmp?

Link to comment

I get to the point of setting the transcode path and set it to /transcode with volume mapped as /transcode mapped to /tmp and when I go to save the settings I get "Your changes could not be saved." This is on needo version 0.9.11.7

 

could this be due to me not having a plus or pro license preventing me from mapping a volume to /tmp?

No license involved because /tmp is in RAM. Everyone has /tmp regardless of license.

 

Are you sure you did the mapping correctly? /tmp should be on the host and /transcode should be on the container.

Link to comment

I get to the point of setting the transcode path and set it to /transcode with volume mapped as /transcode mapped to /tmp and when I go to save the settings I get "Your changes could not be saved." This is on needo version 0.9.11.7

 

could this be due to me not having a plus or pro license preventing me from mapping a volume to /tmp?

 

Did you "Edit" your Plex docker to add a mapping :  /transcode to /tmp ?

Link to comment

 

Triple SSD life!!! That's the best news ever! Kidding everyone - don't hurt me :)

 

And now for the real reason I'm posting. If I test option A above for my VM, then I am using the ram left for unRAID correct? And not the ram allocated to the ubuntu VM?

 

P.S. Option C is no fun so I think I'll just continue to use my VM's and stick my head in the sand... Thank you very much!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.