NAS

Docker container developer best practice guidelines for unRAID

160 posts in this topic

This is  a work in progress. It is currently unofficial. Nothing is set in stone. Nothing is a requirement, only a set of suggestions. Eventually this will be ported to the wiki; until then it will be heavily edited as we debate.

 

Some points may never reach a general consensus as they are personal preference. That’s fine don’t worry about it... it’s only a set of suggestions anyway. If this list proves impractical or a development hurdle it may be deprecated later.

 

I would really appreciate some help on this as it should make things easier for everyone.

0

Share this post


Link to post
Share on other sites

Guidelines

 

Please choose the following base OS and version " phusion/baseimage:0.9.11" unless you have a specific reason you cannot.

 

Releases with multiple versions (e.g. stable/testing/development) should be handled in a single dockerfile with the appropriate tags/scripts to allow the user to choose.

 

The use of "git clone master" and "OS:latest" type commands that result in different code as time passes should be avoided unless the implications are understood. Usually it is better to clone a specific git/svn build number or avoid git completely using tarball downloads which are typically also available.

 

Run your containers as the user "nobody" and group "users" and NOT root. Pay attention of UID and GID differences between OSs (uid=99(nobody) gid=100(users))

 

Decide and explicitly set within your scripts the UMASK you want to your applications with (typically umask 000)

 

Develop your Dockerfile on github and follow git commit best practices so that users can track you e.g.

 

commit only tested code

commit comments should be short and descriptive

add a .gitattributes file to the root of your project that contains these lines:

	# Auto detect text files and force unix-style line endings
	* text eol=lf

 

Include a license file at github so that users are free to consume your work and developers fork if needed.

 

Timezones can be handled with

-v /etc/localtime:/etc/localtime:ro

 

Dealing with locales (TBC) http://lime-technology.com/forum/index.php?topic=34168.msg317791#msg317791

 

Lightly comment your Dockerfile with relevant information.

 

Verified Dockerfiles only TBC

 

The dock should not contain any data or settings specific to the builder (i.e. quirks of their specific setup). All user specific data should be passed in from the command line.

 

Future updates of Dockerfiles that cause an upgrade of setting files and/or databases should be highlighted where possible (i.e. if an upgrade makes it impossible to roll back as non Docker protected config files have been upgraded)

 

apt-get update  autoremove autoclean recomendations

 

Where possible use the traditional port for a service within your container e.g. www port 80. Mapping to bypass port clashes is the job of the user not the container.

 

 

 

Your forum announcement for your Dockerfile should contain a maintained list in the first post:

 


  • A link to your docker repo
    The application version numbers you are providing
    The base OS if you have chosen to diverge from stock and why
    A sample start command
    Any other relevant information e.g. why you need to run as root etc.

 

 

Deprecating

==========

 

Where possible use the "--" version of all command switches within your Dockerfile as they are far easier for the less Linux fluent to review. e.g. --quiet rather than -q

*No one is doing this and its not valuable enough to waste time trying to enforce it.

 

A link to your github repo

*Link to docker repo is enough

 

0

Share this post


Link to post
Share on other sites

Guidelines

 

Please choose the following base OS "FROM phusion/baseimage:0.9.11" unless you have a specific reason you cannot (This is a big debate in itself and is NOT decided yet)

 

Consider using docker tags to allow both a stable and development build within a single dock.

 

The use of "git clone master" and "OS:latest" type command that result in different code as time passes should be avoided where possible. For example clone a specific point in time from git or avoid git completely using tarball downloads which are usually also available. This will allow users to mix and match Docks from different developers without surprise application version clashes.

 

Run your containers as the user "nobody" and group "users" and NOT root. Pay attention of UID and GID differences between OSs (uid=99(nobody) gid=100(users))

 

Develop your Dockerfile on github and follow git commit best practices so that users can track you e.g. commit only tested code, commit comments should be short and descriptive etc.

 

Include a license file at github so that users are free to consume your work and developers fork if needed (unless this is covered within the Docker repo by default?)

 

Dealing with timezones (TBC)

 

Dealing with locales (TBC)

 

Where possible use the "--" version of all command switches within your Dockerfile as they are far easier for the less Linux fluent to review. e.g. --quiet rather than -q

 

Lightly comment your Dockerfile with relevant information. (TBC comments vs. docker layer caching)

 

Future updates of Dockerfiles that cause an upgrade of setting files and/or databases should be highlighted where possible (i.e. if an upgrade makes it impossible to roll back as non Docker protected config files have been upgraded)

 

Your forum announcement for your Dockerfile should contain a maintained list in the first post:

 


  • A link to your github repo
    A link to your docker repo
    The application version numbers you are providing
    The base OS if you have chosen to diverge from stock and why
    A sample start command
    Any other relevant information e.g. why you need to run as root etc.

 

 

Reason for not choosing phusion/base, i don't trust/agree with alot of his choices. why the f.. do we need a base os anyway ? to save a few megs on our terrabytes of storage ? python runs way better on centos then on ubuntu/debian, while mysql runs best on debian. blah blah blah. also it requres none debian users to deal with debian.

 

 

Docker tags: Agree

 

Git clone and From OS:latest is not a bad thing it means that you can Easly refresh your suddenly out-of date code with 99% chanse of success, if it fails, just dont use the build.

 

User/group: Agree

 

Dockerfile on github: Agree (But i wanna sugest only useing containers developed with dockers autobuild (prev. "verified")

 

Licens: Agree

 

Timezone\Locales: Agreed

 

-- rather than -: Nope not gonna happen, sorry i write the code so its easy for me to read not the user

 

The Post List: github repo is not needed as its referenced in the docker repo, Application numbers where necessary (i.e. i still like git installs), The base os can be mentioned, but i don't care for a stock, a sample command is always great ;) May i suggest a list over required (usable params, e.g. ports mappings etc...)

 

0

Share this post


Link to post
Share on other sites

Reasonable enough.

 

I was considering that the git/latest thing be restricted to the experimental tag docker. My fundamental concern with docker and git is the python apps that everyone here seems to use. They invariably all have complex config files and sqlite database that are NOT backwards compatible. So the meta effect is some poor smho runs a one line docker command because they are a user with no intimate back end knowledge end up with no option to use git form then on because they have altered their non docker files. Also I think Docker misses the trick in that there is almost no way to know what version of an app you are being offered without trying it.

 

Happy to remove the git URL requirement if required. The reason i include this is IMHO the docker repo page sucks at presenting a user with meaningful change control messages and I want to actively encourage review by pushing this link to users.

 

"-- rather than -" I would still like to leave the recommendation in for those that dont care. If in time no one does it we can remove the recommendation.

 

"verified" - can you give me a one liner i can include here

 

As for phusion. My stance has been I dont think it is the right way to go either but for long term support reasons. Having just read this "http://blog.docker.com/2014/06/why-you-dont-need-to-run-sshd-in-docker/" I am more convinced than ever that its not as sexy a choice as the project summary page makes out.

 

Why do we need a base OS? Because every dev has to choose one and in many cases it makes zero difference which one they choose. So really the base OS recommendation is simply to limit differentiation for differentiation sake. If i had two docks to choose from and one used the OS i already had docks off i would choose that one. Its not a big deal but might as well make it if only to inspire ongoing conversations like this.

0

Share this post


Link to post
Share on other sites

As for phusion. My stance has been I dont think it is the right way to go either but for long term support reasons. Having just read this "http://blog.docker.com/2014/06/why-you-dont-need-to-run-sshd-in-docker/" I am more convinced than ever that its not as sexy a choice as the project summary page makes out.

 

[sigh] that link is painful to read ... I can see the next holy war coming.

0

Share this post


Link to post
Share on other sites

Reason for not choosing phusion/base, i don't trust/agree with alot of his choices. why the f.. do we need a base os anyway ?

 

Not to question your opinion, but for clarification only, why do you not trust/agree with his choices, specifically?  With me being pretty incompetent in all this still, his explanations sounded reasonable/solid to me.  What do you disagree with, and what would you recommend instead?

 

Just curious.

 

Thanks.

0

Share this post


Link to post
Share on other sites

I can't speak for Ionix (Lonix?  damn fonts), but if you read the link NAS posted you'll see what seem to be some reasons and methods to not do things the way phusion does [shrug].

0

Share this post


Link to post
Share on other sites

I can't speak for Ionix (Lonix?  damn fonts), but if you read the link NAS posted you'll see what seem to be some reasons and methods to not do things the way phusion does [shrug].

 

I didn't read it all, but it sure seems like a long article on why SSH isn't needed.  It doesn't sound like SSH is bad, nor does it give any other 'problems' with what Phusion is doing.  Even if SSH isn't needed, it doesn't seem to be 'wrong' to enable it, so I still don't see the 'problem' with using Phusion.

0

Share this post


Link to post
Share on other sites

I think their points apply more to the enterprise than the home user. I'm certainly not going to setup one container for an app itself, another to edit the config of that app, another to view the logs, and another to be able to restart the app. That seems like a lot of complexity for a home media server. For a situation with lots of developers, that approach makes much more sense.

 

I don't think there is one "best way" to use docker because the ways it can be used vary so much.

0

Share this post


Link to post
Share on other sites

Not to question your opinion, but for clarification only, why do you not trust/agree with his choices, specifically?  With me being pretty incompetent in all this still, his explanations sounded reasonable/solid to me.  What do you disagree with, and what would you recommend instead?

 

In English...

 

1. Debian and Debian based Distros like Ubuntu, Mint, etc. as they stand today and for the foreseeable future is a very poor choice for a Docker OS due to several technical reasons. The two big ones... You will have phantom services and eventually you will blow up your filesystem because when your app stops / crashes... There is no graceful shutdown. It's as if you yank the powercord from it. There are several more reasons why but those are the two big ones. Phusion took Ubuntu and then hacked / tweaked it to address many of those issues and to make it play better with Docker.

 

2. It's big. You can run a "Dockerized" Tiny Core Linux OS and go from 400+mb down to 30mb or so. If you install Docker in Windows or Mac, it uses Tiny Core Linux for docker2boot.

 

3. There are a great many people here who do not know Linux or how it works and believe their Apps (Plex, Couchpotato, Sickbeard, etc.) will not work in Tiny Core Linux. This is not true.

 

4. There are a great many people here who feel comfortable and have experience with Debian Based Distros like Ubuntu.

0

Share this post


Link to post
Share on other sites

Phusion took Ubuntu and then hacked / tweaked it to address many of those issues and to make it play better with Docker.

 

Which is why I was asking why Lonix didn't like it as a base.  It sounds like Phusion has 'resolved' most/all the problems with ubuntu as a base, other than being 'too big'.  Whether or not we can or need to shave a few hundred MB off the size of the base image can be debated, but for me, a few hundred MB on a 500GB drive isn't something I'm worried about.

 

His comments indicated that he didn't like things specific to what Phusion did, and I was just curious what specific things he disagreed with, or doesn't like about the Phusion image.

0

Share this post


Link to post
Share on other sites

@JustineChase: well it isn't just about SSH, and it provides details on alternatives, hence the length.

 

@jkm9000: yeah that did seem like a lot to me but I don't think it is as bad as they describe.  I think the key is to have as much of your stuff you need to mess with in the volume and then have a single docker capable of touching that volume.  But I admit to being way too under-skilled to fully appreciate the nuances of their advice.  But ya know, they are like ... the docker devs [shrug]  And it also gives their perspective on the question of how / why to deal with not having a service manager.  I'm not saying that is The Best Way, but it explains it.

 

And brings me back to something I said in another thread somewhere else: I have a feeling docker in unRAID will ultimately only satisfy the most basic of needs / users.  Like on a scale of 0 (grandma who thinks the computer has run out of internets) to 10 (a neck beard coding FORTRAN) anyone above about a 4 is probably going to want just run a virtual machine of their OS / Distro of choice.  I feel like folks like phusion are trying to get docker to do things it isn't meant to do, things done much easier via a full VM.

 

Or, they are taking the ball and running at the next level ... I don't know ... I'm just pontificating while I wait till the dust settles just a little more and I move off my beta5a ArchVM (or not).

0

Share this post


Link to post
Share on other sites

Which is why I was asking why Lonix didn't like it as a base.  It sounds like Phusion has 'resolved' most/all the problems with ubuntu as a base, other than being 'too big'.  Whether or not we can or need to shave a few hundred MB off the size of the base image can be debated, but for me, a few hundred MB on a 500GB drive isn't something I'm worried about.

 

His comments indicated that he didn't like things specific to what Phusion did, and I was just curious what specific things he disagreed with, or doesn't like about the Phusion image.

 

If you want my professional opinion based on what I have seen Enterprises doing with several 1,000+ Docker App Deployments...

 

Use a smaller Linux Distro that is designed for Docker and is a rolling release. You are running a fork of a fork (Debian, Ubuntu, Phusion) with version releases. All the issues with Debian based Distros aside... A new version of Ubuntu every 6 months is going to catch up with your apps / image eventually.

 

Having said all of that... People do not seem concerned nor do they want to use best of breed practices that I mentioned above and want to continue to use Ubuntu or a fork of it. That being the case, you have a solution to the problems that standard Ubuntu causes.

0

Share this post


Link to post
Share on other sites

Thanks again grumpy.  I've seen you mention a couple of smaller linux distros for use with docker, but I didn't notice that they were specifically designed for docker.  If you were going to create a docker for yourself, to run SABnzbd, what would you, personally, use as a base?

0

Share this post


Link to post
Share on other sites

Thanks again grumpy.  I've seen you mention a couple of smaller linux distros for use with docker, but I didn't notice that they were specifically designed for docker.  If you were going to create a docker for yourself, to run SABnzbd, what would you, personally, use as a base?

 

I'd look at something like Alpine Linux or Tiny Core (there are plenty of small linux distros to choose from). Both have package managers and neither run upstart (which is where most of the problems come from with debian based distros). In Phusion, they replace it. There are some other things you have to deal with like logs, kernel messages, ssh, etc. I would have to play with some of them to pick a favorite. From what I hearing from all the Linux Dorks in the various Communities... I wouldn't be surprised if you see a lot of the Linux Distros create / release "Dockerized" versions.

 

People do not have an issue with the size of the image... I get that. What they do not understand / realize the bigger your image is and the more things it contains (which 90% of it you will not use for your Docker App)... The more things that can go wrong when updates and new releases come out for the Distro.

0

Share this post


Link to post
Share on other sites

As i posted previously I like the elegance of TCL but I worry that the skills required them develop one out weight the benefits they bring to the unRAID community.

 

I think the proof of this is that so far no one has produced even one. Until we see at least a proof of concept I suspect it will always stay in the "wouldn't it be cool to do pile" and I dont think we should dwell on it.

0

Share this post


Link to post
Share on other sites

As i posted previously I like the elegance of TCL but I worry that the skills required them develop one out weight the benefits they bring to the unRAID community.

 

I think the proof of this is that so far no one has produced even one. Until we see at least a proof of concept I suspect it will always stay in the "wouldn't it be cool to do pile" and I dont think we should dwell on it.

 

Lol! You are laying down the gauntlet and challenging us Linux Dorks to prove our manhood.

 

I have a feeling your tatic will work. If someone doesn't do one by Sunday, I will dive on the grenade and create a few Sunday.

 

 

0

Share this post


Link to post
Share on other sites

As i posted previously I like the elegance of TCL but I worry that the skills required them develop one out weight the benefits they bring to the unRAID community.

 

I think the proof of this is that so far no one has produced even one. Until we see at least a proof of concept I suspect it will always stay in the "wouldn't it be cool to do pile" and I dont think we should dwell on it.

 

Lol! You are laying down the gauntlet and challenging us Linux Dorks to prove our manhood.

 

I have a feeling your tatic will work. If someone doesn't do one by Sunday, I will dive on the grenade and create a few Sunday.

 

lol wasnt my intention but yeah it kinda looks that way reading back. It kinda hints at the crux of it though... even a new docker user could cobble together a POC dock based on Debian as fast as I type this reply. I strongly suspect TCL will present some hurdles. Lets wait and see if anyone wants to try it.

0

Share this post


Link to post
Share on other sites

I think their points apply more to the enterprise than the home user. I'm certainly not going to setup one container for an app itself, another to edit the config of that app, another to view the logs, and another to be able to restart the app. That seems like a lot of complexity for a home media server. For a situation with lots of developers, that approach makes much more sense.

 

Yeah I don't think we'll need all those extra containers either.  But I think the key thing they are recommending is that you store all your data and config in volumes, which are simply directories on your unRAID array.  Then you can access the config/logs/data directly from unRAID and don't need to SSH into the container to make changes. 

0

Share this post


Link to post
Share on other sites

I'm all for using cool and awesome things to do a inanely simple task requiring a doctorate to do. But let's ask ourselfs, why would we? Why would we run a sash server to manage a python script ? Therefore. I throw phusion  out the Window. what is There to manage dokker pull/start/stop/kill should take care of 99% of our problems. Who do you want to develop containers ? Who do you want to use it. When we are talking unraid. using a correctly setup tlc with bells and whistles for something that can be solved with while true. Can be done by 4-5 peps. This is not nor will it ever be a enterprise environment and should not be treated as souch. And why would you update your working docker ?  Anyhow I would never use a docker I didn't make myself. That's how one learn.

 

 

Sent from my iPad using Tapatalk

0

Share this post


Link to post
Share on other sites

Yeah I don't think we'll need all those extra containers either.  But I think the key thing they are recommending is that you store all your data and config in volumes, which are simply directories on your unRAID array.  Then you can access the config/logs/data directly from unRAID and don't need to SSH into the container to make changes.

 

Technically Sickbeard, CouchPotato, etc. are not what I describe as docker friendly apps. They do not update via a package manager (instead use GIT) and they are not a client / server type of app either. If you could just have sickbeard be a dumb front end but have all the data in a mysql database (which runs elsewhere in a VM, on the host, etc.), that would be better.

0

Share this post


Link to post
Share on other sites

Do those apps you named not allow you to designate where its config / databases are stored?  Plex has a default location but you can easily change it (I assume it would have to be defined when the container is built) to be anyway where you want.

 

I'm not bringing up a debate point, I'm asking for science because I don't use / know about those other apps.

0

Share this post


Link to post
Share on other sites

using a correctly setup tlc with bells and whistles for something that can be solved with while true. Can be done by 4-5 peps.

 

This is not true. We are talking about probably 10 lines in a cfg file that is very easy to understand and I bet when it's all said and done... There will be 100s of people who can create / edit / manage a Docker App.

 

When you cut through the BS... We are using Docker to install a Linux Distro to have access to it's package manager to run "apt-get install plexmediaserver". People can tell you how it's in a container, it's safer, it's better, it's this, it's that... Sure maybe to a few Dorks / Nerds care about that but for a majority of us, it's access to a package manager since Shitware doesn't have one (why you have 1,500+ lines of XML code for plugins that only 4 or 5 people can make).

 

This is not nor will it ever be a enterprise environment and should not be treated as souch.

I hate to break it too you but Docker is specifically design for the enterprise, DevOPS, Software Developers, infrastructure, cloud, etc. Not Joe Sixpack with his unRAID machine.

 

Also, you have to look at this from unRAID's point of view... To them (or at least it should be) it is an Enterprise Environment (They have 10,000+ unRAID servers sitting all over the world). They should being thinking of how enterprises do things and applying those best of breed practices to unRAID so it makes their life easier along with ours.

 

And why would you update your working docker ?

 

1. The Apps we use are constantly updated (through git no less).

 

2. They have a TON of dependences which are going to force you to have to upgrade whether you like it or not.

 

3. A lot of people hang these Apps on the Web to access remotely so you will want to make sure your app and all their dependences have all the latest security patches / updates. (Which is another thing we should discuss in this thread. Harding the Docker OS, Apps and the container)

 

3. Linux is real quick with end of life with previous versions (software and OS).

 

Anyhow I would never use a docker I didn't make myself. That's how one learn.

 

That's great but if Docker is the way that unRAID wants us to use / install apps in unRAID 6.0... I can assure you 90% of unRAID users are not going to do what you suggest or even know how. They are dependant on others (since unRAID doesn't provide it themselves) to have an easy way to install / use the Apps they want / need.

0

Share this post


Link to post
Share on other sites

We are getting off topic here interesting as it is there are plenty of other threads for this debate.

 

Lets come back to looking at a best practice recommendations list as set out in the opening posts.

 

ta :)

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.