Docker Selenium Networking - "connection refused"

[Sn3akyP3t3]

I've been running Selenium headless under docker for about a year or so on Unraid without any networking issues.  After upgrading Unraid recently to 6.3.5 from 6.3.2 I discovered that I'm now getting this error message when I run the previously known functional script:

"Error during browser start: Message: connection refused"

After a few days of attempted troubleshooting and trying to understand what could be wrong I feel I'm not any closer to the actual cause, but I did uncover a few things I thought to be worth mentioning.  First, I believe that the problem must be somewhere in how Unraid is managing networking with Docker containers, but I have no proof other than I've tried just about everything I can think of and I'm getting the same failure on every attempt.

 

I'm not running this particular Docker container in the manner it probably was intended, but as I've said before it does work.  Basically, I'm running a bash script that launches a Docker selenium container which launches a Python script which conducts it's work via Selenium web driver using Firefox, saves work to a Docker mapped volume, then implodes.

 

I realized that there are a lot of variables involved that may be the cause so I've tried a few things.  Here is what I tried to get around this issue:

• Newer versions of Selenium began experimentation with a new Firefox driver.  Firefox updated with Unraid 6.3.5 to version 52 or so.  According to what I've read the new Firefox driver is still quite buggy and the Nightly Builds are recommended.  Instead of trying to pretend to be a good matchmaker of Firefox version to Selenium version that works I opted to downgrade to known good version of selenium/standalone-firefox:2.52.0.  I downloaded 64 bit Linux binaries of Firefox 45.0.2 which was the last known non-new Firefox driver featured version and paired it up with geckodriver version 0.11.1.  I made some changes to the python script to incorporate the new firefox binary and geckodriver and still got the same error, "Error during browser start: Message: connection refused".
• Against better judgement I tried pairing various more recent versions of Docker Selenium to Firefox with no success.
• Tried adding --privileged Docker flag
• Tried changing networking from bridged to host, --net=host
• Tried taking a drastic changed route while host networking mode enabled to use a remote method and also met failure with a different error message that I didn't record.  This was the code snippet:

  browser = webdriver.Remote(command_executor='http://' + currentIpAddress + ':4444/wd/hub', desired_capabilities=DesiredCapabilities.FIREFOX)

   

 

The geckodriver log revealed something rather interesting about the matter.  I'm not familiar with the material I'm afraid.  I don't know if this is related to the error and failure.

1495958386007    geckodriver    INFO    Listening on 127.0.0.1:59303
1495958386992    mozprofile::profile    INFO    Using profile path /tmp/rust_mozprofile.u3IslBf9J3YK
1495958387004    geckodriver::marionette    INFO    Starting browser /usr/bin/firefox
1495958387022    geckodriver::marionette    INFO    Connecting to Marionette on localhost:43711
Xlib:  extension "RANDR" missing on display ":1001".
process 41: D-Bus library appears to be incorrectly set up; failed to read machine uuid: UUID file '/etc/machine-id' should contain a hex string of length 32, not length 0, with no other text
See the manual page for dbus-uuidgen to correct this issue.

 

[Sn3akyP3t3]

This is still broken using Unraid Version 6.3.5.  This script was working without error in previous version 6.2.* versions of Unraid.  I think it broke about version 6.3.1, but I don't exactly remember.

[Sn3akyP3t3]

It turns out that this has nothing to do with Docker or UnRaid updates or changes.  I was unable to identify the cause of the sudden failure on my own, but I was suspecting a wide variety of things including potential security changes with seccomp, which turned out to be not the case.

 

The root of the problem is the sudden changes that the marionette driver brought, but mainly that headless mode appeared to be crippled before Firefox version 55 was released.  Even after that, there appears to be extra work required to get it to perform without networking errors.

 

This led me to a working solution: https://www.alexkras.com/running-chrome-and-other-browsers-in-almost-headless-mode/

 

This bug identifies one of many usability issues that the marionette driver caused: https://github.com/SeleniumHQ/selenium/issues/2559