Sign in to follow this  
Followers 0
marshy919

append isolcpus rc7 issues

7 posts in this topic

I thought this was just maybe a bug with cAdvisor at first, but noticed significantly higher DPC latency (from 500 to now 15000)

None of the CPU pinning seems to be working.
Without changing anything besides upgrading to rc7 last week, I've noticed this happening.

Anyone else having same problems?

 

cAdvisor showing (this should be only 0,6)

  • Allowed Cores 0 1 2 3 4 5 6 7 8 9 10 11

 

My flash file: 

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=1912:0015
  append pcie_acs_override=downstream
  append isolcpus=1,2,3,4,5,7,8,9,10,11 initrd=/bzroot

 

cAdvisor showing (this should be all but 0,6)

  • Allowed Cores 0 1 2 3 4 5 6 7 8 9 10 11

 

VM config: 

  <vcpu placement='static'>9</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='7'/>
    <vcpupin vcpu='5' cpuset='8'/>
    <vcpupin vcpu='6' cpuset='9'/>
    <vcpupin vcpu='7' cpuset='10'/>
    <vcpupin vcpu='8' cpuset='11'/>
    <emulatorpin cpuset='3'/>
  </cputune>


 

0

Share this post


Link to post
Share on other sites
17 hours ago, marshy919 said:

cAdvisor showing (this should be only 0,6)

  • Allowed Cores 0 1 2 3 4 5 6 7 8 9 10 11

 

17 hours ago, marshy919 said:

cAdvisor showing (this should be all but 0,6)

  • Allowed Cores 0 1 2 3 4 5 6 7 8 9 10 11

 

AFAIK Not a bug per se.

 

What you're seeing is that you're only allowing unRaid to execute on certain cores (and by extension, the docker system).  But, the system knows that you've got 12 cores to play with, and that you're not limiting the particular application to only execute on certain cores via the extended parameters.  

 

cAdvisor tells you the app is allowed by docker to execute on those cores.  When its actually executed, linux overrides everything and executes it on the cores that's actually allowed.

 

0

Share this post


Link to post
Share on other sites
46 minutes ago, Squid said:

 

 

AFAIK Not a bug per se.

 

What you're seeing is that you're only allowing unRaid to execute on certain cores (and by extension, the docker system).  But, the system knows that you've got 12 cores to play with, and that you're not limiting the particular application to only execute on certain cores via the extended parameters.  

 

cAdvisor tells you the app is allowed by docker to execute on those cores.  When its actually executed, linux overrides everything and executes it on the cores that's actually allowed.

 

 

@Squid Are you sure about this? It might just be the wording of your reply that has me confused. You said that what you're seeing is that you're only allowing Unraid to execute on certain cores. and not limiting the particular application by extended parameters.

 

The syscfg isolcpus setting is not allowing Unraid to access certain cores or any application unless allowed by an extended parameter. I would have expected cAdvisor to only show the cores Unraid is allowed on. 

 

I'm trying to find a command I used to check this when I first applied isolcpus setting as I've only just started using cAdvisor, and can't say what it used to show.

 

EDIT - Well based on this it does seem to be working.

root@Tower:~# cat /sys/devices/system/cpu/isolated
8-15
root@Tower:~# cat /sys/devices/system/cpu/possible
0-15


root@Tower:~# taskset -cp 1
pid 1's current affinity list: 0-7
root@Tower:~# lscpu | grep CPU.s
CPU(s):              16
On-line CPU(s) list: 0-15
NUMA node0 CPU(s):   0-15 

 

root@Tower:~# cat /proc/$$/status|tail -6
Cpus_allowed:    00ff
Cpus_allowed_list:    0-7

Edited by Tuftuf
0

Share this post


Link to post
Share on other sites
2 minutes ago, Tuftuf said:

 

@Squid Are you sure about this? It might just be the wording of your reply that has me confused. You said that what you're seeing is that you're only allowing Unraid to execute on certain cores. and not limiting the particular application by extended parameters.

 

The syscfg isolcpus setting is not allowing Unraid to access certain cores or any application unless allowed by an extended parameter. I would have expected cAdvisor to only show the cores Unraid is allowed on. 

 

I'm trying to find a command I used to check this when I first applied isolcpus setting as I've only just started using cAdvisor, and can't say what it used to show.

I've personally never seen the point in my VM usage to ever isolate cores, so yes its theory.

 

But, it makes sense.  Otherwise the dashboard would only ever show you cores 0 & 6 as an example, and not all 12 of them.

 

The docker subsystem allows you to further limit the cores that any particular app can execute on.  cAdvisor sees that the app isn't limited to executing on any particular core, so it reports that it can execute on all of them.  Where it actually executes on is then determined by the isolcpus parameter.

0

Share this post


Link to post
Share on other sites
Just now, Squid said:

I've personally never seen the point in my VM usage to ever isolate cores, so yes its theory.

 

But, it makes sense.  Otherwise the dashboard would only ever show you cores 0 & 6 as an example, and not all 12 of them.

 

The docker subsystem allows you to further limit the cores that any particular app can execute on.  cAdvisor sees that the app isn't limited to executing on any particular core, so it reports that it can execute on all of them.  Where it actually executes on is then determined by the isolcpus parameter.

 

I managed to prove myself wrong, so I have to agree its just how it's displayed.

0

Share this post


Link to post
Share on other sites
1 hour ago, Squid said:

 

 

AFAIK Not a bug per se.

 

What you're seeing is that you're only allowing unRaid to execute on certain cores (and by extension, the docker system).  But, the system knows that you've got 12 cores to play with, and that you're not limiting the particular application to only execute on certain cores via the extended parameters.  

 

cAdvisor tells you the app is allowed by docker to execute on those cores.  When its actually executed, linux overrides everything and executes it on the cores that's actually allowed.

 

 

Hmm, I think I understand what you are saying.

You mean because cAdvisor isn't limited to any cores that's why it is showing the above?

Because I have extra parameters for cAdvisor.

--cpuset-cpus=0,6


It used to look like this:

This is my Krusader docker viewed through cAdvisor.

Allowed Cores 0 1 2 3 4 5 6 7 8 9 10 11

 

1 hour ago, Tuftuf said:

 

@Squid Are you sure about this? It might just be the wording of your reply that has me confused. You said that what you're seeing is that you're only allowing Unraid to execute on certain cores. and not limiting the particular application by extended parameters.

 

The syscfg isolcpus setting is not allowing Unraid to access certain cores or any application unless allowed by an extended parameter. I would have expected cAdvisor to only show the cores Unraid is allowed on. 

 

I'm trying to find a command I used to check this when I first applied isolcpus setting as I've only just started using cAdvisor, and can't say what it used to show.

 

EDIT - Well based on this it does seem to be working.

root@Tower:~# cat /sys/devices/system/cpu/isolated
8-15
root@Tower:~# cat /sys/devices/system/cpu/possible
0-15


root@Tower:~# taskset -cp 1
pid 1's current affinity list: 0-7
root@Tower:~# lscpu | grep CPU.s
CPU(s):              16
On-line CPU(s) list: 0-15
NUMA node0 CPU(s):   0-15 

 

root@Tower:~# cat /proc/$$/status|tail -6
Cpus_allowed:    00ff
Cpus_allowed_list:    0-7

 

Thanks for those commands.

 

cat /sys/devices/system/cpu/isolated = 1-5, 7-11. Perfect :)

cat /proc/$$/status|tail -6 = cpu_allowed_list is 0,6

 

So it looks like it is working...

59 minutes ago, Squid said:

I've personally never seen the point in my VM usage to ever isolate cores, so yes its theory.

 

But, it makes sense.  Otherwise the dashboard would only ever show you cores 0 & 6 as an example, and not all 12 of them.

 

The docker subsystem allows you to further limit the cores that any particular app can execute on.  cAdvisor sees that the app isn't limited to executing on any particular core, so it reports that it can execute on all of them.  Where it actually executes on is then determined by the isolcpus parameter.

 

The reason I isolate CPU is I try to game on my VM.

When you get 15 second lockups every minute due to a docker unpacking a 5GB file, its very annoying.

Only when nothing is happening at all my DPC latency is okay (500ish), but even the parity check absolutely kills it now.

Without changing anything, any activity on the server absolutely kills my VMs DPC latency

The driver that is killing it now is wdf1000.sys.

 

Essentially I had previously used cores 0,6 to run all of my unraid server and dockers.

Core 3 for emulatorpin.

Every other core for VMs (and it worked...)

Without changing any of this, DPC latency is through the roof and now cAdvisor is showing that is isn't working. 

But commands in putty are?

 

 

0

Share this post


Link to post
Share on other sites

I think we have come to the conclusion that cAdvisor is showing the correct information but it does not mean the cores are isolated. Also I expect you can see on the dashboard the cores are not doing anything when your VM is stopped. If they are then something is using it.

 

Something else may be the cause of the increased latency. 

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.