I am wondering as to: * Why 6274 cpu refuses to turbo to the second turbo state with half of the cores in use (it stops around 100mhz before the first turbo state)
* Is it possible to force enable second turbo state for all cpu cores, assuming adequate cooling? Or is it controlled on the CPU itself?
Off topic but would rather not make another thread:
* Does NUMA ram alignment matter performance wise with only one physical CPU? (RAM being split half and half per core set at the moment)
* Does anyone know where I could get a reasonably priced 6287SE or 6284SE? or (even better) an engineering model of 62xx?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/08/2016 11:20 PM, Taiidan@gmx.com wrote:
I am wondering as to:
- Why 6274 cpu refuses to turbo to the second turbo state with half of
the cores in use (it stops around 100mhz before the first turbo state)
The CPU will not turbo to the highest frequency unless half the cores are in the CC6 power state. Have you verified that the CC6 power saving mode is enabled via nvramtool?
- Is it possible to force enable second turbo state for all cpu cores,
assuming adequate cooling? Or is it controlled on the CPU itself?
This is controlled by a black box on the CPU die itself. There is no way to override the settings of this black box unless you have an engineering sample (obtaining one at this point is illegal in at least the United States). We do have some experience overclocking the Opteron systems through other methods, but won't discuss further outside of a development contract.
Off topic but would rather not make another thread:
- Does NUMA ram alignment matter performance wise with only one physical
CPU? (RAM being split half and half per core set at the moment)
Yes! The G34 CPUs are actually two dies in one package, with 4 sticks of RAM attached to each node (making 8 sticks per package for the KGPE-D16).
- Does anyone know where I could get a reasonably priced 6287SE or
6284SE? or (even better) an engineering model of 62xx?
Those are rare chips and very expensive. See above for engineering samples; they're not legal here so I would have no idea.
I hope this helps!
- -- Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com
On 12/09/2016 09:51 AM, Timothy Pearson wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/08/2016 11:20 PM,Taiidan@gmx.com wrote:
I am wondering as to:
- Why 6274 cpu refuses to turbo to the second turbo state with half of
the cores in use (it stops around 100mhz before the first turbo state)
The CPU will not turbo to the highest frequency unless half the cores are in the CC6 power state. Have you verified that the CC6 power saving mode is enabled via nvramtool?
- Is it possible to force enable second turbo state for all cpu cores,
assuming adequate cooling? Or is it controlled on the CPU itself?
This is controlled by a black box on the CPU die itself. There is no way to override the settings of this black box unless you have an engineering sample (obtaining one at this point is illegal in at least the United States). We do have some experience overclocking the Opteron systems through other methods, but won't discuss further outside of a development contract.
Off topic but would rather not make another thread:
- Does NUMA ram alignment matter performance wise with only one physical
CPU? (RAM being split half and half per core set at the moment)
Yes! The G34 CPUs are actually two dies in one package, with 4 sticks of RAM attached to each node (making 8 sticks per package for the KGPE-D16).
- Does anyone know where I could get a reasonably priced 6287SE or
6284SE? or (even better) an engineering model of 62xx?
Those are rare chips and very expensive. See above for engineering samples; they're not legal here so I would have no idea.
I hope this helps!
Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJYSsThAAoJEK+E3vEXDOFbSdkH/RJ5dJrN4a3dGopOqYWt3FSr OeIk/SYh6JtiuyUeLV4XMwSbXOTgAEsRGKj2Dt/o0LZZnjkVOm1OaXKfn5K9+0iZ Ec4ae142KjIolNbgl1re3BY7MqW2QCiSqBCsOBmqzYXu1Ypi9IevzfpXDnilToJW sBzwA33kp3Xk7ldtMAIYjtp9ys9z6KSwAJwPJHd6f9aIGmpzh1Gdw2AQeF9wbplf DJLY9xup4SRCns922r/Z93eat34QGb7PZy+1oz8nWAojk+y5JFbRNMTCtjdzAbM5 tPVQykNTfBGQoeV+hL0NmukAJmJDWaCnkOsJTD01WpCKDLIIKf8N6ShxS6Qi1Ds= =DUK+ -----END PGP SIGNATURE-----
Thanks! helpful as always >:D
Yes c states and cc6 states are enabled, 1 or 2 cores can get up to around 100mhz or so less than turbo 2 however I cannot get the whole advertised 8 or even 4 (i get max around 2.5ghz as reported by "cpupower monitor" ), my temp is 35C with full load cpu usage and my fans nearly off (tower cooler) so I don't think that could be the issue.
- Do mismatched cpus work at their native clock speeds? What would happen if I inserted for instance a 6220 in to the second cpu socket.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/11/2016 07:52 PM, Taiidan@gmx.com wrote:
Thanks! helpful as always >:D
Yes c states and cc6 states are enabled, 1 or 2 cores can get up to around 100mhz or so less than turbo 2 however I cannot get the whole advertised 8 or even 4 (i get max around 2.5ghz as reported by "cpupower monitor" ), my temp is 35C with full load cpu usage and my fans nearly off (tower cooler) so I don't think that could be the issue.
Interesting. Are these the results of the turbostat program? What does powertop show in the idle stats?
- Do mismatched cpus work at their native clock speeds? What would
happen if I inserted for instance a 6220 in to the second cpu socket.
Yes, they should work at the native clocks. Don't even try mixing different generations of Opteron even with the same socket, it won't work, but a minor clock difference shouldn't be a problem in the same overall family (e.g. mixing an 82xx with an 82yy should be OK).
- -- Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com
y'all helpful as always >:D
On 12/12/2016 03:00 PM, Timothy Pearson wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/11/2016 07:52 PM, Taiidan@gmx.com wrote:
Thanks! helpful as always >:D Yes c states and cc6 states are enabled, 1 or 2 cores can get up to around 100mhz or so less than turbo 2 however I cannot get the whole advertised 8 or even 4 (i get max around 2.5ghz as reported by "cpupower monitor" ), my temp is 35C with full load cpu usage and my fans nearly off (tower cooler) so I don't think that could be the issue.
Interesting. Are these the results of the turbostat program? What does powertop show in the idle stats?
Powertop, around 96-98% or so. Turbostat doesn't show turbo frequencies for some reason under tsc_mhz, and average cpu freq is lower than wcpupower monitor reports.
With a single 100% cpu thread I can get to 3ghz (still a little short), however with every thread I add the clock speed goes slightly down. Weird stuff.
Is there a way to view and force c-states? I tried using turionpowercontrol but it didn't work, the cpu simply clocked up again and it only has a max cstate and not a min cstate option. I am curious is there a way to disable the second half of the 16 core CPU optys in coreboot?
- Do mismatched cpus work at their native clock speeds? What would
happen if I inserted for instance a 6220 in to the second cpu socket.
Yes, they should work at the native clocks. Don't even try mixing different generations of Opteron even with the same socket, it won't work, but a minor clock difference shouldn't be a problem in the same overall family (e.g. mixing an 82xx with an 82yy should be OK).
I see, thanks!
Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJYTwHhAAoJEK+E3vEXDOFbJmYH/A6UnubucxZQSbqfdvEHX1Nm 8n4yJQYIqQkehdI6Ojyw2WWUBNTF9hBrS/ZbXWWC3lFn1UlohEBPp5kLRDInZ52A rUohi9M2qQv5vnQn1g8SD5Tj6BpNr3GZaqHsK2MDJ+cJJlCXz9UfOoPzl2f9W0c/ jmADXI0/UQnpk2HCYJ/wtO3o+3EtvEfHkw/AYFpZLID6hILNw3kSb84LpqYBU3tj 71xI8DXhhwNFJkV9icX2FHLo3a9opz2ZZ3DXwJ7Jj+Hrm9Zq3nZFZosrnxxLHFIy hvbCdu8bWiJPMrs9fk6DK7+AMwQDM89YnkGt9lbJlnVMqKIm1bDu++2HaJ6YEMk= =nKbX -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/16/2016 07:13 PM, Taiidan@gmx.com wrote:
y'all helpful as always >:D
On 12/12/2016 03:00 PM, Timothy Pearson wrote:
On 12/11/2016 07:52 PM, Taiidan@gmx.com wrote:
Thanks! helpful as always >:D Yes c states and cc6 states are enabled, 1 or 2 cores can get up to around 100mhz or so less than turbo 2 however I cannot get the whole advertised 8 or even 4 (i get max around 2.5ghz as reported by "cpupower monitor" ), my temp is 35C with full load cpu usage and my fans nearly off (tower cooler) so I don't think that could be the issue.
Interesting. Are these the results of the turbostat program? What does powertop show in the idle stats?
Powertop, around 96-98% or so. Turbostat doesn't show turbo frequencies for some reason under tsc_mhz, and average cpu freq is lower than wcpupower monitor reports.
With a single 100% cpu thread I can get to 3ghz (still a little short), however with every thread I add the clock speed goes slightly down. Weird stuff.
I strongly suspect Linux is scheduling threads on the other cores, causing the black box inside the CPU to momentarily take the highly loaded CPU(s) out of maximum boost for brief intervals. turbostat and friends work by measuring the TSC over a period of time to detect boosted operation, and being "a little short" of maximum boost is consistent with this theory.
Is there a way to view and force c-states? I tried using turionpowercontrol but it didn't work, the cpu simply clocked up again and it only has a max cstate and not a min cstate option. I am curious is there a way to disable the second half of the 16 core CPU optys in coreboot?
No, you cannot force C-states with a stock kernel. Coreboot itself has no control over this; the Linux kernel controls all power saving states across all cores once started. One thing you can try is to move all of your processes (including system processes) onto one group of cores with cgroup functionality; Linux should then idle the unused cores via the C3 (subsequently hardware-transitioned to CC6) power saving state.
- -- Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com
On 12/19/2016 01:03 PM, Timothy Pearson wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/16/2016 07:13 PM, Taiidan@gmx.com wrote:
y'all helpful as always >:D
On 12/12/2016 03:00 PM, Timothy Pearson wrote:
On 12/11/2016 07:52 PM, Taiidan@gmx.com wrote:
Thanks! helpful as always >:D Yes c states and cc6 states are enabled, 1 or 2 cores can get up to around 100mhz or so less than turbo 2 however I cannot get the whole advertised 8 or even 4 (i get max around 2.5ghz as reported by "cpupower monitor" ), my temp is 35C with full load cpu usage and my fans nearly off (tower cooler) so I don't think that could be the issue.
Interesting. Are these the results of the turbostat program? What does powertop show in the idle stats?
Powertop, around 96-98% or so. Turbostat doesn't show turbo frequencies for some reason under tsc_mhz, and average cpu freq is lower than wcpupower monitor reports. With a single 100% cpu thread I can get to 3ghz (still a little short), however with every thread I add the clock speed goes slightly down. Weird stuff.
I strongly suspect Linux is scheduling threads on the other cores, causing the black box inside the CPU to momentarily take the highly loaded CPU(s) out of maximum boost for brief intervals. turbostat and friends work by measuring the TSC over a period of time to detect boosted operation, and being "a little short" of maximum boost is consistent with this theory.
Is there a way to view and force c-states? I tried using turionpowercontrol but it didn't work, the cpu simply clocked up again and it only has a max cstate and not a min cstate option. I am curious is there a way to disable the second half of the 16 core CPU optys in coreboot?
No, you cannot force C-states with a stock kernel. Coreboot itself has no control over this; the Linux kernel controls all power saving states across all cores once started. One thing you can try is to move all of your processes (including system processes) onto one group of cores with cgroup functionality; Linux should then idle the unused cores via the C3 (subsequently hardware-transitioned to CC6) power saving state.
Timothy Pearson Raptor Engineering +1 (415) 727-8645 (direct line) +1 (512) 690-0200 (switchboard) https://www.raptorengineering.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJYWCEHAAoJEK+E3vEXDOFbURUIALclOMMi/Kbk46eF+OmYiSGY b9A3V6F0xmjiVxU4Mh17YGWmhsh2S608GvMsonggUQrjWRw5Nr8FQmtNcqOuNaKh tfm2MT3NmSb//cv2nA6a5HavbV7z+oNtVP4y4D2pKLhmN57QAy+XxnuMt9YYzVZz W0Lb0e/Q7vHNsknuea1F5+qU+T3uuu3NWUImdiiq9IhUSQGBNCSpwX+m1RYrC4Dt nKcC1oMYAsvrCa/4xpvfINnYgWKi9yg4TKGCurlPhDCU9crGP5luSCNE9WwMG2EJ yUN/uWx9kSo5F4Frzisuff54Fv7WwysAlvo58Yn9jrAv/cEZFXZiW/IBMGRMBrA= =LoK6 -----END PGP SIGNATURE-----
I posted to the mailinglist about my turbo problem a few months ago and after an hour of testing I finally have an improvement.
I thought that a tickless kernel reduces performance, so I had put: nohz=off In my kernel command line, but this means that all the ticks will wake up the cores so they can't sleep enough for turbo 2 "half core turbo" to enable - so you need nohz=on.
This also saves 10W of power per 16 cores, because the cores/modules not in use will be parked instead of waking up to do absolutely nothing.
I now am able to get 8 cores at 2.65ghz vs before with only 2.5ghz (turbo 1 6274)
It seems there is still a blockage stopping full turbo 2 (3.1ghz) due to half of the CPU not being in C2 100% of the time but I wanted to share this interesting development.
Before C0 was around 1.5 per CPU and there were 4K wakeups idle, now there are only around 50 idle.
|Mperf || Idle_Stats CPU | C0 | Cx | Freq || POLL | C1 | C2 0| 18.93| 81.07| 1742|| 0.00| 34.38| 49.28 8| 19.43| 80.57| 3093|| 0.00| 0.01| 80.56 1| 25.47| 74.53| 1742|| 0.00| 12.81| 61.98 9| 82.67| 17.33| 3095|| 0.00| 0.00| 17.27 2| 0.15| 99.85| 1394|| 0.00| 0.00| 99.85 10| 1.10| 98.90| 1399|| 0.00| 0.00| 98.94 3| 0.00|100.00| 1179|| 0.00| 0.00| 99.99 11| 1.13| 98.87| 1399|| 0.00| 0.00| 98.91 4| 0.00|100.00| 1402|| 0.00| 0.00| 99.99 12| 0.00|100.00| 1778|| 0.00| 0.00| 99.99 5| 0.00|100.00| 1404|| 0.00| 0.00| 99.99 13| 0.00|100.00| 1377|| 0.00| 0.00| 99.99 6| 0.00|100.00| 1442|| 0.00| 0.00| 99.99 14| 0.00|100.00| 1393|| 0.00| 0.00| 99.99 7| 0.00|100.00| 1391|| 0.00| 0.00| 99.99 15| 0.01| 99.99| 1411|| 0.00| 0.00| 99.98
Damn I really wish I could OC opterons - inside every 6274 is a 6287SE screaming to get out.
-
Taiidan＠gmx.com -
Timothy Pearson