[Mckernel-users 30] Re: McKernel and Turbo

[Balazs Gerofi]

Hello Scott,

On Mon, May 22, 2017 at 2:07 PM, Scott Walker <walker8 at email.arizona.edu>
wrote:

> I have attached a tarball with some results in it. The 5th column and the
> 7th and 8th columns are the interesting results. The 5th column is the
> measured frequency. The 7th column is one of the turbo disable bits, and
> the 8th column is another turbo disable bit. The 7th and 8th column should
> be non-zero if turbo is disabled.
>

I took a look at the tarball, but I cannot find any results file in it.
Which file should I look at?


> In the linux results the frequency stays in turbo and the turbo bits are
> both off, as expected. In the McKernel results, the processor is not in a
> turbo frequency but the turbo disable bits do not stay off, sometimes both
> of them are zero.
>
> I have noticed the following bugs regarding turbo:
>
> The "-t" flag to mcreboot does not enable turbo. It appears to do nothing.
>

This is strange, I just double-checked it by printing out the MSR and it
does toggle the turbo bit.
Turbo is bit 32 of MSR_IA32_PERF_CTL (0x199) and is configured in
init_pstate_and_turbo() function in arch/x86/kernel/cpu.c, I actually have
the text from the Intel manual copied there:

    /* Turbo boost setting:
     * Bit 1 of EAX in Leaf 06H (i.e. CPUID.06H:EAX[1]) indicates
opportunistic
     * processor performance operation, such as IDA, has been enabled by
BIOS.
     *
     * IA32_PERF_CTL (0x199H) bit 32: IDA (i.e., turbo boost) Engage. (R/W)
     * When set to 1: disengages IDA
     * When set to 0: enables IDA
     */

Is this the MSR you are looking at?

When turbo is supposedly disabled in McKernel, I sometimes see the
> processor in turbo frequencies.
>

What platform are you using and how do you configure Linux and McKernel
CPUs exactly? Do you split HW threads of the same CPU core between the two
kernels?


> mcstop+release sometimes does not re-enable turbo. I'm not sure how to
> replicate this bug, it just seems to happen sometimes.
>

Thanks for reporting this, I am adding a bug report.


> As for trying these experiments yourself, it may be a little tricky. I
> modified the McKernel source code to provide two system calls which allow
> me to read and write MSRs. I can send you a diff of the modifications if
> that helps.
>

I do have a patch in a development branch that does similar things
actually, but yes, send me your code please!

Thanks,
Balazs


> thanks,
> Scott
>
> On Thu, May 4, 2017 at 1:20 PM, Balazs Gerofi <bgerofi at riken.jp> wrote:
>
>> Hello Scott,
>>
>> could you let me know what is this test exactly?
>> If you can share it, I could try to run on one of our machines to
>> investigate what's going on.
>>
>> Balazs
>>
>> On Thu, May 4, 2017 at 12:01 AM, Scott Walker <walker8 at email.arizona.edu>
>> wrote:
>>
>>> Hi Balazs,
>>>
>>> I tried executing the same test in Linux and I didn't see it modifying
>>> those registers at all. I am currently either partitioning one or two cores
>>> to McKernel, and never core 0.
>>>
>>> I'll let you know if I find out more.
>>>
>>> Thanks,
>>> Scott Walker
>>>
>>> On May 3, 2017 10:50 PM, "Balazs Gerofi" <bgerofi at riken.jp> wrote:
>>>
>>> Hi Scott,
>>>
>>> that sounds strange, McKernel touches those MSRs only during boot.
>>> How do you partition your CPUs?  Isn't it possible that someone else
>>> (e.g., Linux) makes modifications to some MSRs on the fly?
>>>
>>> Balazs
>>>
>>>
>>> On Wed, May 3, 2017 at 10:26 PM, Scott Walker <walker8 at email.arizona.edu
>>> > wrote:
>>>
>>>> Hi Balazs,
>>>>
>>>> Thanks, that's exactly what we need.
>>>>
>>>> For "sometimes enabled" here is what I observed:
>>>>
>>>> If I run a bunch of mcexec jobs where I check to see if those 2 turbo
>>>> disable bits are set to 1, I noticed that they independently change states.
>>>> Sometimes both bits would be true, sometimes only one or the other would be
>>>> true, and at other times they would both be false.
>>>>
>>>> I'm not sure if Turbo actually becomes active during the latter
>>>> observation, I could find out if you want. Also, if I try to change the
>>>> values of those bits then McKernel locks up.
>>>>
>>>> Thanks,
>>>> Scott Walker
>>>>
>>>> On May 3, 2017 10:14 PM, "Balazs Gerofi" <bgerofi at riken.jp> wrote:
>>>>
>>>>> Hello Scott,
>>>>>
>>>>> On Wed, May 3, 2017 at 5:45 PM, Scott Walker <
>>>>> walker8 at email.arizona.edu> wrote:
>>>>>
>>>>>> The experiments we are doing require Intel Turbo to be enabled but we
>>>>>> noticed that McKernel is disabling it. I am seeing that the Turbo disable
>>>>>> bits in the PERF_CONTROL MSR and MISC_ENABLE are sometimes enabled.
>>>>>>
>>>>>> This is not happening with the same system running linux. Is there a
>>>>>> way we can disable this in McKernel? I've been unable to track this down in
>>>>>> the McKernel source code.
>>>>>>
>>>>>
>>>>> Turbo mode is disabled by default. If you want to enable it please
>>>>> pass -t to the mcreboot script.
>>>>> Also, what do you exactly mean by "sometimes enabled"?
>>>>>
>>>>> Best,
>>>>> Balazs
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Scott Walker
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pccluster.org/pipermail/mckernel-users/attachments/20170523/aad3a0e0/attachment-0001.html>