<div dir="ltr">Hi Scott,<div><br></div><div>I will look into the 64 bit return value issue hopefully sometimes later this week.</div><br>For the time being, you could just try:<br><br>unsigned long *addr = ihk_mc_syscall_arg1(ctx);<br>*addr = __rdmsr(reg);<div><br></div><div>in your sys_rdmsr() implementation, because on x86 it is possible to access userspace from the kernel (assuming that your are in the given process' context). This should enable you to proceed with your experiments.<br></div><div>Note that copy_to_user() is slower because it goes through various verification steps to make sure the address is valid.<br></div><div><div><br></div><div>Best,</div><div>Balazs</div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 23, 2017 at 3:51 PM, Scott Walker <span dir="ltr"><<a href="mailto:walker8@email.arizona.edu" target="_blank">walker8@email.arizona.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Balazs,<br></div><div>When I went to get my code to send to you, I noticed something which may better indicate the problem.<br><br></div><div>When I first made my read/write MSR system calls, I was passing and returning the register values to the syscall. This was resulting in a bug where 64 bit values passed and returned had strange behavior. I did not realize that I left debug code in the system call when I was trying to figure this out, and that was causing the readings to come back indicating the aforementioned strange turbo behavior.<br><br></div><div>I changed my code to use the only way I could get the read/write MSR syscalls to work, which is using copy_to/from_user to handle all 64 bit values. This shows that turbo is indeed enabled when I use the "-t" flag (and off without the -t flag). However, the problem still remains where the 64 bit values being passed and returned have odd behavior. It looks like returned 64 bit values have the sign bit flipped sometimes, and passed 64 bit values have the top 32 bits truncated.<br><br></div><div>I have attached a test program which shows the strange behavior this, along with my two McKernel read/write system calls. In addition to this program's output, you can uncomment the "kprintf" and check the McKernel klog, which shows the problems with 64 bit values being passed to the system call. Please let me know if there is a fix for this problem as using copy_to/from_user has too high overhead for our experiment.<br></div></div></div><div><div><br></div><div>Additional info:<br><br></div><div>All of these tests have hyper threading off.<br><br></div><div>In my test program I am comparing the value of a system call reading the TSC, and using the "RDTSC" instruction in the user program. Both counters still increment at the same rate but the TSC value returned from the kernel has strange behavior.<br><br></div><div>I tried using my own rdmsr/wrmsr assembly functions instead of the one in "registers.h" but they both had the same result. However, you can verify correct values within the kernel with kprintf.<br></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">thanks,<br></div><div class="gmail_extra">Scott<br></div><div><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 23, 2017 at 12:13 PM, Balazs Gerofi <span dir="ltr"><<a href="mailto:bgerofi@riken.jp" target="_blank">bgerofi@riken.jp</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr">Hello Scott,<div><br></div><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, May 22, 2017 at 2:07 PM, Scott Walker <span dir="ltr"><<a href="mailto:walker8@email.arizona.edu" target="_blank">walker8@email.arizona.edu</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div><div><div><div>I have attached a tarball with some results in it. The 5th column and the 7th and 8th columns are the interesting results. The 5th column is the measured frequency. The 7th column is one of the turbo disable bits, and the 8th column is another turbo disable bit. The 7th and 8th column should be non-zero if turbo is disabled.<br></div></div></div></div></div></blockquote><div><br></div></span><div>I took a look at the tarball, but I cannot find any results file in it. Which file should I look at?</div><span><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div>In the linux results the frequency stays in turbo and the turbo bits are both off, as expected. In the McKernel results, the processor is not in a turbo frequency but the turbo disable bits do not stay off, sometimes both of them are zero.<br><br></div><div>I have noticed the following bugs regarding turbo:<br><br></div><div>The "-t" flag to mcreboot does not enable turbo. It appears to do nothing.<br></div></div></blockquote><div><br></div></span><div>This is strange, I just double-checked it by printing out the MSR and it does toggle the turbo bit.</div><div>Turbo is bit 32 of MSR_IA32_PERF_CTL (0x199) and is configured in init_pstate_and_turbo() function in arch/x86/kernel/cpu.c, I actually have the text from the Intel manual copied there:</div><div><br></div><div><div> /* Turbo boost setting:</div><div> * Bit 1 of EAX in Leaf 06H (i.e. CPUID.06H:EAX[1]) indicates opportunistic</div><div> * processor performance operation, such as IDA, has been enabled by BIOS.</div><div> *</div><div> * IA32_PERF_CTL (0x199H) bit 32: IDA (i.e., turbo boost) Engage. (R/W)</div><div> * When set to 1: disengages IDA</div><div> * When set to 0: enables IDA</div><div> */</div></div><div><div><br></div><div>Is this the MSR you are looking at?</div><div><br></div></div><span><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div>When turbo is supposedly disabled in McKernel, I sometimes see the processor in turbo frequencies.<br></div></div></blockquote><div><br></div></span><div>What platform are you using and how do you configure Linux and McKernel CPUs exactly? Do you split HW threads of the same CPU core between the two kernels?</div><span><div> <br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div>mcstop+release sometimes does not re-enable turbo. I'm not sure how to replicate this bug, it just seems to happen sometimes.<br></div></div></blockquote><div><br></div></span><div>Thanks for reporting this, I am adding a bug report.</div><span><div> <br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div>As for trying these experiments yourself, it may be a little tricky. I modified the McKernel source code to provide two system calls which allow me to read and write MSRs. I can send you a diff of the modifications if that helps.<br></div></div></blockquote><div><br></div></span><div>I do have a patch in a development branch that does similar things actually, but yes, send me your code please!</div><div><br></div><div>Thanks,</div><div>Balazs</div><div><div class="m_6039433928869890357gmail-m_-3562019356465458502h5"><div><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div><br></div><div>thanks,<br></div><div>Scott<br></div></div><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-HOEnZb"><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 4, 2017 at 1:20 PM, Balazs Gerofi <span dir="ltr"><<a href="mailto:bgerofi@riken.jp" target="_blank">bgerofi@riken.jp</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr">Hello Scott,<div><br></div><div>could you let me know what is this test exactly?</div><div>If you can share it, I could try to run on one of our machines to investigate what's going on.</div><span class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370HOEnZb"><font color="#888888"><div><br></div><div>Balazs</div></font></span></div><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370HOEnZb"><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 4, 2017 at 12:01 AM, Scott Walker <span dir="ltr"><<a href="mailto:walker8@email.arizona.edu" target="_blank">walker8@email.arizona.edu</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="auto"><div dir="auto">Hi Balazs,</div><div dir="auto"><br></div><div>I tried executing the same test in Linux and I didn't see it modifying those registers at all. I am currently either partitioning one or two cores to McKernel, and never core 0. </div><div dir="auto"><br></div><div dir="auto">I'll let you know if I find out more.<br><br>Thanks,<br><div dir="auto">Scott Walker</div><div><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370m_8848839898212243554h5"><div dir="auto" class="gmail_extra"><br><div class="gmail_quote">On May 3, 2017 10:50 PM, "Balazs Gerofi" <<a href="mailto:bgerofi@riken.jp" target="_blank">bgerofi@riken.jp</a>> wrote:<br type="attribution"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370m_8848839898212243554m_5127136209023769263quote"><div dir="ltr">Hi Scott,<div><br></div><div>that sounds strange, McKernel touches those MSRs only during boot.</div><div>How do you partition your CPUs? Isn't it possible that someone else (e.g., Linux) makes modifications to some MSRs on the fly?</div><font color="#888888"><div><br></div><div>Balazs</div><div><br></div></font></div><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370m_8848839898212243554m_5127136209023769263elided-text"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 10:26 PM, Scott Walker <span dir="ltr"><<a href="mailto:walker8@email.arizona.edu" target="_blank">walker8@email.arizona.edu</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="auto">Hi Balazs,<div dir="auto"><br></div><div dir="auto">Thanks, that's exactly what we need.</div><div dir="auto"><br></div><div dir="auto">For "sometimes enabled" here is what I observed: </div><div dir="auto"><br></div><div dir="auto">If I run a bunch of mcexec jobs where I check to see if those 2 turbo disable bits are set to 1, I noticed that they independently change states. Sometimes both bits would be true, sometimes only one or the other would be true, and at other times they would both be false.</div><div dir="auto"><br></div><div dir="auto">I'm not sure if Turbo actually becomes active during the latter observation, I could find out if you want. Also, if I try to change the values of those bits then McKernel locks up.<br><br>Thanks,<br><div dir="auto">Scott Walker</div></div></div><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370m_8848839898212243554m_5127136209023769263m_-5028997867798451419HOEnZb"><div class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370m_8848839898212243554m_5127136209023769263m_-5028997867798451419h5"><div class="gmail_extra"><br><div class="gmail_quote">On May 3, 2017 10:14 PM, "Balazs Gerofi" <<a href="mailto:bgerofi@riken.jp" target="_blank">bgerofi@riken.jp</a>> wrote:<br type="attribution"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr">Hello Scott,<div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 5:45 PM, Scott Walker <span dir="ltr"><<a href="mailto:walker8@email.arizona.edu" target="_blank">walker8@email.arizona.edu</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="auto"><div dir="auto"><div dir="auto">The experiments we are doing require Intel Turbo to be enabled but we noticed that McKernel is disabling it. I am seeing that the Turbo disable bits in the PERF_CONTROL MSR and MISC_ENABLE are sometimes enabled. </div><div dir="auto"><br></div><div dir="auto">This is not happening with the same system running linux. Is there a way we can disable this in McKernel? I've been unable to track this down in the McKernel source code.<br></div></div></div></blockquote><div><br></div><div>Turbo mode is disabled by default. If you want to enable it please pass -t to the mcreboot script.</div><div>Also, what do you exactly mean by "sometimes enabled"?</div><div><br></div><div>Best,</div><div>Balazs</div><div> <br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="auto"><div dir="auto"><div dir="auto"><br>Thanks, <span class="m_6039433928869890357gmail-m_-3562019356465458502m_-835775587783023695gmail-m_2752198956228544370m_8848839898212243554m_5127136209023769263m_-5028997867798451419m_-4315423277648846579m_-5660631635574264550gmail-HOEnZb"><font color="#888888"><br><div dir="auto">Scott Walker</div></font></span></div></div></div>
</blockquote></div><br></div></div>
</blockquote></div></div>
</div></div></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>