- Devel - lists.tockos.org

Re: Looking for Insights on Tock's Context Switch Latency

by Leon Schuermann

Hi Zhiyao, "zhiyao.ma.98--- via Devel" <devel(a)lists.tockos.org> writes: > Update to include some additional observation from my end: Compiler > optimization significantly affects the speed. Currently Tock's kernel > uses "z" which is the slowest. > > On STM32F412G Discovery board @ 96 MHz, performing 10,000 > ping-pong. When changing kernel compilation to use different > optimization flags, the measured time and code size are shown below: Thanks for posting this, this is really quite interesting. I did not expect there to be such a drastic difference between optimization level and overall performance. Especially on the simpler microcontroller platforms, conventional wisdom would suggest that fewer & smaller instructions should correlate with better performance. I'd guess that the compiler may choose more efficient algorithms for certain primitives and performs more aggressive inlining on higher optimization levels, which may serve as a gateway to enabling other optimizations. I'm curious whether we could pinpoint these performance gains to some few subroutines. I did a similar exploration a while back for Tock on RISC-V, which I've posted to the old mailing list: https://groups.google.com/g/tock-dev/c/FPTmNe4BAq0 This highlighted the `memcpy` intrinsic as being particularly inefficient on the `-Oz` optimization level. Supposedly this is fixed in upstream Rust / LLVM since a couple months or years now, but it might be good to verify that. For RISC-V we can use the LiteX Sim target (compiling its HDL to a Verilated simulation) to generate an instruction trace with cycle-accurate information on the instructions executed by the CPU. I have a set of patches for this which I can dig up if you're curious. I don't know whether we have an equivalent for ARM Cortex-M (yet). I'm Ccing @Alex who had Tock running on Renode (emulating an STM ARM chip) once, which may also be able to generate such a trace. -Leon

Oct. 17, 2024

Re: Looking for Insights on Tock's Context Switch Latency

by zhiyao.ma.98＠gmail.com

Update to include some additional observation from my end: Compiler optimization significantly affects the speed. Currently Tock's kernel uses "z" which is the slowest. On STM32F412G Discovery board @ 96 MHz, performing 10,000 ping-pong. When changing kernel compilation to use different optimization flags, the measured time and code size are shown below: O3: 6831 ms text data bss 151042 32 76320 O2: 6954ms text data bss 135682 32 76324 O1: 8528ms text data bss 139778 32 76352 Os: 8528ms text data bss 106498 32 76324 Oz: 10748ms text data bss 100866 32 76320

Oct. 17, 2024

Re: Looking for Insights on Tock's Context Switch Latency

by Samir Rashid

Hello Zhiyao and Amit, Sorry for the misinformation, ignore my previous message. The kernel indeed returns immediately to the process after a command(). I measured that, from board startup, the kernel regains control 100,014 times for 10,000 IPC ping-pongs (Measurement Methodology: counted how many times do_process reached the match). Doesn't this mean that the user/kernel barrier is crossed 20 times per IPC loop (10 round trips each)? @Amit What is the benchmarking story for Tock? There are no code speed benchmarks in the Tock repo. I found tock/TockBenchmarking, but it never got upstreamed to Tock. I will be benchmarking Tock this quarter as part of the same class as Gabe (Pat's student, the guy who made the aforementioned repo). It will be interesting to compare my results to his. What should I do if I want to get some benchmarks upstreamed? I can wrap them with an Action like the code size check (https://github.com/tock/tock/blob/fb33e5f672411318ce9ca4700d9fb3c4a08babe0/…) Secondly, I am having a hard time trying to explain the cause of IPC's performance. I am not sure how to go about profiling within the kernel to figure out where all the overhead is going. Any tips? Thanks, Samir Fingerprint: DE65 F61B 7AD6 69C8 3972 1530 A81C A0CA 8957 AC94 On 2024-10-14 07:12, Samir Rashid via Devel <devel(a)lists.tockos.org> wrote: > Hello Zhiyao, > > Unfortunately, I also do not have a great explanation for you. At 64MHz (on the NRF52840DK), your benchmark takes 4846 ms. > > I would like to point out some things. Your context switch diagram is underestimating the amount of context switches. The sequence you present is not correct because > If a syscall does not immediately error (which IPC is not doing in this case), then the kernel triggers a context switch (ContextSwitchReason::SyscallFired). To be more precise, the kernel's process loop is going to mark that a syscall has fired and ask the scheduler whether it should immediately handle the syscall. The default round robin scheduler will always choose to switch to the other process. > > To revise your diagram: > On the "client" side: - calls command() -> context switch to the kernel > - (NEW) kernel defers processing the command() -> kernel round robin scheduler context switches to server's main() > - (NEW) server app main loop immediately yields -> server context switches to kernel > - kernel processes the client's command -> kernel context switch to the client app > - client immediately runs yield_for -> context switch to kernel > > I empirically verified that the `loop { yield(); }` runs 10,000 times in the server. > > It is possible there is more overhead from the server calling `ipc_notify_client(pid);`, but I did not trace further in your timeline. I think the best method to get more info about this bug would be to do some debugging inside the kernel scheduler (`round_robin.rs`). Let me know if I got anything wrong, Professor Levy. > > Samir > Fingerprint: DE65 F61B 7AD6 69C8 3972 1530 A81C A0CA 8957 AC94 > > > > On 2024-10-11 18:23, zhiyao.ma.98--- via Devel <devel(a)lists.tockos.org> wrote: > > Hi Professor Levy, > > > > Thank you so much for looking into the problem. Please let me know if there is anything I can help on my side. > > > > Best regards, > > Zhiyao > > _______________________________________________ > > Devel mailing list -- devel(a)lists.tockos.org > > To unsubscribe send an email to devel-leave(a)lists.tockos.org _______________________________________________ Devel mailing list -- devel(a)lists.tockos.org To unsubscribe send an email to devel-leave(a)lists.tockos.org

Oct. 15, 2024

Re: Looking for Insights on Tock's Context Switch Latency

by Samir Rashid

Hello Zhiyao and Amit, Sorry for the misinformation, ignore my previous message. The kernel indeed returns immediately to the process after a command(). I measured that, from board startup, the kernel regains control 100,014 times for 10,000 IPC ping-pongs (Measurement Methodology: counted how many times do_process reached the match). Doesn't this mean that the user/kernel barrier is crossed 20 times per IPC loop (10 round trips each)? @Amit What is the benchmarking story for Tock? There are no code speed benchmarks in the Tock repo. I found tock/TockBenchmarking, but it never got upstreamed to Tock. I will be benchmarking Tock this quarter as part of the same class as Gabe (Pat's student, the guy who made the aforementioned repo). It will be interesting to compare my results to his. What should I do if I want to get some benchmarks upstreamed? I can wrap them with an Action like the code size check (https://github.com/tock/tock/blob/fb33e5f672411318ce9ca4700d9fb3c4a08babe0/…) Secondly, I am having a hard time trying to explain the cause of IPC's performance. I am not sure how to go about profiling within the kernel to figure out where all the overhead is going. Any tips? Thanks, Samir Fingerprint: DE65 F61B 7AD6 69C8 3972 1530 A81C A0CA 8957 AC94 On 2024-10-14 07:12, Samir Rashid via Devel <devel(a)lists.tockos.org> wrote: > Hello Zhiyao, > > Unfortunately, I also do not have a great explanation for you. At 64MHz (on the NRF52840DK), your benchmark takes 4846 ms. > > I would like to point out some things. Your context switch diagram is underestimating the amount of context switches. The sequence you present is not correct because > If a syscall does not immediately error (which IPC is not doing in this case), then the kernel triggers a context switch (ContextSwitchReason::SyscallFired). To be more precise, the kernel's process loop is going to mark that a syscall has fired and ask the scheduler whether it should immediately handle the syscall. The default round robin scheduler will always choose to switch to the other process. > > To revise your diagram: > On the "client" side: - calls command() -> context switch to the kernel > - (NEW) kernel defers processing the command() -> kernel round robin scheduler context switches to server's main() > - (NEW) server app main loop immediately yields -> server context switches to kernel > - kernel processes the client's command -> kernel context switch to the client app > - client immediately runs yield_for -> context switch to kernel > > I empirically verified that the `loop { yield(); }` runs 10,000 times in the server. > > It is possible there is more overhead from the server calling `ipc_notify_client(pid);`, but I did not trace further in your timeline. I think the best method to get more info about this bug would be to do some debugging inside the kernel scheduler (`round_robin.rs`). Let me know if I got anything wrong, Professor Levy. > > Samir > Fingerprint: DE65 F61B 7AD6 69C8 3972 1530 A81C A0CA 8957 AC94 > > > > On 2024-10-11 18:23, zhiyao.ma.98--- via Devel <devel(a)lists.tockos.org> wrote: > > Hi Professor Levy, > > > > Thank you so much for looking into the problem. Please let me know if there is anything I can help on my side. > > > > Best regards, > > Zhiyao > > _______________________________________________ > > Devel mailing list -- devel(a)lists.tockos.org > > To unsubscribe send an email to devel-leave(a)lists.tockos.org _______________________________________________ Devel mailing list -- devel(a)lists.tockos.org To unsubscribe send an email to devel-leave(a)lists.tockos.org

Oct. 15, 2024

Re: [Spam/Phising] Notification:- Password Update

by Leon Schuermann

"lists.tockos.org or Email Admin via Devel" <devel(a)lists.tockos.org> writes: > The password to your mailbox devel(a)lists.tockos.org has expired. Apologies for letting this email through, this is obviously phishing. The "Accept" and "Reject" buttons in the webinterface were too close together. :/

Oct. 14, 2024

Re: Looking for Insights on Tock's Context Switch Latency

by Samir Rashid

Hello Zhiyao, Unfortunately, I also do not have a great explanation for you. At 64MHz (on the NRF52840DK), your benchmark takes 4846 ms. I would like to point out some things. Your context switch diagram is underestimating the amount of context switches. The sequence you present is not correct because If a syscall does not immediately error (which IPC is not doing in this case), then the kernel triggers a context switch (ContextSwitchReason::SyscallFired). To be more precise, the kernel's process loop is going to mark that a syscall has fired and ask the scheduler whether it should immediately handle the syscall. The default round robin scheduler will always choose to switch to the other process. To revise your diagram: On the "client" side: - calls command() -> context switch to the kernel - (NEW) kernel defers processing the command() -> kernel round robin scheduler context switches to server's main() - (NEW) server app main loop immediately yields -> server context switches to kernel - kernel processes the client's command -> kernel context switch to the client app - client immediately runs yield_for -> context switch to kernel I empirically verified that the `loop { yield(); }` runs 10,000 times in the server. It is possible there is more overhead from the server calling `ipc_notify_client(pid);`, but I did not trace further in your timeline. I think the best method to get more info about this bug would be to do some debugging inside the kernel scheduler (`round_robin.rs`). Let me know if I got anything wrong, Professor Levy. Samir Fingerprint: DE65 F61B 7AD6 69C8 3972 1530 A81C A0CA 8957 AC94 On 2024-10-11 18:23, zhiyao.ma.98--- via Devel <devel(a)lists.tockos.org> wrote: > Hi Professor Levy, > > Thank you so much for looking into the problem. Please let me know if there is anything I can help on my side. > > Best regards, > Zhiyao > _______________________________________________ > Devel mailing list -- devel(a)lists.tockos.org > To unsubscribe send an email to devel-leave(a)lists.tockos.org

Oct. 14, 2024

Re: Looking for Insights on Tock's Context Switch Latency

by zhiyao.ma.98＠gmail.com

Hi Professor Levy, Thank you so much for looking into the problem. Please let me know if there is anything I can help on my side. Best regards, Zhiyao

Oct. 12, 2024

Re: Looking for Insights on Tock's Context Switch Latency

by Amit Levy

> current RTC counter) takes about 42uS (10,000 in ~2.1 seconds), so about > 21uS each way or 1680 instructions at 64MHz. That's about 13x off of Correction, I meant 50,000 here, not 10,000.

Oct. 11, 2024

Tock meeting agenda 2024-10-11

by Amit Levy

Meeting Link: https://meetings.dialpad.com/room/tock/1vnk9r3e1b No agenda items today. We can add some on the call. Otherwise, I'll suggest briefly talking about clarifying invariants of the alarm stack, followed by pull request triage. Agenda - Updates - TBD

Oct. 11, 2024

Call for Tock meeting agenda items 2024-10-11

by Amit Levy

Please response with proposed agenda items and +1s for tomorrows core meeting

Oct. 11, 2024