- Devel - lists.tockos.org

Re: Looking for Insights on Tock's Context Switch Latency

by Amit Levy

Hi Zhiyao, Thank you for including code. I haven't done so just yet, but will replicate this soon to both validate and try to debug/explain in more detail. Meanwhile, there area few things going on here, I think. First, to confirm some general numbers. At 64MHz (on the NRF52840DK), an almost no-op system-call (a context switch back and forth that reads the current RTC counter) takes about 42uS (10,000 in ~2.1 seconds), so about 21uS each way or 1680 instructions at 64MHz. That's about 13x off of what you're seeing _if_ no work is being done between those context switches, though of course there is. So on average, there is something like 12k instructions left to explain, which is indeed a lot. Second, it's worth noting that IPC in Tock is so unoptimized its not even funny. I added IPC in 2016 originally in service of a paper deadline, when there was no real use case for IPC for any Tock users, but as a proof of concept that, yes, this kind of interaction could also be supported. Since then it's hardly been touched. Recently there are some actual use cases that use and/or require IPC, so it's a good time to design a half-way decent system... Finally, IPC in Tock is quite different from FreeRTOS task notifications (as I understand them anyway), and as a result does quite a bit more. FreeRTOS task notifications directly pass control from the sender to the receiver rather than queue a notification to be handled asynchronously, which is what Tock IPC does. So there are a bunch of subsystems that get traversed in Tock (process scheduler, systick reset, hardware event loop, MPU reconfiguration, etc) that shouldn't need to be traversed in FreeRTOS. MPU reconfiguration is expensive (not _that_ expensive), but should only be happening twice here. Because the IPC is asynchronous, other things _could_ be happening between yieling in the client and receiving the notification in the server (or vice versa), such as handling hardware interupts, that that seems unlikely. In summary, while I don't have an off-the-cuff explanation for the high latency, I'm not _that_ surprised that IPC is very slow. To be continued... -Amit "zhiyao.ma.98--- via Devel" <devel(a)lists.tockos.org> writes: > Hello Tock developers, > > I am recently testing Tock's context switch speed with an STM32F412G-Discovery board, and I noticed that the latency is longer than expected. I hope to confirm that my settings are correct and gain some insights behind the data. > > To test the context switch speed, I implemented two apps called "server" and "client." The server sets up an IPC service that simply notifies completion back to the client. The client invokes the IPC service 10,000 times in a loop and then calculates the average ping-pong time. > > On my board, the 10,000 ping-pong takes 10,748 ms. That is more than 1ms for each ping-pong. > > I believe the CPU clock is set to 96 MHz by the default kernel configuration for the board. With a similar clock speed, a ping-pong implemented through task notification on FreeRTOS takes around 13 μs, which is almost two orders of magnitude faster than Tock. > > Professor Campbell pointed out to me that there are multiple context switches for each ping-pong: > > On the "client" side: > - calls command() -> context switch to the kernel > - command returns -> context switch back to the client app > - client app calls yield -> context switch back to the kernel > > Next, on the “server” side: > - upcall scheduled -> context switch to the server app > - calls ipc_notify_client -> context switch to the kernel to schedule the upcall > - command returns -> context switch back to the server app > - yield -> context switch to the kernel > > Finally, on the “client” side again: > - upcall scheduled -> context switch to the client app > > However, even if there are 8 context switches per ping-pong, on average each context switch still takes around 135 μs, which is around 13k instructions executed per context switch on a 96 MHz CPU. The number still seems to be high. > > Below is the source code for the “server” and “client” app. > > Server: > > ```c > #include <libtock/kernel/ipc.h> > #include <libtock/tock.h> > > static void ipc_callback( > int pid, > __attribute__ ((unused)) int len, > __attribute__ ((unused)) int buf, > __attribute__ ((unused)) void* ud) { > ipc_notify_client(pid); > } > > int main(void) { > ipc_register_service_callback("server", ipc_callback, > NULL); > > while (1) { > yield(); > } > } > ``` > > Client: > > ```c > #include <stdio.h> > #include <libtock-sync/services/alarm.h> > #include <libtock/kernel/ipc.h> > > size_t service_id = -1; > > char buf[64] __attribute__((aligned(64))); > > bool done = false; > > static void ipc_callback(__attribute__ ((unused)) int pid, > __attribute__ ((unused)) int len, > __attribute__ ((unused)) int arg2, > __attribute__ ((unused)) void* ud) { > done = true; > } > > static uint32_t get_time_ms(void) { > struct timeval tv; > libtock_alarm_gettimeasticks(&tv, NULL); > return (tv.tv_sec * 1000) + (tv.tv_usec / 1000); > } > > int main(void) { > int ret; > > ret = ipc_discover("server", &service_id); > while (ret != RETURNCODE_SUCCESS) { > printf("No server, retry in 1 second.\n"); > libtocksync_alarm_delay_ms(1000); > } > > printf("Server discovered.\n"); > > ipc_register_client_callback(service_id, ipc_callback, NULL); > ipc_share(service_id, buf, 64); > > uint32_t start_time = get_time_ms(); > > for (int i = 0; i < 10000; ++i) { > done = false; > ipc_notify_service(service_id); > yield_for(&done); > } > > uint32_t end_time = get_time_ms(); > > printf("Ping-pong 10000 times take %lu ms.\n", end_time - start_time); > > return 0; > } > ``` > _______________________________________________ > Devel mailing list -- devel(a)lists.tockos.org > To unsubscribe send an email to devel-leave(a)lists.tockos.org

Oct. 11, 2024

Looking for Insights on Tock's Context Switch Latency

by zhiyao.ma.98＠gmail.com

Hello Tock developers, I am recently testing Tock's context switch speed with an STM32F412G-Discovery board, and I noticed that the latency is longer than expected. I hope to confirm that my settings are correct and gain some insights behind the data. To test the context switch speed, I implemented two apps called "server" and "client." The server sets up an IPC service that simply notifies completion back to the client. The client invokes the IPC service 10,000 times in a loop and then calculates the average ping-pong time. On my board, the 10,000 ping-pong takes 10,748 ms. That is more than 1ms for each ping-pong. I believe the CPU clock is set to 96 MHz by the default kernel configuration for the board. With a similar clock speed, a ping-pong implemented through task notification on FreeRTOS takes around 13 μs, which is almost two orders of magnitude faster than Tock. Professor Campbell pointed out to me that there are multiple context switches for each ping-pong: On the "client" side: - calls command() -> context switch to the kernel - command returns -> context switch back to the client app - client app calls yield -> context switch back to the kernel Next, on the “server” side: - upcall scheduled -> context switch to the server app - calls ipc_notify_client -> context switch to the kernel to schedule the upcall - command returns -> context switch back to the server app - yield -> context switch to the kernel Finally, on the “client” side again: - upcall scheduled -> context switch to the client app However, even if there are 8 context switches per ping-pong, on average each context switch still takes around 135 μs, which is around 13k instructions executed per context switch on a 96 MHz CPU. The number still seems to be high. Below is the source code for the “server” and “client” app. Server: ```c #include <libtock/kernel/ipc.h> #include <libtock/tock.h> static void ipc_callback( int pid, __attribute__ ((unused)) int len, __attribute__ ((unused)) int buf, __attribute__ ((unused)) void* ud) { ipc_notify_client(pid); } int main(void) { ipc_register_service_callback("server", ipc_callback, NULL); while (1) { yield(); } } ``` Client: ```c #include <stdio.h> #include <libtock-sync/services/alarm.h> #include <libtock/kernel/ipc.h> size_t service_id = -1; char buf[64] __attribute__((aligned(64))); bool done = false; static void ipc_callback(__attribute__ ((unused)) int pid, __attribute__ ((unused)) int len, __attribute__ ((unused)) int arg2, __attribute__ ((unused)) void* ud) { done = true; } static uint32_t get_time_ms(void) { struct timeval tv; libtock_alarm_gettimeasticks(&tv, NULL); return (tv.tv_sec * 1000) + (tv.tv_usec / 1000); } int main(void) { int ret; ret = ipc_discover("server", &service_id); while (ret != RETURNCODE_SUCCESS) { printf("No server, retry in 1 second.\n"); libtocksync_alarm_delay_ms(1000); } printf("Server discovered.\n"); ipc_register_client_callback(service_id, ipc_callback, NULL); ipc_share(service_id, buf, 64); uint32_t start_time = get_time_ms(); for (int i = 0; i < 10000; ++i) { done = false; ipc_notify_service(service_id); yield_for(&done); } uint32_t end_time = get_time_ms(); printf("Ping-pong 10000 times take %lu ms.\n", end_time - start_time); return 0; } ```

Oct. 10, 2024

Re: Discussion on u32

by Leon Schuermann

Hi Lawrence, Thanks for posting this! Apart from the other posts, I wanted to briefly summarize the next steps we discussed on the core call on Friday, the notes of which are up in a PR [1]. There are three types of changes to the system call interface that we'll have to think about for supporting a CHERI and/or 64-bit platform: 1. The system call driver return values. Possible changes are to - extend the `SyscallReturn` enum with additional variants. We'll have to make sure that we did not promise this enumeration to be exhaustive in our documentation (likely TRD104) and ensure that none of our current userspace libraries rely on the current enum definitions to be exhaustive. We can likely implement this in a TRD that extends TRD 104, and as part of a minor version bump. - change certain system call's return values, to better match the semantic meaning of their return value and to avoid switching between variants on different platforms. For this change to be rolled out on 32-bit platforms, we'll need to wait for a major (breaking) release. Depending on where this is documented, we probably want a full revision of TRD 104 and other applicable reference documents for this. 2. System call arguments. Driver and subdriver (command number) arguments should always be 32-bit. We should check if we state something different anywhere, and then manifest these changes in the kernel code base. Notably, libtock-rs already assumes them to be u32s. For the two command arguments, we need to decide whether we want to encode more typing information into a driver's interface. Currently, these arguments are `usize`. We very much intended these to be only `u32`s (such that driver interfaces are identical across platforms). The kernel still uses `usize` for these as we didn't want to change interfaces when practically, given we only supported 32-bit platforms, these types were identical. It is clear that restricting these to `u32`s will be prohibitive of some applications, such as passing length parameters over this interface. However, we'd probably still want to document and enforce which type a driver actually expects to be passed as a parameter. One such soltion would leave the raw interface types as `usize`, but extend a proposal like https://github.com/tock/tock/pull/4112 to add type information per command-number and implicitly cast this `usize` per sub-command. 3. Upcall parameters. Upcall parameters should be able to return buffer lengths, etc. Therefore, they also should be `usize` wide. TBD: do we want more typing information here? It seems harder to enforce the relation between upcall-number and types passed. Maybe we can extend the Grant type to encode this information? In summary, we'll likely want to review these interface changes as part of either a minor extension or revision of TRD 104, to ensure that we stay backwards compatible. A full interface revision can then be proposed as a full TRD. Finally, I want to echo Hudson's and Johnathan's statements from the call: as long as we don't change the existing 32-bit non-CHERI interface, it should be fine to introduce a new ABI for these new platforms. However, we want to make sure that this doesn't cause much overhead or churn for kernel & capsule developers. This interface should probably also be proposed in a TRD. It seems best to continue figuring out these open questions and details on the mailing list. Hope this summary is useful! I hope I didn't mischaracterize anyone's opinion, please let me know if you disagree with anything. -Leon [1]: https://github.com/tock/tock/pull/4189

Oct. 7, 2024

Re: Discussion on u32

by Johnathan Van Why

I don't think (2) is acceptable in Tock 2, because it would be a breaking change. However, I don't think we need to add as many new return variants as (1) proposes. I don't think we care about any platforms where address offsets may exceed 2 GiB (correct me if I am wrong), so we can pack offsets into i32/u32 even on our 64-bit platforms. Therefore, I think the only return variants we need to add are: 1. SuccessWithMetaPtr: Used by Memops 1-9. 2. SuccessWith2MetaPtr: Used by Subscribe and all three Allows. 3. FailureWith2MetaPtr Used by Subscribe and all three Allows. We would keep the current return variants for all currently-stabilized system calls on 32-bit non-CHERI platforms, and use the new return variants on all other platforms (including all 64-bit and CHERI). That keeps backwards-compatibility on platforms that Tock 2 currently supports while providing better semantics on 64-bit and CHERI platforms. For Tock 3, it might make sense to go somewhat further than just u32, u64, usize, and MetaPtr. How about signed integer, floats, bool, etc? It would be nice if the system call layer could indicate that the button upcall <https://github.com/tock/tock/blob/65dda58c893e5b1fb49da7e4389552584758d5e1/…> receives a (u32, bool) and the temperature reading upcall <https://github.com/tock/tock/blob/65dda58c893e5b1fb49da7e4389552584758d5e1/…> receives an i32 (this isn't in the documentation), rather than pretending that all upcalls receive (u32, u32, u32). On Thu, Oct 3, 2024 at 3:51 PM Amit Levy via Devel <devel(a)lists.tockos.org> wrote: > > The current set of return codes include a type. e.g. calls like allow/brk > currently return SyscallReturnU32U32 or SyscallReturnU32. Clearly this is > not true for 64-bit platforms. Or for CHERI where the size of an address is > not the size of a pointer. > > Excellent point, and this is definitely something to address. Thanks for > noticing and bringing this up. > > I think given this is in the context of switching from only 32 support to > adding 64 bit and/or CHERI, I don't think we should be overly concerned > about breaking backwards compatibility. Instead, this is potentially a fine > reason to think towards a 3.0 system call interface---one which shouldn't > be overly difficult to port to, but wouldn't be binary compatible with > applications compiled for 2.0. (we may lump in fully relocatable binaries > and/or first-class non-XIP support if changing some expectations about > binaries helps us accomplish that, as as potentially changes to core system > interfaces, such as IPC) > > >From that perspective, I think (1) is untenable. I think we should be > much more concerned with portability of, e.g., capsules and a unified > system call interface than with backwards compatibility. I don't think > there is any advantage to (1) other than backwards compatibility. > > I think (2) is the answer. It follows the general idea in the MetaPtr PR > (#4174) of converting naming things what they actually are. > > (3) I understand the expediency for a downstream port, but I don't see an > advantage to this if we can do 2 (which I think we can). > > (4) Similar to 3. I think it would be _reasonable_, if we went with 3, to > bump the major version number and redefine "success" to have the current > value of successu32, such that it just happened to be the case that 2.0 > binaries also turned out to work unmodified. > > -Amit > _______________________________________________ > Devel mailing list -- devel(a)lists.tockos.org > To unsubscribe send an email to devel-leave(a)lists.tockos.org >

Oct. 5, 2024

Re: Tock meeting agenda 2024-10-04

by Amit Levy

FYI I am under the weather today but hopefully someone else will moderate.

Oct. 4, 2024

Tock meeting agenda 2024-10-04

by Amit Levy

Meeting Link: https://meetings.dialpad.com/room/tock/1vnk9r3e1b Agenda - Updates - Pointer returns from syscalls on non-32-bit platforms (https://lists.tockos.org/hyperkitty/list/devel@lists.tockos.org/thread/ACZB…)

Oct. 4, 2024

Call for Tock meeting agenda items 2024-10-04

by Amit Levy

Please response with proposed agenda items and +1s for tomorrows core meeting

Oct. 3, 2024

Re: Discussion on u32

by Amit Levy

> The current set of return codes include a type. e.g. calls like allow/brk currently return SyscallReturnU32U32 or SyscallReturnU32. Clearly this is not true for 64-bit platforms. Or for CHERI where the size of an address is not the size of a pointer. Excellent point, and this is definitely something to address. Thanks for noticing and bringing this up. I think given this is in the context of switching from only 32 support to adding 64 bit and/or CHERI, I don't think we should be overly concerned about breaking backwards compatibility. Instead, this is potentially a fine reason to think towards a 3.0 system call interface---one which shouldn't be overly difficult to port to, but wouldn't be binary compatible with applications compiled for 2.0. (we may lump in fully relocatable binaries and/or first-class non-XIP support if changing some expectations about binaries helps us accomplish that, as as potentially changes to core system interfaces, such as IPC) From that perspective, I think (1) is untenable. I think we should be much more concerned with portability of, e.g., capsules and a unified system call interface than with backwards compatibility. I don't think there is any advantage to (1) other than backwards compatibility. I think (2) is the answer. It follows the general idea in the MetaPtr PR (#4174) of converting naming things what they actually are. (3) I understand the expediency for a downstream port, but I don't see an advantage to this if we can do 2 (which I think we can). (4) Similar to 3. I think it would be _reasonable_, if we went with 3, to bump the major version number and redefine "success" to have the current value of successu32, such that it just happened to be the case that 2.0 binaries also turned out to work unmodified. -Amit

Oct. 3, 2024

Re: Discussion on u32

by Brad Campbell

At least some of the complexity results from system call arguments/return values which are sometimes arbitrary bit-limited values, and sometimes pointers. For example, memop 6 returns a pointer, whereas memop 7 returns a quantity. In contrast, allow only has pointers as arguments and returns pointers. One idea to consider is restructuring the system call interface to remove this ambiguity; arguments per system call are either pointers or values, but never both. - Brad On Wed, Oct 2, 2024 at 5:53 PM Lawrence Esswood via Devel < devel(a)lists.tockos.org> wrote: > I think this is a good discussion for Friday, but I wanted to send out an > advance email to give time for people to gather their thoughts in advance > > The current set of return codes include a type. e.g. calls like allow/brk > currently return SyscallReturnU32U32 or SyscallReturnU32. Clearly this is > not true for 64-bit platforms. Or for CHERI where the size of an address is > not the size of a pointer.I think there are a few options: > > 1. 1) They should return different codes on different platforms: > SyscallReturnu32 on 32-bit, SyscallReturnu64on 64-bit, > SyscallReturnMetaPtr on CHERI etc with some meta constructors to help > choose. Userspace should expect the same and be If-Defed appropriately. > 2. 2) They should return a new more specific code SyscallReturnPtr / > SyscallReturnUsize on all platforms, and user space should check for > just that one.e > 3. 3) These calls are already made type safe because of the wrappers > in libtock-c / libtock-rs. Just use SyscallSuccess for these built in > syscalls. Wrappers know what they are getting. > 4. 4) A true evil one. Option 3, but SyscallReturnu32 is the code for > legacy reasons. You know what you are actually getting. > > What I actually did on my fork was just 3 (because all these different > codes were getting in the way), but this does break compatibility unless > there is a matching change in libtock-c/libtock-rs and it would probably be > better not to do so. In fact, both 2 and 3 are breaking API changes. This > means that possibly we should prefer (1), simply for backwards > compatibility? This is roughly what the consttructor I added for > SyscallReturnUsize does: selects one of SyscallReturnu32 or > SyscallReturnu64 as appropriate. But possibly we will need a few more as > soon as CHERI comes into play.As for types for commands:Currently in the > kernel commands are:fn command(&self, cmd_num: usize, arg1: usize, arg2: > usize, appid: ProcessId) > Which is already usize-ified.In libtock-c: syscall_return_t > command(uint32_t driver, uint32_t command, size_t arg1, size_t arg2); > There is clearly a mismatch there on command number. I am of the mind that > the driver/command being u32 is fine, and args should really be usize. For > instance, a command might realistically take an offset to a previously > allowed buffer. This is usize, not u32.Upcalls: in the kernel were all > u32, I changed most of the arguments usize apart from data/fn pointer which > are MetaPtr. i.e., the type in C should be:typedef void > (subscribe_upcall)(size_t, size_t, size_t, void*);where currently it istypedef > void (subscribe_upcall)(int, int, int, void*);And I think its worth some > discussion whether people are happy with this. It means nothing for 32-bit > libtock-c because the ABI for passing usize is the same as int on 32-bit > platforms. In libtock-rs I have the patches ready to go to define the > upcall trait on both u32 and usize so it won't break any code there. > > Lawrence > > _______________________________________________ > Devel mailing list -- devel(a)lists.tockos.org > To unsubscribe send an email to devel-leave(a)lists.tockos.org >

Oct. 3, 2024

Discussion on u32

by Lawrence Esswood

I think this is a good discussion for Friday, but I wanted to send out an advance email to give time for people to gather their thoughts in advance The current set of return codes include a type. e.g. calls like allow/brk currently return SyscallReturnU32U32 or SyscallReturnU32. Clearly this is not true for 64-bit platforms. Or for CHERI where the size of an address is not the size of a pointer.I think there are a few options: 1. 1) They should return different codes on different platforms: SyscallReturnu32 on 32-bit, SyscallReturnu64on 64-bit, SyscallReturnMetaPtr on CHERI etc with some meta constructors to help choose. Userspace should expect the same and be If-Defed appropriately. 2. 2) They should return a new more specific code SyscallReturnPtr / SyscallReturnUsize on all platforms, and user space should check for just that one.e 3. 3) These calls are already made type safe because of the wrappers in libtock-c / libtock-rs. Just use SyscallSuccess for these built in syscalls. Wrappers know what they are getting. 4. 4) A true evil one. Option 3, but SyscallReturnu32 is the code for legacy reasons. You know what you are actually getting. What I actually did on my fork was just 3 (because all these different codes were getting in the way), but this does break compatibility unless there is a matching change in libtock-c/libtock-rs and it would probably be better not to do so. In fact, both 2 and 3 are breaking API changes. This means that possibly we should prefer (1), simply for backwards compatibility? This is roughly what the consttructor I added for SyscallReturnUsize does: selects one of SyscallReturnu32 or SyscallReturnu64 as appropriate. But possibly we will need a few more as soon as CHERI comes into play.As for types for commands:Currently in the kernel commands are:fn command(&self, cmd_num: usize, arg1: usize, arg2: usize, appid: ProcessId) Which is already usize-ified.In libtock-c: syscall_return_t command(uint32_t driver, uint32_t command, size_t arg1, size_t arg2); There is clearly a mismatch there on command number. I am of the mind that the driver/command being u32 is fine, and args should really be usize. For instance, a command might realistically take an offset to a previously allowed buffer. This is usize, not u32.Upcalls: in the kernel were all u32, I changed most of the arguments usize apart from data/fn pointer which are MetaPtr. i.e., the type in C should be:typedef void (subscribe_upcall)(size_t, size_t, size_t, void*);where currently it istypedef void (subscribe_upcall)(int, int, int, void*);And I think its worth some discussion whether people are happy with this. It means nothing for 32-bit libtock-c because the ABI for passing usize is the same as int on 32-bit platforms. In libtock-rs I have the patches ready to go to define the upcall trait on both u32 and usize so it won't break any code there. Lawrence

Oct. 2, 2024