OpenBIOS August 2011

openbios@openbios.org

19 participants
88 discussions

Re: [OpenBIOS] Grand Unified Tracing
by Stefan Hajnoczi 23 Aug '11

23 Aug '11

On Tue, Aug 23, 2011 at 3:34 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: > On Tue, Aug 23, 2011 at 5:35 AM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: >> On Mon, Aug 22, 2011 at 9:59 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: >>> On Mon, Aug 22, 2011 at 8:23 PM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: >>>> On Mon, Aug 22, 2011 at 7:14 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: >>>>> OpenBIOS uses traditional DPRINTFs for debugging. I was considering >>>>> replacing those with tracepoints, which would output to serial device >>>>> or whatever DPRINTFs are using currently. This would not be extremely >>>>> useful by itself, except maybe that configuration for debugging would >>>>> be concentrated to single 'trace-events' file, but this would not be a >>>>> major improvement over current XML configuration. >>>>> >>>>> But developing this further, maybe OpenBIOS could also pass the >>>>> tracepoint data back to QEMU? Then all tracepoint data would be >>>>> synchronized, coherent and all gathered to single place. >>>>> >>>>> The implementation could be that fw_cfg would be used to pass >>>>> simpletrace style data. An offset should be added to event IDs and >>>>> data would then be output as usual. On OpenBIOS side, the >>>>> implementation would be pretty similar to current QEMU tracepoints but >>>>> in place of file output there would be fw_cfg output. >>>>> >>>>> Syntax for trace-events file should be augmented with include >>>>> directive, so that QEMU knows also OpenBIOS tracepoints. I think the >>>>> only change to simpletrace.py would be to parse this directive. >>>>> >>>>> Controlling OpenBIOS tracepoints from QEMU monitor would be cool too. >>>>> >>>>> Going even further, other targets like kernels could use something >>>>> similar, probably not using fw_cfg though. >>>>> >>>>> What do you think? >>>> >>>> Dhaval showed me a demo of unified host/guest Linux tracing last week. >>>> He is doing something similar except using a hypercall to pass a >>>> string to the host kernel. In his case kvm.ko handles the hypercall >>>> and qemu is not involved. >>> >>> So the events flow directly from guest kernel to host kernel? >> >> Yes. > > How is that used then, with dmesg? When the host processes the trace hypercall it simply forwards the trace event on to host Linux's trace. It currently logs the string from the guest and prefixes it with the guest program counter, I believe. You can dump out the trace using the usual Linux ftrace debugfs files. But the longer term goal, I believe, is to really integrate guest and host tracing so that host userspace (e.g. perf(1)) can pull out trace events from the guest and just sees them tagged as belonging to a particular qemu-kvm process. > >>>> One issue with QEMU tracing is that trace call sites are static. You >>>> need to compile in a trace_*() call, which means that there are two >>>> choices for how to tunnel OpenBIOS trace events: >>>> >>>> 1. Define a tunnel trace event: >>>> openbios_event(uint64_t event_id, uint64_t arg1, uint64_t arg2, ...) >>>> >>>> QEMU only has one trace event to tunnel OpenBIOS trace events. Then >>>> the host is unable to pretty-print OpenBIOS traces automatically and >>>> the max arguments becomes 6 - 1 (for the openbios_event tunnel event >>>> id). >>>> >>>> 2. Generate a switch statement to demultiplex trace events: >>>> void hypercall(uint64_t event_id, uint64_t arg1, ...) >>>> { >>>> /* This is auto-generated by tracetool */ >>>> switch (event_id) { >>>> case TRACE_EVENT_OPENBIOS_FOO: >>>> trace_openbios_foo(arg1, arg2, arg3); >>>> break; >>>> case TRACE_EVENT_OPENBIOS_BAR: >>>> trace_openbios_bar(arg1); >>>> break; >>>> ... >>>> } >>>> } >>>> >>>> With this approach the user can toggle trace events at runtime and it >>>> works out much nicer. >>> >>> Maybe I'm missing something, but why would we have to multiplex >>> anything? Since the trace IDs are 64 bits, we can easily allocate a >>> static range (starting from 0x8000000000000000) to non-native events >>> so that QEMU IDs and OpenBIOS IDs do not overlap. QEMU would add this >>> offset to IDs coming from fw_cfg. Then QEMU would just pass the data >>> (8*64 bit words) to tracing back end. There obviously something needs >>> to be changed so that OpenBIOS messages can be generated. For >>> simpletrace this should be easy, for stderr and other back ends that >>> may be more complicated. >> >> I meant that the interface generated by tracetool is a header file >> with trace_*() functions. If you want to support any tracing backend >> then you need to use this interface and cannot call simpletrace.c >> trace0(), trace1(), ... directly. >> >> Are you thinking of only supporting the simple trace backend? > > That would be simplest, then fw_cfg.c (or some intermediate entity) > could just call trace6(). > > But stderr would be useful too. I think that could be done by adding > code to read trace-events file to grab the format string. Then a > modified vfprintf would parse it but instead of va_arg(args, type) it > would just grab the uint64_t arg. Your approach would be much simpler. I see. I think we should avoid introducing dependencies on specific trace backends, so I would go for the generated switch statement above. That way you can even use DTrace/SystemTap or other backends. Also if we called traceX() directly we'd have to duplicate the logic to check whether the trace event is enabled/disabled, so it becomes a bit messy. Stefan

1 0

Re: [OpenBIOS] Grand Unified Tracing
by Blue Swirl 23 Aug '11

23 Aug '11

On Tue, Aug 23, 2011 at 5:35 AM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: > On Mon, Aug 22, 2011 at 9:59 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: >> On Mon, Aug 22, 2011 at 8:23 PM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: >>> On Mon, Aug 22, 2011 at 7:14 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: >>>> OpenBIOS uses traditional DPRINTFs for debugging. I was considering >>>> replacing those with tracepoints, which would output to serial device >>>> or whatever DPRINTFs are using currently. This would not be extremely >>>> useful by itself, except maybe that configuration for debugging would >>>> be concentrated to single 'trace-events' file, but this would not be a >>>> major improvement over current XML configuration. >>>> >>>> But developing this further, maybe OpenBIOS could also pass the >>>> tracepoint data back to QEMU? Then all tracepoint data would be >>>> synchronized, coherent and all gathered to single place. >>>> >>>> The implementation could be that fw_cfg would be used to pass >>>> simpletrace style data. An offset should be added to event IDs and >>>> data would then be output as usual. On OpenBIOS side, the >>>> implementation would be pretty similar to current QEMU tracepoints but >>>> in place of file output there would be fw_cfg output. >>>> >>>> Syntax for trace-events file should be augmented with include >>>> directive, so that QEMU knows also OpenBIOS tracepoints. I think the >>>> only change to simpletrace.py would be to parse this directive. >>>> >>>> Controlling OpenBIOS tracepoints from QEMU monitor would be cool too. >>>> >>>> Going even further, other targets like kernels could use something >>>> similar, probably not using fw_cfg though. >>>> >>>> What do you think? >>> >>> Dhaval showed me a demo of unified host/guest Linux tracing last week. >>> He is doing something similar except using a hypercall to pass a >>> string to the host kernel. In his case kvm.ko handles the hypercall >>> and qemu is not involved. >> >> So the events flow directly from guest kernel to host kernel? > > Yes. How is that used then, with dmesg? >>> One issue with QEMU tracing is that trace call sites are static. You >>> need to compile in a trace_*() call, which means that there are two >>> choices for how to tunnel OpenBIOS trace events: >>> >>> 1. Define a tunnel trace event: >>> openbios_event(uint64_t event_id, uint64_t arg1, uint64_t arg2, ...) >>> >>> QEMU only has one trace event to tunnel OpenBIOS trace events. Then >>> the host is unable to pretty-print OpenBIOS traces automatically and >>> the max arguments becomes 6 - 1 (for the openbios_event tunnel event >>> id). >>> >>> 2. Generate a switch statement to demultiplex trace events: >>> void hypercall(uint64_t event_id, uint64_t arg1, ...) >>> { >>> /* This is auto-generated by tracetool */ >>> switch (event_id) { >>> case TRACE_EVENT_OPENBIOS_FOO: >>> trace_openbios_foo(arg1, arg2, arg3); >>> break; >>> case TRACE_EVENT_OPENBIOS_BAR: >>> trace_openbios_bar(arg1); >>> break; >>> ... >>> } >>> } >>> >>> With this approach the user can toggle trace events at runtime and it >>> works out much nicer. >> >> Maybe I'm missing something, but why would we have to multiplex >> anything? Since the trace IDs are 64 bits, we can easily allocate a >> static range (starting from 0x8000000000000000) to non-native events >> so that QEMU IDs and OpenBIOS IDs do not overlap. QEMU would add this >> offset to IDs coming from fw_cfg. Then QEMU would just pass the data >> (8*64 bit words) to tracing back end. There obviously something needs >> to be changed so that OpenBIOS messages can be generated. For >> simpletrace this should be easy, for stderr and other back ends that >> may be more complicated. > > I meant that the interface generated by tracetool is a header file > with trace_*() functions. If you want to support any tracing backend > then you need to use this interface and cannot call simpletrace.c > trace0(), trace1(), ... directly. > > Are you thinking of only supporting the simple trace backend? That would be simplest, then fw_cfg.c (or some intermediate entity) could just call trace6(). But stderr would be useful too. I think that could be done by adding code to read trace-events file to grab the format string. Then a modified vfprintf would parse it but instead of va_arg(args, type) it would just grab the uint64_t arg. Your approach would be much simpler.

1 0

Re: [OpenBIOS] [PATCH 10/10] Adding filll Forth primitive (v2)
by Segher Boessenkool 23 Aug '11

23 Aug '11

>> This isn't going to work with -fstrict-aliasing, but I guess >> openbios has that disabled anyway? > I don't think it breaks strict aliasing rules. A cell is defined as a > 32bit integer on 32bit architectures and this code will only be > compiled on ppc32. I just wrote it as a uint32_t for clarity. It breaks aliasing rules if whatever reads the data you write does not read it as a type compatible with the type you wrote it as. If everything reads it as a 32-bit int, a cell, or a character type, it is fine; if not, not. Don't worry about it too much, I'm sure there are hundreds of other places in openbios that do icky things like this. It's in the nature of emulators/ interpreters :-) Segher

1 0

Re: [OpenBIOS] [PATCH 10/10] Adding filll Forth primitive (v2)
by Andreas Färber 23 Aug '11

23 Aug '11

Am 23.08.2011 um 04:45 schrieb William Hahne: > On Mon, Aug 22, 2011 at 7:19 AM, Segher Boessenkool > <segher(a)kernel.crashing.org> wrote: >> This isn't going to work with -fstrict-aliasing, but I guess >> openbios has that disabled anyway? >> > > I don't think it breaks strict aliasing rules. A cell is defined as a > 32bit integer on 32bit architectures and this code will only be > compiled on ppc32. ppc64? Andreas > I just wrote it as a uint32_t for clarity. > > William Hahne

1 0

Re: [OpenBIOS] Grand Unified Tracing
by Stefan Hajnoczi 23 Aug '11

23 Aug '11

On Mon, Aug 22, 2011 at 9:59 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: > On Mon, Aug 22, 2011 at 8:23 PM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: >> On Mon, Aug 22, 2011 at 7:14 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: >>> OpenBIOS uses traditional DPRINTFs for debugging. I was considering >>> replacing those with tracepoints, which would output to serial device >>> or whatever DPRINTFs are using currently. This would not be extremely >>> useful by itself, except maybe that configuration for debugging would >>> be concentrated to single 'trace-events' file, but this would not be a >>> major improvement over current XML configuration. >>> >>> But developing this further, maybe OpenBIOS could also pass the >>> tracepoint data back to QEMU? Then all tracepoint data would be >>> synchronized, coherent and all gathered to single place. >>> >>> The implementation could be that fw_cfg would be used to pass >>> simpletrace style data. An offset should be added to event IDs and >>> data would then be output as usual. On OpenBIOS side, the >>> implementation would be pretty similar to current QEMU tracepoints but >>> in place of file output there would be fw_cfg output. >>> >>> Syntax for trace-events file should be augmented with include >>> directive, so that QEMU knows also OpenBIOS tracepoints. I think the >>> only change to simpletrace.py would be to parse this directive. >>> >>> Controlling OpenBIOS tracepoints from QEMU monitor would be cool too. >>> >>> Going even further, other targets like kernels could use something >>> similar, probably not using fw_cfg though. >>> >>> What do you think? >> >> Dhaval showed me a demo of unified host/guest Linux tracing last week. >> He is doing something similar except using a hypercall to pass a >> string to the host kernel. In his case kvm.ko handles the hypercall >> and qemu is not involved. > > So the events flow directly from guest kernel to host kernel? Yes. > >> One issue with QEMU tracing is that trace call sites are static. You >> need to compile in a trace_*() call, which means that there are two >> choices for how to tunnel OpenBIOS trace events: >> >> 1. Define a tunnel trace event: >> openbios_event(uint64_t event_id, uint64_t arg1, uint64_t arg2, ...) >> >> QEMU only has one trace event to tunnel OpenBIOS trace events. Then >> the host is unable to pretty-print OpenBIOS traces automatically and >> the max arguments becomes 6 - 1 (for the openbios_event tunnel event >> id). >> >> 2. Generate a switch statement to demultiplex trace events: >> void hypercall(uint64_t event_id, uint64_t arg1, ...) >> { >> /* This is auto-generated by tracetool */ >> switch (event_id) { >> case TRACE_EVENT_OPENBIOS_FOO: >> trace_openbios_foo(arg1, arg2, arg3); >> break; >> case TRACE_EVENT_OPENBIOS_BAR: >> trace_openbios_bar(arg1); >> break; >> ... >> } >> } >> >> With this approach the user can toggle trace events at runtime and it >> works out much nicer. > > Maybe I'm missing something, but why would we have to multiplex > anything? Since the trace IDs are 64 bits, we can easily allocate a > static range (starting from 0x8000000000000000) to non-native events > so that QEMU IDs and OpenBIOS IDs do not overlap. QEMU would add this > offset to IDs coming from fw_cfg. Then QEMU would just pass the data > (8*64 bit words) to tracing back end. There obviously something needs > to be changed so that OpenBIOS messages can be generated. For > simpletrace this should be easy, for stderr and other back ends that > may be more complicated. I meant that the interface generated by tracetool is a header file with trace_*() functions. If you want to support any tracing backend then you need to use this interface and cannot call simpletrace.c trace0(), trace1(), ... directly. Are you thinking of only supporting the simple trace backend? Stefan

1 0

Re: [OpenBIOS] [PATCH 10/10] Adding filll Forth primitive (v2)
by Segher Boessenkool 23 Aug '11

23 Aug '11

> +static void > +filll(void) > +{ > + uint32_t value = POP(); > + int count = POP(); > + uint32_t *dest = (uint32_t*)cell2pointer(POP()); > + > + int i; > + for (i = count; i > 0; i -= sizeof(uint32_t)) { > + *dest++ = value; > + } > +} This isn't going to work with -fstrict-aliasing, but I guess openbios has that disabled anyway? Segher

2 1

Re: [OpenBIOS] Grand Unified Tracing
by Blue Swirl 22 Aug '11

22 Aug '11

On Mon, Aug 22, 2011 at 8:23 PM, Stefan Hajnoczi <stefanha(a)gmail.com> wrote: > On Mon, Aug 22, 2011 at 7:14 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: >> OpenBIOS uses traditional DPRINTFs for debugging. I was considering >> replacing those with tracepoints, which would output to serial device >> or whatever DPRINTFs are using currently. This would not be extremely >> useful by itself, except maybe that configuration for debugging would >> be concentrated to single 'trace-events' file, but this would not be a >> major improvement over current XML configuration. >> >> But developing this further, maybe OpenBIOS could also pass the >> tracepoint data back to QEMU? Then all tracepoint data would be >> synchronized, coherent and all gathered to single place. >> >> The implementation could be that fw_cfg would be used to pass >> simpletrace style data. An offset should be added to event IDs and >> data would then be output as usual. On OpenBIOS side, the >> implementation would be pretty similar to current QEMU tracepoints but >> in place of file output there would be fw_cfg output. >> >> Syntax for trace-events file should be augmented with include >> directive, so that QEMU knows also OpenBIOS tracepoints. I think the >> only change to simpletrace.py would be to parse this directive. >> >> Controlling OpenBIOS tracepoints from QEMU monitor would be cool too. >> >> Going even further, other targets like kernels could use something >> similar, probably not using fw_cfg though. >> >> What do you think? > > Dhaval showed me a demo of unified host/guest Linux tracing last week. > He is doing something similar except using a hypercall to pass a > string to the host kernel. In his case kvm.ko handles the hypercall > and qemu is not involved. So the events flow directly from guest kernel to host kernel? > One issue with QEMU tracing is that trace call sites are static. You > need to compile in a trace_*() call, which means that there are two > choices for how to tunnel OpenBIOS trace events: > > 1. Define a tunnel trace event: > openbios_event(uint64_t event_id, uint64_t arg1, uint64_t arg2, ...) > > QEMU only has one trace event to tunnel OpenBIOS trace events. Then > the host is unable to pretty-print OpenBIOS traces automatically and > the max arguments becomes 6 - 1 (for the openbios_event tunnel event > id). > > 2. Generate a switch statement to demultiplex trace events: > void hypercall(uint64_t event_id, uint64_t arg1, ...) > { > /* This is auto-generated by tracetool */ > switch (event_id) { > case TRACE_EVENT_OPENBIOS_FOO: > trace_openbios_foo(arg1, arg2, arg3); > break; > case TRACE_EVENT_OPENBIOS_BAR: > trace_openbios_bar(arg1); > break; > ... > } > } > > With this approach the user can toggle trace events at runtime and it > works out much nicer. Maybe I'm missing something, but why would we have to multiplex anything? Since the trace IDs are 64 bits, we can easily allocate a static range (starting from 0x8000000000000000) to non-native events so that QEMU IDs and OpenBIOS IDs do not overlap. QEMU would add this offset to IDs coming from fw_cfg. Then QEMU would just pass the data (8*64 bit words) to tracing back end. There obviously something needs to be changed so that OpenBIOS messages can be generated. For simpletrace this should be easy, for stderr and other back ends that may be more complicated. > Remember that QEMU tracing also supports DTrace (SystemTap) and LTTng > Userspace Tracer. Simpletrace would be the interface between the > guest and the host for passing trace records. In Dhaval's case we > were talking about passing binary Linux tracing events to the host and > quickly realized there are host/guest ABI compatibility issues to > consider. For OpenBIOS debugging you may be able to get away with > demanding that the OpenBIOS is built for this particular QEMU binary > though :). I'd expect that OpenBIOS needs to be compiled so that it can pass the data to QEMU in a format that both understand and that its trace-events file should be #included by QEMU's trace-events file. This binds both together. In case simpletrace backend is used, I think any QEMU would do except monitor needs event names.

1 0

Re: [OpenBIOS] Grand Unified Tracing
by Stefan Hajnoczi 22 Aug '11

22 Aug '11

On Mon, Aug 22, 2011 at 7:14 PM, Blue Swirl <blauwirbel(a)gmail.com> wrote: > OpenBIOS uses traditional DPRINTFs for debugging. I was considering > replacing those with tracepoints, which would output to serial device > or whatever DPRINTFs are using currently. This would not be extremely > useful by itself, except maybe that configuration for debugging would > be concentrated to single 'trace-events' file, but this would not be a > major improvement over current XML configuration. > > But developing this further, maybe OpenBIOS could also pass the > tracepoint data back to QEMU? Then all tracepoint data would be > synchronized, coherent and all gathered to single place. > > The implementation could be that fw_cfg would be used to pass > simpletrace style data. An offset should be added to event IDs and > data would then be output as usual. On OpenBIOS side, the > implementation would be pretty similar to current QEMU tracepoints but > in place of file output there would be fw_cfg output. > > Syntax for trace-events file should be augmented with include > directive, so that QEMU knows also OpenBIOS tracepoints. I think the > only change to simpletrace.py would be to parse this directive. > > Controlling OpenBIOS tracepoints from QEMU monitor would be cool too. > > Going even further, other targets like kernels could use something > similar, probably not using fw_cfg though. > > What do you think? Dhaval showed me a demo of unified host/guest Linux tracing last week. He is doing something similar except using a hypercall to pass a string to the host kernel. In his case kvm.ko handles the hypercall and qemu is not involved. One issue with QEMU tracing is that trace call sites are static. You need to compile in a trace_*() call, which means that there are two choices for how to tunnel OpenBIOS trace events: 1. Define a tunnel trace event: openbios_event(uint64_t event_id, uint64_t arg1, uint64_t arg2, ...) QEMU only has one trace event to tunnel OpenBIOS trace events. Then the host is unable to pretty-print OpenBIOS traces automatically and the max arguments becomes 6 - 1 (for the openbios_event tunnel event id). 2. Generate a switch statement to demultiplex trace events: void hypercall(uint64_t event_id, uint64_t arg1, ...) { /* This is auto-generated by tracetool */ switch (event_id) { case TRACE_EVENT_OPENBIOS_FOO: trace_openbios_foo(arg1, arg2, arg3); break; case TRACE_EVENT_OPENBIOS_BAR: trace_openbios_bar(arg1); break; ... } } With this approach the user can toggle trace events at runtime and it works out much nicer. Remember that QEMU tracing also supports DTrace (SystemTap) and LTTng Userspace Tracer. Simpletrace would be the interface between the guest and the host for passing trace records. In Dhaval's case we were talking about passing binary Linux tracing events to the host and quickly realized there are host/guest ABI compatibility issues to consider. For OpenBIOS debugging you may be able to get away with demanding that the OpenBIOS is built for this particular QEMU binary though :). Stefan

1 0

Grand Unified Tracing
by Blue Swirl 22 Aug '11

22 Aug '11

OpenBIOS uses traditional DPRINTFs for debugging. I was considering replacing those with tracepoints, which would output to serial device or whatever DPRINTFs are using currently. This would not be extremely useful by itself, except maybe that configuration for debugging would be concentrated to single 'trace-events' file, but this would not be a major improvement over current XML configuration. But developing this further, maybe OpenBIOS could also pass the tracepoint data back to QEMU? Then all tracepoint data would be synchronized, coherent and all gathered to single place. The implementation could be that fw_cfg would be used to pass simpletrace style data. An offset should be added to event IDs and data would then be output as usual. On OpenBIOS side, the implementation would be pretty similar to current QEMU tracepoints but in place of file output there would be fw_cfg output. Syntax for trace-events file should be augmented with include directive, so that QEMU knows also OpenBIOS tracepoints. I think the only change to simpletrace.py would be to parse this directive. Controlling OpenBIOS tracepoints from QEMU monitor would be cool too. Going even further, other targets like kernels could use something similar, probably not using fw_cfg though. What do you think?

1 0

[PATCH] sparc32: add sbus-intr>cpu for sun4m
by Bob Breuer 22 Aug '11

22 Aug '11

Add an implementation of sbus-intr>cpu for sun4m to map sbus interrupt levels (1 to 7) into cpu interrupts, and make use of it for bpp. Assumes false is the same as -1. Signed-off-by: Bob Breuer <breuerr(a)mc.net> --- Index: arch/sparc32/tree.fs =================================================================== --- arch/sparc32/tree.fs (revision 1046) +++ arch/sparc32/tree.fs (working copy) @@ -1,4 +1,16 @@ +\ map sbus intr level to cpu intr +: sun4m-sbus-intr>cpu + \ map 1 2 3 4 5 6 7 other + \ to 32 33 35 37 39 3b 3d 0 + dup 1 7 between if + 2* dup 2 <> + h# 30 + + else + drop 0 + then + ; +['] sun4m-sbus-intr>cpu to sbus-intr>cpu + " /" find-device 2 encode-int " #address-cells" property 1 encode-int " #size-cells" property @@ -61,7 +73,7 @@ new-device " SUNW,bpp" device-name h# 4 encode-int h# 0c800000 encode-int encode+ h# 0000001c encode-int encode+ " reg" property - h# 33 encode-int 0 encode-int encode+ " intr" property + h# 2 sbus-intr>cpu encode-int 0 encode-int encode+ " intr" property finish-device [THEN]

3 5

← Newer
1
2
3
4
5
6
7
8
9
Older →