writing a Linux and macOS Callback Timer.

[kdschlosser]

In a project I am working on there is a need to have a periodic callback timer and that timer needs to work on both Linux and also on macOS. The timer needs to be semi accurate +- 5ms or so. This functionality is not available in the "unix" port and I have been trying various ways to add it.

I cannot use create_timer from the stdlib because macOS doesn't have this function available. I have attempted to use 'dispatch_source_set_timer' for macOS but I did not have any luck with it working.

I am using SDL2 in my project as well and I realized that this has the functionality I am looking for. It works partially. It makes the callback to a C function when it is supposed to without any problems. The problems occur when calling the python callback function associated with the timer. Now I know that the C callback is called from a different thread and I have used the code located here

micropython/ports/unix/mpbthciport.c

Line 97 in 338df1a

while (mp_bluetooth_hci_active()) {

as an example of how to call a python function from a different thread. The callback to the python function is never made but the callback to the C function that schedules the python function callback is made.

I have tried calling the python function directly using mp_sched_schedule and also calling a C function like what is done in the link above and neither the C function nor the python function gets called. It is like the scheduler is not working like it should. No errors are seen at all.

can anyone shed any light as to why the function that is supplied the MicroPython scheduler never gets called?

[andrewleech]

Hi, I've needed a "mock timer" on unix a few times, generally to unit test code that normally runs on hardware.

Lately I've just used an async task for this as if you're careful with async functions to not block badly you should be able to easily get reponsive timing (especially on unix) but they are easy to break if a different async task blocks unexpectedly.

Previously I've written a unix Timer class that uses a background thread:

class Timer:
    def __init__(self, timer_id):
        self.id = timer_id
        self._callback = None
        self._freq = None
        self._stop = False

        import _thread
        _thread.start_new_thread(self._timer, ())

    def init(self, freq, callback):
        self._callback = callback
        self._freq = freq

    def deinit(self):
        self._stop = True

    def _timer(self):
        while True:
            if self._stop:
                break

            if self._freq is None:
                continue

            if self._callback:
                try:
                    micropython.schedule(lambda _: self._callback(), None)
                except Exception as ex:
                    import sys
                    sys.print_exception(ex)
            utime.sleep_ms(int(self._freq * 1000))

This isn't directly answering why your scheduled functions aren't working though, that's a tricky one. When referencing the C snipped you link, have you used those ENTER/EXIT defines and ensured they are actually defined properly to the atomic guard functions? Ensuring C thread safety is a big part of the complexity here.

Separately, what's your python application doing during this time? scheduled functions aren't pre-emptive as such, the scheduler is serviced during times when the VM is "idle". This is typically during a time.sleep, time.idle, async sleep, stream read etc. In C it's whenever MICROPY_EVENT_POLL_HOOK / mp_event_wait_*() is called.

[kdschlosser]

I consider macOS to be a separate port due to it not fully supporting the C stdlib. so it needs to have special code to handle it. While it is mostly like the unix port there are differences that make it easier to deal with by making it a separate thing. The lack of a Timer is one of them tho I have come up with a way to deal with it in a manner that should work with both unix and macOS...

Question about the main loop and the REPL. Can the loop be made to be non blocking and simply add a usleep for 1 millisecond to not end up with a spinning wheels? That seems like an easier solution in order to be able to work and it would not need a huge amount of additional code to handle it. You should be able to change the access to the file so it's a non blocking call.

[kdschlosser]

I have also gotten into the habit of not using looping code in the main thread. If I am going to have looping code I will run it in a separate thread. The reason why I do this is so the REPL stays available for a serial connection.

[andrewleech]

Ah, I always use asyncio and aiorepl so as to keep things non-blocking but still have repl available at all times, but I just find asyncio much safer than threads so I don't have to worry about thread-safe coding elements.

The "solution" to the repl/no-schedule issue on unix port should be to add the event poll hook (or newer mp_event_wait call) into

micropython/ports/unix/unix_mphal.c

Line 157 in 197becb

int mp_hal_stdin_rx_chr(void) {

similiar to at the bottom of

micropython/ports/rp2/mphalport.c

Line 79 in 197becb

int mp_hal_stdin_rx_chr(void) {

But currently the non-dupterm branch of that (which is probably the default) is blocking rather than looping so doesn't have anywhere convenient to just drop the line in. It wouldn't want a sleep of any actual length here else would cause performance issues, but just needs an event yield.

[kdschlosser]

the problem with asynchio is most people fail to use it for what it is indented for. Asynch IO. Asynchronous IO. It is not intended to be used for anything other that handling IO. It is not meant to be used as a thread alternative. asynchio has a HUGE amount of overhead especially on MicroPython. It is also slow as hell on MicroPython because a large portion of it is written in Python code.

So on LVGL land the only part that asynchio should be used on is the flush function and that is if and only if you are not using DMA memory with double buffering. If you are then asynchio is completely useless.

This change is for the LVGL binding I am working on. I didn't like the way the timer was done for the unix port and the timer didn't work for macOS. so I needed to write one for macOS I may as well write one that will also do unix as well. Which is what I did.

[kdschlosser]

I have to figure out how to make stdin none blocking with terminos. It was easy to do for the "prompt" REPL.

in main.c

char *mp_repl_get_ps3(void) {
    return "";
}


static int do_repl(void) {
    mp_hal_stdout_tx_str(MICROPY_BANNER_NAME_AND_VERSION);
    mp_hal_stdout_tx_str("; " MICROPY_BANNER_MACHINE);
    mp_hal_stdout_tx_str("\nUse Ctrl-D to exit, Ctrl-E for paste mode\n");

    #if MICROPY_USE_READLINE == 1

    // use MicroPython supplied readline

    vstr_t line;
    vstr_init(&line, 16);
    for (;;) {
        mp_hal_stdio_mode_raw();

    input_restart:
        vstr_reset(&line);
        int ret = readline(&line, mp_repl_get_ps1());
        mp_parse_input_kind_t parse_input_kind = MP_PARSE_SINGLE_INPUT;

        if (ret == CHAR_CTRL_C) {
            // cancel input
            mp_hal_stdout_tx_str("\r\n");
            goto input_restart;
        } else if (ret == CHAR_CTRL_D) {
            // EOF
            printf("\n");
            mp_hal_stdio_mode_orig();
            vstr_clear(&line);
            return 0;
        } else if (ret == CHAR_CTRL_E) {
            // paste mode
            mp_hal_stdout_tx_str("\npaste mode; Ctrl-C to cancel, Ctrl-D to finish\n=== ");
            vstr_reset(&line);
            for (;;) {
                char c = mp_hal_stdin_rx_chr();
                if (c == CHAR_CTRL_C) {
                    // cancel everything
                    mp_hal_stdout_tx_str("\n");
                    goto input_restart;
                } else if (c == CHAR_CTRL_D) {
                    // end of input
                    mp_hal_stdout_tx_str("\n");
                    break;
                } else {
                    // add char to buffer and echo
                    vstr_add_byte(&line, c);
                    if (c == '\r') {
                        mp_hal_stdout_tx_str("\n=== ");
                    } else {
                        mp_hal_stdout_tx_strn(&c, 1);
                    }
                }
            }
            parse_input_kind = MP_PARSE_FILE_INPUT;
        } else if (line.len == 0) {
            if (ret != 0) {
                printf("\n");
            }
            goto input_restart;
        } else {
            // got a line with non-zero length, see if it needs continuing
            while (mp_repl_continue_with_input(vstr_null_terminated_str(&line))) {
                vstr_add_byte(&line, '\n');
                ret = readline(&line, mp_repl_get_ps2());
                if (ret == CHAR_CTRL_C) {
                    // cancel everything
                    printf("\n");
                    goto input_restart;
                } else if (ret == CHAR_CTRL_D) {
                    // stop entering compound statement
                    break;
                }
            }
        }

        mp_hal_stdio_mode_orig();

        ret = execute_from_lexer(LEX_SRC_VSTR, &line, parse_input_kind, true);
        if (ret & FORCED_EXIT) {
            return ret;
        }
    }

    #else

    // use simple readline

    for (;;) {
        char *line = prompt((char *)mp_repl_get_ps1());
        if (line == NULL) {
            if (errno != EWOULDBLOCK) {
                return 0;
            } else {
                while (line == NULL && errno == EWOULDBLOCK) {
                    mp_handle_pending(true);
                    usleep(1000);
                    line = prompt(mp_repl_get_ps3());
                }
                if (line == NULL) return 0;
            }
        }
        while (mp_repl_continue_with_input(line)) {
            char *line2 = prompt((char *)mp_repl_get_ps2());
            if (line2 == NULL) {
                break;
            }
            char *line3 = strjoin(line, '\n', line2);
            free(line);
            free(line2);
            line = line3;
        }

        int ret = execute_from_lexer(LEX_SRC_STR, line, MP_PARSE_SINGLE_INPUT, true);
        free(line);
        if (ret & FORCED_EXIT) {
            return ret;
        }
    }

    #endif
}

in input.c

char *prompt(char *p) {
    int stdin_fd = fileno(stdin);
    int flags = fcntl(stdin_fd, F_GETFL);
    flags |= O_NONBLOCK;
    fcntl(stdin_fd, F_SETFL, flags);

    // simple read string
    static char buf[256];
    fputs(p, stdout);
    fflush(stdout);
    char *s = fgets(buf, sizeof(buf), stdin);
    if (!s) {
        return NULL;
    }
    int l = strlen(buf);
    if (buf[l - 1] == '\n') {
        buf[l - 1] = 0;
    } else {
        l++;
    }
    char *line = malloc(l);
    memcpy(line, buf, l);
    return line;
}

easy to do and it still has a proper exit of the buffer is NULL because I am checking the errno to see if the NULL is from stdin getting closed or if it is from no user input.