Added on_gain_focus, on_lose_focus, on_show & on_hide handlers on toga.Window

[proneon267]

Implements the APIs described in #2009.

on_gain_focus, on_lose_focus, on_show& on_hide handles are available both as properties and also as initialization parameters in toga.Window.

Only tested on WinForms and gtk. This will take sometime to complete for all backends.

Fixes #2009

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[freakboy3742]

Thanks for the contribution - this is another feature that is going to hit up against #2058 (and #2075); as such, I'm hesitant to merge it without tests.

I'm also hesitant because it doesn't currently have a Cocoa implementation (that should be easy enough, but it's worth flagging); and it's not 100% obvious how this would behave on mobile. My immediate reaction (and the one raised in #2006) is that gain/lose focus might link into application lifecycle hooks (so when the app comes into the foreground, that's the "gain focus" event for both the window and the app), but there's an open question of how those signals interact with tablet platforms that allow split-screen and other multi-app modes. This is an area where some additional design is required.

[proneon267]

Yes, I agree with you. This should not be merged currently. I will need some time to write the implementation for other backends. Like #1930, I will wait until the audits are merged, and will write the tests thereafter.

As for additional design for tablet modes, I agree with you. I will research more about it and will discuss with you while implementing for mobile platforms.

[proneon267]

@mhsmith I need your guidance on android side. According to: https://developer.android.com/guide/components/activities/intro-activities#onpause

The system calls onPause() when the activity loses focus and enters a Paused state. This state occurs when, for example, the user taps the Back or Recents button.

On the Emulator(Android 12):

When starting the app, the following events are triggered: onCreate()->onStart()->onResume() . But, when I press either the Back or Recents button, onPause() event is not triggered. No other events are triggered.

When I select the app by pressing the Recents button, only onStart()->onResume() are triggered. onRestart() is not triggered.

D/MainActivity: onStart() start
I/python.stdout: Toga app: onStart
D/MainActivity: onStart() complete
D/MainActivity: onResume() start
I/python.stdout: Toga app: onResume
D/MainActivity: onResume() complete
I/OpenGLRenderer: Davey! duration=18074ms; Flags=1, FrameTimelineVsyncId=10773, IntendedVsync=2419137327998, Vsync=2419137327998, InputEventId=0, HandleInputStart=2419139817470, AnimationStart=2419139847710, PerformTraversalsStart=2419139888290, DrawStart=2419223580770, FrameDeadline=2419170661330, FrameInterval=2419139788630, FrameStartTime=16666666, SyncQueued=2419225272400, SyncStart=2419226783620, IssueDrawCommandsStart=2419227470230, SwapBuffers=2419249275450, FrameCompleted=2437213784760, DequeueBufferDuration=21300, QueueBufferDuration=360800, GpuCompleted=2437213784760, SwapBuffersCompleted=2419251729810, DisplayPresentTime=0,

When I press the home button, neither onPause() or onStop() events are triggered. No other events are triggered.

On a Physical Device(Android 13):

When starting the app, the following events are triggered: onCreate()->onStart()->onResume() . But, when I press either the Back or Recents button, onPause() event is not triggered. Instead the following events are triggered:

D/VRI[MainActivity]: onFocusEvent false
D/VRI[MainActivity]: dispatchAppVisibility visible:false
D/BufferQueueProducer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:1,p:21956,c:21956) disconnect: api 1
D/BufferQueueConsumer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:0,p:-1,c:21956) disconnect
D/VRI[MainActivity]: setWindowStopped stopped:true
D/ActivityThread: do gfx trim 40 success

When I select the app by pressing the Recents button, only onStart()->onResume() are triggered. onRestart() is not triggered. Instead the following events are triggered:

D/VRI[MainActivity]: dispatchAppVisibility visible:true
D/VRI[MainActivity]: setWindowStopped stopped:false
D/MainActivity: onStart() start
I/python.stdout: Toga app: onStart
D/MainActivity: onStart() complete
D/MainActivity: onResume() start
I/python.stdout: Toga app: onResume
D/MainActivity: onResume() complete
I/Quality : Skipped: false 0 cost 5.253646 refreshRate 8332212 bit true processName com.example.helloworld
D/BufferQueueConsumer: [](id:55c400000001,api:0,p:-1,c:21956) connect: controlledByApp=false
E/IPCThreadState: attemptIncStrongHandle(57): Not supported
D/BufferQueueProducer: [VRI[MainActivity]#1(BLAST Consumer)1](id:55c400000001,api:1,p:21956,c:21956) connect: api=1 producerControlledByApp=true
D/VRI[MainActivity]: registerCallbacksForSync syncBuffer=false
D/VRI[MainActivity]: Received frameCommittedCallback lastAttemptedDrawFrameNum=1 didProduceBuffer=true syncBuffer=false
D/VRI[MainActivity]: draw finished.
D/VRI[MainActivity]: onFocusEvent true

When I press the home button, neither onPause() or onStop() events are triggered. But, the following events are triggered:

D/VRI[MainActivity]: onFocusEvent false
D/VRI[MainActivity]: dispatchAppVisibility visible:false
D/BufferQueueProducer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:1,p:21956,c:21956) disconnect: api 1
D/BufferQueueConsumer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:0,p:-1,c:21956) disconnect
D/VRI[MainActivity]: setWindowStopped stopped:true
D/ActivityThread: do gfx trim 40 success

As, you can see, onFocusEvent, dispatchAppVisibility visible and setWindowStopped stopped show consistent behaviors.

Why are the documented Activity lifecycle events not being triggered as per the documentation?

[proneon267]

Regarding the gain/lose focus on mobile platforms like android:

From my testing, the app will lose focus when either the Home or Recent App List buttons are pressed. On pressing the back button, the app also loses focus and sends the user to the home screen.

In split screen mode (like dual app mode), suppose there are two apps A and B. App A will gain focus when the user touches the A app's screen. The focus is lost when the user touches B app's screen or interacts with the system launcher or presses the Home or Recent App List Button.

In floating window mode, the app will gain focus when the user touches the app's screen. The focus is lost when the user touches anything outside the app, like interacting with the system launcher or another app.

In iOS like the cocoa, there exists UIApplicationDelegate, which has methods like: applicationDidBecomeActive and applicationWillResignActive

But, there needs to be another handler to differentiate between the states when (the app is not visible to the user & is not receiving inputs) and (when the app is visible to the user & is receiving inputs).

Hence, I would like to propose other additional handlers, on_background/foreground to call the handler when the app is in background/foreground and not visible/ visible to the user.

What do you guys think?

Also, without confirmation from @mhsmith regarding the Activity life events triggering behavior, I cannot proceed with the android implementation. Hence, I was thinking about working on the iOS implementation first.

[mhsmith]

When I press the home button, neither onPause() or onStop() events are triggered.

That's because those methods aren't included in the Android template, either in IPythonApp or in MainActivity. They should be added to both places, but ideally with an emptydefault implementation in order not to break any third-party implementations of IPythonApp. And all the existing methods should become default as well.

In split screen mode (like dual app mode), suppose there are two apps A and B.

See this page for how this is notified on Android.

But, there needs to be another handler to differentiate between the states when (the app is not visible to the user & is not receiving inputs) and (when the app is visible to the user & is receiving inputs).

Every API has a maintenance and testing cost, so I'd prefer not to add additional events unless there's a clear need for them, especially if they're only applicable to certain platforms.

[proneon267]

Thank you for helping. I will add default implementations for the remaining methods in the Android template and will submit a PR there after getting a stable behavior.

I agree with you that additional events will incur more maintenance. I feel that the on_background and on_foreground events are mostly needed for mobile platforms. But there is a need for distinction between [not visible state] and [visible but not receiving inputs state] of an app.

For example, the app should be put to a sleep mode(not updating the layout or text) when it is in background or [not visible state].
But when it is in a [visible but not receiving inputs state](like in a multi app screen mode), the app should update the layout or text, so that the user can get the latest updated information.

What do you think?

[proneon267]

I have tested android implementation both on a physical device and on the emulator.

I have submitted a PR at beeware/briefcase-android-gradle-template#69 so that the app focus event can be detected.

[proneon267]

Completed implementations of all the platforms and also added a test in the window example app.

I will write the tests after the audits are merged. But I think this PR is ready for a review.

Also, the CI android testbed is failing on its own for some reason.

[freakboy3742]

What you've done here looks like a good pass at implementing the API as proposed in #2009; however, I think we're hitting an area where we need more design before we proceed.

The detail you've dug up as part of the Android implementation has opened a bunch of design questions about what "focus" even means at the App/Window level. What are we actually trying to achieve with these signal handlers? Is "visibility" a better metaphor than "focus" in this case? Is there any use case for a literal "focus" event on a window? Do we need to differentiate between an app that is "visible", but isn't currently accepting input events, and an app that isn't accepting input events? How does the rest of the app lifecycle map into these events (on all platforms)?

Rather than pressing forward with an implementation, I think we need to step back and come up with a consistent design for these app/window lifecycle events, and work out how they map onto all the platforms we're targeting.

[proneon267]

Researching some more on the topic, it seems like we need 3 categories of events:

• Input Focus -> [Receiving Inputs] or [Not Receiving Inputs]
• Visibility -----> [Visible to User] or [Not Visible to User]
• Hover

The following are the states associated with the event categories mentioned above and their implications for other event categories:

Input Focus ---> [Receiving Inputs]
  |                               | ->[Visible to User]
  |
  |---------------> [Not Receiving Inputs]
                                 | ->[Visible to User] or
                                 | ->[Not Visible to User]

Visiblity -> [Visible to User]
  |                      | ->[Receiving Inputs] or
  |                      | ->[Not Receiving Inputs]
  |
  |---------> [Not Visible to User]
                         | ->[Not Receiving Inputs]

Hover ->[Interacting with a Pointing Device or Mouse] & [Visible to User] & [Not Receiving Inputs]

Use Cases:

• The use case for Input Focus might be to show a highlight effect indicating to the user that it is receiving input.

• The use case for Visibility might be to:

  • Put the app to a sleep mode(not updating the layout or text) when it is [Not Visible to User].
  • Update the layout or text when it is [Visible to User], so that the user can get the latest information.

• So far, I am not sure what the use case for Hover might be. Maybe showing effects that would imply to the user, clicking on it, will make it to receive input focus.

Who should have which event categories:

• Toga.Window should have:
  • Input Focus: Since, multiple windows may be present, hence each window should trigger Input Focus event when they receive/lose input focus.
  • Visibility: Same reason as above
  • Hover: Same reason as above
• Toga.App doesn't really need any of the event categories since it can just iterate through the Toga.App.windows list and add handler methods to the event handlers of the windows. But, we can add them if it would make intuitive sense for the user.

APIs are not much of a problem as the available platform APIs can be properly mapped onto the above described event categories.

[proneon267]

Ok, I have removed the gtk's visibility events(i.e., on_show() and on_hide()) triggering implementations from this PR, and have updated the tests accordingly.

So, both the focus events i.e., on_gain_focus & on_lose_focus are implemented and tested on all platforms(includiing gtk). However, both the visibility events i.e., on_show and on_hide are implemented and tested on all the platforms, except on gtk(I will implement them in a separate PR for gtk).

Also, another issue is testing of the events on mobile platforms, where the app cannot be sent to background and brought forward, switch between apps, or switch to the "App switcher" screen. All of these are required to automate the testing of focus and visibility events on the mobile platforms. Since, these automated tests are currently not possible, hence I have manually tested them on a physical Android 14 device, on the Android emulator and on the iOS simulator.

[freakboy3742]

I started doing a full review; but two things

1. There's a couple of outstanding comments from my previous review pass
2. It seems like there's some bigger picture API design issues that still need to be resolved - in particular, the interaction of minimise and hide.

The interaction between Show/Hide and Minimize is clearly a lot more complex than just #3105 - there's also the inconsistent behavior on macOS. On that basis, it seems like it might make sense to make the two mutually exclusive - state changes are already prevented on non-visible windows, so it doesn't seem unreasonable that visibility changes should be prevented on windows that aren't in a state that can accept them. What does it mean to hide a window in presentation mode? What does it mean to show a minimized window? At the very least, the test suite indicates that there are inconsistencies; I'd argue that the best approach for now would be to prohibit all those interactions, and add them in the future if it makes sense to do so.

[freakboy3742]

It's somewhat frustrating to be asked for a review when previous review comments have been left without a response.

[freakboy3742]

So - this is simultaneously extremely specific, and extremely generic.

This assertion method is in the top level testing utilities... but can only be used for on/off versions of 2 very specific events. It allows the developer to check 1 specific event has been triggered... but then asserts that completely unrelated events haven't been triggered.

What I was suggesting was a very literal assert_window_gain_focus method that asserts on_focus is fired, on_hide isn't, and then resets both mocks. It's then clear what the assertion is asserting, and requires almost no logic in the assertion.

Yes, this means there's a second assert_window_lose_focus, and another pair for show/hide... but that's easy code to write, and easy code to validate - but it dramatically simplifies the test case making it clear what is being tested, and ensures the mocks are consistently reset. As I've noted in previous reviews on other PRs - test code should avoid being smart, because if test code is smart, it can go wrong, and therefore the test code needs to be tested.

[proneon267]

I have removed the assert_window_event_triggered assertion utility and replaced it with 4 separate assertion utilities: assert_window_gain_focus, assert_window_lose_focus, assert_window_on_hide, assert_window_on_show.

[freakboy3742]

The behavior of show() on a minimized window should either be consistent on all platforms, or show()/hide() should raise an error if invoked on a minimized window. We shouldn't need a test workaround here - it's an area where we're in control of the API design.

If we're going to make the behavior consistent, I'd argue the Cocoa behavior is the better interpretation - if someone is calling "show()", they're indicating they want the window to be visible, not just minimized. But I suspect the "show/hide is an error if minimized" approach might be better as it's more explicit about intent.

[proneon267]

f we're going to make the behavior consistent, I'd argue the Cocoa behavior is the better interpretation - if someone is calling "show()", they're indicating they want the window to be visible, not just minimized. But I suspect the "show/hide is an error if minimized" approach might be better as it's more explicit about intent.

The inconsistency is that on macOS, calling window.show() unhides the window and also deminiaturizes it. This is different from the behavior of other platforms, where calling window.show() on a MINIMIZED window only unhides it(i.e., only shows the window icon in taskbar or app tray, and doesn't deminiaturize it).

But this inconsistency is only in the calling of window.show() on a previously MINIMIZED and hidden window, not on calling of window.hide() on a previously MINIMIZED window.

Since, hiding a MINIMIZED window is supported by every platfform, so disabling show/hide on a MINIMIZED window, seems like a bit too restrictive. So, I think the best approach would be to produce the Cocoa's interpretation of window.show() as the default behavior on all platforms. This would help to eliminate the inconsistency in the test suite that you have described.

[freakboy3742]

Since, hiding a MINIMIZED window is supported by every platfform, so disabling show/hide on a MINIMIZED window, seems like a bit too restrictive. So, I think the best approach would be to produce the Cocoa's interpretation of window.show() as the default behavior on all platforms. This would help to eliminate the inconsistency in the test suite that you have described.

I can't argue it wouldn't be restrictive to disable show/hide on MINIMIZED. The question is whether that's a better option than the alternative - whether implementing macOS's behavior on GTK and Windows is problematic.

The major argument again using macOS's behaviour is that it couples two otherwise unrelated features "show/hide" and state are now closely related features. I guess there's an argument that they already are, in that you can't change the state of a hidden window; but implementing macOS's behavior doubles down on that relationship in a way that I'm not sure is helpful.

Another argument against macOS's behavior is a question I raised in my previous comment - what about other window states? What does it mean to hide a window in presentation mode? In fullscreen mode? We already restrict changing window state on non-visible windows; do we need to restrict changing visibility on some window states? This will require some more experimentation to determine what is even possible.

Another complication that came up this morning in a conversation with @mhsmith - macOS has an app-level concept of "Hide all windows". It's not completely clear how this would interact with window-level hide - but it clearly needs to.

One last thing that might be worth exploring - are we sure that makeKeyAndOrderFront is the right API for the macOS implementation of show() (or, at least, for show() on any window that has previously been displayed)? The name of the API itself hints that it's a combination API - it's making the window the key window and ordering the window to the front. It wouldn't surprise me if making the window the key window is what is triggering the unminimize action. It's possible that orderFrontRegardless might be a better match for show() on a window that already exists, or a minimized window.

[proneon267]

are we sure that makeKeyAndOrderFront is the right API for the macOS implementation of show() (or, at least, for show() on any window that has previously been displayed)? The name of the API itself hints that it's a combination API - it's making the window the key window and ordering the window to the front. It wouldn't surprise me if making the window the key window is what is triggering the unminimize action. It's possible that orderFrontRegardless might be a better match for show() on a window that already exists, or a minimized window.

I have tested the following methods:

• orderBack:
• orderFront:
• orderFrontRegardless
• orderWindow:relativeTo:

All of them deminiaturize the window after unhiding the window. So, it seems like there is no other way to unhide a window while not deminiaturizing the window.

a question I raised in my previous comment - what about other window states? What does it mean to hide a window in presentation mode? In fullscreen mode? We already restrict changing window state on non-visible windows; do we need to restrict changing visibility on some window states? This will require some more experimentation to determine what is even possible.

I have tested by hiding the window from every state and then showing the window. The following are the results:

From state	Result on calling window.hide()	Result of then calling window.show()
NORMAL	Window is not visible and window icon is not visible in the dock	Window becomes visible and window icon becomes visible in the dock
MINIMIZED	Window is not visible and window icon is not visible in the dock	Window becomes visible & un-minimized, and window icon becomes visible in the dock
MAXIMIZED	Window is not visible and window icon is not visible in the dock	Window becomes visible and window icon becomes visible in the dock
FULLSCREEN	Screen becomes black	Window becomes visible in fullscreen like before
PRESENTATION	window content is still visible on the screen, but NSWindow.isVisible reports False	window content is still visible on the screen, but NSWindow.isVisible reports True

Although calling window.hide() does produce weird behavior when called on FULLSCREEN and PRESENTATION, but the behavior is normal when window.hide() is called on MINIMIZED state. So, disabling show/hide on a MINIMIZED window wouldn't be justified.

Another complication that came up this morning in a conversation with @mhsmith - macOS has an app-level concept of "Hide all windows". It's not completely clear how this would interact with window-level hide - but it clearly needs to.

Yes, I think this is the one: https://developer.apple.com/documentation/appkit/nsapplication/hide(_:)?language=objc

Hides all the receiver’s windows, and the next app in line is activated.
This method is usually invoked when the user chooses Hide in the app’s main menu. When this method begins, it posts an NSApplicationWillHideNotification to the default notification center. When it completes successfully, it posts an NSApplicationDidHideNotification.

I also found its unhide counterpart: https://developer.apple.com/documentation/appkit/nsapplication/unhide(_:)?language=objc, which invokes: https://developer.apple.com/documentation/appkit/nsapplication/unhidewithoutactivation()?language=objc

Restores hidden windows to the screen and makes the receiver active.
When this method begins, it posts an NSApplicationWillUnhideNotification to the default notification center. If it completes successfully, it posts an NSApplicationDidUnhideNotification.

Since, there are 2 reliable delegate notifications: NSApplicationDidHideNotification and NSApplicationDidUnhideNotification, so there should be no problem in detecting them and triggering the appropriate events.

The major argument again using macOS's behaviour is that it couples two otherwise unrelated features "show/hide" and state are now closely related features. I guess there's an argument that they already are, in that you can't change the state of a hidden window; but implementing macOS's behavior doubles down on that relationship in a way that I'm not sure is helpful.

As you have noted previously about the behavior of window.show():

if someone is calling "show()", they're indicating they want the window to be visible, not just minimized.

I totally agree with this and think that this interpretation of window.show() is the correct one. After all on_hide event is also triggered when the window is minimized, so it is natural that calling window.show() also un-minimizes it and sets the window in a visible-to-user(i.e., not minimized and not hidden state) state.

[freakboy3742]

I have tested the following methods:
...
All of them deminiaturize the window after unhiding the window. So, it seems like there is no other way to unhide a window while not deminiaturizing the window.

Well that sucks... Back to the drawing board, I guess :-)

a question I raised in my previous comment - what about other window states?

I have tested by hiding the window from every state and then showing the window. The following are the results:

Thanks - that's really comprehensive for macOS; to me, it looks like the summary is that this is all as expected, with show() under minimise has the odd behavior that we've been discussing, and hide() of presentation/fullscreen apps behaving in a way that is at least conceptually consistent, albeit a little weird in practice.

Since, there are 2 reliable delegate notifications: NSApplicationDidHideNotification and NSApplicationDidUnhideNotification, so there should be no problem in detecting them and triggering the appropriate events.

I figured there would be.

if someone is calling "show()", they're indicating they want the window to be visible, not just minimized.

I totally agree with this and think that this interpretation of window.show() is the correct one. After all on_hide event is also triggered when the window is minimized, so it is natural that calling window.show() also un-minimizes it and sets the window in a visible-to-user(i.e., not minimized and not hidden state) state.

My hesitation here is that while I think I agree, I'm not 100% sure - and if I'm wrong, walking it back will be difficult without breaking backwards compatibility.

In the interests of moving things forward, my suggestion is to treat this as a future problem. For now, prohibit the interaction, and raise a ValueError for any of the "weird" edge cases (i.e., show/hide when minimized, full screen or presentation). As I noted previously, we already prohibit changing state while hidden; so there's at least some symmetry in having an error for these cases.

Prohibiting the behaviour may seem extreme - but the cases we're talking about are edge cases. Honestly, if you're leaning hard on window.hide() in your UX, I'm already raising my eyebrow. However, if we explicitly prohibit these cases for now on the basis of the inconsistencies we're seeing, the implementation is simple, there's less testing, and it's guaranteed cross platform.

If at some point in the future, we decide that it makes sense to allow show()/hide() in any or all of these situations, we can introduce new behavior that replaces the ValueError. It's a lot harder to do the other way - removing or changing behavior that previously existed - because someone will have built an app that depends on that behavior.

[proneon267]

Since, #3109 is merged, so the implementation has become much more simplified, so I have also re-added the gtk implementation to the PR.

I think that there is overlap between the core tests and testbed tests. But, I am not sure if they should be excluded from either the core tests and the testbed tests, as it would lead to missing coverage on the dummy backend, and on other backends.

I believe that I have gone through all the previous review questions. If I have missed any of them, then do let me know, I'll try to respond to them quickly.

[freakboy3742]

A couple of comments about possible simplifications to some of the logic

[freakboy3742]

Is this if statement needed? The contract for the backend should be that it will only be invoked in "valid" states

[proneon267]

I agree, I have removed it.

[freakboy3742]

As above - is this if needed?

[proneon267]

I have removed it, Thanks.

[freakboy3742]

Again, are these ifs necessary?

[proneon267]

I have removed it.

[freakboy3742]

Can't these be simplified as:

if previous_state != current state:
    if current_state == MINIMIZED:
        on_hide()
    elif previous_state == MINIMIZED:
        on_show()

?

[proneon267]

I agree, I have changed it to use simplified logic. The complex logic was written before #3109 was merged.

[freakboy3742]

Why do this as a completely new test? We're already doing a bunch of transitions as part of the window state changes - why not add event notification checks to that existing test?

[proneon267]

Thanks, I have added the event notification checks to the test_window_state_change test on the testbed. I have also done the same at the core tests.

[freakboy3742]

As with the GTK implementation - can this be simplified? The only state that affects on_show/on_hide is minimized.

[proneon267]

Thanks, I have changed it to use the simplified check.

[freakboy3742]

Awesome - thanks for the PR!

[proneon267]

Thanks for keeping patience with me :)