Input matters for Chrome OS | Session

Key Takeaways

[Music] hi my name is emily roberts and i am a developer advocate for chrome os today i'm going to be discussing input and why it matters to you as an app developer there is a whole world of device form factors out there foldables rollables convertibles tablets laptop devices in addition to phones with all of these form factors come different sizes and shapes of screens we have a number of talks this year specifically about large screens and layout designs so check them out however i would like to talk about the other 50 of the user interaction equation the half of i o that is sometimes neglected if output is the o then the i is input so what's the big deal should an input just work sadly not always many standard ui controls will work as expected automatically for users across devices but there's a limit to the extent the framework can guess how your app should respond without proper input support your app could seem broken could be no fun to use could be inaccessible to some users requiring accessibility tools and lastly your app could be missing out on creating compelling engaging and differentiating user experiences so my call to action today is really to think through input when designing your app right from the base the reality is that users are already using your app with mice keyboard stylus and more and not just touch embracing input as part of the design means your app will be more intuitive and delightful to use okay let's start simple and this is a situation that arises often when an app designed for mobile is run on a chromebook or on an android phone or tablet with a bluetooth keyboard connected there is the text box the user types a bit the user presses enter and nothing happens when a user's hands are already on a keyboard they will assume the enter key will work it is not intuitive to lift your hands off and poke the laptop screen and don't forget some android capable chromebooks do not have touch screens the fix is easy and can make a huge difference in terms of a user's app experience let's take a look here's the code you need to handle the enter key press first of all notice we are overriding the activity's on key up method anki up listens for when keyboard keys are released there is a corresponding on key down for when a key is pressed but in many cases it is easier to use on key up this is because anki down will detect if a key is held down and will send multiple events next look at the event's key code to see which key was pressed and released in this case we're looking for the enter key then take action on that key code in this case calling the send message function this will probably be the exact same function you would have called if the onscreen button was pressed so there is no need for a new code here and that's it for triggering app functionality but you will need to do two more things first be sure to indicate to the system that the enter key was handled by your app by returning true likewise in the event that your app did not handle a given event pass it up to super to allow the system to take action on it for more information check out the android documentation at developer.android.com so it's 2021. if users have their hands on a keyboard they will expect basic keyboard functionality so the spacebar should work for pausing and playing media and undo and redo shortcuts should exist when it's appropriate like in a text editor however i'd like to invite you to think more deeply about your app's input and think about how you could fundamentally reshape the user experience here is an example of mwm's edjaying app [Music] [Music] hey [Music] let's talk about what we saw edjang works with touch on phones and tablets which is great for casual users or even power djs who may want to use their phone or chromebook while riding the metro home to build out their set for later that day for someone who wants to get farther into djing or who wants to do the music for a party or a wedding midi controllers are a great option they offer a nice low latency tactile feel and makes it easy to crossfade scratch and apply different effects and filters and this is all possible with the exact same app as before on the same device just now with the midi controller attached even pro djs might appreciate this flexibility when they can't carry all of their vinyl records around with them and this is my favorite part a keyboard is essentially a low latency tactile input device right kind of like a midi controller edjang took the time to think about chromebook users and realize that they often have a keyboard and trackpad already attached so why not turn those into a built-in dj controller they included keyboard mappings for all the major actions and effects as well as and this is cool track pad based scratching and crossfading it's not quite as fun as a full midi controller but it's a lot more portable let's look at how to support these different input methods in your app midi coding can be daunting if you've never worked with it before luckily there are some handy open source samples in the android media samples library that can help you get started check out the midi synth project at the link here and in particular the synth engine module in that project also just a note if you use c plus plus starting in android 29 so coming soon to chrome os you can also use the ndk midi api adding keyboard functionality is just the same as we looked at before with the enter key here you can see checking for the w a and l keys and the corresponding app actions as demonstrated in the app trackpad input can add really cool two-dimensional tactile experiences to apps here's a simplified example of some code that would take trackpad input and convert it into a control signal for a record scratching function first look for generic motion events then record the change in the x and y position of the pointer then calculate the hypotenuse to get the total distance traveled by the pointer and then do something with that info here we are scratching the record and that's it we are going to improve this code later on so stay tuned for that with so many great input options another issue arises what happens when users switch between them maybe i start playing a game in laptop mode then move over to the couch and flip it into tablet mode and then later on tint the device put it on the coffee table and attach a game controller or another situation and i was fortunate enough to have taught high school and witnessed this first hand with my students some people use their devices very creatively twisted sideways using the trackpad and touchscreen at the same time and then occasionally hitting keyboard keys too it was quite wonderful to see but the question is how does your app handle something like this the number one thing is support your users let them use whatever input devices they want whenever they want this means your app should always respond appropriately to any supported input if a user has a game controller connected they're clicking the keyboard and the touchscreen all at the same time no problem respond to it all as expected things get more complicated however when it comes to ui there is a distinction between input events coming in and what is currently being shown on the screen for example some touch-based games might have an on-screen joystick like this pretend racing game here if the user is using the keyboard to play and not touch you probably don't want to use up screen space with that on-screen joystick and could just fade it out there may also be situations when an app or game has different prompts that are dependent on input for example press m for nitro with the keyboard attached but press x with the game controller and click here for a touch interface here's a look at a game that handles this well dead cells when using a keyboard and mouse the prompts and text show the keystroke and mouse button indicators for touch input you can see the on-screen joystick on the left a touch icon to interact with items and characters in the middle and on the right the jump crouch and item touch targets finally for a game controller all the ui elements correspond to the appropriate controller buttons as the user expects cool but how do you implement something like this one approach is what i call a lazy state machine it is a prioritized state machine that feels lazy because although input events from all devices will always respond immediately the ui changes may be slower to transition let's get concrete here you can see a flowchart for three input states touch keyboard and game controller decide on the priority of each input the number 1 2 and 3 here for the race car game example from before or like with dead cells if the user is using touch at all they need the joystick in buttons on screen or else the game might be hard to play even if keyboard events are being received if the user is touching the screen that on-screen joystick needs to be shown so the touch state gets the highest priority number one when does the game move out of the touch state if the keyboard is receiving input but there hasn't been any touch events for a while let's say five seconds it'd be nice to fade out the unused touch joystick and buttons to maximize screen real estate so keyboard input plus no touch input for 5 seconds equals transition the state machine to the keyboard input state again the moment there is any touch input we should immediately move back to the touch state likewise if the game controller is receiving events and neither the touch screen nor the keyboard has received input for a while the ui can move into the game controller state the instant there is any keyboard input it should move to the keyboard state or if there's any touch input it should move to the touch state this 5 second lazy delay before transitioning to lower priority input states prevents the ui from flickering back and forth in the event of someone using multiple input methods at the same time a last observation in this model the app is only reacting to actually received events so you're not trying to check if there are keyboards or bluetooth controllers attached or in any way trying to guess how the user might be interacting trying to do that will not always work and it will be slower to respond to changes it is better just to act on real received input events okay that was a pretty quick overview of this concept for more detail and code check out our documentation on chromeos.dev on the subject of game controllers chrome os supports all the button mappings that android does including xbox xbox 360 ps3 and ps4 based controllers and in addition we have added mappings for other popular controllers like 8bitdo and logitech to chrome os we then work to get these new mappings ported back to the android framework so other android devices can benefit actually handling gamepad events in code looks a lot like the keyboard handling code from before the difference is just the key code here's some code looking for the x button and the left arrow on the d-pad also with games you often want to know if a button is being held down or get that extra bit of responsiveness by not waiting for a button release in these cases you might choose to use anki down instead of anki up as mentioned before handling game controller joysticks uses a different method override than for buttons and d-pads for that check out the game controller documentation on developer.android.com an android feature that really shines on chromebooks is pointer capture this is when the mouse cursor is captured by the app meaning it is no longer visible on the screen input events go directly to the app and the cursor won't get stuck on the side of the screen if it goes too far in one direction one game that demonstrates pointer capture wow is minecraft education edition which launched on chromebooks for schools last august during gameplay you can see the user is able to direct their point of view with the mouse or trackpad without a visible mouse cursor if you act with the building and learning menus however the mouse pointer reappears as you'd expect here's some code showing how to implement pointer capture you'll recognize our dj scratching code from before in the middle there with our previous code imagine scratching a track and seeing the mouse cursor going all over the screen or potentially getting stuck on the edge and throwing off your groove pointer capture is perfect for this use case to hide that cursor let its movement be unrestricted and still get us the motion events we need instead of on generic motion event like before let's use a captured pointer listener to respond to the motion events the only difference here in the actual scratching code is that the x and y motion event values are relative to the last motion events received instead of being absolute screen coordinates like before which makes sense as the pointer is no longer bound by the screen this means that the x and y values are already relative deltas to the last motion event so there is no need to maintain and subtract the previous values the rest of the scratching code is the same when scratching should be triggered call request pointer capture and when it is finished call release pointer capture that's it for more information and samples check out the android pointer capture documentation the last type of input i'd like to talk about is stylus for drawing and painting apps especially low latency stylus input can make for an incredible experience an app that does a great job of stylus input is concepts they have low latency and tons of great features here you can see how the user is able to use the stylus to have precise control over the drawing and use some really nice brush and nudge effects do you want your app to have low latency stylus support too well i am really happy to announce that our low latency api is now available in alpha it has built-in configurable prediction and supports both cpu and gpu-based rendering paths please check out the api in the demo app available on github and file any issues or feature requests on the github tracker so much input so little time that brings us to the end of this talk and i hope i have convinced you to really think deeply about input when designing your app for more information on all of these topics and others such as supporting large screens getting your app running well on chrome os building web apps optimizing game performance developing on a chromebook and more check out chromeos.dev with that please enjoy the rest of io and i hope to see you again really soon [Music] you