App Actions | Q&A

Key Takeaways

[Music] hello everyone welcome to ask the anything app actions i'm your host dishow developer relations engineer at the google assistant and i'm here with doug and adam from the app actions team so here to answer all your burning questions so please don't forget to post your questions head over to the i o website click on the q a button right next to the live stream video and we'll be choosing the top voted questions so make sure to upload your favorites so let's get started over to doug and adam yep hello everyone my name is doug stevenson i am also a developer relations engineer with the app actions team at google and i'm adam quimber i'm a product manager on app actions for the google assistant all right let's get started then our first question is from stephanie and it's how much work is it to convert my existing app action that uses actions.xml to shortcuts.xml doug do you want to take that yeah it's not a whole lot of work you will have to learn the shortcuts.xml file format good news is that it already exists in the android platform so you can read the android documentation to learn how to get started with that and then you can layer in your app actions on top of that so there's a relatively straightforward process to get your actions from action.xml into the shortcuts.xml format um you can test it with the app actions test tool in the exact same way that you were testing it previously so nothing changes there it's just a matter of migrating the xml format from one to the other anything you want to add there nope that's exactly right and you know we just just just one thing to say would be that you know it's a little bit uh inconvenient definitely to have to make make the change and and you know we're we're sorry for that but you know we think it's really well worth it uh because uh we're really invested in in this api and and in the kind of benefits that it creates an assistant and as we release more and more features uh in the roadmap which which we're we're investing in uh you know they'll be supported in this new format and in this new api so you'll be able to take advantage of them sounds good our next question is from huan and it's how do android shortcuts get used by google assistant maybe adam you can take a shot at that yeah my pleasure um so we're really excited about about this announcement uh you know as you heard um android shortcuts which have been around for a long time are now going to be a bit more useful for the users of your app because uh sort of if you've created shortcuts or if you create shortcuts they will now be usable as voice commands in the assistant and specifically where we're starting is uh with the shortcuts gallery in the assistant so if you uh this is something we announced the last year if you say uh to your assistant uh hey google uh show my shortcuts it'll take you to this gallery where you can see all the different uh shortcuts on your phone that can be set up as voice commands and now uh any shortcut created as an android shortcut will appear and be recommended in that gallery and you know your users can set up your shortcuts as voice commands they can even customize uh the command that they use and uh you know that we're that's just the beginning uh we're excited uh you know soon to be seeing more and more features and assistant uh based on shortcuts uh the one that i'm really excited about that's coming up next is proactive suggestions of of shortcuts so if you use kind of the the the library that we've released for shortcuts and you do everything in the right way uh assistant will actually be able to proactively recommend your shortcuts to when users come to assistant so that's a pretty exciting uh feature uh that's coming soon cool that should help a lot with the discovery definitely right and next question is from bill that's how does shortcut in-app promo work and what are the limitations doc do you want to talk about that uh yeah i can say a little bit about that so the way in-app promo works is there's an sdk that you add to your app a pretty straightforward api you use it when the the user does something in your app that you want to voice accelerate so um if they had just completed some task you could use it to remind them that they can create a shortcut for that an assistant shortcut which is a different than an android shortcut and what that does is it clues assistant into understanding that this is something that the user will want to do again and that there's a voice query associated with that so you're constantly reminding people that there is assistant integration that they can use voice queries to accelerate some of their tasks but it handles all of the ui for that so all you have to do is just um basically pop up that reminder to actually create the shortcut and assistant will do the rest and maybe adam you have something to add to that i'm not sure what the limitations are um that that might be relevant but i'm sure you have something to say about that uh yeah i i'm also you know i can say a bit about it in general and speak to some possible limitations um i i think um the way that it works maybe from a product or user experience uh standpoint like i would just add that um we really think that the best time or one of the best times for users to discover and learn about voices in the action in the moment when they've just completed a task inside of your app and they've just ordered that drink or finished that that you know exercise tracking that at that moment when they really understand sort of what the action is to tell them okay next time you do this action say this voice command like to really take that moment and that's that's what in-app promotion lets you do is is to create that in context discovery with your user and use that perfect moment to um both educate and get them a bit invested in in in this voice experience um so that's that's just really the benefit but everything that doug said is is correct about um about sort of how how you do it and how you integrate with it and and such um in terms of limitations i you know i also would have been very curious uh specifically what the uh the the you know what we had in mind there but um uh you know some things to watch out for you know is that whatever you specify in the api as to what the voice command should be and what the the metadata description of the command uh is uh you know those things will just sort of be faithfully passed over into the assistant ui that that asks the user to save it so please be careful and and you know follow the the kinds of best practices that that will uh document uh as you uh as you specify what what the user should say think about what is the you know command that a user might you know really most likely want to use which might be different than how you would represent it visually in the ui and also think about you know the the metadata where we describe what the action does and how best to uh you know uh call up to the in the user's mind uh you know that action that they've done in your app and understand you know that this command is gonna sort of re-uh re-engage with that with that same action so just you know be thoughtful about that because it will be you know what you specify will be passed over into the into the assistant ui and and what's presented to the user um obviously you know i think there are some other uh email ways to to uh reach out uh um about uh you know the specific limitations so please feel free to do that and and we will respond yeah i think uh one limitation i do know of is that it requires android api level 21 or higher um the sdk can be used on lower uh api levels but it won't work unless you're on api level unless the user's on api level 21 so that's something to keep and keep in mind um but api level 21 is pretty old these days so a vast majority of your users should have support all right sounds good um our next question is from sarah when should widgets be used and why not just open the app so yeah adam do you want to take a yeah yeah uh happy to take that one and uh and and doug i'll ask you to to supplement as as always um so um when should widgets be used why not just open the app so what as we begin bringing widgets into the assistant you know as we started thinking about that the opportunity that we really saw with widgets and voice was that as an app developer there are only certain times when you can engage with your user you know the user has to open up their phone have the phone unlocked and be in a context where they're looking at the screen and they can they can interact sort of in depth with you with your app but that is not all of the contexts where you know your app might be able to serve your users there's other cases where a user might be hands-free a user might their phone might be locked and ended it's sort of oh they only have one hand and their other hand is busy they might be in the car there's there's all these different contexts where there might be an opportunity or a value that you can bring your users you know i'm on the way to uh to pick up some coffee and i want to place the order when i'm on the way or i'm just putting in headphones and going for a run and and i want to start tracking uh the run without sort of unlocking my phone and and whatnot um and this is the vision of the the kind of the value that widgets can bring to you as an app developer and to your users uh is is that in these contexts where you know you can't reach users today widgets is a is a kind of a portable ui a portable experience from your app that can come into assistant uh so that assistant can present it in these in these sort of voice forward uh contexts um so there's definitely a time to open the app this is something that you generally have control over if it's possible to open the app and that's what you prefer you can set it up in the api so that it will open the app but by adding widgets in as well you can now access these new these new contexts so um so that's generally how we think about it but uh doug please uh please hop in uh with your thoughts yeah um i like what you said about a voice forward context um i think that's where which it's really shiny here because if the uh if the user is interacting with assistant they might want to continue to interact with the system it might not just be one query it might be a series of queries or follow-on um queries uh and so if you launch your app the user's clearly left assistant and is interacting with your app now but if you present a widget then the user stays an assistant they get their answer immediately and widgets also have a a text-to-speech option so you can speak out an answer so you're staying within this entirely within this voice forward context which might make sense for queries that just have a simple answer for other things like if you want to kick off a task opening the app makes more sense because you want them to go to completion you know maybe the query is just the beginning of something more they've begun a journey within your app so you'd want to launch the app to continue that journey so widgets are really good for these one-off requests that have spoken responses immediate answers and um everything else would be uh good for opening an app yeah and i think that example of um i guess your dunkin donuts where you're just reordering your uh you know your drink and so forth also like i guess repeating a task would be a good example i guess for widgets as well yeah i love the way that both of you said that it's it's you know one way to think about it is is is this idea of a a one shot or a quick answer response um and being able to stay in context and just get that response from within the assistant and then there's other other experiences other use cases that are more like a session where you're going to be doing a long running kind of task inside of an app in there the full app ui can be used so in an ideal world you know whenever the user just wants a quick answer we'd always have a widget and we could give them the widget and whenever the user wants to do a more complex task we could assistant could jump you into the right place in that task and you know in the ideal world that's a kind of would be a great breakdown to have all right so our next question is from early and see what's the difference between android intent and built-in intents doug do you want to answer that yeah uh so you'll probably notice as you read through app actions documentation you do encounter two different things that are both called intense but they're different things so uh our documentation should make this clear in every context so um when you're reading about an android intent if you're an android developer that is exactly what you know it to be the intents that do things like watch activities start services trigger broadcasts uh things like that those that that's an android intent now assistant has another concept that sits adjacent to that called a built-in intent which is a very different thing uh basically represents sort of a a a unit of semantic meaning so a built-in intent represents something that the user wants to do with a voice command and so all of the uh parameters of that meaning are sort of baked into these built-in intents and you can see there's a there's a list of like 60 plus intents and we're adding new built-in intents all the time these are not android intents these are again assistant specific concepts but uh when the user does trigger a built-in intent with their voice that ends up being translated into an actual android intent that you have to handle within your app so the built-in intent is effectively encoded inside an android intent so when you're reading through the documentation it should be clear but it's good to keep in your mind that these are two different things uh we hope that is not too intense that was a play on words that was wonderful that was a good one all right our next question is from andrew and what are the different ways app actions can hook into my android app adam maybe that's a good way to yeah so um this is a good question so you know last year we we launched sort of the first uh part of the answer which is you know app actions uh api uh called actions.xml which allows you to hook up you know the assistant and voice queries to android intents as doug was just explaining or deep links in the app uh so it so what the way that it's hooking in uh so to speak is by you know connecting these these um you know semantic built-in intents which represent you know voice uh voice interaction to your app activities via android intents via deep links and this allows these experiences that we have where you know you say query like search for coffee shops on google maps and we deep link into google maps and we paste coffee shops into the search box so that's what we launched uh last year and uh as you heard you know we are um you know moving to this new capabilities api but it really is the same uh same fundamental idea of hooking up voice to android intents now what we've announced uh you know this year at i o um is two more ways to uh that app actions lets you hook uh hook up assistant and voice to your app so the first one is android shortcuts um so android shortcuts are also generally you know ultimately an android intent uh but they have some different properties that make them quite powerful and i think the main one that excites me is uh dynamic shortcuts uh which are user specific and and obviously as the name implies dynamic um so this means that whereas what we can put in a static file like actions.xml or shortcuts.xml are things that apply to all users through dynamic shortcuts assistant can hook into the specific actions and specific uh content that users are engaging with in your app um for example uh you know a a app like like the android messages app uh exports shortcuts for the top contacts that you're chatting with so those uh things can then be accessible through through voice queries uh you know and those are very unique to you and personal to you which makes them we think uh more compelling and and uh and and and rich for users so that is uh shortcuts as another way to hook in a kind of a user specific android intent that uh assistant can can hook into through app actions and then the uh the other thing that we announced as we were discussing before is widgets and uh widgets are another way that uh assistant can hook into your app and in this case as we described um you know the widget is sort of a remote ui so a ui that is presented from outside of the the boundaries of your app's activities and can be brought inline into the assistance ui um and and so so you know each of these uh uh android intents and deep links or capabilities um shortcuts and widgets each of these uh are are the ways you know as of now that um that assistant can can hook into your app through through app actions and uh and yeah doug is there uh anything that i missed um no that was pretty comprehensive i don't have anything to add cool all right so our next question is from paul i think doug used to be able to answer this how can i test my app action integration yeah so um so during the android application development life cycle you probably want to be able to cycle very quickly and see if the thing that you wrote the app action that you integrated works as you expect uh we have tools for that it's worth noting though that in order to kick off that process you do need to upload your app to the play console um that's that right now is a required step uh you the good news is you don't have to actually publish your app you just have to upload it to maybe a test track that upload to the test track lets google know that you have an app and it has a particular unique application id once we know that application id then you can use a android studio plug-in called the google assistant plug-in you can download that just like any other plug-in and it has a tool inside it called the app actions test tool once you have that installed into android studio you can use it to test trigger your app so you it basically reads your shortcuts.xml uh creates what's called a preview on our server so this preview basically lets you your google user on your device test the app actions integration driven through this google assistant plugin um so that's how you that's basically how you accelerate your testing so uh one you have to have a device or an emulator with your google account signed in two you have to be signed into android studio with that same account three you have to have your account registered to upload that app in the play console um again with the same google account once you have all that set up then that google account can then be used with the app actions test tool to test your integration then when you're when you're satisfied with how it works you can publish your app promote it fully to productions for all of your users to get a hold of it but you don't have to complete that final step until you're happy with the way your app actions work using the plugin sounds good all right um next question is from jen and it's there isn't a bii for my use case bi being a built-in intent what can i do instead uh adam do you wanna answer that yeah um that's a that's a that's a great question um and i'm glad we have a good answer for it uh which is that you know you are not constrained by the uh set of built-in intents that we have published uh although if we have published a built-in intent you definitely should use that instead of what i'm about to say uh so if there is a built-in intent uh please use that uh but uh if not then uh instead of you know waiting for us to publish it we do support custom intents uh and so through custom intents uh you can sort of create your own uh semantic intent that represents that particular use case you use a string resources file to specify uh you know the different possible example query patterns that uh basically requests that users might say and tag them where there's a a parameter and uh from there it actually looks very very similar to implementing a built-in intent so it's the same uh format of declaration in the shortcuts.xml file you test it in the same way everything is is the same just you were in control of describing the uh the metadata around the intent and uh and of course the the user's uh uh potential utterances and then really the the idea is that the google assistant will kind of take those example queries that you specified and um and try to uh you know eventually expand them into more queries this is something that we're relatively early uh on but it's something that we are investing in to to make your life a bit easier and to make the experience a bit better for for your users i also uh sorry uh one last uh thought there i would also encourage you to um file a uh uh request on our on our public facing um you know uh uh issue tracker um when uh when there is a uh a built-in intent missing for a use case because this is how we can capture that feedback and add it to our our pipeline you know as i mentioned at the beginning we always prefer to use a built-in intent if at all possible there's obviously a uh you know a trade-off where we can't create everything which is why we introduced custom intents but if we do see multiple people requesting something then we there is a good chance that we would invest in creating it so we need to hear that feedback from the community and i'd encourage you to uh to do that sounds good all right our next question is from david and it's so with the new shortcuts.xml capabilities api when is it best to extend an app action with slices versus extending an app action with an app widget is it possible to declare both and if so when does one take precedence over the other so doug do you want to answer that uh yeah so uh if if you were an app actions developer prior to google i o and you wanted to do a uh an inline response to a query kind of like what we were talking about before with widgets you would have used something called a slice and so a slice is basically that just like a widget where you have this bit of remote ui that embeds into the assistant um ui um we're basically trying to move away from that we're moving towards widgets we find that widgets give an overall more pleasant user experience the api has been around since forever and android so it's something that developers are already familiar with uh you probably you won't say probably but you might already have some widgets in your app and you can maybe repurpose those for the purpose of voice fulfillment so we're leaning into the widgets api uh we're moving away from the sizes api but the slices api isn't going to stop working um that's you know if you have a slice already that's going to keep working and we don't have any near-term plans to get rid of that um though i imagine i don't maybe adam you want to say a little bit more about what the future of this is yeah happy too so as doug mentioned uh if you've implemented a slice and connected to the assistant news nationals actions xml that will continue to work um if you are considering implementing shortcuts.xml and capabilities you know we are not uh we're we're we're going to support widgets there uh you know and so and and also sort of the improvements where we're making in this space things like adding tts outputs uh you know support which which is a great feature of widgets these things we're focusing more on widgets now the reason for this is is as you heard in the in the android uh keynote and and throughout you know the through across google we are doubling down on widgets and you know it just you know we we believe that it makes most sense to kind of consolidate on one solution for uh for this kind of remote ui presentation inside of assistant and and widgets is the one that you know we're all kind of leaning into uh so so you know uh that that's that's where we'll be sort of adding new features like tts output and new ui components and and lots of things like not just for assistance but also you know broadly in android um so highly encourage you to as i said before move to shortcuts.xml from actions xml move uh you know implement a widget but as doug said we you know totally value the the any work that has been done to create a slice uh before and it will continue to work and serve users in the assistant sounds good for yeah widgets for the win uh renee that's our next le this is probably our last question for the session so we'll see if we can squeeze in one more but the question is can you provide some steps how i can implement with app actions a shopping list i know from dialogflow already how i can create utterances etc but the google assistant does not recognize my intents since it seems that they collide with your internal system intents for shopping can you help yeah i can start and uh and then uh doug please chime in so um so it's a good question um we actually support built-in intents for for lists for app actions so there are a set of productivity built-in intents i believe that the the relevant ones here are uh called uh you know create item list update item list and and get item list if i if i am remembering right and uh these will actually give you you know uh built-in support for all those same sorts of queries like you know add this you know uh to my to my list and uh you know and so on and so forth uh so you know you don't you know dialogflow is not actually something that is uh integrated to app actions it's it's it's part of the uh you know conversational actions um experience uh but for app actions uh in this case you can really just set up really easily set up um you know these productivity built-in intents and connect them into your app and and users will be able to say all those sorts of queries and be uh deep linked into your app for to fulfill the request yeah i don't that was pretty good i don't have anything to to add in there but if we have another question i think that would be great if we have time for one yeah let's see maybe like one more question all right so from russ we have uh does actions the intent open up feature work via google assistant now or just using the testing utility i haven't gotten it working yet and i've seen issues posted that haven't had a completed resolution um doug do you want to take that um it should work um i can't think of any reason why i i don't think there's any technical limitations why it wouldn't work um adam maybe you can correct me on that but i think it's just a standard uh common bii that should work both in the testing tool and for published apps as long as the trigger phrase matches something that open up uh feature actually recognizes so i think that's i think that's where some people might sort of go wrong is that they might use a phrase that the bi doesn't support and it might it might have some problems there because so just to back up the test tool doesn't work with english language phrases it works with the contents of the bi so when you use the test tool you're actually composing the full bii sending that off to assistant then letting it know hey pretend the user said something that would that would match this bii however when you're doing a voice query you're actually saying or typing something assistant has to then match that turn that into a bii then ship that off to your app so um if if it's not working with a voice query it's very possible that the voice query just doesn't support it for whatever reason um and since i don't know what your voice query is it's hard for me to say from here um but that's the first thing i would think of yeah so i would just add you know i'm quite certain that open app feature does work uh you know i know that there's queries like open youtube history uh or um you know other other partner queries that that definitely are are working and serving users uh in in production uh i know that there was a bug in the testing uh of open app feature specifically a few months back i am quite certain that it was resolved uh but as doug mentioned you know there are several steps along the way of testing an app action that can lead to to problems so so you know please please keep trying and and we'll we'll respond to all those uh issues raised but this should be working and thanks for the feedback yeah and uh oh i had one more thing if you have um if you have a very specific case where something isn't working the way you expect i would encourage you to compose a question on stack overflow and be sure to show the actual code and the steps that you're taking that reproduce the problem so that anyone including myself would be able to reproduce it and see what's going on so um it might be something simple and someone can just point out hey you know you're not doing something right or you know so what you're doing isn't supported something along those lines uh but if you yeah please do ask us questions if something isn't working the way you expect yeah all right so actually i think we've actually run out of time and that wraps ups our live q a thank you so much for joining us today for app actions and we hope that you learned a little bit about you know app actions a little bit more and we have follow-up resources at developers.google.com assistant slash app so thank you again thank you doug thank you adam