Utilizing Mobile First Device Capabilities for Microsoft Teams Apps

Key Takeaways

[Music] i'm anuj bahil and i work as a program manager on teams mobile platform in the last six months or so we've been investing heavily on mobile-first innovation and capabilities we're not just investing on your features for mobile users but also making sure that the platform constructs are easier to use and apps are more discoverable and simpler to acquire on mobile today we are going to specifically talk about device capabilities and and these are essentially hardware capabilities that come on your phone uh we want to showcase the various scenarios or to you uh on how extensible app developers can utilize these capabilities to build richer mobile first applications i want to briefly talk about the various platform constructs um and how they essentially look on mobile i'm sure most of you would be uh familiar with this but we have bots which are conversational entities that perform a certain specific set of tasks so bots can register with the ms bot framework and can and then act as essentially any other member as part of a chat or a team and bots can communicate with users respond to their questions proactively notify them about changes etc then we have tabs these are canvases that can be used to share content whole conversations and third party services and these are essentially nothing but web views that load a website url messaging extensions is a construct that users can use to aid themselves while they're composing a message and on mobile it sort of comes up when you open up the the compose extension tray and lastly we have personal apps and these are similar to tabs which just exists in the personal scope so while you have tabs that can be configured and designed to be more collaborative in nature as they can be pinned to a particular chat or channel users can use personal apps to increase their own productivity and that's what comes up when you swipe up or on the home page in the in the more menu all right so before i hand over to shubham i just want to talk uh a bit about um why we're really investing heavily in mobile first capabilities before the pandemic the usage of teams is sort of restricted to north america but in the last year or so we've seen the usage grow in other parts of the world uh students front line workers and other blue collared workers that don't have access to large screen devices use mobile on an everyday basis and mobile only users today account for close to 20 of our teams users and this is one of the fastest growing user segments and we do expect this growth sort of continue in the near future as well so hence we're like focusing on empowering teams developers to build create mobile first apps by enabling device capabilities in the in the js sdk uh over to use shubham uh to talk more about the different apis we've exposed thanks thanks uh here one uh work with android on the platform uh so i'll just jump right into the device capabilities which we have developed over the past couple of months as well as we'll also look into some capabilities that we are currently working and will be releasing in the future uh so here's a quick road map of how things have been over the past and how it looks in the future we started off uh with the image capture and you know photo gallery uh image picking kind of scenarios first so the first two there uh the first two items here on the timeline represent the same uh we have used office lens which is like a powerful tool that microsoft leverages to power scenarios around capturing images through a camera then the third one here is recording audio using a microphone this again is a kind of a very common uh scenario if you talk about mobile especially for mobile first scenarios and then the fourth one here is a location detection interface which basically means uh if you're familiar with say how we kind of share our location uh for while while booking a ride on uber or lyft so that kind of map interface can directly be leveraged using the aps that we have recently introduced um then the fifth one here is around barcode qr code scanner again a very mobile first kind of scenario uh so we again use office lens sdk to power our uh control here we'll go into details uh of this i'm just quickly going through the timeline for now uh then uh till now we have only talked about the mobile capabilities but some of these capabilities uh we thought would also make sense to have the desktop counter counterparts supported as well so then we kind of work to gain parity uh in terms of functionality on desktop so around this quarter we have been working on uh the first two set of capabilities around images camera location and making sure that it also works in desktop so this is our current work that we are doing then at the end the last two are around a people picker hvc hvc meaning a high value control uh so there are very common scenarios in which you would want you know a control that allows an end user to select a user say from a chat or channel or within the org in order to say assign a particular task to a user or just add the user in a particular workflow we believe that's a very common workflow and we also wanted to make sure that this kind of control is added to the client sdk itself so that the developers can directly leverage it instead you know using their own uh putting in their own effort and building such a control that can be easily leveraged directly then the last one is around video capture capability which we'll be picking up sometime next quarter i'll be going into quick demos of how these capabilities currently work uh so the first one here is around um image apis uh so here we just wanted to demonstrate uh the power that office lens provides uh so june if you can just play the demo uh i'll just uh this doesn't have an um voiceover so i'll just uh walk you into what's happening so here we just created a sample app to demonstrate the power that uh office less provides uh in terms of the edit functionalities and kind of high high level uh you know image capabilities so for example the scenario here is this an app a parent is trying to share the image of his or her child working or with the classroom in a particular app so as you saw here uh the app could directly uh you know access certain functionalities which are natively available and the parent is able to either click an image from camera or just select one from gallery and then office lens provides a host of functionalities such as you know you can add a text annotations you can add filters you can add free uh free hand drawing on drop of the images basically you can have all the functionalities which everyone is kind of very familiar with now and that can be added to the image before sharing it right back to the underlying web app so this is more about it uh then um it's kind of standard stuff we can we can pause this demo uh it's just submitting the image to the web app and uh move to the next one so this demo was around demonstrating how uh powerful office lenses in terms of high level edit functionalities over images uh the next demo we have that's a common uh demo that kind of demonstrates three of the capabilities including the one that we talked about uh which is image and then location and mic so this one has a voiceover so i'll just mute myself for a minute or so yeah june you can just play the demo with the help of number of device capabilities now enabled on teams platform contoso has enabled a tab app that lets field agents record information easily on teams itself that is instantly viewable on the contoso repair channel the field agent can quickly create a new inspection item and raise the ticket the feed agent can use the attach picture option to capture images of the car using office lens they have the option of choosing these images from the gallery or clicking them with the camera using the multiple image support the field agent can upload multiple images of the car at the same time they also have the option to choose and pick the current location of the customer by dropping a pin at the map interface instead of typing out a description of the car the field agent can record an audio description of the car itself on recording the audio successfully the feed agent can submit the ticket and finally the information is added to the tab app in the channel yep uh so that was a short demo of a simple app we built that leverages three of our capabilities around image location and audio or mic the last demo that we have at least for this area is around a qr barcode scanner capability we just went live uh last month uh so yeah this again has a voice over so i'll mute myself june uh you can play the uh demo here's a demo showcasing qr and barcode scanner capability that's being added to the team's client sdk max a contoso employee needs it support to fix his laptop he uses the incident reporting app to raise support request earlier entering asset code manually used to be really inconvenient with the code being lengthy and typed usually in small font also there was always a risk of making mistakes now max uses his phone's camera to scan the asset barcode which automatically fills in the asset info fields this makes the process of raising support ticket much simpler and after entering the issue description max just hits submit to generate a fresh support ticket max's life is easier now thanks to qr and barcode scanner capabilities present in the team's client sdk so that was around the qr barcode scanner capability which just went live around last month so so this is a control that we are currently working on um so this is again something that we have been hearing a lot from our developer community and especially our 3p partners who all of them had had a requirement a control which can be used leveraged to you know pick people uh it's it's not an easy process to do and if if something like this can be extended right out of the team strand sdk then that would add a lot of value to the developer community in general so we listen to that feedback and yeah he is his uh kind of the i work that we are currently doing and uh these are just a box for now but it's under progress so as you see uh here's a sample a purchase approval kind of uh workflow wherein one of the uh steps is to select the approver so the second and the three third screens that you see are here these represent the experience that would again uh be coming in in form of an api in decline sdk so this entirely would be just one epa call away so developers can just leverage that call uh that capability and you'll have you don't have to invest as much in building a people picker on your own so here on the second and third screens uh the end user selects the respective approver and as you see in the last screen uh the name of the approver and relevant details get shared back with the underlying app and the end user hits send and the workflow gets completed yep so so these were the demos uh now coming to what api is to leverage so uh i'm glad to tell that all our documentations have now been updated and published uh i will provide the link to all everything that we just demonstrated all the apis and how they work and uh even uh sample code snippets to make sure that uh you don't get stuck anywhere so i'll just quickly go through the apis uh the corresponding apis uh to the relevant uh api uh capabilities we just discussed so the first one uh is around images uh image capture and also we have another api for uh providing an image viewer uh control as well so select media is the api uh that you will need to uh invoke or call to kind of capture uh or let users capture and attach images to the uh underlying web so uh so i just wanted to cover the configurations also really quick uh so we kind of understood that there are different requirements or use cases coming in uh from uh from mobile-first scenarios such as for example there was a scenario which one of our customers was asking that you know we want to create a check-in kind of app wherein the user is restricted to capture image only using the front camera we don't want users to use rotate the camera at all or there were scenarios like you know this we have the scenario that we want users to scan a picture instead of capturing a picture and then there were scenarios around you know uh in in um quality check or quality uh inspection kind of scenarios they wanted to uh do a free hand annotation over the image before submitting it to the app so different scenarios uh kind of translate here to different configurations that we have kind of provided so you can change this configurations as per your need of the app and respectively the uh control would make sure that it opens in a relevant configuration so the first second one here is a number of medias that can be attached pretty straightforward third one is the source source means uh whether you like to let users capture an image or you know pick an image from the gallery or you know let the users choose which one they want to do uh then the fourth one here is the start mode so um office lens supports a couple of uh scan modes as well instead of like standard uh image capture kind of mode so it supports document scanning whiteboard scanning and business code scanning so if you want if you have a scenario wherein the control should open in a particular scan mode then that too is available in form of configuration then the fourth last one is around cameras which are if you want to default to front or back camera or want to restrict users to use only a particular camera then this is the configuration to use uh the last three are around uh editing so inking applying text stickers and applying filters so these are just boolean uh boolean configurations if you set them to true uh these capabilities which will just show up in the uh camera control uh then the last thing i want to cover is around a view images view image is another api that we have added to the sdk which can directly directly be leveraged uh to you know have a image viewer kind of control so that you can just use an inbuilt control instead of building your own image viewers to preview the attached images so the coming to the location uh apis uh there's just two pretty straightforward apis here one is get location one is show location so get location is you would be used to capture users location uh we have configurations which kind of let the developers decide whether they want to show a map or not show a map and if they want to show a map then would the user be allowed to move around the map and pick any location on the map or are they restricted to submit they only their current location so all of them are the configurations that can be created by using allow choose location and show map variables which are the configurations that we currently support so allow choose location would mean you know a map would open uh this again is coming from a mobile first scenarios that we have heard a lot for example in some scenarios the apps just want to show the map as a confirmation screen and they want the user to submit only the current exact location for verification kind of scenarios so listening to those kind of feedback we have this configuration that restricts our user to either submit the current location only or maybe enables users to pick any location on the map and share with the underlying web app and show map is is just a com like as the name suggests it it's a configuration if set to true it would show them up if set to false then this is kind of a scenario which is also currently enabled by the html5 uh get current position kind of api so it just uh kind of fetches the current coordinates from the gps of the device and shares with the underlying web app without showing any particular map on the ui then the other api show location it would just open a particular location uh that you provide to this api on on the uh native mobile supporting app so coming to the last capabilities that we have rolled out for now a microphone is again very simple there's just one api which you have already discussed so select media is kind of an umbrella uh api so we have just extended select media which we discussed earlier to also support my capability so basically the interface is same as as seen in the demo using this api a small kind of bar would appear at the bottom of the screen wherein you'll the end user will have the option to stop the recording whenever the user feels like ending the recording and the recorded audio stream gets shared with the underlying gap so the only configuration here is max duration which kind of is limited currently at 10 minutes and if you set this duration to say one minute then the recording would automatically terminate after that set period then the last capabilities around barcode qr code scanner again just one api which is called uh scan barcode the only configuration we have is timeout intervals in second it's currently marked at 60 second and default uh is 30 seconds but it can be configured for the needs of of the end user uh all the experiences were kind of demonstrated back uh in the demos so yep i hope everyone can relate which apis correspond to which capability and we just post we just shared a blog post recently on all the four capabilities we just talked about which is to summarize which is around image apis which is capturing an image or picking an image from gallery or viewing image then secondly around location which is uh getting a user's location or showing a location on a map third is around audio or mic which is recording audio on using the device microphone and the final one was around qr barcode scanner which is again uh being added being added to the client sdk as a ready to use control right out of the box uh so you can just check out this blog post uh the link is ak dot ms device capabilities and also uh respective documentations so that's all we had from the device capabilities area uh i hope everyone did find some value in what we have been developing from especially uh looking at the mobile first vision that the team currently has do share our feedback on the chat and yep also share any questions on the chat if you have any thanks what do you choose awesome thank you chevron thank you anders and the ship mentioned on that blog you can find all of the the resources um and the documentation um that they mentioned as well so be sure to check out that link i'll let me go back so that you guys can have it again in front of you if you haven't copied it already but aka dot ms slash device capabilities thank you guys so much