Suraj Rampure, Christopher Pyles Otter Grader: A Lightweight Solution for Creating and Grading Jup

Key Takeaways

my name is sanyari I'm the sessions chair for this second talk on education chaptercon 2020 it is my pleasure to welcome Suraj and Christopher who will give us a overview of Auto grades to grading Jupiter assignments without further Ado you have the stage thank you good thank you welcome everyone uh you're about Ultra grader at first some brief introductions so my name is Surah drampore and I am a lecturer in the halei geolu data Science Institute at UC San Diego where I teach undergraduate courses and I will probably primarily be speaking about otter from the perspective of an instructor who uses otter but the person who built otter is not me that's Christopher uh hello everyone uh thank you for being here my name is Christopher piles um I am the lead developer of otter and I guess currently it's sole maintainer uh currently I'm a software engineer at Google but before that I went to UC Berkeley uh where I majored in computer science and built otter as a student with a team of other student developers um so without further Ado um I'd like to start off by talking a little bit about Auto grading and what it is uh and how it works basically Auto grading is an automatic method that we can use to grade students code um so that we don't need to have graders reading through all the code and trying to determine you know having fun what would python print exercises uh instead we can use unit testing like things to assign grades based on how it works against some public test cases or private test cases and basically all it does is it re-executes the student's code in some kind of grading environment and runs a bunch of unit tests against the global environment resulting from that execution and the reason for this is scale Auto grading takes you from the ability to manually grade you know 20 to 30 to however many graders you have uh submissions for an assignment to you know scaling quite quite large some of the courses we have at Berkeley are over 2 000 students that are using Auto auto grader to grade assignments at quite large scale so autograder itself is an auto grader and it's basically a lightweight modular Open Source Auto grader designed to grade programming assignments at any scale it is a successor to another another Auto grader called okay Pi that was also developed at UC Berkeley um this one was originally developed in mind with instructors who don't have access to large-scale compute resources in mind its original Inception what the idea behind its Inception was that you could collect a bunch of submissions or programming assignments download them all onto your laptop and then run all of them in Docker containers and then just get like a CSV of grades that has since expanded quite a bit into the way it's used now at UC Berkeley in conjunction with an LMS called grade scope um but that was the original idea behind it so otter itself works with both Python and our assignments it was designed for data science Jupiter notebook assignments but it is compatible with other format like Scripps and our markdown files the package itself includes tooling for developing and distributing assignments and you can run as I said assignments locally or with third-party services like grade scope and the core abstraction again is that you as the instructor provide compute to run assignments and Otter will take care of all of the rest so to set up otter it's actually very simple all you need to do is PIP install the python package and then if you're going to be grading locally in Docker containers there's also a Docker image that you need to download as well so in today's talk we have kind of three main parts we'll do a short demo of the assignment authoring release and collection process we'll discuss various use cases and extensions and then finally we'll discuss some shortcomings of otter areas we hope to improve and future plans for development so to begin student-facing notebooks kind of look like this once they come out of otters assignment developer assignment distribution tool the kinds of questions that work best for Otter are where you provide some Exposition and then skeleton code or students are going to be filling in the blanks so for example we have this a cube plus b squared function and if I just run this right now obviously I'm just returning um uh Ellipsis which is going to throw an error because you can add ellipses together and then inside the notebook the student sees we have these cells that call this grader.check function and what this is doing is it's taking the a cube plus b squared function the student defined running it again some test cases and then uh if those test cases pass it's going to say it pass but here in this example obviously this throws a type error because we can add ellipses so what I do if I want to say actually fill in the function and then we will see that I have passed the test cases because I've actually done it um in this example though this is just checking at the moment some public test cases so there's only one public test case so if I say let's just return the value expected by that public test I'm still going to be passing even though I have not correctly implemented the function and so this is kind of the level of granularity you can get where you have public tests that allow students to check you know the reasonableness of their answers but then you can have hidden tests that are run only by the autograder when it's determining the final grade and are obfuscated from the students that actually check correctness so to get a question like this you may have some snippet in a source notebook some instructor facing source code that looks like this where we've defined a question with a name we have our Exposition here in markdown the solution to the the example or the the question and then some test cases down here and then all you would do is have these run save this notebook run it through order and you would end up with that student facing version okay um so this is one workflow this is kind of the the main workflow that uh we see used at UC Berkeley and Suraj uses at UC San Diego um where you start by creating a source notebook which I will show an example of one in a second and by the way if you'd like to follow along with this demo you can open up this URL in your browser it is a Jupiter light instance that I'll be showing in this demo then you would run auto assign which is the tool I've been talking about you then distribute the student-facing notebooks collect um all of the submissions and then grade them on whatever your grading platform is so The Source notebook itself consists of all the skeleton code with the solutions as I've just shown all of your test cases some question level metadata and assignment level metadata so let me show you an example of what that looks like so this is an Abridged version of a of a project that we wrote for another course of Berkeley called Data 88 that also uses otter as you can see here at the top we've got a little bit of assignment config like you know we say oh we required this file to be distributed with the notebook because students will need it some other configurations and then the rest of this is just Exposition for the assignment some imports Etc and then finally get into the first question and we've demarcated that I've begun the question with our notebook raw cell then we have the question Exposition the solution and then these test cases as an aside this is a different test case format from the one that was in the slides um but it does work the same under the hood and then again in the question some more Expedition we've got another question and this is basically what the notebook looks like uh you've just got all of your Exposition you demarcate the questions the solutions and the tests and then for some cases where you want to have students write out long form answers that you want to grade manually Auto supports that as well you just note that the question itself is manually graded in the metadata and then you can provide markdown Solutions and do whatever you need to do so uh with this notebook the specially formatted notebook that you have what you'll then do is run auto assign which is the tool that takes in the source notebook and creates the auto grader version of The Notebook this is a notebook that has all the Solutions in it and then the tests for this directory have public and hidden tests included and so this is what's used to generate this ZIP file that we're going to use to configure the autographer which I'll talk about in a second and then you have a student directory which contains the student-facing notebook where we have all those prompts instead of solutions and only public tests so if we go back out into the demo you can see here in this outdirectory we have an auto grader directory in the notebook if we open it we've got um all the solutions included but you'll note that we have this cell to initialize otter and all of our tests have just been replaced with this one cell that calls grader.check and if we go into an example test you'll see that I have the hidden test and the public tests in this in this example but if we go out into the student directory here again we've got all the test cells gone we've got only the input the initialization of otter and all of our Pro all of our Solutions have been replaced with those Ellipsis prompts and if we look at a test that same test uh the hidden test is gone so that is what Auto assign does and then so what you can do then is you have that student directory all you can do you just take that whole directory put it in your GitHub repo however you're Distributing it to students and you ship that and and it's it's all there students can go in they know exactly where their code needs to go they have the public test cases and then audit takes care of for example generating the submissions of file but for Distributing I'm going to hand it back over to Suraj who's going to talk about his workflow thanks Chris so as Chris mentioned once you run auto assign you have the student folder that somehow you need to distribute to your students and there are a variety of choices you could make like one thing you could do is just submit a or upload that zip file or to make a zip of the student directory and put it on canvas or whatever learning management system you're using one thing you could do if you want your students to get familiar with Git is to I think I have to stand a little closer here uh if you want them to get familiarized with Git is to have them clone a course GitHub repository but one thing we often do in our introductory courses uh is use NB git puller and so here's an example screenshot taken from the course I'm teaching this quarter which I should be doing lecture for right now but I'm here instead um and you can see there's a link that just says data frames and data visualization and um when students click that link really this is the link that they're opening and you can see if you're not familiar with Envy get puller what this is saying is open our UC San Diego data Hub another term for Jupiter Hub uh open our course GitHub repository DSC courses dsc10 2023 SP and then open this homework to uh file and this homework slash homework O2 folder that's just a renamed version of a student folder that was the output of otter assign so we're just taking that literally and giving it to students and as Chris showed it's all good to go okay then how do we get things set up so that students can actually submit it once they have access to the student folder well first we take the auto grader configuration zip file that we get from the auto grader folder from autoresign and we um you know one thing we can do and one thing we often do in our workflows is put this on our learning management system that is configured with auto grading capabilities such as grade scope so we'll talk briefly about how you can do this without grade scope in just a minute but often what we do in these courses is just take that zip file and upload it as the autograder on grade scope and that's really all you have to do you don't have to tweak with any configurations there it's all handled for you in the zip file that otter assigned gives you so it's really nice there once you've done that you need to collect your student submissions and so in my workflow where I use gradescope what students do is they just submit their finished notebook files directly to gradescope and when they submit all they will see is the results of their code on the public test cases and the public test cases are the ones they already had access to in their notebook right so really what they're seeing is just a validation that what they did in their notebook while they were working on Jupiter Hub or locally notches uh sort of the public tests in our grading system because the public test will factor into their grade however when they submit right when they submit the hidden tests are also run on their code but they can't see the results of those hidden tests until we decide to release grades now one model could be to release graves right away or just have it so that hidden tests are always you know your score on hidden test is always shown and so I've seen instructors do that where in order to see your results on hidden tests you have to go through the hassle of submitting in some instructors and this is typically what we do we don't show the results on hidden tests until after the deadline so students can't just repeatedly resubmit and game the system until you know they've essentially made the hidden test public test at that point right and you can also manually grade questions kind of like you can in and be greater so let me show you some screenshots here so here I made a burner account on my course grade scope and I just submitted an assignment and in this case you can see that this submission passed all of the public test cases okay and so this is the view of a student and you can see it says all test cases passed and from the student's perspective all this is confirming to them is that they passed all the public cases which they already knew when working on the assignment but when this exact same student submitted here's what I would see on the instructor side so sure it would show me that they passed all of the public test cases but it would show me that they failed several of the Hidden test cases and it would show me the instructor exactly which hidden test cases they failed and so when I released grades to students this is exactly what they would get to see as well they would show them their score on every single question both public and hidden and it would show them the specific test cases that they failed okay and then here's the UI on grade scope at least when you want to manually grade submissions so for instance um you know Auto grading plots that result from like matplotlib or plot lead that can be a little finicky and so we often have humans grade students work what a concept and you know this is what that looks like on gradescope okay so now you may be saying hey well I don't have an Institutional license to grade scope or I want to use something else what do I do instead and as Chris mentioned at the beginning the you know conception of otter is that it's meant to be platform agnostic so you don't need to have access to grade scope to use it um instead what you could do is just ask your students to submit their completed notebooks to whatever learning management system you're using canvas you could even just tell them to send you links to Google Drive files whatever and then once you get all of your student submissions downloaded on your computer you can just run that auto grader configuration zip file locally using the otter grade command and he provided otter Docker image so you can do all of this locally on your machine without having to have access to platforms like raidscope and what you get back when you run auto assign or sorry not Auto assign but other grade is a CSV of assignment scores for each student so like you know what you see here you'd have one column for each of these and one row for each student and then you can upload that into whatever learning management system you want to use so this is still useful even if you don't you know know or have access to gradescope okay so now that we kind of get what otter is all about and how you would use it who actually uses it uh so as Chris mentioned it was kind of developed at UC Berkeley with these large data science courses in mind and so some of the largest courses on campus are called Data 8 and data 100 and they have often over a thousand students per semester and of course we can't manually grade every single thing they do and so otters you know quite powerful there though otter is not restricted to just grading python assignments so there are courses in political science and public health that use both our notebooks and our markdown uh they're a bit smaller but they still use otter and we've linked an example our markdown file from a public health course um if you if you want to take a look at that now most of these courses use a campus hosted data Hub server uh though again there's nothing about otter that requires that you do that you could have students work locally data 8X is a massively open online course version of data 8 that runs on edx that also uses otter and the folks at Berkeley have also put together a sort of cloud-based grading system that does what otter does but doesn't require you to have access to grade scope and so this is for schools that are adopting Berkeley's curriculum but don't have institutional licenses to this other software okay um so at UC San Diego which is where I am now our intro course dsc10 works very similarly to Berkeley's data eight so we use our campus hosted data Hub server and students use NB get puller though the course I want to talk to you a bit more about now is a second year course where we want students to get familiarity with pulling and pushing to get well really in this course just pulling and also the work flow is slightly different in that students actually write their code and submit their code not in notebooks but in Standalone Python scripts even though the assignment descriptions are distributed in the form of a notebook so I'll show you a little bit more about that as well um but several other institutions also use otter I I was in some talks yesterday of schools internationally that are already using otter and you know we're working with high schools and community colleges that are adopting it as well okay so the course I wanted to tell you a little bit more about it's called dsc80 at UC San Diego they're the philosophy taught is that notebooks are meant for experimentation and when you're like prototyping your code but when you when you're you know building a library of code that should all exist in separate.py files that you can import into your notebooks for more experimentation and so in that course each assignment is consists of both a notebook which contains the descriptions of the problems and imports of the data sets as well as a standalone.py file which when distributed to students just has stubs of functions but students only actually submit the dot py file so they aren't allowed to you know rely on anything in the notebook when they submit okay and so by default otter takes a notebook in and gives you a notebook out but for the purposes of this course we built an extension around otter that calls otter but then extracts all of the functions that we want students to uh to implement and put that in a separate.py file and that all uses NB convert as well so let me show you a little bit about how that works so here's a screenshot from an assignment in this course you can see that it says completely implementation of the function extract personal it you know reads some messy user data and you can see here extract personal isn't defined in this notebook right so we import the relevant data and then we test that students have you know implemented the function correctly again you can do the same public test in the notebook hidden test on grade scope thing but the actual function extract personal is not here instead it's in a separately distributed python file and so students will complete this here and then using a a cell level magic I believe you can have it so that this file is automatically reloaded in the notebook every two seconds or something like that and so this file is imported here and so students can still experiment with the code in their dot py file but uh ultimately all they submit is that dot py file and not the notebook okay and so the point I wanted to make here is that while this behavior of separating a notebook uh into a notebook in a DOT py file is not native to Otter uh the assignment authoring process in this course was significantly more complicated before otter came around essentially I got to UCSD I had used otter at Berkeley before and then I was scheduled to teach this course and then every assignment was made up of like 20 different files and I'm like I don't have the patience to work with this I'd rather just use otter which I know is easy to use and then just build a wrapper around that and now it's much easier to develop assignments in this course so just like one of the possible things you could do with otter and so I'll pass it back to Chris to discuss some of the shortcomings and future plans with otter okay um so one of the things that you experience with otter and indeed I think this is true of all auto graders is that writing test cases for auto grading is both an art and science um there are a lot of things a lot of stumbling blocks uh and an otter doesn't necessarily have uh the facility to alleviate all these issues so one of these is Randomness um otter has some capabilities that help you work with Randomness better there are some things you can do around seating Randomness uh between like whenever a student it's going to use Randomness in their assignments to make sure that you always get a deterministic answer the visibility of public tests so there's like for example if you want to have public tests attributed with a notebook but you don't want students to be able to like go and read the public test as I did earlier in the example and say oh this is looking for the number 129 why don't I just return that you can do things like hashing you can also have uh success and failure messages that are shown to students along with your test cases um but this is you know still an issue and there's also some issues around the way that otter does tests because it's using internally doc tests around the fact that you're comparing the string representations of objects and not their actual values and things like that um other things to think about um all auto graders tend to encourage guess and check behavior and auditor doesn't really do anything about that um uh we do have some issues integrating with other notebook front ends things like uh Google collab and Jupiter light um there are a couple feature gaps when running on those environments these are things that we are working on um they're actively issues on our GitHub repo right now about increasing our compatibility with these front ends um and uh you know it is it is on our to-do list we don't support question randomization so you can for example write you know unique exams for each student using Auto assign um it's you're going to get one notebook out and it's going to be the same for every student and there are some Jupiter more advanced Jupiter features like cell and line magic that are currently unsupported but I'm happy to say that is being fixed in version 5 which I hope will be released this summer um we will be supporting these kinds of more advanced features as we're migrating to executing notebooks using nbcon xqp processor these are plans um these are kind of broad goals that we have for Docker for Otter at the moment we'd like to expand our ability to integrate with Learning Management Systems Beyond grade scope um our kind of tethering to grayscope at the moment is an artifact of the fact that this was developed at Berkeley uh where we do use grade scope quite heavily but we'd like to expand the range of lms's that we integrate with we like to support common patterns for image grading image creation like repo to Docker files and we'd also like to start building a community of Auto users we kind of have already done that we have a slack by the way if you find your way to the auto GitHub repo there's a link to join the auto slack there so please do join um but we like to expand this community even further and try and build out like a system of plugins um and so there are a lot of topics we didn't get to cover in this talk plugins is one of those there are ways of like altering otters Behavior otter has a few built-in plugins um that's one of those topics there's a bunch of them like working with our assignments different test formats seating all these different things so if you're interested in any of these topics please do look up on Arts documentation which is linked here um you can find all of the this notebook the slides the demo on Jupiter light notebook in this GitHub repo and then again I have the link to that Jupiter light demo and that is it so uh we're happy to take any questions now thank you uh Christopher and Suraj please a round of applause for both we have time for questions uh before you answer the questions please repeat for the recording uh any one questions hello Nicole Terry from Parisa clay uh I'm currently using quite heavily and be greater and one of the features I like is that it works with any programming language I'm doing C plus plus classes uh do you see opportunities given how autograder is built that it could be made to to be language agnostic like NB grader is mostly yeah um so the question was are there opportunities to make arter grader more language agnostic um based on the way that it's implemented and I will say yes um so originally it was python only completely Python and then the ability to grade R was something that I kind of hacked together one summer just because I was a little bit bored um and so the there's like two main parts to Otter um there's the kind of otter grading server quote unquote it's not really a server but it's like the whole grading mechanism and that is in Python but it doesn't need like it doesn't need to be in whatever language you're grading in then there's like a client package and that's the part that students use to check their answers and so as long as there's a client package available in whatever language and that client package handles like uh the execution and checking test cases and like reporting results back in a uniform format um it's Auto can just kind of read that in so that's what otter does right now for R is that the python starts in our process calls executes the students our notebook and they are just kind of generates this Json that auto reads back and reports the results uh from so yes um there would be some work required to add any language to Otter um which is mainly building that client package but uh yeah it is very possible to expand Beyond just by an arm great question we have time for another question um yeah great great topic um a great presentation uh I was I was thinking is there uh any easy way to sort of integrate this in with GitHub classroom have you looked into this so that uh just the students submit the to their directory and then maybe just a workflow runs at some time and out pops the grades on the other end and the instructor does nothing and except fold the grades yeah um uh yes I do think it would be very it's very possible to integrate with GitHub classroom I haven't done it yet I did work on another Auto grader that was trying to do that called piperite which is something I did for Microsoft but um yeah so otter is just basically a python package it would be very easy to just have a get of action that pip installs it runs you know otter has a non-containerized version of grading where you just do auto run this thing and it gives you backup results it would be very easy to just write I think it would take some hacking to put together such a GitHub action workflow but yeah I could definitely see that being you know a weekend's worth of work uh thank you uh for the questions and for your great presentation Christopher and saraj please uh last round of applause for our three speakers