Music21

Key Takeaways

[Music] data skeptic features interviews with experts on topics related to data science all through the eye of scientific [Music] skepticism Michael Cuthbert has a PHD in musicology from Harvard and is presently an associate professor of Music at MIT his scholarship and research explores Medieval music as well as minimalism in addition to his lectures and writing on music he is also the primary investigator of the music 21 project a python library for flexible computerated musicology analysis Mike welcome to data skeptic thank you it's great to be here so I've been looking forward to this because I've started playing around with music 21 I'm excited to hear a lot of your perspective on it and get some uh further details maybe we could start there what exactly is music 21 so music 21 is a library of modules for being able to represent music and to manipulate it and it's primarily focused on symbolic music that's to say musical scores things like music theory ideas chords and keys and scales and rhythms that can be precisely represented in a computational framework and then we can use this to analyze large quantities of music or to make comparisons across different musical genres or times and places the music 21 motto is listen faster and so it allows you to do the types of analytical procedures that that people who are really interested in music professional musicologists do all the time but between a thousand and a million times faster than we can do it by hand what's an example of one of those analyses a musicologist might be interested in one of the things that we looked at is uh how did the Baseline of pop songs change over time so there used to be a period where all the baselines you know classical music tends to have a lot of jumping around by what we call fourths and fifths mhm and then we have a period where we have a lot of moving down by thirds so [Music] um MH Etc and now we're in a period of a lot of moving up by thirds or things like that um da Punk Etc so how can we see this Evolution over time you know we can pick out a few representative pieces and do it by hand but with music 21 we can actually look at the distribution of interval uses by year by decade over a longer period of time what are some of the insights that have come out from that for example like is is music evolving faster than it did in the Classical period or are we repeating the same things we've seen before what do we learn from some of those analysis I think it's unfair to talk about Evolution as being faster or slower in any given period because we tend to think of music in the past as not evolving as fast and as more static because the things that we're most interested in changing aren't happening at that time and things that we can't detect like you know the vocal tomber the color of something in the 17th century that's basically lost to us because all we have are the scores but what we can find out from this is what qualities of Music make it sound of a particular time and don't the other thing that we can learn from this is what aspects of our music theory education what are we teaching people that really aren't nearly as relevant as they once were uh for instance we still teach in almost every music history class that composers and songwriters avoid the parallel fifth something [Music] like and that's basically gone in the data sets that we can see this is a a rule that at some point was very helpful to people and that people today are not finding it helpful when I think about music and maybe this is my own bias at least modern music I can kind of label it by decade and say oh that's what music was like in the 80s and in the 70s but then I have these broad brush Strokes of you know classical baroke romantic periods as you were saying it's that those are things that I see as static if I were to really spend the time investing and looking at something like let's say Medieval music would I see just as much variation over a 10-year period as I perceive in my own time period even for a specialist like me it's hard for me to really hear wow that's definitely from 1370 not from 1360 they they totally weren't doing that at you know 1373 in part because we don't have enough labeled data you know there aren't even our what we call the ground truths aren't absolutely correct but also it's so hard for me even as a specialist in this music to get rid of all the biases that I have that make it hard for me to hear the subtle variations but for music 21 and U machine learning program that's not hard it doesn't come in with any of the bies about oh well music from you know 1960 sounds more different from 1970 than 1360 does from 1370 I've been able to use Music 21 to point out big problems in some of the dating of pieces some of the things that we thought were well this absolutely sounds German there's no way that a French composer was writing like that well actually the computer system says you know well this piece has a 90% probability of clustering with the other French pieces and that's been really revelatory for me and my colleagues so I'm familiar with midi and I'll take for granted that listeners kind of know at least generally what that is that it's a digital format that records the inputs given to an instrument different from like MP3 that's recording the sound waves mid's recording sort of the events that describe a piece of music I have a midi Library I'm familiar with in Python I could use that to read all those events and presumably do some music analysis why do I need music 21 if I know midi if you know midi you'll definitely be able to with any sort of programming package or any Library come up with the count of how many notes are above the treble cluff staff or what's the most commonly used Rhythm you don't need music 21 for that and I I'd encourage people if you know if you want to go forward and this use whatever is most beneficial there but music 21 has a lot of the algorithms that other researchers have developed for things such as analyzing the keys and key changes or figuring out of all the notes that you're hearing at that point what chord is it and how does that chord function in this larger context and you don't need to reinvent the wheel if you'll load your midi file into music 21 or collection of MIDI files in addition there's a lot of richer data formats than midi such as music XML which encodes a lot of information from the score is this thing that sounds loud is it Forte or is it fortisimo loud or very loud in the score and music 21 can read all that information and work with it in a machine learning context as well so yeah if if I wanted to ask myself a question like I don't know what is Box's favorite chord I can see where that would be a daunting task if I just wanted to do it with midi if I ask you to read some source code on air that might qualify us as the most boring podcast ever but at a high level could you walk me through what would it take in music 21 to write some code to answer that question what's Box's favorite chord sure the first thing that I would do is we would load up you know in a for Loop uh let's load each piece in because they're all independent of each other and then we're going to go through and do something called Salami slicing that is to say every single time that any musical event changes we will call it a new vertical sority or chord so for instance if I play a chord like this at the moment when those two notes change we would call that two different vertical sonorities and then music 21 has some analysis packages that can reduce the number of chords of vocabulary reduction to be able to say well a lot of the things that if you look at the salami slices the thinnest slices of Music look immediately like chord changes don't actually function that way they're a melody or some kind of passing tone so then with music 21 we can reduce the size of each piece of 100 melodic elements into maybe 10 chords per minute from there we can reduce this thing that looks like a chord is actually or um can be thought of as a restatement of another chord so that these notes c e g spread widely apart are the same as ceg close together so we'll reduce the number of chords in that way so ceg written with the c and the E and the g written very widely spaced could be thought of as essentially the same chord as C EEG written closely or maybe even c and e without the G and these are things that music theorists do all the time unconsciously or through training from there it would be easy to go through and transpose all the chords so we might say well ceg has a very different function in the key of c as it does in the key of D Minor so we could then rewrite each of these chords not as letter names but as a function so that one is the tonic in one key and the other's the subtonic in another and you technical things like that but from there it would be really easy just put it into a counter object and put it all out and say oh I I don't know probably it's going to be G major is box favorite chord um and you know these are things that come up and then then the fun begins because as a musicologist I want to know why you know I I can suddenly have this data that would have taken me years to get before and I can have it in probably half an hour an hour and then I can start asking well you know why is it that his favorite chord is different from monto's favorite chord who composed at the beginning of the broke era Bach dying in 1750 Mont Verity living around 1600 so in this 150 year period that we tend to just throw together as the broke what changed and how is that interesting where we started out that for Loop where we load all the bo pieces can music 21 help me find those pieces music 21 doesn't have that built-in capability but I've been working with another project called Elvis the electronic locator vertical interval sonorities another one of these cases I think where the acronym preceded the expansion the Elvis project has a large online database of all the machine readable symbolic formats that it was able to find on the internet um midi and music XML ABC I believe and other formats and so they've all been collected there there are other large repositories of huge numbers of MIDI files from the internet I believe there's a Cornell researcher who's just acquired over a million MIDI files that are right for this kind of large scale symbolic analysis wow so it seems like a pretty rich data set to work on then absolutely so I'm familiar with a number of different music formats we've mentioned midi and you had mentioned music XML I'm also a fan of a project called Lily Pond can music 21 interact with all of these interchangeably music 21 can interact with both music XML and midi and there's some other formats capella noteworthy composer ABC which is a large repository of tunes for the Lord of the Rings Online project and other folk music music 21 can read in all of those formats and it can also produce lilyon format it's a text based format that produces absolutely beautiful musical scores it can't to actually read in Lily Pond except the most basic files because Lily Pond files are actually a form of scheme source code macros and so I would need an entire scheme interpreter to read in everything that could possibly be there makes sense one of the other formats that I'm really proud that music 21 works with on the output side is Braille music code and Braille music notation which I I hope will be able to open up the world of scores to a much larger audience of people with site difficulties who will be able to see exactly what this music looks like on paper interesting I'm unfamiliar with that if I saw a printed page of it would it look anything like sheet music it would look to us just like text Braille but for the most part each cell encodes both a pitch name and a duration there's various ways of encoding chords that are very different from how sheet music encodes it and it doesn't uh waste space on needing the vertical component and the horizontal component together to tell us the duration and the pitch and so it's actually in some ways much more compact than conventional sheet music notation there's always a chance that if we were starting over today we might use something much closer to how brailing codes how about on the visualization side of course when I have all my analysis there's going to be a point when I perhaps I I detect a funny cord or something like that and I'd like to see a a printed you know five staff version of it can music 21 help me see music as well oh absolutely since it's a computer-based project one of the other things that we can do is as you tweak parameters of course uh we can generate scores in real time and it's all a process of do you want a very high quality score that you're going to then publish well we can send it out through Lily Pond or Muse score finale some of these music notation software packages that take a little bit of time to render the score or if we want something that looks pretty good and it's incredibly fast we can send it out through a Java Script library and render it in the browser so there's a lot of things that we can do to get immediate feedback in score notation if you didn't read score uh we can also generate plots for instance of a piano roll style where the horizontal component is time and the vertical is completely just which note on the piano is being played and you can see each note there's a lot of YouTube visualizations so this type and you know I read music notation pretty fluently but every once in a while I'll put out a piano rooll plot mainly to demonstrate something to somebody who doesn't read music and all of a sudden I will see repetition patterns in theic music that were never obvious to me before and so a lot of these visualization tools are even used by people who can read the notation so you had mentioned earlier some of the uh built-in algorithms and processes that are available that music 21 gives a user for free could you give us maybe a teaser of what some of those are what kind of maybe the depth and breadth of stuff that's available well one of the things that my students find really valuable is the ability to take a single melodic line and realize it into a fully realized a harmonic package and they find that very valuable because it's generally the things that their other teachers are making them do for homework another really useful set of tools that music 21 has is a large collection of feature extractors so these are uh little tools that change some part of the musical score into a numeric representation and a guy named Corey McKay developed about 100 of them and we've added about 50 more that can be applied to MIDI in music XML files and really be able to look at you know well what's the rhythmic variance between one section and another how smoothly do key changes happen and so on and so forth and from these feature extractors we're able to use off-the-shelf packages like orange to really see how pieces of music relate uh one to another when you mentioned feature extractors and you've already mentioned machine learning previously my head's going directly there that I can do some interesting classif apption or regression type problems given these data sets and tools do you have any favorite examples of work that other people or perhaps yourself have utilized music 21 to do along those lines yeah I had a graduate student who I thought this was really fascinating trying to figure out with these classical pieces that tend to have four movements maybe they're fast slow dance and finale and the student used music 21 to figure out how small of an excerpt of a pie do we need to have in order to know exactly Which movement are we in or even we could extend this to where in a piece are we sometimes we walk into a concert late or we walk into a friend's house and we turn on the radio and a piece is already in progress and we might not know the piece at all like the computer who hasn't been trained on the score and yet we have a really good sense of oh that should be nearing the end it's too bad I missed it or oh good this is near the beginning and how do we know that and how good can the computer be with that and I thought that was a really fascinating project was music 21 started out of uh just the the sheer obviousness of like hey there should be a cool library for working on music projects or did it arise out of some particular need you had to uh solve a problem you were working on independently actually a little bit of both I was working a lot on small fragments of Medieval music at that point and they you know we don't save that much from the 15th century from The 14th Century so a lot of what does survive are you know little ripped up tubs they're tucked Into The Binding of another book or something like that and quite often we'll have you know six notes from one staff and then four notes from the next staff and three notes and I just felt like you know we should be able to identify these pieces that should be enough of a fingerprint for me to be able to understand you well where did this come from and so I did what anybody else would have done is I Googled to see if there was an analysis package out there and there was a very good one called hum drum which was developed in the late ' 80s and early '90s by David hiron at Ohio State University but it was definitely from a different era it was a series of cell scripts where everything was piped together and there were no objects to represent various parts of the musical score and I thought well this would be very difficult because I would also have to encode everything in humdrum format and so I thought well I'm just not Googling well enough so I kept looking around and I thought this was as as you said it's it's a very obvious idea and I thought somebody else had to have done this and it had to have been out there there and I just couldn't find it and so I think that was around 2004 and I waited a year and couldn't solve my problem and I did a very thorough web search again and nothing came up and finally around 2006 I was teaching at MIT so my students wanted to work on projects like this so I said ah heck I'll just go ahead and write it myself and about I guess 10 years later I fully understand why nobody else did it because it's ended up being a lot of work and there's a lot of tricky Corner cases in almost everything in music and that's why I think at this point there's only one project like music 21 yeah definitely so I know there's this stereotype and I know it's wrong but it's useful for conversation of we have sort of left and right brain people um if I uh stick to my I guess it's right brain my more more analytical side uh music 21 is obviously the tool I want to pick up and it was very intuitive for me to learn but when I think about the The Stereotype of the left brain person you know which perhaps where a lot of musicologists lie it bears with it this whether it's right or not The Stereotype of someone who's less technical or or uh doesn't want things as rigidly structured probably not computer programmers I bet the intersection of musicologists and programmers is rather small how have you find that that Community has uh adopted to taking on this technology and maybe having to learn to code just to use Music 21 I will say when I use Music 21 and it ends up confirming something that we've long thought in music history everybody's very happy to have a new technical tool in their toolkit but when music 21 ends up refuting something that we've always been thinking you know then the the difference between the left brain I I don't want to uh have a computer tell me the right way to analyze things uh versus uh versus the you know more technical side then it really does come out but there are more and more people who are going into liberal arts Fields like musicology especially like music theory who are interested in programming and who know that this is important and even if they don't want to personally do it themselves they're interested in learning the basics of a toolkit like this to at least be able to do something like a you know a note counting routine or a search for a particular figure um so there's been quite a bit of adaptation and at least enthusiasm in the field but as we all know who are programming and who are working with these kinds of data sets it's a lot of work to get to the level of fluency with a programming language to be able to really take an idea for something you would like to do or something you'd like to search for approve and actually execute it because of that one of the most important things we've done in the music 21 project is really try to make it so that the naming of objects is very close to what musicologists and musicians would think they should be and things like the first measure of a piece is measure one and not measure zero in um you know with zero indexed uh lists and all these little things like this to try to make it so that we can bridge the gap between the the left brain and the right brain people so that a musicologist can you know he can call up his technical programmer friend and she can write the code that you can make this possible so that two people can work together with a shared vocabulary and that's been very important for me so in doing a lot of my research for the interview um I came across something I didn't quite understand I I thought hopefully I could ask you this maybe you could educate me um there was some mention of uh the fact that we've well studied Melody and Harmony we understand these ideas but we don't understand Counterpoint as well could you tell me what Counterpoint is and and why would it would be the case that we don't have as good of an understanding of it as we do of Melody and Harmony Melody you know is um you [Music] know something like that and Harmony so um we're thinking of something along if we're looking at a musical score that that's really oriented on the vertical sorry on the horizontal axis then Harmony would be a succession of chords something like you [Music] know counter point is the intersection of the horizontal and the vertical so while we're moving from chord to chord on an instrument like a piano or a guitar or in an orchestra especially useful in a choir to think that the chord isn't moving but individual notes within the chord are moving to other notes in the next chord how one note moves in one chord chord to its position in the next chord we call Voice leading and the interaction of all the notes in one chord that move to all their new configurations in the next chord is CounterPoint a point being a Melody originally in uh medieval Latin so a punctus contra punctus Point against Point what's challenging about Counterpoint from a computational standpoint is that there's lots of ways that we can think of analyzing one-dimensional data so a Melody we can essentially put it in a list or in a array or something of that sort and a chord we can do the same except that our array is sort of oriented 90° in the other direction but with Counterpoint we're really thinking of well how do we look at the simultaneous movement of one verticality to the next and the way that composers solve the problem of well I want every note within a chord especially in something like a coral piece I want every note within a chord to move to the next chord in such a way that has a beautiful melodic line at the same time I want all of the chords to move together in a progression that has a beautiful Harmony so the melody and the harmony have to be solved at the same time it's a little bit like Rubik's cube solving where you know you think you get one face done and oh no you've screwed up all the other faces the techniques for being able to analyze Counterpoint require new algorithms they're not especially difficult algorithms compared to some of the things that the other data scientists that you've brought on to your podcast are doing but they're new and they need to be thought of in musically cotent ways so I know music has changed quite a bit stylistically you know like we we hear a lot of seventh chords today and there was a time when that was considered discordant and and not a I guess proper part of Music um I know a lot of your research goes into Medieval music how does Medieval music compare to uh modern music I mean I think that's a general and and maybe too open-ended question obviously it's different and has certain properties but does it follow the same rules and structures it's just maybe parameterized differently than modern music or has everything changed and evolved well a lot of things have changed and when we're talking about Medieval music we're really talking about an era that's you know maybe from 500 to 1,500 or 1400 so it's about as long as you know three times as long as everything that's come after it that we listen to so I can't really say about you know one thing that's happening there but I'll say in the periods that I mostly work on toward the end of the Middle Ages the 1200 1300s 1400s we're seeing a change going from the two consonant sonorities are the fourth so I'll count 1 2 3 4 fourth and the fifth to the two consonant sores being the third and the fifth and the third and the fifth together what we call Simply a Triad or just a chord and the chord began as an interesting dissonant moment that you could put in the middle of a phrase like sort of an ellipses to say hey more is going to come because obviously we can't end on a sound like that and then over time the number of Triads the number of these three note chords that were being used in a row were you know more and more and more and I can plot uh the change in that over time and eventually that Triad became thought of as the stable sority that didn't need resolution and the same thing happens in the 1600s and 1700s with as you mentioned the seventh chord so 1 3 5 [Music] 7 and those chords originally had a high amount of tension and they could only be used in a place where they would be resolved to something more stable and that's basically true up until the late 19th century with composers like Vagner and Strauss and debc Who start to see that as a stable sority but I think really it changes with Jazz that suddenly a jazz composer will put in you know a piece that ends um with with what would have been considered so unstable sority [Music] [Applause] before and and you know sounds like that I'm not a very good pianist in general or a jazz pianist but you know something like that and you hear oh yeah yeah that that's where you put down your Gonic your scotch and you go and you you know applaud and that sounds like an end to us now and so a lot has changed over time and I a lot of times I wish that I could be put into cryogenic freeze and and listen to what the music would be like 500 years from now because I don't think it's changing I think you know a lot of the best music is still to be written it's being written today and will be written I don't think that even if I could use the computer to predict what types of things would come next uh which some people think that music 21 can be used for and it's possible but it's a question that interests me because I don't just want to know you know what are the main trends that are going to happen next but also how are people going to react to this how are people going to feel what's the actual piece that's going to be written with the things that are going to change in the future I'm curious about the data set you have available to study Medieval music you were noting that some of it's coming through in just fragments of papers pressed between books and things like that A lot's been lost I presume if I wanted to look at the complete discography of The Beatles I might have some copyright issues but I know all the information is there how much of a struggle is that to study that time period yeah well actually let me answer the something that wasn't a question at the end first you said A lot's been lost actually one of the first papers that I did that was very mathematical is actually showing that a large percentage of Medieval music survives so that's a very that was sort of the first thing that got my colleagues very you know got got their thorns out or whatever the expr is so there's a large percentage maybe 40 maybe 50% maybe uh maybe even more of the music that I'm studying actually does survive and we can look at statistical tests for being able to assert that and so I I think that hey this is the time to do it because what we have is very representative of what was once written down out there and one of the ways I know this is usually uh these small fragments end up being a piece we already know so we have us we have a small piece of music and we think that you know well uh I can't I can't think of the piece that goes with that so so much medieval music's lost yeah it's probably a new piece and I and once I had the math behind me in this article tipping the iceberg because everybody said well we only have the tip of the iceberg and I say no now we have most of the iceberg once I had that math behind me I said well okay we're going to find that most of these fragments are pieces that we already know so because most of the pieces were probably the ones that surviv we can go to the printed scores that musicologists have been making since the 19th century but especially since World War II and um there's a 24 volume beautiful green set of books called polyphonic music of the 14th century and that's I think it's about 8,000 Pages or something like that and I had a project that we entered it all into the computer there was just no substitute if you want highquality data for uh entering it all and I'll thank my collaborator on on a grow for really organizing this whole project so I have that as a data set I'm planning on releasing that set as soon as I can get what copyright uh concerns for 14th Century Music are still out there there there's there's a few with you know we need to take out things that have been heavily edited but once I can get that out there it will be the only data set of every known piece of music written in an entire Century uh that's that's out there and so that's pretty exciting we can't do that with every piece of 20th century music it's just too big too many copyrights but we can do that with another era and um some of the things I found are that at least 40 of the pieces that we thought well were were different pieces were actually the same piece as some other piece with new text and maybe a little bit of different decorations and you know moved up a key or something and uh there's at least 10 20 cases of composers that we thought their most important piece was such a unique piece of Genius actually seems to be copied or plagiarized from somebody else so that that should be pretty exciting I'm presenting that uh at a at a conference coming up in Vancouver the big musicology and music theory conference in November oh what a treasure that's a great data set so it's it's obvious to me that a musicologist would want to use Music 21 uh I have course see the immediate use for someone with some machine learning Talent who wants to get in and and do some neat projects are there any other groups I'm not thinking of who would benefit from leveraging the library definitely composers use it a lot and while I'm a musicologist primarily I also compose and so I try to think when I'm designing a new module well you know how would a composer use this repetition expander module or repetition finder module to make something new how can we you know manipulate a particular scale algorithmically so that I'll use that in my analysis I'll manipulate a scale to figure out well is there something happening in this section that's actually just a slight manipulation of a common scale but a composer can use that in the reverse way to generate all the different permutations of a scale or manipulations that have a particular property and put it into her or his piece so composers are big users of the of the system oh very interesting yeah that makes sense um so in terms of getting involved is this an open source project is there room for a contributor absolutely I would love to have more contributors uh it's open source we can find it on GitHub uh just music 21 and it's um very cool and if I wanted to started playing around with it where's the best place for me to get the hello world equivalent in music 21 sure I think the the best thing is just to um you download the package and then you'll find docs at my website at MIT web.mit.edu music21 I think SL do if you want to go directly to the docs and there's a tutorial users guide you'll see it starts petering out around chapter 15 but the you know 15 chapters are very good examples of what you can do and you know I hope people will contribute more excellent I'll have links to all that in the show notes for anyone who wants to give it a whirl before we go any other uh things You' want to highlight around music 21 maybe interesting use cases I came across score follower for example yeah yeah well so some of the things that we're trying to do with music 21 a score follower party Chima is an attempt to automatically reduce a large complicated score into the most important elements so you can follow along and have it synced with an AIO file and that's something I've done with Vladimir Vero of peach note another project I've done with with a great young researcher named MOA church is to use music theory the examples embodied in the music 21 library to improve what's called omr it's the musical equivalent Optical music recognition the musical equivalent of optical character recognition and um and the output for that is very promising but it's not yet at the level that you know that you can immediately use it in any particular project so we can use music theory to improve the output of the omr and you know maybe a third of the time we can take what was definitely wrong and make it absolutely right in what was written on the page depending on what the problem is higher or lower well Mike this has been fascinating thanks so much for coming on to share some of the story about music 21 and ways people can use it I hope everyone goes and checks out the website thank you very much it's a pleasure for more on this episode visit datas skeptic.com if you enjoyed the show please give us a review on iTunes or Stitcher