Designing Recommender Systems for Digital Humanities

Key Takeaways

Welcome to Data Skeptic, a podcast [music] exploring the methods, use cases, and consequences of recommener systems. Welcome to another installment of data skeptic recommener systems. Today we're getting into recommenders in the digital humanities. That's the interdicciplinary field that combines traditional disciplines like history, literature, philosophy, art, cultural studies, and so on with modern digital tools, methods, and computational approaches. Our guest today, Florian, works in this space, particularly on a platform called monasterium.net, mast.net, if you want to check it out. It is the largest online collection of historical charters and it has millions of medieval and modern historical documents from all across Europe. With a deluge of data like that, you're going to need a good recommener system to find the things that you're interested in. We talk about how they're building that in today's interview. [music] I'm Florian Atenhofa and I'm a PhD student at the Gratz University of Technology and in addition to that I'm a developer and IT project manager at the University of Gats in Austria. >> Can you share a few details on what you're studying in your PhD? In my PhD, I'm focusing on recommener systems for the digital humanities. And I'm focusing on three specific aspects of recommended systems in the digital humanities with a specific focus on digital archives. And that is multistakeholder recommended systems and problems associated with this. Then the information seeking behavior of specific groups of users. And third, I'm focusing on the specificity of cultural heritage items and the repercussions on recommended systems of that. >> Could we zoom in on what the humanities need that's, let's say, unique and you couldn't just use a vanilla off-the-shelf recommendation approach? Um well first of all I must say that there is a huge lack of recommended systems in what you can deem a digital humanities application because naturally uh the digital humanities have been concerned a lot with preservation of cultural heritage data um and they are only slowly gliding into more innovative ways of how to present the data to users so other platforms and users. This means that there have not been so many different applications of recommended systems in digital humanities and for cultural heritage especially for digital archives and we are disregarding here the digital libraries aspect which includes u for instance scholarly items such as uh research articles when we focus on primary sources so more historical material we're completely in a new field and is not yet tested whether offthe-shelf models also work well or not. And I am arguing already before I'm beginning my work in my thorough work uh that there are several hindrances to just using an off-the-shelf model. You have to deal with a more complex data model. So the way of how you structure the data, the way how you serialize the data and also the routes to that data are wholly different than for instance in more commercial applications. In addition to that, you have different user needs. You're not dealing with a, as I said, a commercial application. You're not dealing with a typical user item relation, but you're dealing with people and users that are much more invested in the application as you would usually be. So, the stakes are maybe not that high as in other domains, but they are definitely complicated and we are only slowly grasping how to uh formalize them and how we integrate these needs into the system. when you work in the digital humanities, who is your user? >> First of all, I would say that the digital humanities cater to researchers in the first place and and generally to the public domain. So they definitely target any user that would also be let's say quote user of any glam institution that includes galleries, libraries, archives and museums. So it could be any other citizen that would be a valid target user >> in your domain. You have these experts that have I presume a lot of deep knowledge not necessarily so much surf well they have surface level knowledge but they don't need to query for it and maybe you have only a few people working in areas. I guess I'm just speculating as to why user item might not be perfect. Could you expand on what the challenges are? We do have a a specific relation that is driven by the fact that we have so selected a handful of experts that is very knowledgeable about a specific field and there is a discrepancy between those and the more novice users. We need to exploit that relation a bit. uh for instance leverage the knowledge of the experts for instance for cold start problems in recommended systems that is the problem that you have new users um and you have a lot of items and many of them and the very large majority of them have not been interacted with. This means that you have a very sparse user item interaction matrix. It is even in other domains where you have more interaction between them. it is quite hard to predict what what item a user might like and the more sparse it gets the harder it gets. This means that um we need to leverage specific methodologies for instance uh how you deal with cold start problems or how you could lead novice users and do that in a grounded way. This means that we will consult with the experts and we will also consult a potential novice users in how they would like to be guided, how they would like to receive recommendations. >> Very interesting because yeah, it's a journey. I could see a firstear researcher asking very different questions than a fourth year researcher and maybe the system could become aware somehow. >> Yes, exactly. And also, if I may add here, we're talking about charters here. Uh that's where I'm coming from. That's where my project is working in or working with. And charters are relatively encompassing for medieval times and modern history. And we have hundreds of thousands. No, we have millions of them spread over Europe. Yes, there's a a huge bias towards Europe. And one of the major problems considering this is that we are not talking just about historians or auxiliary historians who could be interested in this source because the source is so important to understand past for societies. But this is also used by genealogologists by art historians because we're dealing with illuminated material pictures and imagery in these in these objects and also you would have linguists who are interested in the languages that they carry. So we would have to branch or consider the interests of these different let's say types of researchers as well if we want to be a successful recommended system. >> For listeners who haven't yet visited and we'll put a link in the show notes to monasterium.net. Is that correct? >> Yes, that's correct. >> Could you share some details and what they'll find there and maybe what your contributions are? >> So monasterium.net has been around for more than 20 years. It was created in the heart of Austria and lower Austria by a set of monasteries who just wanted to provide a better way for interested researchers or just any other person to look at their very nice material. So they used the traction of let's say the world web at that time and the expertise of some IT people and came up with a very first version and a basic version that is a digital archive for charters. It started small and over the years it grew up to be the largest online collection of these materials. The project DIP which I'm currently working in and in conjunction to which my PhD is done rebuilds currently this platform and we are up to releasing the next version towards the end of the year. So when you visit monasterium.net net currently you will see the old version and there will not be a recommended system yet implemented and also not any of the new fancy semantic search features those will be available by the end of the year. >> Can you maybe uh tease a little bit about the user experience? How will um someone who's coming to the site once those features roll out, how will the experience evolve for them? Well, we differentiate between uh an exploration page and a more controlled way of looking for let's say similar items because we know that we need to differentiate as we discussed earlier between more expert users and more novice users. And we found the best way to enable this is to separate the functionalities a little bit to components. And this has results on the user experience. Which means if people are noviceses um they are more guided let's say or probably more prone to just using the exploration page. They will be able to select different profiles that we have preconfigured and which are based on priorities of research or looking up items that we have found are important based on just our experience with working with these different people with the different users and the different stakeholders. And then there will be other features such as selecting individual items or groups of items and building new sets of data based on that. So you could have one or more let's say charter baskets and they are called baskets and based on these subselections you can exploit them to find more similar ones. This search is more based on reranking and less of a recommendation but rather a controlled exploration. >> Could we zoom in a little bit there? I think most listeners who have a surface level understanding of recommener systems will know things like the OP priority algorithm or user item similarity like we're matching maybe jacard distance and these sorts of things. Are any of those techniques useful or do you have to adopt different methodologies? In essence for a recommener system the most important thing that we have to consider here is that we have this sparse matrix of user and item interactions and we have a very cold starting problem here which means that we leverage a lot the knowledge base behind this data set and the content of this data set. And what this means for those that are not familiar with it is that we leverage different modalities of the charter data and we embed them in different spaces. In addition to that, we make it so you can combine them and also weight them a little bit. So for instance, we know that one major use case is based on finding uh new or similar items just based on the text content of one charter. Assuming here that we do have a transcription. if it is not created automatically by our HDR pipeline which is the automatic text recognition. If that's the case then a similarity search based on text could be very very beneficial for them and this modality of text similarity could be of interest and this can be combined with other modalities such as a subspace selection of imagery. For instance, if people would be interested in a very specific imagery of, for instance, animals that are depicted in these charters, and there are several of them, they would weight that feature more. Depending on how you prepare these um embedded spaces, you can allow the user to exploit them. Given that we are a project and we have a specific timeline, we do make a selection of features or feature spaces. So a selection of modalities that we find most important and give the power to the user to weight them a bit. This means that they can exploit these modalities and by this influence which things are being recommended to them and the side effect of this is that we can then fill this interaction m matrix and make it easier for later users to use the whole system. >> Am I correct in saying that a charter is the item that you're ultimately going to recommend? Yes, exactly. At the very heart of it, actually we focus on images, so scans of chargers which are usually very high quality. And based on our pipelines, we create let's say the text. And based on this pipeline of the combination of a scan plus text recognition plus metadata that we either get from edition or the archive directly, we create a very structured document which is in a XML encoding called the charter encoding initiative. This is an homage to the text encoding initiative and has also been born 20 years ago and it it's still being used not just by our system and through this uh serialization we use one document and that is also one interpretation of such a manuscript in order to recommend it as an item. So yes, one charter is one item. So, uh, a charter, it sounds like it's frequently an image, but there could be metadata about it. Maybe you've transcribed something and then perhaps even that got translated. I don't know how it works, but metadata like the uh the date it was created or where it was found. These sorts of things. What's the field of metadata available to you? >> So, it it depends on which pipelines have been applied to this specific original data. We assume that we either have an XML serialization already provided with metadata from the archive because we're also working a lot with material that has already been digitized or has been around for some time or we get just the scans of images and apply different pipelines on it and depending on which pipelines we have different metadata fields available and they can then be embedded. So for instance we can assume that we have for all of them some form of automatic text recognition also in connection to this we have abstracts and they are called regesta in the proper terminology. So essentially these are short summaries about what the charter is about and this is really important because charters can be very very lengthy and also in the past scholars of the 18th 19th century have employed writing abstracts or short forms of it just to be able to sort them better and we can leverage this tradition as well uh for similarity search and for recommendation. In addition to a text, an image, an abstract, we have much more information and this is in connection to the tradition and the discipline that is dedicated to the research of charters and this is called diplomatics and the field of diplomatics has come up with some features that are important such as who is the issuer and in connection to this to NLP applications we can leverage named entity recognition and a proper serialization for this. This is not so easy because we're talking about historical material here. We can leverage named entity recognition and a semantic labeling. For instance, this person is the issuer and the other person is the receiver. And we can leverage this uh technology to create new embeddings as well. And we can make them either implicit or explicit depending on the use case depending on the downstream task. Delete Me makes it easy, quick, and safe to remove your personal data online. At a time when surveillance and data breaches are common enough to make everyone vulnerable, delete Me isn't just a one-time service. Delete Me is always working for you, constantly monitoring and removing your personal information that you don't want on the internet. Their team of privacy experts will remove your data from hundreds of data broker websites, giving you comprehensive coverage where it matters most. As someone who values data privacy, I've been alarmed by how my information keeps reappearing on broker sites even after I've manually requested removal. That's why I appreciate Delete Me's ongoing service. They don't just remove your data once, they keep monitoring and removing it regularly, saving me countless hours of frustration. Removing your data will help you protect yourself from potential identity theft and unwanted exposure. Take control of your data and keep your private life private by signing up for Delete Me now. At a special discount for our listeners, get 20% off your Delete Me plan when you go to joindeleteme.com/data and use the promo code data at checkout. The only way you get the 20% off is to go to joindeleteme.com/data. Enter code data at checkout. That's jointdeiteme.com/data code data. And do you have any thoughts on how you'll measure the success of the system? I guess ultimately you want people to find novel recommendations, but how will you know if you're achieving that or to the degree you're achieving it? >> Yes. I mean there's very classic approaches to the evaluation and there are different metrics and there are things like root mean square deviation or error and you could look at metrics that are more important for aspects like fairness where you essentially look at the propensity or the amount of of charters based on a ranked list which have been liked or preferred by one group of users versus another. So if you look at multistakeholder recommendation and this is important here, we have to look at the overlap of preferred items versus dispreferred items for a specific subset. And we are only slowly figuring out what the best way of evaluating the recommendation system is. We do have evaluation procedures for every other pipeline that goes before the recommendation. But for um coming up with a full evaluation scheme uh we are not quite there yet. This is in the works and this is also where my uh PhD or parts of my PhD come into play. And what I've done over the past few weeks and months actually is I've talked a lot with different stakeholders and I've done structured interviews and focus groups with several of them. uh that was a selection of 25 experts from a pool of over 70 plus and I've talked with upstream provider system consumer and downstream stakeholders and that would include a whole range of archavists specialists the developers also of of this system or rather similar systems but also the researcher educator students that use this material and use this platform for teaching And also lastly to produce scientific output and that's where also the downstream stakeholders come into play because at the end of the day what is being shown in such a platform what users look at and as such what is being recommended to them could end up in a published article could end up being edited for school books even. So essentially we need to consider different perspectives on what is being recommended and this can be leveraged for evaluation as well and I'm convinced that this is necessary for a public and less commercial domain. I'm curious if you got any specific insights from those user interview sessions. I've been on a few situations like that and projects I've worked on and I've always found it very insightful to directly connect with my users and hear their needs. Definitely definitely many findings and I had the chance to also put this into a paper but what we found is that it's really not easy at all to structure the let's say the preferred values associated with the recommendation and we came up with what we call the research funnel or alignment to the research funnel where we differentiate between different phases of such a interaction between user and item. So this begins with the discovery stage. Yeah, this is just about the exploration, the first contact so to say. It goes over to a interaction stage where the users u let's say are a bit more cautious and hesitant about how this item might be useful to them in that very moment or even later. Then it goes over to the integration stage where this question is already being settled and users think more deeply about the item. For instance, taking one item plus comparing it to another. And we came up with also the fourth that is the impact stage. Here things come into play that ask how useful is this for further exploration of sources? How useful is this for actual publications and how important is this to other stakeholders. We propose based on these stages different let's say metric directions. This is not a full evaluation framework but rather it is a a pathway towards it. So we call them um evaluation setups and we propose different metrics such as research path quality. This is a subjective measure of the user how it integrates with their specific research journey and we know that these researchers do have very individual journeys at times. Sometimes you don't feel like that. um sometimes you do and what we also found is that a lot of the stakeholders they're not only interested in explanability but they are also interested in the biases prevalent in these systems. So what we found is that essentially all of them somehow find that we should consider collection representation as one measure. That is we should be able to consider that one overly represented archive does not overshadow others. And with this risk that some things some charters are just lost and being behind in the being lost behind in the shelves being put away and not seeing the daylight anymore. So the digital daylight so to say. This means collection representation is also important to them. Yeah. And we proposed several other metric directions which we want to explore and as soon as we have operationalized them as soon as we um formalized them um and implemented them we can also evaluate. So many years ago, I saw a presentation by someone. I believe they worked at amazon.com uh on the recommener system team and they said, "The recommener will be complete when you visit Amazon and there's a single product on the homepage and you immediately buy it." Which [clears throat] I hope they meant as a joke cuz it's sort of silly, but you know, an extreme version of what's the goal. So I guess my question for you is what's the goal? Discovery is a big part of it. Like if I'm exploring these things and I'm a researcher, a little bit of serendipity would be good. Learn things I didn't know I was looking for, but I also need to get my research paper done. What uh how do you think about the best experience to create for a user in terms of what they're finding maybe versus discovering or getting right to it? >> Well, certainly uh just based on the nature of this domain, serendipity is very important. We're talking about research here and the context of cultural heritage. So, it's about finding stuff and finding some form of identification with it. And [snorts] this was also reflected in the focus group interviews that I've done for this past study. Uh, and it was quite interesting because stakeholders inter interpreted serendipity in different ways. I've had one um upstream stakeholder that it was an archavist uh and they argued the most beautiful finds you make are the findings that you would not expect. Okay. And then they go on and you would rather find them by equal representation which is quite interesting because you would rather have a notion that disentangling the system from biases would not have an effect on the serendipity. and another consumer stakeholder that was a developer I think no that was a a specialist they emphasized that they think that we need to constantly be challenged so that we don't pursue the wrong and where we think we're going so there are several notions of being challenged while also being able to trust the system and the consence was also that at the end of the day this platform and this recommendation system should not be a co-author in a publication but at least a specific version of the recommended system should be acknowledged in a dissemination in a scientific output. So when people have used them for research they should acknowledge that they did so and also which specific version which is kind of hard to grasp because usually we assume that a recommendation scenario is very subjective. So the question is does it really make sense to let's say acknowledge the system where it was rather a very specific and individual situation of a single user? I'm not quite sure yet. >> That's very interesting. Yeah, I've worked with some companies who will store a uh unique identifier telling you exactly which version of their software in the git repository is running at the time that some observation was made. You could make some similar approach where you give your users a receipt with an ID, but I don't know what they do with that ID next. Although it's interesting. >> Yes, exactly. That's a very good point. We're also including let's say the DOI, so the builds of the specific software stack at that time and they very well can include that for any output or any citation. But still the question is whether the users have really understood it and also if the user base can generally understand let's say the tech stack and the technology behind it because most of the stakeholders lamented that they find the technology very interesting and they can imagine it is extremely useful to digital archives this one specifically but they also generalized to cultural heritage in general but they lamented as I said that they would like to understand the system as much as possible. while it should be most effective and also carry explanability. So we believe there must be some tradeoff and the most easy form as you put earlier would be just a single item recommendation and nothing else which of course takes away all the agency all the understanding process that scholars uh set to desire. Well, text similarity is something that has evolved quite a bit or your options have grown at least let's say over the last two decades. Sometimes something simple like TF will do the job for you and that's okay. But we've also got you know uh neural network based things and embeddings and stuff like that. What was the methodology that worked for you? >> That's a very good question. My thesis itself, the work I've done was both theoretical and practical. I've had a very very thorough introduction into different aspects into what I call formulacity or what is called formulacity that is essentially the formulaic language aspect that is prevalent in charters and that's very crucial and I showcased in my thesis theoretically and practically exactly that you have different ways of looking at similarity and similarity means different things to different folk. Um and this is a let's say a a critical topic to this day in our team and uh to the associates. Um it is really hard to measure it and you it it seems that you have to show them a lot especially the non-technical people what can be understood by a specific kind of text similarity as you've put earlier. Um it's true there are more simple measures but there are also more complex ones and at the end of the day it really depends on the downstream tasks and this means that we are bound to these tasks and they require us to to embed our data in a different way. For instance there of course you can take TF and this is rather still rather effective. It is also the case for historical data it seems and this has several reasons also due to the pecularity of certain languages or certain terminology or certain regions. And then you can go over to more let's say more sophisticated but still still very understandable embeddings based on or features. For instance, just skip grams or leave an out grams, which means uh which is a a similar form to conk grams. And what you do is you have certain phrases, you leave words out and you still consider them to be part of a specific phrase or you also consider order of these words and still subsume it under the same group and then you do the embeddings. Well, my hope for you as the recommener system rolls out is that you become a victim of your own success, that everyone's very happy with it and they're demanding a little more. Hey, how do we take this to the next level? What's the next level? >> Uh, well, certainly I hope that the platform that we've built and renewed grows and grows and grows and things are looking very good. Our project is also engaged in getting more data into the system. It's becoming more, let's say, international. We are filling the gaps that we have geographically in Europe such as more towards western of Europe and towards the north and actually we're talking about not just a handful of charters that is more in the system but tens and and hundreds of thousands of charters more and further down the road we could imagine that yeah recommendation systems as a means of exploration are being picked up by not just um our digital archives but also to other digital archives that contain cultural heritage material. Um, we are convinced that or rather it is a fact that a lot of repositories that are currently concerned just with the preservation of already digitized material are CEOs. They are yeah they are bunkers. They are really closed and it's sometimes hard to get the data out without just flattening them and making them unusable. I hope that exploration through recommener systems will catch on with other archives. And we could also if we go down one step further, we could also imagine recommendation on a more aggregated level assuming that the data is not too flat. So if you think about aggregation services be it on a national level or for instance on a European level such as with the oil pana we will definitely see in the next let's say 2 years that exploration through recommendation will catch on and will be implemented. >> Well my personal bias is always towards like open things open source open data maybe a federated system open standards open protocols things like that. Um, you're free to disagree with me, but uh either way, do you have a sense of the state of openness and standards in digital humanities? Like, uh, how advanced is that? And, uh, would you like to see it evolve in any way? >> I think, uh, you mentioned it especially in connection to the point I made earlier about the repositories. But in general software stacks that are being used as repositories are mostly open source already but they're also changing technologies and it depends on for instance the institutional support also which software is being chosen as a repository software and whether this is a closed source um closed source software or not. So the researchers or the technologies uh technologists who working in digital humanities or cultural heritage glam or repositories they don't have the sole say in designing this but they definitely favor open source solutions and if they had the chance they would always go for open source. I see the digital humanities community as a very open one and they have always been and they showcase this over many decades now that they can very easily mutually influence with other disciplines uh be inspired by them and this is also the case with let's say the latest traction it gathered from computer science applications being more mainstream also LLMs being taken up in their research and just showing people that you can do a lot with them. >> I know you mentioned you're currently involved in your PhD and we talked a little bit about where you're headed with it. Could you remind listeners uh what you're doing and I guess what your next steps are? >> So I'm a second year PhD student and my progress is quite good. I've already chosen the the specific fields or the specific subtopics of my PhD. So these are three strands and I'm focusing on the multistakeholder perspectives of charter recommendation as well as the information seeking behavior of the specific users and third I'm focusing on the multimodel similarity um on different multimodel similarity approaches that you can leverage based on cultural heritage data. So currently I'm focusing on the multistakeholder problem uh on its evaluation. I'm presenting a paper on this at the recommender systems conference in Prague which is in September and also I will be part of the PhD symposium where I'm again presenting and discussing my ideas and whether I'm on the right track here and later down the road I will do some more literature research and try to find similarities and dissimilarities between the information seeking behaviors of different recommended system user groups and I want to find out how from that perspect perspective what I could improve in my application domain further down the road or rather in parallel since this is also very important in our project we're focusing a lot on multimodality and how we can leverage this and how we can use multimodal representation to give power to users in doing search and we really believe in this so that's where I'm heading currently >> well exciting stuff a lot of areas I hope we can follow up on I imagine there'll be some interesting progress along the way to look at. >> Yes, I hope so. >> And is there anywhere listeners can follow you online? >> I've put a Twitter handle in the description. I think I'm not very active in it, but you can still follow the project DDP that I'm still in and any other follow-up project on that one. And the URL would be dip.eu. >> Very cool. We'll have a link in the show note as well. Well, Florian, thank you so much for taking the time to come on and share your work. >> Thanks again for having me. It was fun talking to you. [music] >> You as well.