Comprehensive Overview | Obsidian Block References & Transclusion | Sorry Roam!

Key Takeaways

[Music] welcome back everyone if you're new here my name is brian jenks and today it finally happened obsidian has block level referencing for those of you who are interested in personal knowledge management you're aware that there are two really popular applications nowadays in this space one is rome research and the other is currently obsidian i made a video prior about rome versus obsidian some pros and cons to both which there are pros and cons to either application and you can catch that up there or wherever i can put a card and what we're going to talk about today is block level referencing block level referencing was one of the last hold out pieces of functionality that rome research really was holding over obsidian a lot of people who were on the fence about rome and obsidian really wanted to have block level referencing in obsidian and at the time until just today obsidian had no way of doing block level referencing to individual lines or paragraphs all we could really link to was headings or entire files and you know for some people this might work if you kept your atomicity of each individual idea or note into that level you know just using headings or the name of the file that might be good enough but for some people they even want more granular referencing where you can in you can reference individual blocks of text inside of a document the obsidian developers were working with the people who have developed foam which is a personal knowledge management system in vs code i believe and there was some black magic going on with uh some text references for uh markdown where you were actually using like the upper carrot symbol and a randomly generated uid for blocks of text and this has worked it's not part of the markdown standard but we have block level referencing in obsidian so all the people who are holdouts for rome research because of the block level referencing we have it now so that is what we are going to talk about today in detail so let's jump right into obsidian so first of all block level referencing is not currently available to the general public block level referencing is in version 0.95 which is an insider's release as of today october 19th 2020. if you're interested in getting insider releases and early access to a lot of these new releases in obsidian you can support the devs at obsidian.md as a quick aside i remember watching this video about the development of foam's block level referencing back in early september by uh yani i'm just going to guess it's pronounced but i remember watching this and then using hashes right here for the references that appear at the end of individual blocks of text these are treated as metadata and not really displayed on preview but these references were actually using these hashes to point at individual blocks and lines of text and if i'm not mistaken this is actually what has been implemented in obsidian so this is a little bit of crossover work from foam and the obsidian developers now it is important to know that this is not part of any markdown standard or specification so using this feature is not really going to play nice with a lot of other tools unless they are specifically designed to use this feature implemented in this way now there are some cool like tricks we can get around that and i'll go over a little bit about that later but uh for now we're actually going to see this live implemented and begin using these references so block level referencing is going to give us a whole new world of functionality in obsidian where we can now reference granular pieces of text not just headings and not just file names this opens a lot of new doors and there were a lot of new options added in this latest release of 0.9.5 and we're going to go over pretty much everything in gross detail and some of my thoughts about these things so let's jump right in so first and foremost you're probably aware by now that we can reference individual headers and the names of an entire file so for instance i'm in my note for andy matushak and i have a note for evergreen notes obviously these are two connected notes because andy matushak is the one who coined evergreen notes but in any case i'm going to reference uh let's say evergreen notes and i don't really have any headings in here so actually let's do um andy's note here for socials so if i do andy and then you know we all can also autocomplete these things now so i don't need to type out or tab this note i can just do a hash symbol and that will auto complete and then tell me okay hash symbol so you're looking for headings so which heading would you like to reference in this note and let's just say socials cool so that's that's all there is to it i can you know preview that and it just shows that that's it that's it so now if i actually preview and hold in my case for mac command it'll actually show me a preview of that entire block now my little reference here is part of that block and let's see if we can actually change that all right so now because i have another heading separating this i'm only referencing this heading and this entire block right here so this goes based on heading blocks everything underneath that heading at the level that that heading is at is going to be included in this reference so i can preview that it just shows this i can hold command for mac and then hover over and it show me it will show me that entire block for socials which is the second level header if i had a top level header above here and i just said you know andy and then i can change this reference to you know the h1 header here now andy and then go back to preview command and hover it'll show me the entire file everything that is under the first level heading so in some cases this is actually beneficial because you can capture multiple swaths of information from a file but it's not quite as granular as individual line and paragraph block referencing now it's also useful to know that pretty much all of these you can add a bang or exclamation point to the beginning and that will actually transclude that piece of information transclude meaning you are including it but it's from a different location or in this case it can be different files we want to preview and show that contents embedded inside of another document so i can preview this and now you can see the entire note is actually referenced within itself which is why you get this you know endless inception up to five levels of transclusion here which uh it's kind of fun to play with but in any case that bang symbol to transclude the piece of text or you know reference to that document can work for document level header level and now even the block referencing level all right that's cool whatever i don't care brian show me the block level referencing you got it so block level referencing instead of using a hash symbol we're going to use a caret symbol we use the hash to reference document names or actually use the hash symbol to reference document headers we can just reference the document name with the link and then we use the pipe symbol to change what the preview text of a link looks like but now we have a new symbol so now i can do andy and i can still autocomplete but now we're going to do a carrot symbol the carrot symbol little upward arrow will now show you this menu now you saw before when i was referencing headers we actually had h1 h2 it was telling you the level of the header there in a little you know more aesthetically pleasing graphical kind of way i already talked to the devs about changing the graphics of this this is still showing the hash symbols for the levels of heading and you know that's cool but we can see line by line individual bullet points i have here are all referenced in this menu because the caret symbol is saying hey which block of text ie a line or a paragraph would you like me to reference individually now you can see i have a little gray something right here like what is that little piece of metadata we're going to get to that but what this looks like in practice is i want to say that you know good old andy matushak here is a knowledge worker so i'm going to say uh good old andy is a and then we're going to reference that little block of text so if i just do this as a reference it's just going to be a link i want to actually include it so i'm going to add a bang to the beginning all right so now carrot what do we want to reference that he's a knowledge worker you can watch the preview over here what happens but when i click knowledge worker now you have this little generated hash over here and over here you have a hash symbol like we're referencing a header but instead it's actually referencing that hash number or id this is a randomly generated unique identifier or uid this is a very critical piece of maintaining references to an infinite amount of blocks but in a unique fashion because with each uid you have the possibility to be all 10 numbers 1 through 9 and 0 and all 26 letters so that's a good amount of options for each digit of this which gives you a large swath of permutations so in that case keeping these uids means that you have a unique id for every single individual block that you will ever reference in your system all right cool we made the reference the metadata has been added to the the source file and so now let's see what it looks like in its destination here when i preview that we can see good old andy is a knowledge worker now obviously he didn't display andy's name but this is a way of including an entire block of text as if we were just saying hey give me the file embed the file or embed everything underneath this header in a file and so this way we can easily embed little pieces of text information entire paragraphs and individual lines if we so chose so you've seen it before that if i just did two brackets to open a link i can reference any file in my entire vault doesn't matter if it's in a folder it'll show me the file path but i can reference any file anything so i can reference andy matushak here great that's it that's the entire file but now you can also have two other different kinds of global searches so now just by opening two brackets i can search for the heading inside of any document without having to specify what document i'm looking at first where you normally will reference a document and then it's headers like andy matushak here hash now i can see its headers well instead of this we now have a global search for headers which we can do with hash and now it's looking through every single header in every file in my entire vault this is really great for discovery because now you can reference to individual headers and everything below them so let's just say um i'm going to say oh bash and let's do streams okay cool let's transclude that preview and now i can see everything under the stream's heading in my bash map of content really neat so that's really cool but that's just headings we just got block level referencing and a nice new shiny toy of bracket bracket and now with double carrot symbols because we used one carat to reference a block of text in a file we can do carrot carrot and now we can reference any block of text in any file in our entire vault every file every line and the reason this works is that every single line of your file as long as it's you know following the markdown specification so you know you have a blank line between lines of text or you're just referencing an entire block of text or a unit like a image or a table then you can reference that entire block with one of these uids and so by using this block reference with that double caret symbol we can see whatever file we're looking at the headings as well as the individual lines within those files and because it will assign a unique id as soon as we make that selection we can then reference that block in any other document and it will still use that existing uid alright so let's see the global block reference in action i'm just going to reference another thing inside of this document so i'm going to reference knowledge worker here again so we already have a uid for it and so we don't really need to do anything about that it's just that exists now that block is forever until i change it uniquely known as that hash id so let's reference knowledge worker and you can see that every time knowledge worker is referenced i have the name of the file and the block that is being referenced now i can see right here and even 2 shack and knowledge worker i'm going to click that and now it just gives the hash symbol and the block that's it we don't have the name of the file we don't have any sort of heading it's just the block reference and if i you know preview it we can see that now the hash symbols have been are being replaced by the uh greater than symbols and this will be essentially pointing to the block or the reference so if i hover over this you can see it's just that bullet point even though this bullet is part of this list it still will only show me that bullet and if we take this block reference add a bang now it's transcluded and we can see just that single piece of information now some of the really cool things we can do with these block level references are we can move them so this is the little piece of metadata the uid that is pointing at this particular block knowledge worker now you can see that that preview referencing that block 573 e3a 573 e3a cool those reference each other so i can say hey that's knowledge worker that's knowledge worker that's where that uid is that's cool what would happen if we just cut and paste so let's preview again different block reference now so what this is doing is that this little piece of metadata is pointing at whatever block it's next to or line that it's on whatever piece of information this is pointing at is that uid's block so if i say hey show me that block reference 573 e3a wherever i put that darn thing is what's going to be displayed or referenced to so in this way we were actually able to move these blocks if we really wanted to and it's an easy way of maintaining them if you needed to move something around or quickly change a reference now one thing you can do with these little uids and little pieces of metadata is we can actually define our own if we so chose personally i don't think i would ever do this because i like the randomization of the hash and knowing that i'm never going to accidentally reference something in a duplicative way and it always is what i expect it to be so in any case you can do this if you so choose but we can actually define our own custom block level references i'm just going to keep it simple knowledge worker is now the block level reference uid of test cool so now we can use the no the syntax now for these block level references which is a beginning hash and then the actual uid so let's do bang because we're going to we're going to transclude and show the result hash and then it doesn't give me an auto complete menu you know okay but if i do test now it will display knowledge marker so the block level reference uids if you do them custom like this will display but personally how i reason why i would not do this is if you use this hash and you're referencing the name of a header or something that's common you know common language english for me in this case i might get that confused with something accidentally reference that as a heading or some other weird thing and it's just not worth the headache or worth taking up you know cognitive load or mental mental thought space for that i'd rather just keep it random keep it a uid randomly generate a hash and it's forever randomized and perfectly reference referenceable now block level referencing for me anyways was released today hours ago it's still beta so not everything works perfectly yet and that is a-okay so one of the things that foam currently supports that block level referencing and obsidian does not yet is the updating of the names of links so for instance you can see i have at andy matushak here if i add a one to the end of this you can watch the link as i press enter the links will update automatically everywhere that's a really cool feature because i never need to worry about those links being updated and always current and i believe that's a setting use a feature that you set in your settings of when you want those to all be randomly updated automatically or if you want to be prompted for that but in any case so we had the block level reference i'm going to set this back up here we have the block level reference to test i can preview it knowledge worker knowledge worker test yes so i had that set but what happens if i rename test here and i just say testing okay uh oh i broke the block level reference okay that didn't work so what if i change the name of the reference id here testing preview nope that didn't work either okay so the block level references unlike the names of links and unlike block level references in phone which i believe is what this implementation is mirroring is the uids do not update across all references of themselves like the names of these files do in every time they're linked in any other file that might be functionality that will be coming soon but as of today hours after release of course there not everything is perfect and you know 100 so for now the uids do not auto update themselves okay and another good reason to in my opinion hash the to keep the hashes is that and this is going to be one of the hacks i'm going to talk about later of keeping up on all these things is if it's unique enough like these then you can easily grip them and then do said to search and replace if you've been following my channel from when i'm i'm really into command line utilities doing regex and search and replace on these would be a really great way of updating and maintaining a lot of these uids until better functionality exists in you know the gui interface okay cool we actually have block level references to little pieces of text this is incredibly useful and practical so what are some other cool things that we can reference by block not just a line of text there are some other cool things let's test some of them out so i'm going to add a bang because we want to transclude and then i'm going to say i want to search for c plus plus data type variables and data types and i want to look for the blocks and let's look for the table yes this is the table you can't tell here because the pipes are not showing but i'm going to reference that table and let's see what this transclusion looks like when previewed so markdown tables the entire table can actually be referenced as a block if we actually go to that file and look at it you know and i clicked on the link it'll actually highlight in green that entire referenced block now if i go to uh the actual raw text we can see that the uid is right here e35560 cool and it's on the line below now i like to think of this as the carrot is pointing up at its block in this case the other uh uids are on the same line i just look at it as like footnotes you know how the footnote syntax is just referencing you know upward carrot and here's the the footnote but you know that's fine and i like to imagine this one is actually pointing up at the block it's referencing but you don't actually need this on its own line i found this out by playing around with this today that for these tables in any case i can put actually put it at the end of that table because this isn't something displaying so these tables and markdown are a bit sensitive when it comes to extra characters after the pipes but because this is metadata that is not really displayed you can actually have like these table references like this with the metadata right after the last pipe you know still displays correctly go back stills displays correctly and everything's good alright cool so so far we're able to reference entire files any heading and everything underneath that heading in a file individual lines of text markdown tables we can also reference images so this is an image i have embedded in my space repetition note which shows the learning curve the projected forgetting curve where you're going to need to keep prompting yourselves for repetition so you keep that information in working memory and we're going to reference this image so if i preview it you know there's nothing there just the actual transclusion of the png file which shows the image so now i'm going to reference this image by doing space repetition and i'm going to auto finish with a upward carrot for the block level reference i can scroll all the way down to the bottom you know here's a lot of my reference information and metadata i don't want any of that what i want is right here space repetition.png that is this file so if i click that now it references test which in this case is actually something i set on here test it would be a randomly generated uid if you chose but we can see that ah it will reference the image and now i can also add a transclusion and now the image is referenced in this transcluded block so in this case you can even pull in an entire image block just like the table now if you've seen my workflow video and i can link that up there in a card but i have a whole comprehensive workflow for my obsidian settle costing system and you can check that out but one thing i do in there is i make prolific use of iframes so if i went to look at let's say let's pull out a podcast so let's say getting things done yeah so this is actually a podcast player now this would work with the youtube video it'll work with anything any iframe because when you can see this is rendered as a podcast player but if i show it raw it's just an iframe interesting let's go back to our andy note now let's let's actually remember that this is you know percent sign 2020 getting things done all this stuff this is all stuff i mentioned in my workflow video about why i name files this way and yeah so let's go back to andy and let's actually look at a block reference to an iframe so if i do transclusion double bracket percent so i can see my podcast references getting things done and now after getting things done i want to reference a block of ah there's the iframe now if i reference that iframe you can see now it gets a unique uid or unique identifier uid if i preview that it'll actually show me the transcluded block of the iframe which is an embedded podcast player this is really neat because now if it can be shown through an iframe in your markdown file if you have an internet connection then i can also transclude that block level reference to that iframe in any other document and so this will work for anything that can be held in an iframe this works for air quotes tweets youtube videos podcast players other websites anything you can embed in an iframe which is a lot of things you can do block level referencing to that iframe tag and then show it in any other document and just as another example of what you know might look like if i was going to reference an iframe using a tweet i can actually do all my tweets which i only have one in here right now and i can say hey show me a block level reference to that iframe show me and a transclusion it'll take a little bit to load because it's going through twit frame but eventually it will show the tweet itself awesome cool little tweet by jenny bryan so this is a really really cool implementation of block level references because it can show tables images anything that you can embed with iframes individual lines or paragraphs of text and this is just all in addition to what we already had headers and file names so the obsidian developers made it very clear and this is common knowledge for probably most developers that deal with markdown is that these uids and doing block level references in this way is not a part of any particular established markdown specification what this means is this is not a portable feature so if i take all of my now still plain text documents that are being looked at by obsidian it's just a directory filled with plain text files if i throw that into any other system none of these block level references will work unless that system supports it foam might make it work but who knows yeah i probably will but not a lot of other systems are currently using this implementation of this blockable reference feature with markdown files so it's not a very portable feature or very future proof if you plan to take full advantage of this functionality but one thing you can do is if you're sort of familiar and comfortable with the command line if you want to make sure that you can always see where your block level references are and you for some reason obsidian no longer exists and none of no features are supported with block level referencing and markdown anymore you just want to be able to see your references you can just use a simple tool you know if you want to you know search for every instance of let's say i have this block id for andy's notes and andy matushak's notes i could just do rg which is rip grip it's a command line tool you can download from pretty much any package managing service i'm on mac you can do brew install rip graph or i think it's just bruinstor rg either way rip graph and i can search for that block id so i can paste that in search for that and rip graph searches recursively by default which is why i prefer it better than grep in this instance because grep needs a lot of other command line flags flag options to tell it to do this but really it's just rip grap give it an item run it in the directory where your vault is so i'm just in that vault directory all my files are just there in a pile and i'm searching for one little bit of text that text is the uid you can see over here if i actually move that over for you can see that i'm searching rip grab and just the name of the uid if i run that doesn't find anything why because i actually need to escape that special character and if i run it again now it'll show me ah there it is in andy's notes interesting so we also referenced this in the evergreen notes and i referenced that in the upper portion here d3501 but it doesn't find the carrot because the carrot symbol isn't in here the carrot symbol gets changed to the hash mark so that's one reason why i personally would prefer to keep these as hash symbols or hash it hashes because when they're so distinct and unique like this and not anything related to english they won't be mistaken for a heading and maybe you're not using a command line like this but if you are this might be something you might want to take advantage of because this will easily find every block reference in a moment's notice and you know sometimes just using command line options and utilities on plain text files is just infinitely more powerful than using any sort of gui interface so rip graph the uid without the carrot symbol and it'll show me all the references it'll tell me the name of the file the line it's on and pretty much the the block with it with it on whatever line it's on because this works on lines basically or blocks of text so in this case using rip graph will actually return the block itself in some instances and i think this is a really important and critical thing especially why i think it's really important to be familiar with the command line is this is somewhat future proof yeah you may not have the ability to easily maintain and update and change and deal with block references like this but options exist for you to still utilize them in a way that makes sense as long as you could easily randomly generate a hash or some weird combination of symbols which you could easily do with a bash script or python script or whatever you want and establish the syntax like this and then search for it by using something like rip grab or regular expressions really makes it easy to just keep up on all your plain text notes especially with these tools and just to wax poetic a little bit more about command line utilities let's say you needed to actually update one of these uids across your entire obsidian vault because currently there's no way to update all the links from one instance like i can change the name of a file every reference link to that file is automatically updated if i choose to do that but no such functionality currently is implemented for these block level references in obsidian maybe in foam but not in obsidian not yet personally stuff like that doesn't really bother me why because i have perfect workarounds such as said or stream editing you can ignore the dash i dot back and all that stuff because it's just some weird thing with the bsd version of these utilities on mac os but really the important thing is the regular expression and again why i think it's better to just keep the random hashes if it's not something that is you know commonly used plain english plain text whatever and it's something completely random like a hashed uid you're not going to accidentally replace anything that you cared about so in this instance i'm replacing that hash for the andes note links well that currently that one will actually go to andy's notes right here and we can actually see on both of these actually let me open up both of them side by side so we can actually see so we have the uid over here we have the uid over here and now i'm actually going to do a said regex replace and so what that means is i'm going to replace that uid right here with the words hello youtube from set so you can ignore everything else going on here just that regular expression is important let's run it awesome so now you can see both of those uids have now been changed to my text that's it across the entire vault anytime that reference that unique hash id occurred it would have then been replaced and this is why i think keeping the hash is random is really useful and important for protecting yourself and your files if you're going to do something like this as a workaround to easily just update and change everything and as if there weren't enough shiny toys that were just given to us in this brand new release we now have a hotkey key binding that we can remap to complete and basically toggle check boxes in obsidian this is really neat because now i can easily just say i want to view a checkbox so show me a check box okay empty show this on youtube neat now i remapped my key binding to control space which for mac the control key isn't used very often so yeah i just did that so if i do control space you know that's all i'm going to press ctrl space ah it completes itself and if i keep pressing it it now toggles very useful really really appreciate this feature lycat is the name of one of the devs great work thank you like hat silver for the great work that you guys do on obsidian it's really awesome how far this application has come in just the brief time that i've been using it and that's all i have for you on update 0.9.5 block level referencing wow so many new things to take advantage of and for all of the holdouts that were stuck on rome because of the block level referencing yeah it's not a perfect implementation it's not perfect yet it just got released today and it's not exactly the same as rome's implementation but you know what we're still in beta things are still developing and at the rate of development on this application who knows where this is going to go for now obsidian just keeps getting better and better there's so much we can do with this and i'm excited to play with this more so thank you all for watching i hope you found this very interesting and let me know what you guys think about this let me know how you're using block level referencing and what your thoughts are about this and obsidian's implementation of it if you'd like to talk more to me about this type of stuff or just see what i'm up to you can join my discord server link to that is in the description below and on my channel page and before i go a quick shout out to the patrons who support my channel thank you guys very much for your support devin alberto klaus and brandon thank you guys for just being there and always providing your support for this channel and i'll catch you in the next one [Music] block level referencing was rome research bleh