Note: Would you like to give feedback on or ask a question about this research? Be part of the Open Peer Review! Please visit the comments section at the bottom of this page to participate.
[Music plays then fades down]
Voice Over [0:00]
This is Open Peer Review. A podcast that explores how podcasting itself can be used as a way to conduct open peer review of scholarly research. Each episode features a researcher, and a peer reviewer — an expert in the same field. Their conversation gives insight into the topic and the research process. And the researcher gets valuable feedback they can use to refine their work. You’re invited to give your feedback and join the conversation at OPRpodcast.ca.
Taylor MacLean [0:33]
Hello, I’m Taylor MacLean program lead at the Center for Communicating Knowledge here at Ryerson University. Today we are joined by researcher Lori Beckstead, to discuss her project titled “The Genetic Code of Podcasting: a new approach to thinking about discoverability“. Lori is an associate professor of audio and digital media in the RTA School of Media at Ryerson where she teaches storytelling via sound media such as podcasting and radio. Welcome, Lori.
Lori Beckstead [1:02]
Hi, Taylor. Thanks.
Taylor Maclean [1:04]
Also joining us in the role of peer reviewer is Dario Llinares. Dario is a principal lecturer in contemporary screen media at the University of Brighton. He has published work on a range of topics including cinema studies, audiences, and podcasting as an academic practice. He is the co-editor of the book Podcasting: New Aural Cultures and Digital Media. And he’s also the co-founder and co-host of the Cinematologists podcast and the New Aural Cultures podcast. Thank you for being here, Dario.
Dario Llinares [1:38]
Thanks for inviting me.
Taylor MacLean [1:40]
Okay, so let’s get started. Lori, can you tell me about your research?
Lori Beckstead [1:44]
Yeah, basically what I set out to do was a pretty large undertaking. I wanted to find out whether we could determine a sort of genetic code of podcasts. So we tend to classify podcasts by genre, and of course, we add keywords to podcasts in the RSS feed so we can locate it, you know, if we want to hear a true crime podcast. But I wondered whether we could come up with a set of characteristics that were common to most, if not all, podcasts. And so what I did was I asked listeners to rate a given set of podcasts. I took the top podcasts from certain lists and asked them to rate them on certain characteristics. So some of these characteristics included things like authenticity, you know, how authentic do the hosts seem; sound design, was there use of music and sound effects and other sound design in there; how much storytelling was involved in the podcast. So we came up with 10 characteristics [anchor/pop up box] that we wanted people to rate the podcasts on so that we could see if these characteristics were present in most podcasts. So what we found when we analyze the data was that podcasts of similar genres shared similar ratings on all of these characteristics, which makes sense. And then we also discovered that podcasts that were of completely different genres also seemed to share some of the more important characteristics as well, or did, or could share those characteristics.
So that got me to thinking that perhaps this could be a way to improve podcast recommendation engines, which tend to rely on keywords. So if we were to have recommendation engines use these characteristics that we came up with, and the ratings for each of them on the podcast, they could then potentially say, ‘well, if you’d like this podcast from true crime, that was really high on storytelling and authenticity and was very emotional, for example, then maybe you’ll also like this cooking podcast, which is also very high on storytelling’, etc. So we think that this perhaps could be a system for improving recommendation engines for listeners to podcasts.
Taylor MacLean [4:01]
That’s great. Can I ask where or how you came up with those 10 characteristics that you were measuring?
Lori Beckstead [4:09]
Yeah. So this was partly looking at the literature, the scholarship on podcasting. It was partly reading into industry based writing blogs, articles, online producers and people who are involved in the podcasting industry and what they were talking about. And part of it was sort of a gut feeling, just my own, having listened to many podcasts and thinking about things like, I wonder if the podcast is really great at storytelling, I wonder if it’s okay if it doesn’t have a lot of great sound design or maybe the production quality isn’t as high. So that’s why I wanted to try to come up with those characteristics and kind of be able to make comparisons across them.
Taylor MacLean [4:52]
That’s great. That’s so fascinating. So, what was your research question? The main research question that was guiding the study.
Lori Beckstead [5:03]
So, it was a big one. My original research question was: can a framework for defining the underlying characteristics of podcast content be established? And I always felt a little odd about asking that question because if you replace the word podcast with a different medium, like, could a framework for defining the underlying characteristics of books be established, right? It just seems like a weird question to ask, but I guess because podcasting is still so young and we’re still exploring it to a great extent that maybe it was crazy, maybe it wasn’t, I don’t know. But that was one of the underlying questions.
And also, you know, could these characteristics then be used to improve access to listeners to podcasts of interest? Because we know that there are some issues with recommendation engines existing like Apple, for example, we don’t really know how podcasts make it onto their top lists. We think it has to do with sort of ‘buzziness’ around a podcast release, so that might favourite podcasts that tend to release episodes all at once and have a great PR machine behind them. So yeah, it was kind of you know, can we get to the nitty gritty, the genetic code of podcasts and use that to help listeners find podcasts they’ll like?!
Dario Llinares [6:25]
Can I ask Lori if you think there is something about podcasting that requires these sort of more esoteric genetic codes than say, any other medium would? Like, say for example, why is genre, in the traditional sense, not enough for defining podcasting? Because I mean, it’s an interesting sort of move to go to these, like I say, sort of quintessential ideas around podcasting, which could be quite abstract.
Lori Beckstead [6:56]
Yeah, I think I was mostly motivated to do that because a lot of the literature around podcasting today, I mean, it’s beginning to change, but in the early days certainly it was all about “what is this medium” and it was often talking about podcasting in relation to its closest relative, radio. So there was a lot of scholarship about “what is podcasting in relation to radio” and “what are the differences between the two” and a lot of scholarship that was even asking, okay is it, “can we consider it a medium”? I feel like we’ve graduated to yes, it is a medium. And so I wanted to move from the idea of what defines it as a medium to, you know, the next stage in my mind is genre, is another way to classify it. So it’s almost like a pyramid: here’s the medium and here are the defining characteristics of it as a medium. And then the next stage down narrowing the focus is genre. The next stage down perhaps narrowing the focus is format. So is it a chat cast? Is it an interview based thing? Is it a documentary style? Is it highly produced with sound design? And then I feel like I’m trying to get to the next level, even down from there, and what does a podcast have to have in order to be thought of as a podcast beyond what it is as a medium.
Dario Llinares [8:20]
And just, I mean, there’s so much to talk about, we’ll come back to the genetic codes and everything in a bit more detail later, I suspect. But I mean, it’s interesting because there is an argument to say that podcasting is actually maybe even reverting back to more ‘radioness’ if you think of the way in which the industry is perhaps becoming more and more professionalized and turned into networks, you know what I mean?
Lori Beckstead [8:48]
Absolutely.
Dario Llinares [8:49]
So that initial phase of podcasting, maybe its sort of early adolescence, when the idea of it being this cult practice or outside of the mainstream and they have this discrepancy between independent and institutional podcasting. The way that things are going sort of professionally, or in terms of the structuring of the industry, is maybe turning it back to a sort of more radio kind of orientated sound, perhaps. I mean, do you think that there’s a sort of sense that your study may be quite important in making sure that those sort of elements of podcasting’s independent and strangeness and real interest beyond its professional structure, that this could be used to kind of mine those areas still?
Lori Beckstead [9:40]
That’s what the hope is. So if we can kind of get to the nitty gritty of what a podcast is, then we can identify other podcasts. And it shouldn’t matter whether it came from Gimlet or it came from Radiotopia, it could just be somebody who independently put together a podcast. But if any of those podcasts are rated based on these criteria then you should be able to find recommendations from any level, whether it’s behind those sort of more gate-kept, you know, podcasting networks or if it’s purely independent.
Taylor MacLean [10:14]
Lori, I’m wondering if you in your research, if there are any parallels to the music industry and music streaming platforms? I know that there’s been similar, I guess, challenges in terms of the recommendation algorithms and how they make those. Did you come across anything like that?
Lori Beckstead [10:33]
Definitely. So one of the sort of inspirations for wanting to do this research was Pandora. So Pandora is a music streaming service. We can’t access it here in Canada and Dario, I don’t know if you do know whether you have access to Pandora in the UK?
Dario Llinares [10:52]
I have been reading a little bit about it and I don’t think we have access yet. But it’s definitely a kind of interesting idea. But go ahead because I think there’s definitely some questions on the kind of connection to your project.
Lori Beckstead [11:04]
Right! Yeah, so Pandora basically you set up a music station on Pandora. So you log in and you pick a song, your favorite song. So let’s say you pick Alicia Keys. Most recommendation engines might say ‘okay, well if you like Alicia Keys, you know she’s classified kind of as R&B say, so let me give you some other R&B hits’. But what Pandora does, they call it the, what do they call it now? The Music Genome Project. That’s it. So they actually had a number of music experts, I think 25 of them that worked on this for years, who identified the actual sort of genetic code of each song. So instead of just saying, well that’s R&B, it might say something like, okay there’s highly melodic lyrics, it’s a female singer, it has piano and this type of instrumentation, this song is in a minor key etc., etc. So then what it would do is say, ‘well, if you liked those characteristics in this song then here’s another song that could be from a completely different genre, completely different artist, maybe one you’ve never heard of but that has similar genetic codes’, if you will. So that was definitely an inspiration here and funnily enough, I just learned recently that late last year Pandora came out with a press release saying they are working on a genetic code of podcasting. So you know, maybe they’ll beat me to the punch with all of their resources and maybe they’ll have 25 podcasting experts working on that for five years. But here’s where we’re at with the three of, myself and my two research assistants working on this for the past few months. [Laughs]
Dario Llinares [12:47]
Yeah, that’s one of the things I was gonna ask about because just sort of reading Pandora, they’re talking about attaching 1500 potential attributes to an individual thing.
Lori Beckstead [12:27]
[Laughs] Yeah.
Dario Llinares [12:58]
But I was thinking about this. So as you’ve kind of intimated there, it’s like if you’ve got 10 categories, it seems like that their DNA sequence as it were, or their genome sequence is much more in depth and complicated. But it seems to me that they’re just attaching a singular attribute to a podcast or to a piece of music as you highlight. So like you say, that’s very binary, it has it or doesn’t. Whereas I think what you’re doing is interesting because you seem to be having a scale of each attribute. So in a sense, I think that there is sort of crossover here in this kind of methodology. And obviously, it would be very difficult for you to get, you know, every podcast sequenced in that way, so we’ll come up to the scalability issue in a second. But I think that the difference here is the idea that you can have an attribute that has a scale between one and six or you know, between one and 10, or whatever it might be and that gives you that kind of holistic map about what a podcast actually sounds like. And I think getting beyond the idea of it is or isn’t something also could move beyond the idea of something is good or bad in those attributes that you’re pointing to. Which I think is much more, at least from an academic perspective, is a much more interesting way to kind of understand again, in this holistic way, what podcasts are like, you know, that underpin them.
Lori Beckstead [14:27]
And thank you, I appreciate your confidence in me that I may be doing this better than Pandora is doing it [laughs]. Yeah, so we did rate. So for each of the characteristics we came up with we asked listeners to rate it on a scale of zero to five. And one of the interesting things that we did with the data was we used data visualization techniques to have a look at — ironically, we’re using visualization for an aural medium — but it gave us a sense of which podcasts shared characteristics. So what we did was once the ratings were done and we got the data, we created spider graphs or radar charts. So these are charts that start with a zero in the middle and there are spokes sort of radiating out from the zero towards, you know, five on each spoke. So 0,1,2,3,4,5 on each spoke, and each spoke represents one of these characteristics. So if a podcast was rated a five on storytelling, then we would plot a number five on that spoke and then if it was rated four on authenticity, you’d plot a four. And so you’d end up with a geometric shape for each podcast that kind of was a visualization of what the essence of this podcast is. And it was really neat to be able to use those visualizations. What we did was we started layering them on top of each other to see where the similarities were. And what we found was that podcasts that came from the same genre certainly shared these characteristics. So we’ll share some of these radar charts in the show notes because, of course, it’s hard to describe them here. But I’ll do my best.
So for example, we had a couple of sports podcasts that we looked at that happened to be in the top podcast lists that we decided to look at. So one was Pardon My Take and another one was called Spittin’ Chiclets. And Spittin’ Chiclets, I think is, I can’t remember if it’s a Canadian but it’s a hockey based podcast so it’s definitely popular here in Canada [laughs]. And so they were both rated very highly on humor, they were mid rated on celebrity, they were mid rated on being informative, they were highly rated on being authentic and they were also highly rated on being niche because they were both focused on a particular sport. They were not very evergreen, they were fairly topical and timely because they were probably talking about a particular sport that was happening and they were pretty highly rated on production quality and not so much on sound design, and not very high on emotional. So their shapes ended up being very, very similar. So that was interesting to see, that perhaps what this does is validate the idea that there is something to these characteristics that we’ve chosen that gives some reliable feedback about evaluating podcasts.
Note: Would you like to give feedback on or ask a question about this research? Be part of the Open Peer Review! Please visit the comments section at the bottom of this page to participate.
Dario Llinares [17:34]
It’s interesting, because I think I want, I definitely wanted to ask a little bit about some of the genetic codes themselves and in a way how the respondents were asked to answer them. Because I think that could be a sort of, the way that you ask the question I think is important in the kind of information that you get, for example. But just on the genetic codes to begin with, (like something like) it’s interesting how different they are in terms of what they’re alluding to, I think. And so something like authenticity it’s a difficult one because it’s like, that can be dependent on the kind of podcast we were talking about. You know what I mean? Because if it’s a fictional podcast, how do you score it on authenticity? That’s quite difficult to do. I mean, you know, authenticity to me lends itself to a podcast that has a host or an expert, or how authentic is the conversation that’s going on, rather than something that’s produced as an audio drama, let’s say, for example. So that’s quite, — and so therefore, when you’re asking that question of the respondent, how are they scoring it that actually, this is not authentic, but it doesn’t matter, because it’s nothing to really do with this?
Lori Beckstead [18:54]
Yeah. Well, you hit the nail on the head. I mean, we did grapple with that. We had a session with the people who were rating the podcast to go through what do we mean by authenticity. And that is one of the limitations of this, is what is the definition? And, you know, for example, emotional, like, ‘is the podcast emotional?’, that’s gonna be really different for one person than it is for another. But to answer that question, specifically about authenticity, what we decided on was that if it was an audio fiction podcast that we just said, just rate it as inauthentic, as not being authentic. And so what we’re saying is that not all of these 10 characteristics that we came up with are or have to be in every podcast in order for them to be considered a podcast or to be considered good. There’s, you know, a relationship that we’re interested in exploring that relationship. Like, if the podcast doesn’t seem very authentic, then what are the other characteristics that seem to have to be present at a high level in order for it to still have wound up on this list of top podcasts that people seem to enjoy?
Dario Llinares [20:01]
Was there ever any discussion, say for example, (if you are) if you’ve got like a rating of zero to five, then whether you put a word or a DNA, you know, a signifier at one end and a different one at the other. So say for example, if you have niche on one end and then you have mainstream at the other at least it would give you again that sense that it’s positioned somewhere on there and gets out of that kind of good or bad dichotomy like something like storytelling, for example, narrative. That’s just a “is it a storytelling podcast”? And then the second question is, “how good is its storytelling”? So it’s kind of like, some of them are one step and other one are two step questions, it seems to me.
Lori Beckstead [20:47]
Yeah, that’s an interesting point. So I mean, we did only ask people to rate, you know, “does it contain storytelling”? So a five would be, it’s all about storytelling. You know, that’s the primary focus. But yeah, that would be really interesting to have a set of ratings that was part B is “how good are they at storytelling”? Yeah. And the other category that was a sort of spectrum, as opposed to a binary, is evergreen or timely, right? So we asked them to rate them on that scale and yeah, one isn’t better than the other. Obviously, if it was a sort of news or politics based podcast then often it was very timely and not evergreen at all. But evergreeness is definitely something that gets talked about a lot in terms of podcasting as a medium but we are, more and more, finding more podcasts that are very very timely.
Dario Llinares [21:40]
Yeah. Yeah. Just one more and then obviously Taylor can come in anytime. It’s interesting again that you’ve got the informative and educational element to it, which yeah, obviously a signifier of that is easy to kind of, (or you can) define a podcast through that way. And then you’ve got kind of funny or humorous. And it’s interesting because there could be a sense in which there are distinctions being made or boundaries being made by different types of podcasts that do fit to genres. So if something’s political, you may not necessarily think it’s funny, but the one thing about podcasting is it does, it does blend those things together. Like something can be really informative and quite in depth but also have humor behind it as well. So it’s interesting that [it] does offer you the possibility of getting people to listen to something that may be if it was just presented, say in a lecture ‘oh here’s a, you know, here’s a lecture about, I don’t know, really in depth computer science or something like that’. But yet, it has a high score alongside it, as you might say, ‘oh, actually, I mean, you know, that might be quite, quite interesting’. So I think that sort of the way that these things map out do offer something slightly different from, you know, true crime podcast, so I like true crime, I’ll listen to that. So yeah, interesting.
Lori Beckstead [23:04]
Yeah, that’s the hope. And I’ll give you an example of true crime. Let me just find that chart here. Yes. So this was where we started mapping those radar charts on top of each other from podcasts from different genres. And so for example, one that matched up almost exactly, two podcasts, one was from the natural sciences category in Apple and the other one was from true crime. And those two podcasts mapped up almost exactly the same on those characteristics. So one was Patient Zero and the other one was Hell and Gone, that was the true crime thing. So it seems unusual that someone might think, ‘well if I really love Hell and Gone true crime, why would someone recommend to me a natural sciences podcast?’ but these things both rated very highly on being informative, very highly on storytelling, both rated very highly on being emotional, sound design was way up there, production quality was way up there, and authenticity was high. And those are some pretty significant characteristics to have in common, and so there’s no reason why someone who liked that true crime podcast might also really love the natural sciences one. And that might be the next step for this research, is to actually try to validate this by, ‘okay, what recommendations would this system come up with?’ And then ask people, ‘okay, if you like Hell and Gone, we’re gonna recommend this natural sciences podcast to you’. Rate it for us, let us know if you liked it, and then we’ll know whether we’re way off base or not with this.
Taylor MacLean [24:43]
Yeah, I’m also thinking that it might be a way to define kind of unique combinations of attributes that maybe people wouldn’t necessarily think, ‘oh, so people are looking for something that is this and this’. The other question that I have, I guess I’m thinking about kind of the difference between quality and the labeling of these attributes, which you guys have touched on a little bit here, which is that some of these characteristics, like production quality for example, like that when you’re rating that you are rating that on a quality scale, but the other ones you’re not, or not all of them are necessarily based on quality. Something can be, for example, not authentic and still be considered great and I guess I’m wondering sort of where this understanding of how good something is comes into this, whether that’s important, and maybe what sort of implications this might have for critical tradition, like the criticism around podcasting.
Lori Beckstead [25:50]
I think I understand where you’re going. And yeah, I mean, that opens up a whole new venue for this to do that sort of maybe second study where we’re rating it on the quality of the storytelling and the quality of the humor. You know, again, we get into these really difficult things to measure. I see Anna, did you want to say something? Anna Ashitey is my research assistant on this project and she’s an undergraduate just going into her fourth year of Media Production at Ryerson University.
Anna Ashitey [26:25]
Taylor, actually just to touch on production quality, if it’s a lower quality production does that mean that it’s not as good as a podcast? Or if it’s a higher quality production does that mean it’s a better podcast? But Alex Symons actually identified that some podcast creators will go as far as manufacturing authenticity. So if there’s a celebrity who’s the host, they may be inclined to lower the production quality just to try to relate to their audience that of the listeners. So in those cases, it may actually be more beneficial for them to lower their production quality to have a more parasocial relationship with the listener. So that’s another way where production quality can be used to really either heighten authenticity or even some cases heighten intimacy between the listener and the host.
Lori Beckstead [27:14]
And that’s what we’re interested in. Thank you, Anna. And that’s what we’re interested in exploring, the whole combination of, what can you get away with in some of these categories in terms of, you know, having a low rating on that, if you’ve got a really high rating in some other categories. And that’s so interesting about the idea of celebrity and authenticity. We’re seeing now, especially now in COVID times, where people are, you know, the intimacy factor and authenticity factors I think are going up for many podcasts and that’s because people are sitting in their living rooms trying to record their podcasts and their dogs are barking and their spouses are interrupting and that sort of thing. And so we’re getting an even more authentic and personal look into people’s lives, which is something that people have always loved about podcasting and podcasts, that the hosts and guests tend to be very open and personal, and even more so now.
Anna Ashitey [28:09]
Yeah. For sure.
Dario Llinares [28:11]
Yeah, that’s a great point by Anna because I think that it’s one of the things again, that’s always riven the this sort of, not argument, but the clash, say, between what podcasting is as opposed to being radio and who podcasts, and how they do it, and not having that feeling, the need to replicate the studio sound, you know, that you’ve said that’s associated with with NPR in America or the BBC over here. Because you want to do something that is slightly different, you want to have a different sound whether you describe that as intimate or authentic, or whether it is just kind of like raw, or you know, unproduced. And I think Lori, what you were saying there about COVID sort of, in a way it’s leveled the playing field now because in the podcast that I produce I’m always like struggling to make better sound because we have to do, we’ve always had to do it remotely. Our, me and my host, we record across in the same way that we’re doing today and it’s been amazing to me how very highly respected podcasts that have a bigger listenership than ours are just taping through Zoom or taping through Skype, and the audio quality is a lot less and I’m like, you know, how have you guys not figured out how to record better quality? But in a sense, I think that a lot of shows are leaning into that and saying, ‘oh, yeah, it is a bit crackly’ but this is the, you know, we’re in a sort of lo-fi world at the moment, do you know what I mean?
Lori Beckstead [29:53]
Yeah, yeah. And this is a question that comes up for producers a lot. So I’m on a number of Facebook groups for podcast producers and people will, especially newbies will say, ‘How much do I have to spend on a mic’?, ‘Do I have to spend on a mic’? ‘Can I just use the built in microphone’? Like, ‘how important is this’? And so it’s kind of the perennial question. And I think you can get away with lower quality if you’re really high on some of these other categories. Like if you’ve got great storytelling and your hosts are, maybe they’re high, you know, maybe it’s a celebrity hosting. So if Oprah is gonna host a podcast, I mean, obviously, she’s got the resources to pay for the great quality, but I don’t think anyone would really mind if it sounded pretty crummy, because she’s Oprah. Right!? So which of these characteristics can you get away with as being lower if you’ve gotten higher on other ones?
Dario Llinares [30:49]
Could I just ask something about the issue of format when it comes to these characteristics again in your genetic code? Because I think that again you’re obviously sort of focusing on the artifactual nature of the podcast now rather than is it a technology or is it a practice? Or is it a medium in that broader sense? So this is just about the artifacts of the sound, you know, the sound itself. But I think like, key within that is different elements of the structure that may then feed into these other characteristics. So I’m thinking on your scale, something like segmentation versus free form, or monologues versus dialogues or, you know, multi person conversation. And then you’ve got things like atmosphere, whether something is taped live or it’s sound designed in that way, or it’s an informal conversation or it’s a studio. So there’s kind of like structural elements, I think that again could make their own set of characteristics to add on to these, you know, much more sort of DNA or abstract abstract ideas, perhaps.
Lori Beckstead [31:59]
I couldn’t agree more. And in fact, when I set out to do this I originally hoped to find the underlying characteristics of not only the content, as you say, like as an artifact, but also the format, the forms, and modes of presentation that are present in podcasts. And you know what, it was simply by sheer resource and time availability that isn’t part of this study. But of course, for future studies, I would love to overlay that data. Like, how does it being a conversational chat-cast interact with some of these other characteristics? Or if it’s a highly produced documentary style with scenes that involve people doing something in a location, you know, how does that factor into, how does that relate to storytelling, for example? So we did actually ask our participants to collect that data. So we do have it, we just haven’t rolled up our sleeves and dug into that part of it yet. So maybe that will be sort of stage two.
Note: Would you like to give feedback on or ask a question about this research? Be part of the Open Peer Review! Please visit the comments section at the bottom of this page to participate.
Dario Llinares [33:04]
And maybe you could talk a little bit about perhaps some of the issues around demographics here. Because obviously, you know, you’ve got the students, [they] were the test group, let’s say. And that’s its own demographic in and of itself. You know, you can get into all the sub demographics within that. But I mean, it’s interesting with podcasting because they do seem to attract certain types of people if you’re looking at things like levels of education or levels of income and these kinds of things and then I noticed in the sort of methodology you’re asking people to listen to podcasts that they haven’t heard before. And, you know, then there is the sense of what’s interesting to me, this would be really fascinating if you could like, on your phone, have an app where you were rating it as you were listening rather than doing it after the fact.
Lori Beckstead [34:04]
Right?
Dario Llinares [34:05]
You know what I mean? It’s like, having that kind of built into the rating system, I think is really interesting. Anyway, just in terms of the sort of, some of the demographic issues that you might have come across?
Lori Beckstead [34:15]
Yeah, so the people that were rating these podcasts were my students, and I was giving them grades to do it [laughs]. This seemed to be the most expedient way to get the thing done. So obviously, you know, part of the issue was, you know, there’s always going to be students who just kind of fill in a form and haven’t really thought about it. But I did spend time in class with them. And there was more than one student that did each podcast, so if they had differing numbers I had them talk to each other and kind of come to consensus about what the numbers should be. So they were a little bit accountable in that regard. But also, they were only rating podcasts at the episode levels. So I let them choose whatever episode they wanted from the particular podcast, series or season that was listed in the, you know, in the list that we chose from. So it’s possible that these characteristics might change from one episode to another even within the same series, so that could be a limitation too. Also these are students of media production, so their understanding of some of these things might be quite different than an average listener because they’ve been studying it. I noticed that they were fairly, they had very high standards when it came to rating the production quality of something. Whereas perhaps a lay person might not notice some popping p’s and, you know, some peaking here and there, whereas these students were like, ‘wow, the production quality was terrible’. ‘Did you hear the popping p’s?’ So, there is that.
Dario Llinares [35:46]
Yeah, I guess it would be important, wouldn’t it, that one of the next sort of phases of this would be to be non sort of, non-media students who are ready to make those kinds of assertions [and] are just like listening in the way that they would ordinarily if they were on the bus or whatever, you know?
Lori Beckstead [36:04]
Yeah. So you raised a good point about the idea of, you know, could we get listeners to do their own ratings on this scale to go into some kind of crowd sourced database that rates these things? I think one of the things that disenchants me about crowdsourcing information, and this goes back to Pandora’s music recommendation system. I loved Pandora for the golden moment it was available in Canada before they yanked it because of copyright or whatever geo-blocking because I found other systems that said, ‘Well, other people thought’, you know, ‘Other people who liked the same song you liked also like this other song’, and I would look at the recommendation go ‘Well, I think that person is full of crap. I don’t like their tastes’, you know what I mean? So there is that element of crowdsourcing information that doesn’t necessarily, so you know, how do you keep this objective? And that’s even impossible using a group of media students to do the ratings.
But I also wondered whether we could get the producers of the podcasts themselves to use this rating system. So for example, could something like this go into an RSS feed? So an RSS feed has different areas for different types of metadata that can be entered and so what if the producer, and again, is this objective? Probably not. If the producer is asked to rate you know, how high is it on story, they’ll be ‘Great. It’s full of storytelling’. Maybe it isn’t, who knows? But, I mean, that is one of the questions, is how do you actually rate all of the podcasts that are out there? This needed a human touch and we were able to rate 78 podcasts (updated number: 76 podcasts). You know, that was it, so that’s what our data is based on. So how on earth could we rate all of the podcasts out there in the universe? Could this be done through artificial intelligence (AI), for example? I know that there is a newish thing in recommendation engines, is what they call natural language processing, where it’s not just looking for keywords in a transcript, but it’s actually reading. The artificial intelligence kind of reads the mood and the tone of what’s being said and how it’s being said and gets information from that. So AI is kind of coming around, but I don’t know that it would be ready to be able to say ‘This is high in storytelling, or this is really authentic’. I mean, could you train AI to be able to rate authenticity, for example? So this rating system really feels like it really needs that human touch. And so that’s the limitation, like how do you actually rate them all, then?
Dario Llinares [38:43]
Yeah, I guess it as well. It depends a little bit about what the long term aim is around something like a word like discoverability. Because it’s funny when I hear that word, I have this kind of idealized notion of, ‘Oh, somebody’s going to introduce me to something that I never would have found otherwise’, but actually, really, what discoverability is, is big corporations trying to get you to buy stuff similar to the stuff that you’ve bought before. And, you know, it’s whether that is the thing that you’re actively trying to rail against or reject, or whether you’re actually trying to do that, but in a, you know, in a much more holistic sense, or, you know, in a much more rigorous sense, let’s say.
Lori Beckstead [39:24]
Well, I think I share your view on the idealistic view of discoverability that, you know, hopefully you have the opportunity to discover podcasts that don’t have a giant PR machine behind them. But you’re right. And you touched on this before with the sort of corporatization of podcasts and sharing promos across podcasts from the same network and so you tend to just get into this self-fulfilling bubble of, you know. For example, my husband has always loved Malcolm Gladwell’s podcasts and when they cross promote something from the same network, you know, he listens to that, too. And so he seems to be a little bit in this bubble, but he does love them. But I wonder like, is there somebody out there that’s doing something quite similar that no one’s ever heard of yet because they’re not part of a big network.
Dario Llinares [40:10]
Yeah, I like Malcolm Gladwell’s podcast, but I always feel like he’s trying to sell me a Lexus or something.
Lori Beckstead [40:15]
He’s definitely trying to sell you a Lexus! I couldn’t believe when he did that [laughs].
Dario Llinares [40:21]
This is a really interesting area though. That idea of advertising blending into podcasting so you don’t know whether you’re listening to a podcast or just a long advertisement. It’s really, you know, somebody should write something about that.
Lori Beckstead [40:32]
The one you’re referring to though, in my opinion was one big long ad. Yeah, that was a, yeah, that’s a topic for another research study: ‘Is Malcolm Gladwell selling Lexuses?’ or Lexi–how do you say that? [laughs]
Taylor Maclean [40:47]
Yeah, I like it when they have their theme music for their ads that let you let you skip ahead or at least you know, know that’s what you’re listening to. We’ve sort of touched on the ideas of the study Lori, but I’m wondering maybe if we could just kind of, you know, wrap it up and summarize that. You mentioned that format was sort of beyond the scope and I don’t think we’ve actually gone through all of the different categories. So maybe you could just kind of briefly talk about the main ideas and just kind of go through those categories. And I don’t know if Anna wants to jump in, too.
Lori Beckstead [41:31]
Yeah. Anna can help us out on this a little bit. Anna do you want to talk about, maybe just even touch on your literature? Because this literature review that you did, because this was a doozy of a lit review because we were kind of starting from the point of view of let’s just look at all the scholarship on podcasting to see if it can help us figure out what some of these characteristics might be.
Anna Ashitey [41:53]
Right. So, as Lori mentioned, there was a lot of scholarship concerning podcasting as a medium, but we found when it came to those 10 characteristics that we determined to really make the genetic code of podcasting that there was limited scholarly data. So because of this, we decided to consider articles and industry talk. And actually, Dario, as you mentioned, about if we’re entering a new era of authenticity, there actually are articles concerning that, not necessarily scholarly articles, but there are online and industry professionals as well. So that kind of really helped us to get an idea of what we’re looking at. And our main question was what’s happening inside of podcasts. So we came across a lot of recurring characteristics, such as like intimacy, for example. We know that the podcasting medium is a highly intimate medium, but then again, getting a little further into it, actually Dario, again with your work with Richard Berry, you’ve linked this to the modes of listening. So inside of the modes of listening, we have the headphones, earphones, the physicality and the oral mechanics of podcasts listening come together to make a really intimate listening experience for the listener. And then following our understanding of intimacy, we came across a lot of studies to support authenticity as a really important aspect of podcasting. And again, authenticity today may be different than even six months ago considering what we’re going through, with everything going on with Coronavirus and all of these lockdowns. So our tolerance for authenticity is changing. So maybe in six months again it would be good for us to look at the data again and kind of figure out where authenticity is. Another place our literature led us to is storytelling. Storytelling can be used as a tool to create a memorable content, it also really helps captivate audiences and ultimately it also can be used to bridge that gap between the listener and the content, which is really why we love podcasting to begin with. It feels intimate, we know our hosts and we have those parasocial relationships.
Lori Beckstead [44:00]
Yeah, so that touches on a few of the characteristics that I think we’ve identified as well. And, you know, to be quite frank and open about the process here and some of our limitations and challenges, when I put together the forms for the students to rate the podcasts I left off intimacy as one of the characteristics, so we didn’t actually get to rate that [laughs]. But as Anna says, authenticity and intimacy really seemed very correlated and so I think we can maybe infer from the authentic. But ultimately it was authenticity, storytelling, humor, emotion, you know how much emotion the podcast evoked in the listener, that’s how we chose to try to define what that meant; how informative it was, whether there was celebrity involved in the podcast as a host or as a guest; what the production quality was; what level of sound design there may have been, whether it was niche or mainstream on that spectrum, or whether it was on the spectrum of being evergreen, or timely. So those were what we ultimately settled on. And just to kind of further answer your question Taylor, you know, some of the data we did find out was interesting. So, for example, of all of the 78 podcasts we rated (updated number: 76 podcasts), and I should mention which podcasts did we rate. We took the top 20 from the Apple US charts and the top 20 from the Apple Canada charts. We took the top 20 from the Spotify us and Spotify, Canada. And again, we have some issues around knowing exactly how those are determined but because they’re, you know, super prominent in terms of where people find their podcasts, we took those. And then to kind of include critically acclaimed podcasts, I was collecting this data in 2019 so I took Time Magazine’s Best podcasts of 2018 and CBC Radio, the Canadian Broadcasting Corporation, their podcast playlist show did the best podcasts of 2018 as well. So I stuck all those together, got rid of duplicates, came up with 78 podcasts , and so of those 78 (updated number: 76 podcasts), storytelling turned out to be a major factor. It was rated a four or a five out of five in […]
Taylor MacLean [46:14]
Lori, can I just interrupt you for a second? Sorry, can I ask how you define storytelling? Just because for me, that’s one that along with authenticity, is one of the ones that’s a little bit harder to define.
Lori Beckstead [45:25]
Yes, for sure. So does the podcast contain storytelling? Do the hosts or guests recount stories with a narrative such as a personal experience or a story about someone else? So that’s a pretty loose definition. And it covers, I think, a wide range of the types of stories. Because I think sometimes when you say the word storytelling, you know, this is a familiar term to media production students, and, you know, that everything is a story, but I think maybe the lay person might think of, you know, sitting down with a story book and reading it to children, right storytelling. So I think that opens it up to people just recounting personal experiences or telling something that happened to them or even telling a funny joke that kind of involves a narrative. Yeah, but that’s a good question.
But so, storytelling turned out to be really 62% of the podcasts that were rated had storytelling as a huge factor. So that tells us something I think, you know, from this limited sample that storytelling is a big deal in podcasting. 65% of the podcasts were rated as highly informative, either a four or five out of five on being informative; 56% were rated as authentic, being high on authenticity; and production quality from these top podcasts, or critically acclaimed podcasts, fully 78% of them were rated very high on production quality. So perhaps that tells us something too, about what makes it into the mainstream top charts – that production quality maybe does need to be a pretty big factor. And interestingly, for niche 28% of these podcasts were rated as being niche. And niche is talked about as being like one of the great things about the podcasting medium is that, you know, anyone can do it and you can have a podcast that is just about, you know, what type of broom is appropriate to play the game of Quidditch in Harry Potter, for example, right? So it can be extremely niche but that doesn’t mean that niche podcasts are the ones (…) I mean, that makes a lot of sense, we were looking at the top podcasts in charts and so presumably they need to be more mainstream to appeal to a wider bunch of people.
Yeah. So that’s some of the interesting things that we’ve found through doing this and I feel like this approach can take us in many different ways. We could also do things like consider adding other characteristics like the human voice is one that I left out and I think that that would be so important to look at podcasts right now they tend not to be music based, so I mean, there are some that talk about music and have some music in them, but because of the limitations of copyright, being able to clear the rights to put music in podcasts can be very expensive, and so we have a medium that is primarily focused on speaking the human voice conversation. And so what role does the voice have, we know that people have conscious or unconscious biases about gendered voices, about things like vocal fry, about the type of vocabulary or accent that people use, so that could be a really interesting overlay with this as well. But I’m also interested in, you know, I hope that this system could also help listeners understand their own motivations for what they like a little better. So if they get some insight through seeing this sort of transparent rating system that, ‘oh, you know, I never really thought about it, but I really am attracted to things that are really high on storytelling and that are really emotional and so now that I have that insight, maybe I can, you know, not fall prey to the corporate machine that Dario was referring to in terms of quote unquote, discoverability’. So maybe it could be helpful in that way too.
Note: Would you like to give feedback on or ask a question about this research? Be part of the Open Peer Review! Please visit the comments section at the bottom of this page to participate.
Taylor MacLean [50:12]
So you were taking these rating systems and then comparing, were you comparing genres at this point?
Lori Beckstead [50:24]
Well, that’s one of the places we went first with it. [It] was ‘let’s look at podcasts from the same genre’, and that’s where we found that this rating system did show very similar shapes on these charts. So [it] felt like that gave us a sense of this being, I guess, a fairly reliable way of looking at things, because you would expect things of a similar genre to have similar characteristics, right? So that let us know we were on the right track with rating them in this way, rating the podcasts in this way. But then we moved to looking for patterns across different genres to which helps us to think ‘okay, could this actually work as a recommendation engine that moves beyond genre and beyond the keywords’?
Taylor MacLean [51:10]
Right, so did you find that you could kind of make connections across genre and find these characteristics?
Lori Beckstead [51:20]
Definitely. Yeah. So I gave that one example of the true crime in the natural sciences one. Another example would be another science one called The Habitat, and society and culture relationships genre, Why Won’t You Date Me? Those ones shared similar shapes as well. So they were high on storytelling, they were high in humor, they were very authentic, the production quality was good. So, it seems to me that, you know, perhaps there could be something there in terms of going across, outside of genres, beyond keywords, and being able to find things that people may like.
We also identified that some of the characteristics are probably more important than others. So for example, if two podcasts across genres shared high production quality and high sound design, for example, and were also very niche, does that really tell us a lot about what those podcasts are like? You know, if they don’t share ratings, similar ratings on the other characterises, like, so there’s some characteristics that help to shed light on the others or have some kind of correlation with the other characteristics but may not be enough on their own. So we’re looking at which of the characteristics would be more important than others in terms of making meaningful recommendations across genres.
Taylor MacLean [52:48]
And how do you know that? Like, how do you measure that and figure out which ones are the most important?
Lori Beckstead [52:54]
Right? So right now it’s kind of just a gut reaction to, you know, I think if something’s very niche and it’s also good at production quality that doesn’t necessarily mean people are gonna like both, you know? I mean, obviously, a niche is a very personal thing. So I think that might be like stage two of this research is, you know, to validate it is to actually come up with some recommendations and ask people to listen if they liked the first one podcast, will they also like the one that we recommend? And I think that we would have to do that kind of testing to see if this is really a viable way of recommending podcasts.
Dario Llinares [53:30]
There’s the guy who did those visualization studies about podcasting groups, have you seen that? I saw that I forgot the guy’s name. I should have written that down.
Lori Beckstead [53:39]
Dan Meisner. Yeah, from Pacific Content
Dario Llinares [53:42]
Yeah, that’s kind of interesting, because I think there’s a way of kind of maybe cross pollinating the idea of groupings there, which may be kind of useful. And, again, I think, you know, I share sort of what Taylor was maybe sort of alluding to in terms of the readings of the graphs can be very kind of like, you know, non scientific if we’re being disingenuous, you know what I mean? If we’re being supercritical where ‘yeah, it looks a bit like that, and that looks a bit like that’. I’m just wondering (if there’s any) if those could be kind of read in a more systematic kind of way. And then maybe again, what would be really interesting is if, because of the particular podcasts that you’ve selected are all in this all in this bracket of being in the charts or being recommended by, you know, in a sort of top 10 in a national newspaper, so they are defined by, you know, legacy mainstream criteria in that sense. I mean, it would be really interesting if you could get a similar number of podcasts like, say for example, if you went on social media and say, ‘does anybody, any independent podcasts want to subject their podcast to a reading in this way?’ And you’ve got a sort of a variant group, which were not on that [previous] list, and were completely independent, and then sort of looked for correlations in the way in which similar types of podcasts of the same genre, whether it’s true crime or news or whatever it might be, whether the size of the grass is slightly smaller when they’re independent and they don’t have the production values or what things stick out more even though they are the same genre because then there’s again another sort of typology may be on offer depending on what kind of podcasts you’re you’re looking at.
Lori Beckstead [55:28]
Yeah, that would be a great idea. It’s true that, you know, we focused on a specific subset of podcasts and they happen to be mainstream. And to be able to test this across just different podcasts that aren’t, haven’t already been rated as valid, I suppose. I think when I set out to do this I was thinking, you know, can we determine factors, characteristics of podcasts that make them successful? That was kind of my original question. And that’s why I went to the, you know, the top charts to see if there was something we could learn there. But I think applying it across a range of, quote unquote, successful or unsuccessful podcasts might be really interesting to do.
Dario Llinares [56:08]
Success is a very variable factor, though. That’s really hard to test for.
Lori Beckstead [56:11]
Exactly. Yes, let’s just throw in another factor that nobody can define and see what we can do with it. Yeah. [laughs]
Taylor MacLean [56:20]
This sort of leads me to asking about, I guess other limitations and challenges that you experienced Lori, during the study.
Lori Beckstead [56:30]
I think I’ve probably kind of covered most of them. I don’t think I have necessarily anything else to add there. But maybe Dario has some other places to poke at with that.
Dario Llinares [56:44]
The only other thing that I would just sort of mentioned is the idea of, if this, you know, remains within the boundaries of an academic project then, you know, fine, that’s what it is. But I mean, you know, you’re talking about some potential for scalability beyond that. Maybe possible if, like you say, you could connect it to, say, a web hosting site in which instead of having their five stars they have this more, you know, in depth sense of ratings. And not necessarily doing that just in terms of you trying to get open source data, but just in the sense of, you know, you’ve got an alternative that is using this criteria and you could end up seeing where that goes and working with a web hosting platform may offer up sort of slightly different opportunities. Maybe because as you sort of talked about when we’re doing academic studies within the university we are sort of limited by that.
Lori Beckstead [57:45]
Mm hmm. Good point. Thank you.
Taylor MacLean [57:49]
And the only other thing on my list, which I think just would be of interest, I mean to me but I think also to listeners. Maybe both Lori and Dario if you can talk about this, how you got into podcasting as an area of research? What led you to this area?
Lori Beckstead [58:08]
I love sound media, I’m such a sound geek. I grew up on campus radio and then got involved in that. And so I’ve been a producer and a host in campus radio, and I am now a producer and host in podcasting. I teach sound media, I just love it. And so of course this is naturally my area of inquiry and it’s exciting to be inquiring into a medium that is still so young. The first mention of even the word podcast was in 2004 and so we’re looking at a teenage medium here and to be involved in studying it at this point is very exciting because there is so much to be discovered and to be talked about. I love having long discussions with my research assistants about what we’re finding out and yeah, it’s just exciting for me. How about you Dario?
Dario Llinares [59:05]
Well like, I kind of got into it a little bit by accident because I’m from film background, film studies background, and me and my colleague that I was working with at the time, we were listening to, we were starting to listening to quite a lot of film based podcasts and we felt that there was a sort of, it was a really interesting area where there was, it seemed to be getting to the crux of what was going on in film, much more than just film reviewing was doing at the time. And we were just kind of like, ‘yeah, we could do that’ [laughs]. So we just kind of like not knowing anything about audio production or anything like that we just started doing the Cinematologists and it picked up pretty quickly. I sort of taught myself editing and, you know, the sound production sort of improved over time and it really pointed me in directions that began to make me look at what was going on in academia and how we understand communication and how we understand how knowledge is defined by the medium that we’re using. And it was related very much, it became related for me to the limitations of the academic journal, which is sort of a little bit why we’re doing this today where you have this text and then a couple of people read the text and it’s all a very disjointed, not a very fluid conversation or dialectic kind of process. So, it’s not a very dialectical process when you’re trying to sort of be productive of knowledge and I just felt that the way that the conversations were going in the podcast that we were doing, they weren’t just people having a conversation, they were actually productive of knowledge. And because people were, you know, me and my partner and the people we were interviewing, the way that we were talking was not like a, you know, a regular film interview, we were actually discussing ideas and sort of producing new ways of understanding things. It felt to me that it was best that it was kind of happening. So that then suddenly turned into, you know, ‘oh, this is actually a kind of a movie in which the subject and the form are coming together at the same time’. So the subject is podcasting and the form of podcasting is entering into the subject. So, I kind of found it interesting and then, you know, I had to go back and write a book about it so I completely contradicted myself [laughs], so there we are because you can’t get around the fact that universities want books and papers written down.
Lori Beckstead [1:01:48]
Indeed, but we’re trying with this little effort right here. I guess what we’re trying to demonstrate is having a conversation around something that’s pre-publication and, you know, in the attempt that by discussing it in this way, through a podcast, we can improve the thinking around what ends up in the final publication. You know, I’ll hand in that essay, but I had way more fun doing this podcast! [laughs]
Taylor MacLean [1:02:17]
Yeah, I think it also demystifies the process, you know. Speaking as someone who is not so familiar with the research process that academics go through, this is a really great way to get some insight into that and to see kind of how the questions evolve, how the sort of investigation proceeds.
Lori Beckstead [1:02:40]
I’m glad to hear that. It’s hitting the mark then.
Taylor MacLean [1:02:44]
Okay, well, thank you both so much for being here. This has been fascinating.
Lori Beckstead [1:02:48]
My pleasure. Thanks. This has been a productive conversation.
Dario Llinares [1:02:51]
Thanks very much for inviting me and look forward to seeing how this develops.
Voice Over [1:03:01]
Thanks for listening to Open Peer Review. Want to join the conversation? Your feedback would be a valuable part of the research and knowledge-creation process. Visit oprpodcast.ca.
[Music plays then ends]
Note: Would you like to give feedback on or ask a question about this research? Be part of the Open Peer Review! Please visit the comments section at the bottom of this page to participate.
Open Peer Review Podcast 