Yale Psychiatry Grand Rounds: December 16, 2022

December 16, 2022

Information

"Hey Siri, Please Rate My Therapist. The Promise of AI and NLP for Training and Quality Assessment in Mental Healthcare."David Atkins, PhD, Research Professor, Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine

ID9312

To CiteDCA Citation Guide

00:00Um.
00:04Thank you so much Seth and John
00:06for inviting me and hosting me.
00:09Finding out that I'm the last
00:11grand round speaker of the year,
00:12I feel like I'm standing in
00:14between you and your holidays.
00:16Hopefully it doesn't feel that way to you.
00:19But very excited to be here today
00:21and talk about our research.
00:23Let me. Without any.
00:31Further in do we'll get.
00:37We'll get started. It it it is still
00:40early out here on the West Coast,
00:42so you'll have to put up with me being
00:45lit by the glow of my computer monitor.
00:48So. Very excited to be able to
00:50be here and talk about our work,
00:54which is focused on AI broadly.
00:57And we'll unpack exactly what that means
00:59and what we do to try and understand
01:02really at a fundamental level what
01:04happens in counseling and psychotherapy.
01:06How is it that we can capture
01:09some of those active ingredients
01:11using new technology so that we
01:14can understand something about?
01:16What is in the pill of
01:17psychotherapy out in the real world?
01:22Hold on a second.
01:24Before we jump in, I I do want to
01:28acknowledge our research support.
01:30Which has come really in two phases.
01:32There has been foundational
01:34research at the universities.
01:36This is really been about 15 years
01:39of trajectory at this point and that
01:42research is continuing at the startup
01:44that we founded that Seth had mentioned.
01:47And just to disclose,
01:49I am a cofounder and have an
01:52equity stake in that company.
01:55Alright, so where are we headed today?
01:57What am I hoping to cover?
01:59I want to talk a little
02:01bit about the problem.
02:02What is the problem that
02:03we're trying to solve?
02:04And then spend a bit of time at
02:06at at least an intuitive level
02:08trying to provide an understanding
02:11of how the AI technology works.
02:13How is it that we can go from
02:16fundamentally a conversation
02:17to something about the quality
02:20kind of actionable information
02:23fidelity and competence?
02:25Of psychotherapy and counseling.
02:26And then we'll shift and we'll look
02:28at some of the technologies that we
02:30are developing and studying right now.
02:37SO1 slide hitting an issue that we
02:39all are intimately familiar with,
02:41behavioral health problems
02:43are massive and disabling.
02:45This figure of 20% of Americans
02:47is now a couple years old.
02:49I can only imagine,
02:50courtesy of the pandemic,
02:52that that is higher.
02:54If anything,
02:55my colleagues in the Institute for
02:57Health Metrics and evaluation here
02:59at the University of Washington have
03:01been conducting the global burden of
03:03disease study for a number of years.
03:05And one of the findings?
03:06That was provocative when it first came out,
03:09was how large a proportion of psychiatric
03:13and psychological conditions account
03:15for in terms of the global burden of disease.
03:19So again to say something that we all know,
03:22there is a huge need for us to have
03:25effective and widely available treatments.
03:28There are two fundamental problems
03:30that we bump into on a daily basis.
03:34One is access,
03:34and this tends to be front and center,
03:37I think in our minds for good reasons,
03:39this is being.
03:43Exacerbated by the workforce shortage and
03:45how do we train up a new workforce members?
03:50And then the second issue is quality.
03:52And quality can often I think fly under the
03:55radar to a certain extent behind access,
03:58but let me make the case that
04:00it is pretty significant.
04:02That,
04:03as best we can tell,
04:04there's approximately 100 million counseling,
04:07psychotherapy,
04:07behavioral health oriented
04:09intervention conversations each year.
04:12And we basically don't know what happens
04:15in any of them for the most part.
04:18When we do get glimpses into the
04:21actual conversations that are
04:23counseling and psychotherapy,
04:25we can find incredible variability.
04:27And so let me unpack what
04:29we're looking at here.
04:30My colleague John Bayer,
04:32who's at the VA Puget Sound
04:34on several years ago,
04:36did a motivational
04:38interviewing training study,
04:39partnering with a variety
04:41of community agencies,
04:42substance use agencies here
04:44in the Puget Sound region.
04:46As part of that,
04:48we created 200 providers,
04:49got six recordings of
04:52their actual counseling,
04:53and then spent the time effort resources
04:57to actually fidelity code those.
05:00And so this is just the empathy rating
05:02that comes out of the motivational
05:04interviewing treatment integrity system,
05:06their fidelity coding system.
05:07And so each one of those purple
05:10dots is the average empathy
05:12for each one of 200 providers,
05:14most of whom have about 6 sessions.
05:16Going into that,
05:18so based on what we know from this scale,
05:21we have here at the upper end
05:23of this scale some superstars,
05:25these are clinicians that are
05:28out practicing in substance use
05:30settings who are demonstrating
05:31a deep understanding of their
05:33clients worldview and capturing
05:35something about the meaning of what
05:37is being said by their clients.
05:39On the other hand,
05:41down here we have a group of no
05:43stars I guess who are demonstrating
05:45no interest in their clients
05:47worldview and little to no attention
05:50to what the client says.
05:51And if you are like me when
05:53I first saw this thinking,
05:54how is that possible?
05:56If these are actually behavioral health
05:58providers, these sessions tend to
06:00sound a little bit like Doctor Phil.
06:03You know, when are you going to get real?
06:04When are you going to come to grips with
06:06the fact that substance use is ruining
06:08your life and it's causing problems?
06:09And so there really is a sense that
06:11they are not listening to the client.
06:12They have a message that
06:14they're hammering home.
06:15And based on what we know of
06:18the very broad literature on the
06:20association of empathy and outcomes,
06:23this is really toxic treatment.
06:26But the main point that I'm trying
06:28to make here is not that there's
06:30some toxic treatment happening,
06:31but just that there is incredible
06:33variability and we don't have
06:35any line of sight into this.
06:36This is data that has only come out
06:39of a well funded NIH research study.
06:42And so in those well funded
06:44NIH research studies,
06:45we use the traditional evaluation method
06:48for counseling and psychotherapy,
06:50which is behavioral coding or human coding.
06:52We don't usually do it in real
06:55time like this picture.
06:56We record sessions and then have
06:58a team of experts evaluate them.
07:00But it is slow and it is expensive and
07:03it really is not used in the real world
07:06outside of well funded NIH research studies.
07:09If we had AI or or some alternative
07:13rapid means of assessing quality,
07:15there's a variety of uses that we
07:18could use it for performance based
07:22feedback and training for supervision.
07:25One of the best ways we'll we'll
07:27look at an example of using this
07:29in a in a few minutes,
07:30but one of the best ways to learn
07:32is to get specific feedback on
07:34new skills that you're learning.
07:38Similar to the image,
07:40the figure that we just saw.
07:43Having some means of quality and
07:45sure quality assurance or quality
07:47improvement within service delivery.
07:51We think a little more commercially,
07:54payers are writing checks for services
07:56of unknown quality at this point.
07:58So payers could have something to
08:01know what they're paying for and
08:03could potentially be the basis for a
08:05type of value based care arrangement.
08:10Finally, there are in in many ways we
08:14still don't know exactly how it is.
08:17The conversations of of counseling and
08:19psychotherapy lead to behavior change.
08:22And so being able to open the black box
08:25in this sense could help us understand
08:27how is it that those conversations,
08:30those intimate engagements that
08:32we have with our clients lead
08:35to sustained behavior change.
08:39So let, let's tip into.
08:41So over the last 15 years, I have been,
08:45I am a clinical psychologist by background.
08:47I do have an interest in data science,
08:50but this work has really been
08:53enabled by deep and sustained
08:56collaborations with technical
08:58experts across machine learning,
09:01natural language processing,
09:03speech engineering.
09:05And so we'll we'll dip into.
09:08Within this next section,
09:10I'm hoping to give kind of
09:11an intuitive understanding
09:13of how the technologies work,
09:15and if we want to dip into more details,
09:17happy to do that in the Q&amp;A.
09:22So let me start with just thinking
09:24through at at a really basic if we were
09:27to to trying to describe to a layperson,
09:30you know, what are the raw data,
09:32what are the basic building blocks that
09:35go into psychotherapy or counseling.
09:38So first and foremost it's words,
09:40whether that is you know
09:42increasingly now telehealth.
09:43But it could be in person,
09:45it could be telehealth,
09:46could be on a phone, could be video,
09:48could be text or chat based text interaction.
09:51But it is a conversation,
09:53so words are one of the basic ingredients.
09:57For everything except text based chat,
10:02interaction, tone and other types
10:05of paralinguistic information.
10:07So there's tone prosody things like
10:10linguistic disfluencies when someone,
10:12someone,
10:13someone maybe perseverates on a certain word,
10:16and that's indicative of cognitive load.
10:18So all of the types of information
10:21about how something is said versus just
10:24what is said in the words is important.
10:27If it's in person or video,
10:29there's different types of
10:31nonverbal information.
10:31That could be posture,
10:33facial emotion,
10:34gestures.
10:37And then finally, it's not simply
10:39each one of these components,
10:41but the dynamic way in which they
10:43unfold in the interaction itself.
10:46So the we can't just take
10:47a statement such as, wow,
10:49I cannot imagine how difficult
10:51that must have been to lose your
10:53kids and on its own say whether
10:55that is an appropriate or good
10:57high quality intervention.
10:59We need to know what's the context,
11:00how would, where is that being said?
11:04So then let's switch over and think about.
11:06I mean the challenge of measuring
11:09counseling and psychotherapy is
11:11fundamentally that it is a conversation.
11:14It's very unstructured.
11:15We think about other types
11:17of data that we collect,
11:19whether that's lab tests or
11:22questionnaire data like the PHQ 9.
11:25There is inherently nothing
11:27numeric about a conversation.
11:29So how do we actually
11:31quantify this information?
11:35Historically, natural language
11:37processing use what's called ngrams,
11:40and that's really a fancy way of saying.
11:44Quite literally.
11:45They would dummy code create indicator
11:48variables for unique words, for vocabulary,
11:51or for short common phrases.
11:53So two word and three word phrases.
11:57And if you're thinking like,
11:58how is that even possible?
12:00If you have a basic understanding
12:01of a regression model,
12:02wouldn't that mean that there are thousands
12:05and thousands of predictors in these models?
12:07Yes, that that is exactly right.
12:10That is a good intuition.
12:12Increasingly, these models are now
12:14using something called word embeddings,
12:17and this is the idea that
12:19when we see a word there,
12:21it has a certain meaning.
12:23And and these word embeddings are ways of
12:26trying to get at a meaning so that it's
12:29not just a word with an indicator of binary.
12:33Yes, no, this word showed up, but it's
12:35implying something about the meaning.
12:37I won't try to go in to explain that, though.
12:40Happy to go into the weeds if
12:42that were of interest later.
12:43But so there's there.
12:45There are ways of quantifying words.
12:48Similarly,
12:49there are a variety of speech
12:52signal processing methods so that
12:54we can estimate acoustic features.
12:57So these are things like tone,
13:00the vocal arousal that we can hear,
13:03and in someone's voice.
13:04When someone is excited and the pitch
13:07or the tone of their voice goes up,
13:08we can measure that reliably.
13:11And in addition,
13:12we can also measure something called jitter.
13:15This is when someone is really upset
13:16and we say that their voice is shaking.
13:19That is the very extreme form of jitter,
13:21but we can measure that over a broad range.
13:23So again,
13:24point being that there are speech
13:27signal processing methods for
13:30quantifying these types of information.
13:33Similarly,
13:33there is an area of machine learning
13:36called computer vision that is the
13:39reason if you use Google Photos and
13:41you can search Google Photos for
13:43finding your dog or what have you,
13:45that is computer vision is the
13:48AI engine that's enabling that.
13:50Although our team has some expertise in that,
13:52that's not a focus of our current
13:55work as we have felt that language
13:57and words is really the lowest common
14:00denominator across all the different.
14:03Medium of counseling and psychotherapy.
14:07Finally,
14:08there are a variety of techniques
14:10both machine learning and natural
14:12language processing,
14:13but also outside of that dynamic
14:16systems models for understanding how
14:19is it that interactions unfold over time.
14:22So just a couple again,
14:24my goal here is really to provide
14:26some sense for what is this process
14:28and how does this work and so let's
14:30take an example of 1 specific example.
14:32So our work started.
14:35Is motivational interviewing.
14:37In hindsight,
14:37that seemed like an incredibly wise choice.
14:39The reality was, of course,
14:41it was a bit happenstance.
14:43Was collaborating with some
14:45colleagues who were using MRI and
14:48had recordings from some RCT's,
14:50that that was the very first
14:52grant in this work.
14:54But MI is a fantastic place
14:56to start because it
14:58is very linguistic focused.
15:01So am I is interested in things like.
15:05Or is a clinician asking an open-ended
15:08question versus a close ended question,
15:10so really tightly tied to the language itself
15:14in a way that is fundamentally different
15:16than say cognitive behavioral therapy where
15:19they're interested in assessing how well
15:21the clinician set in the agenda. And yes,
15:23there we we can know that from the words,
15:27but it's at a kind of higher level,
15:29it's more of a psychological
15:30construct that's in the words.
15:32So am I was a great place
15:34for us to start this work?
15:36And so we have this brief little
15:38snippet of transcript here.
15:40You know, client says,
15:41I wouldn't mind coming here for treatment,
15:43but I don't want to go to one of
15:44those places where everyone sits
15:46around crying and complaining all day.
15:48The counselor says you don't want to do that,
15:50so you're kind of wondering
15:51what it would be like here.
15:54So this was a an example that we used
15:57in in one of our early research papers,
16:00where the goal was can we
16:03automatically identify when a
16:04therapist is making reflections,
16:07when they're providing a brief summary
16:10and reflecting back to the client whether
16:14they are understanding them correctly?
16:17And so let's just use this to unpack.
16:20How do we actually go from a transcript
16:24of words to a predictive model?
16:27So again, as we talked about, um,
16:29one of the traditional ways is that
16:31we would use what's called Ngram
16:34features where literally it's just
16:36identifying there's particular
16:38words that are in this statement.
16:40And also common two or three word phrases.
16:44So again we would basically be quantifying
16:48those in types of indicator variables.
16:52This is also a conversation,
16:53so something is happening over
16:55time and so the local context,
16:57especially for trying to understand
16:59something like a reflection,
17:00which is inherently something being said
17:03back in response to a previous statement.
17:07So something about the context so
17:09we can look at words in the local
17:12context before or after.
17:13There's also a little bit of metadata,
17:16and when we use metadata in
17:19natural language processing,
17:21it refers to anything that is
17:23not the words themselves.
17:25And in this case,
17:26what we minimally know is that
17:28there are two different speakers
17:30and they have different roles.
17:32So is this the client or is this
17:34the therapist who's speaking?
17:38Finally, we can create other types
17:40of features to include as predictors,
17:43and because a reflection is inherently
17:46capturing something about the,
17:48there should be some similarity
17:49with what the client has just said.
17:51We can identify other types
17:53of similarity features,
17:54whether those are parts of speech
17:56such as are using an adverb or a
17:59direct match in terms of words.
18:01So all of these would be ways
18:04that we could quantify and and,
18:06you know, in a statistic sense.
18:08Create a a set of predictors
18:11for our prediction equation,
18:12trying to identify a reflection.
18:17One of the other things that as
18:20I got into this work and began
18:23to collaborate with computer
18:24scientists and speech engineers is.
18:27I do enjoy data science and there was
18:30a period early in my career where I
18:32was a lot of my time was being spent
18:35really as an applied biostatistician.
18:37As we got into this work,
18:39I quickly realized there's a whole
18:41set of models and methodologies
18:44that I was never exposed to.
18:47And so part of the inter interdisciplinary
18:50work is really that translation,
18:52being able to form a foundation of knowledge,
18:56both clinical knowledge,
18:57so the computer scientist and the speech
19:00engineers needed to learn something
19:02about motivational interviewing,
19:04but also for the rest of the team
19:06to understand something about the
19:07models that are being applied.
19:09And so if you look at these different models,
19:13latent Dirichlet allocation,
19:14conditional random field,
19:16recursive neural networks.
19:18And think I have never heard of any of
19:20those and I at least had some stats.
19:21You are not alone and that's
19:24something about the the work here
19:26able to bridge those gaps.
19:30So the initial phases of our research
19:34focused strongly on this idea of can we
19:37go from the word spoken in a session
19:41to reliably estimating fidelity codes.
19:46And we over the course of about 8 to 10
19:49years, we have probably 25 or 30 publications
19:52capturing different aspects of this,
19:54looking at words themselves,
19:56looking at paralinguistic information
19:58tone itself, combining them,
20:00different types of models.
20:02And and let me,
20:03so let me show you a couple results and let
20:06me tell you what we're looking at here.
20:08So the traditional method for
20:11estimating a fidelity code is to
20:14have a team of raters learn a
20:16well validated clinical system.
20:18So here the motivational interviewing
20:21treatment integrity system or the
20:23motivational interviewing skills code
20:25system and then they make their ratings.
20:29But even well trained humans do not
20:32agree with each other perfectly.
20:34So we call that inter rater reliability
20:37and so that's an important piece.
20:39For training a computer to be
20:41able to do this,
20:43which is that interrater reliability
20:45of functionally sets a ceiling for us.
20:48And so the goal here is really can we
20:51develop an AI algorithm that will be as
20:54accurate as the most accurate human.
20:57And So what we're looking at there on
21:00the X axis each one of these labels,
21:02advice, giving, affirmation,
21:04confront these are specific fidelity
21:06codes within the motivational
21:09interviewing system.
21:10Either things that you should do,
21:12such as asking open questions
21:14and making reflections,
21:15or things that you should not do,
21:18such as confronting your client
21:20or giving up giving them advice.
21:22And that X axis is asking,
21:25out of the reliability of the human raters,
21:28how reliable is the computer estimate,
21:31and so at 100% the computer
21:36is estimating providing.
21:38Fidelity codes that are identical
21:40to our most reliable human coders,
21:44and so we can see that over time
21:46this is not where we started,
21:47but over time we have been able to
21:50develop AI algorithms that would
21:52start with a recording and generate
21:55codes that are highly reliable and
21:57very similar to expert human coders.
22:00The the one other thing that I'll
22:02mention here is that this graph
22:05and all of our results use the.
22:08Additional methods within machine
22:09learning to evaluate models,
22:11which is we take the whole data and
22:13we cut it up into a couple pieces
22:16and there's a set of data that
22:18we use to train models.
22:19And then when we are completely done
22:21with the training of those models,
22:23then there is a separate piece of
22:25data that they never saw in training.
22:27That is our test set or evaluation set.
22:29And so these numbers and every time
22:32we evaluate them and come out of
22:35a test set which and I hope and.
22:38In our Q&amp;A,
22:39we can get into a little bit of
22:41the conversation around what is the
22:43data that trains models and then
22:45where is that model being applied,
22:47because potential AI bias is
22:50inherent in those types of
22:52questions. OK. So that was for
22:55motivational interviewing.
22:56More recently, we have done at a parallel
22:59set of work with cognitive behavioral
23:02therapy that's focused on the CTRS.
23:05And similarly over time,
23:06this was not the where we started,
23:09but over time we've been able to
23:12develop models that reliably replicate
23:14what human experts will do in
23:17terms of generating fidelity codes.
23:22And just to highlight another aspect
23:25of this work, which is I I have
23:28primarily been talking about this
23:31one slice around prediction models,
23:33but the reality is the the entire,
23:36what we call the pipeline starts with
23:40a recording or that spoken language
23:44and there's an incredibly important
23:46and perhaps the most complicated part
23:49of what we do is that a speech signal.
23:52Processing tasks.
23:53So from a recording,
23:55can you tease apart uniquely the
24:00multiple different speakers and can
24:02you identify automatically who the
24:04therapist is and who the client is?
24:06And can you then generate a
24:08highly reliable speech to text,
24:10transcript and and so I never
24:13would have imagined,
24:15as I was getting my PhD
24:17in clinical psychology,
24:18that I would be collaborating
24:20at points on methods for.
24:22Lattice scoring in speech
24:24to text transcription,
24:25but that has been part of the work,
24:28is that to help move this forward I have
24:32needed to move into technical areas.
24:34And my technical colleagues have actually
24:37gone to motivational interviewing
24:38workshops taught by Bill Miller.
24:40And I think that's part of the
24:42magic that has made this work,
24:43is that you have a collaborative
24:45team that's really willing to get
24:47outside of their comfort zone pretty
24:49dramatically in certain cases.
24:53All right. Just to give a snapshot,
24:56so the that AI pipeline that we
24:59were just talking about at the
25:01moment generates around 54 metrics.
25:04And so as we saw for both CBT and
25:06for motivational interviewing,
25:09we generate gold standard fidelity metrics.
25:12These are not systems that we made-up,
25:15but we went to the literature and said,
25:17OK, CBT researchers,
25:18am I clinical developers, what all,
25:21what is the gold standard?
25:23And there's also some other
25:24things that we have kind of baked
25:27into the pipeline over time.
25:29So there are some content codes,
25:31so we can have the goal here was
25:33really to provide a line of sight
25:35into what's this conversation about?
25:37Is it about at a high level,
25:40is it assessment or therapy
25:42or case management?
25:44And then what's the focus
25:45of the conversation?
25:46Is it about mood problems or
25:49trauma or suicide work problems,
25:51intimate partner problems?
25:53So we can capture something about
25:56really what's the conversation
25:58about and then I'll we'll talk a
26:00little bit about this at the end,
26:01but we have some exciting
26:04developments in particular around
26:06suicide risk assessment.
26:07And automatically identifying emotions.
26:12OK. So we, I had started off a couple
26:15minutes ago showing a graph that looks
26:18a lot like this that was courtesy
26:20of my colleague John Bair to give
26:23a sense for the power of the AI.
26:26So that was approximately 900
26:29sessions that took John and his
26:32team about a year to generate.
26:35Now that we have moved this
26:37technology outside of the university,
26:39we have an opportunity to work with.
26:42And partnerships customers and
26:46in partnering with a large
26:50digital telehealth company,
26:52we got access to a a million sessions
26:56on more than 5000 providers.
26:59And so here it's hard to see but
27:01there are actually 5000 purple dots.
27:04And there were summarizing
27:06approximately 1,000,000 sessions
27:07on that same empathy scale.
27:10We see a similar pattern here,
27:12but in addition because we were
27:15partnering with a a real World
27:18Service delivery provider,
27:20we also got access to some KPI's.
27:23And so here we can say not only
27:26something about the empathy but
27:28we can also look at if you saw
27:31providers who are highly empathic.
27:34Turns out that your clients
27:36were much more satisfied.
27:38That's Net Promoter score,
27:40which is functionally a zero to
27:4210 score of how satisfied you are.
27:45You're much more satisfied than if
27:47you saw folks who are less empathic.
27:50It's not rocket science.
27:51If someone is really good at
27:53paying attention to you and trying
27:55to understand your worldview,
27:57it's not surprising that their
27:59clients will be more satisfied.
28:00But we also see this with other effects.
28:05So this what we're calling
28:07active listening here, is,
28:08for those of you who know,
28:10motivational interviewing.
28:11It is a summary of ORS,
28:14how out of all of the language
28:17from the therapist,
28:18how much of that is open-ended questions,
28:22affirmations, reflections and summaries.
28:25And again,
28:27it is those really low level
28:30micro counseling skills.
28:31And what we see is that if you see
28:34a therapist who is really good at
28:37listening at those kind of basic
28:40active listening skills.
28:41You are much more likely to get what
28:43we would consider a full dose of treatment.
28:46Conversely,
28:47if you see someone who does not do
28:49that and is probably giving advice,
28:52giving lots of information and
28:55maybe even confronting,
28:57then you don't stay around
28:58in treatment very long.
29:00So the the key point here is that
29:02both there is a level of validation
29:04that's coming as this gets out
29:07into large real-world data.
29:08It's also demonstrating that these
29:11tools can to a certain extent begin
29:14to open up the the black box that
29:17is therapy in the real world and
29:20provide a line of sight and some
29:22reliable indicators of what's happening.
29:25So let's now kind of shift gears
29:27a bit and we'll look at some
29:30of the specific technologies.
29:32So really everything that I've
29:34talked about thus far is kind of
29:37describing the engine of the car.
29:39And then now let's look at some
29:41different technologies and ways
29:43that it's getting deployed and
29:45some related research as
29:47well.
29:49And so kind of just repeating what I said,
29:52that that university based research
29:54really laid the kind of AI engine,
29:57the AI foundation that now is getting
29:59evaluated in a variety of technologies.
30:05So let's take a look at how we
30:08might use this for training.
30:11One of the things that I love about
30:12Bill Miller and the MIT community is
30:15that they're incredible empiricists.
30:16And so Bill was one of the first people
30:19to do fairly rigorous training studies.
30:22And what he found is that the ways
30:25that we traditionally do training,
30:27which is 1/2 day or a full day
30:31workshop don't actually work very well.
30:33And when when I say they don't work
30:35very well, I mean they don't have
30:38durable effects on provider behavior.
30:40And so he, he is raising this
30:43question of how,
30:44how is it then that we could train to
30:47get broader and more durable effects?
30:50This is particularly critical right
30:52now because we have such a workforce
30:56shortage and so being able to train
30:59rapidly and highly effective ways
31:02it would be incredibly useful.
31:05Fortunately,
31:05we haven't yet really moved past
31:09our traditional methods.
31:11We now might do them online,
31:14but our professional training
31:15often includes a lot of content
31:18and so that's either slides and
31:20written content will have lectures.
31:22You can see some examples which
31:25would be good,
31:26but in terms of practice probably limited
31:29to role plays and that's probably a very,
31:32very small part part of the training.
31:35And in terms of assessment or
31:37demonstrating what has been learned,
31:38we're probably limited to some
31:41some type of knowledge quiz.
31:43And so at the end of traditional training,
31:47oftentimes what we can measure are
31:49things like what content was offered,
31:52how many providers access the content,
31:54to what degree, and again,
31:57maybe something about the demonstration
31:59of of knowledge or attitudes.
32:01But critically,
32:02we have not had good ways of actually
32:06assessing skills or skill development.
32:09If we look at other skills,
32:12and I think we can make a very strong case
32:15that counseling and psychotherapy is a skill,
32:18behavioral skill,
32:19much like other types of
32:21skills that we would learn.
32:22A really key component is practice,
32:25lots of practice, ideally with feedback.
32:29And this is some of the ways in
32:31which I think things break down,
32:33which is if you are learning to play tennis,
32:36you know whether you're shot,
32:37went in or out.
32:38And I don't know that that that there
32:41is a strong parallel at that level.
32:43So feedback often is going to
32:46require a coach or outside input.
32:49The other thing that's interesting and
32:51has motivated some of our training work
32:54is that we practice small components.
32:57So professional musicians
32:58still practice scales.
33:00Professional athletes will practice
33:02their backhand over and over.
33:05And it's it's interesting to to think about,
33:07as far as I know,
33:09we don't do that within
33:10counseling and psychotherapy.
33:11We don't.
33:12Practice.
33:13We might do it a little bit
33:16in our degree based training.
33:19And uh,
33:20but we don't.
33:23We we certainly don't do
33:25that in a professional way.
33:27We don't practice.
33:28I'm going to just practice
33:29making reflections or asking
33:31good open-ended questions.
33:35And sorry, I had a little thing pop up.
33:39And, and this fits with what we know from
33:41the learning sciences community community.
33:44So Danny Kahneman said this in his
33:47book thinking fast and slow that.
33:49The acquisition of skills requires
33:51a regular environment and adequate
33:54opportunity to practice and rapid
33:56and unequivocal feedback about the
33:58correctness of thoughts and actions.
34:01And intuitively,
34:02that makes complete sense for how that
34:04is going to drive good skill acquisition.
34:08And man, those are hard conditions
34:10to meet in terms of training,
34:13counseling, and psychotherapy.
34:16So if we then say OK, if that if
34:19those are the optimal conditions for
34:21skilled development and we wanted
34:24to design A training technology,
34:26these would be some of
34:28the design requirements.
34:29There would be a heavy focus on practice,
34:32many unique practice opportunities.
34:34Varying skills, varying content and feedback.
34:39In particular, immediate feedback
34:41on the correctness of thoughts
34:43and actions according to Kahneman,
34:46and being able to track that overtime.
34:51Excuse me?
34:54So this is these are exactly some of
34:57the design requirements and features
34:59that we have built into several
35:02different training technologies.
35:06And so I want to give you a quick
35:09snapshot of our MIT training platform.
35:13So not surprisingly,
35:14we it's designed around different skills,
35:17so there's different skill modules.
35:20And you know, similar to other
35:22types of training, we are going
35:24to have some expert introduction.
35:26So we'll just hear.
35:28A quick intro from Terry Moyers.
35:31One big reason for burnout in
35:34clinicians is the burden of
35:36trying to convince people to
35:37change when they don't want to.
35:39When you see that as your job,
35:41it can be exhausting to spend all
35:43your time trying to persuade clients
35:45to do what they're fighting against.
35:49So again, this would be similar to
35:51what we find in other trainings.
35:53However, the total video
35:57piece here is 3 minutes long,
35:59so we're going to give you a relatively
36:03light touch in terms of that content.
36:06We're going to give you some
36:08examples of asking good questions.
36:13But then the, the innovation,
36:14the thing that we really
36:16spent the time working on was
36:19developing practice and feedback.
36:21And so in this training tool what we
36:24have are a variety of brief clinical
36:28vignettes from standardized patients
36:30with actors portraying patients.
36:32And so we'll just take a quick listen.
36:35My dad was a drunk and I always
36:37thought I would never be like him.
36:41But lately, all I do is
36:42act like him at his worst.
36:47I get mean when I drink. Just like he did.
36:54And again, this is not a
36:56a long kind of 10 minute,
36:59this is 43 seconds and that's what
37:02these were designed as are like 30
37:04seconds to one minute long and a
37:07variety of them where the point is that
37:10it would give you opportunity to practice.
37:13So we'll look at a couple different
37:16practice examples and the types
37:17of feedback then that is given.
37:19So when you drink a lot it
37:21impairs how your brain works,
37:22it makes your prefrontal cortex.
37:24Part of your brain and the front
37:26of your head work less effectively,
37:28so you end up making really
37:31impulsive decisions like getting
37:32into fights when you don't want to.
37:42And so there's an opportunity to practice.
37:44So you, you. We've just heard our
37:47our trainee listen to Gabriella kind
37:49of talk about her drinking problems.
37:52Our trainee went with a
37:54kind of psychoeducation.
37:56Let me teach you something.
37:58And so the system immediately
38:00transcribes what is said and then
38:02tags it within the motivational
38:05interviewing fidelity coding system.
38:07And so through an MRI fidelity lens,
38:10that was giving information psychoeducation.
38:13And so then we get a brief
38:16encouragement and kind of a redirection.
38:19It looks like you gave information,
38:20let's try asking a question.
38:23And so we'll look at just one
38:24or two more examples of this.
38:29If you want to make better
38:31decisions and get in fewer fights,
38:32you have to consider reducing your drinking.
38:35That's going to help your brain work
38:37better your prefrontal cortex and
38:39allow you to make better decisions.
38:41So you might potentially get in
38:43fewer fights with your partner.
38:45Maybe just talk things out instead?
38:55So again, same idea. Unfortunately,
38:57our trainee has not quite gotten it,
38:59so she's now providing advice.
39:02You know, you should,
39:03you should stop doing this,
39:05you know, kind of hung up on
39:08the prefrontal cortex here, Umm.
39:10And then finally,
39:11example decisions and get in fewer fights,
39:15you have to consider reducing your drinking.
39:18That's going to help your brain work better.
39:21One second. OK.
39:27Have you just considered
39:28trying to drink less?
39:34And we're getting closer,
39:35but the goal here is to ask
39:38an open-ended question,
39:39right one in which our our client
39:41is going to kind of open up and
39:44tell their narrative, their story.
39:47Have you just considered trying to
39:50drink less is a yes or no question and
39:53so that's going to close them down.
39:56So just a couple examples of how we have.
40:01Tried to have you just considered that?
40:03There we go.
40:06A couple of examples of how we're trying
40:09to use AI to provide that practice
40:12opportunities and immediate feedback.
40:14Whereas traditionally if we get
40:17feedback it would be, you know,
40:19having an entire session and then
40:21getting some high level feedback on it.
40:23The idea here is can we provide
40:26small training examples where
40:28practice is coupled with specific
40:30feedback in relatively small bytes?
40:33And then there are dashboards for
40:36providing summative feedback.
40:37How well are you learning the skills
40:40and for then completing additional
40:43practice opportunities over time?
40:47There is a larger RCT of this
40:50training that is underway currently.
40:52The pilot work that laid the
40:55foundation for this showed that
40:58the immediate feedback after each
41:02statement not only led to larger gains.
41:06So it's our red bars down here in our graph,
41:10but at a later testing period
41:13where there was no feedback.
41:16We also saw better retention
41:19and it increases overtime.
41:24So that was a quick snapshot on training.
41:27Let's shift gears and think about
41:30supervision and quality monitoring at scale.
41:35And so here we have been thinking about
41:38that AI pipeline and how we could use it
41:41within a direct clinical service setting.
41:45And so a couple features that we have
41:48built into that technology is that at
41:52a foundational level, we believe that.
41:56Providing a direct contact with the skills
41:59of counseling and psychotherapy are
42:02going to enhance reflection and learning.
42:05So, you know, as opposed to supervision,
42:08that begins with a supervisor asking
42:10what do you want to talk about or
42:13tell me what happened last week?
42:15If we can enhance contact with the
42:17review of the actual skills and practice,
42:20regardless of whether we have any AI
42:24fidelity, that's going to enhance
42:26people's reflection and learning.
42:28And so one piece of it is providing
42:30easy access and easy ways to interact
42:33with the session recording itself.
42:35So there's an automated speech to
42:38text transcript,
42:39it's searchable.
42:40Those little kind of orange bubbles
42:42that you see at the top,
42:43those are our AI identified content
42:47codes kind of what what's happening
42:50in this session.
42:51And then you'll notice that we also provide
42:55a little tags for kind of good good behavior,
43:00if you will.
43:02So particularly empathic moments
43:04within the session get tagged as well.
43:10Each session then gets a
43:12formal fidelity assessment.
43:14So here we're providing kind of a
43:16high level summary of the quality of
43:20motivational interviewing and then some
43:22additional traditional fidelity metrics.
43:25And and so this provides both.
43:28We can drill down really deeply.
43:31Again, MI is is helpful because it
43:34actually will uniquely code every
43:36statement from the therapist in a session
43:39and so our AI will uniquely code every
43:42statement from a therapist in a session.
43:46And then we've also built tools
43:48to try and support really more
43:51of a quality assurance view.
43:53If we now have the ability with that
43:57pipeline to take every session and
44:00generate fidelity metrics or quality metrics,
44:04how then do we interact
44:05with all of that data?
44:07If you are in a large agency,
44:09there's thousands of sessions a month,
44:12if not per week.
44:13And so we have built in this kind of
44:16suite of tools that would allow either
44:19a supervisor or a clinic manager to
44:21be able to have really a population
44:23health view of the quality of services
44:26being provided within their clinic.
44:28And so this,
44:29this GIF is showing that you can select
44:32a particular individual quality metric,
44:34you can select a time frame and then you'll
44:37see summaries of all your providers.
44:40And for Andy,
44:41any individual provider,
44:42you can select them and see something.
44:44Up their caseload and drill down
44:46all the way to individual sessions.
44:52This technology also is getting
44:54evaluated in a current RCT.
44:56Actually I think I've got a
44:58few pieces of information about
45:00that on the next slide in.
45:02In preparation for that work,
45:06we did some user centered design
45:09with publicly funded agencies
45:11in the Philadelphia area.
45:13This is in collaboration with Tori
45:15Creed at the University of Pennsylvania.
45:18And so we spent time talking
45:20with therapists and leadership,
45:22showing them examples of the
45:24technology getting their input both
45:26about implementation feasibility,
45:28how how easy or challenging would
45:30it be to implement these types of
45:33technologies within their setting
45:35and and also measuring kind of
45:37implementation readiness and in
45:39particular after we spent time with
45:41them and showing them the existing
45:44technology all kind of across
45:46the board the acceptability of.
45:48Appropriateness and feasibility increased
45:50for both therapists and leadership.
45:56We are just starting recruitment for a
46:00step wedge design where we will across
46:045 actually due to workforce shortages,
46:07we anticipate it'll be more like
46:097 or 8 clinics to get up to the
46:12number of providers that we need.
46:13But we will randomize clinics to
46:17a sequential essentially turning
46:19on of the technology to support
46:21supervision and quality monitoring in.
46:26Almost 1900 clients and where we
46:29will also be assessing PHQ and GAD
46:32on a weekly basis to assess both
46:35is there an overall effect as well
46:37as what are the particular ways in
46:39which supervisors and clinicians
46:41use the technology that then lead
46:44to improved client outcomes.
46:45So there's a a mediational
46:47hypothesis hypothesis here as well.
46:53All right. I am just got a few more
46:55slides and before wrapping up and so
46:57want to just kind of talk about a couple
47:00things that are in the works right now.
47:05As probably everyone here knows,
47:07there is both a an epidemic of suicide and
47:13suicidality and that the federal government
47:16has been put in has put in place 988,
47:20which is a new crisis care call line.
47:23That is where the idea
47:25is that this mimics 911.
47:27It's a simple 3 digit number
47:30anywhere in the US that can be
47:33used to access crisis services.
47:35As part of this rollout,
47:37which SAMPSA is coordinating,
47:39they are mandating some quality assurance.
47:43I mean, obviously these are some of the most.
47:50Severe interactions within
47:52behavioral healthcare.
47:54As as part of the grant that
47:56we wrote on this topic,
47:57I learned that in crisis
48:00calls in 1% of the calls,
48:02a suicide is taking place during the call.
48:05So these are incredibly important
48:09conversations. And.
48:13But it we're in that same situation
48:15of how do you assess the quality
48:18of these crisis care calls?
48:20Well, the traditional method is you record
48:23them and then they are reviewed manually.
48:26And so we have been working with a partner
48:30to lay the lay the foundation for and
48:33we wrote a grant together to develop an
48:37AI assisted suicide risk assessment.
48:39So these are some of the dimensions.
48:43Of the quality assessment tool that um
48:46Samsa and their partners have put together
48:49and are the focus of our grant work.
48:52So we're we're partnering with protocol
48:55who is provides 988 services and
48:58is also one of the Samsung funded
49:01national backup Centers for 988.
49:04And so as part of our pilot
49:06work with protocol,
49:06we after getting all of the
49:10appropriate IRB in place,
49:12we took some of their existing crisis calls
49:16and put them through our current pipeline.
49:18And one of the things that we
49:21can do right now is identify when
49:23suicide conversations are occurring.
49:25And so as just sort of a proof of concept,
49:27we were able to demonstrate that we
49:30could differentiate both is there
49:31suicide content in this call or not,
49:34as well as how much?
49:35Of a focus,
49:36was it so that X axis is really
49:39kind of what proportion of the call
49:42was focused on suicide?
49:44And then also within that kind of
49:47transcript review phase of the technology,
49:50suicide is one of those items that
49:52gets tagged kind of session content
49:54that gets identified immediately.
49:56And so it's possible to scope into
50:00exactly where in the conversation
50:03suicide is being discussed.
50:04And so as part of what we hope for,
50:07we don't yet have the grant,
50:08but as what we hope for will be our,
50:10our next grant will be developing
50:12some AI technology.
50:14To automatically identify whether
50:16or not suicide risk assessment
50:18is occurring from the call taker
50:20and the quality of that.
50:24Last thing that that I'll mention,
50:26another kind of exciting thing that
50:29is kind of right in the middle of.
50:32You know one of the real world
50:35challenges is implementation and I
50:39think within a training setting it's
50:41you know easier to sell AI based
50:43feedback for learning new skills.
50:45But to you know kind of selling clinicians
50:48in the real world on quality assessment
50:51is a little less straightforward.
50:54And so as we have spent time with clinicians,
50:57I'm talking with them and user
51:00centered design interviews,
51:01you know one of the things that.
51:03We consistently heard from them
51:05was we hate documentation.
51:07If there's any way your technology
51:09could help us with documentation,
51:11we would be more excited about this.
51:14And so we have been in the process
51:18of doing an initial version 1.0 of
51:22a automated clinical documentation.
51:24In particular,
51:25we have done the same basic process
51:28that I described at the beginning,
51:30which is with a clinical partner
51:32we were able to.
51:33Get access to 40,000 clinical notes.
51:38And then we have been training AI,
51:40AI models that well,
51:41here's the recording of the session
51:44and then here is what the human
51:45said is a good summary of it.
51:47And then we can train the AI to
51:50learn that mapping of a recording
51:52to a session summary.
51:54And so this is the type of summary
51:58currently that the system can generate.
52:02So it is if you think about.
52:04Adapt note or a soap note.
52:07It provides that basic initial
52:10discussion summary of what
52:12actually occurred in the session.
52:14And we are still in the process
52:17of evaluating this internally
52:18and with some partners.
52:20But the goal here really is that
52:22it would be a tool that would help
52:25and support providers with not so
52:27much their clinical work per se,
52:29but necessary documentation that
52:31goes along with that clinical work.
52:35And there's some,
52:36some other features that we've
52:38built into it you can include.
52:41The kind of canned phrases that you
52:43might want to include with regularity.
52:47So let me let me wrap up there.
52:48There is a lot of work still to be done.
52:50We have a bunch of these technologies
52:53are currently getting assessed in RCT's.
52:56But it does feel like, you know,
52:59particularly with AI, there can be so,
53:02so much breathless excitement
53:04about AI and what I could do.
53:07And having spent the last
53:0915 years in the trenches,
53:11I feel like we're getting to the
53:13place where there are some practical
53:15tools and we can see some practical
53:17applications of how AI could really
53:20support behavioral healthcare.
53:23So let me wrap up there and happy
53:26to take any and all questions.
53:29Feel free to ask or if they're.
53:33In the chat, I'll try and open the chat.