“Lessons Learned from a Faculty Peer Observation of Teaching Program”- Heather Collette, MD, MHS - 3/20/23

May 23, 2023

Information

Pediatric Educational Learning Community (PELC): Heather Collette, MD, MHS - 3/20/23: Lessons Learned from a Faculty Peer Observation of Teaching Program

ID9955

To CiteDCA Citation Guide

00:00Heather doesn't need an introduction,
00:02but I always love and and plus I had
00:06the opportunity to look over your CD and
00:08your CD Part 2 a couple times lately.
00:12So. So I always love learning about,
00:16you know some of the really,
00:19really wonderful educators in our community.
00:22And so all of you know Heather,
00:25Heather Collette, AKA Dahlquist and so.
00:30Heather did her residency
00:32at University of Minnesota.
00:35I wish she was chief resident,
00:36which I should have known based on
00:38how comfortable you are with kind of
00:41like a B stuff and technical stuff,
00:43as well as your leadership skills.
00:46And then came here in about 2016 and really,
00:49really distinguished herself by,
00:52you know, by her first,
00:54her teaching skills and her
00:57commitment to education.
00:59She very quickly became the
01:03Director of the Pediatric resident
01:05Community Hospital rotation,
01:07followed by the pediatric sub internship,
01:12became one of our seals at Hospital
01:15Medicine and then Associate Fellowship
01:18Program Director in our newly
01:22established Hospital Medicine program.
01:25So.
01:25On top of that,
01:28she did complete her MHSMED and
01:31one of her areas that she had
01:35focused on was on initiating EPA
01:39assessment in the Subintern and
01:42that has actually become kind of a
01:47paradigm in UME and so you know,
01:51crossing potentially other departments.
01:56Heather has also emerged as
01:59an educator of our educators.
02:02She has certainly focused on
02:06increasing active teaching skills and
02:09then very quickly became our go to
02:13person leader in virtual teaching,
02:16which certainly coincided with the
02:18COVID pandemic when we all have to do it but.
02:22Heather walked many of us through
02:25it as well as really became an
02:28expert in the institution.
02:30And then over the last,
02:32let's see,
02:33probably about two years or so really
02:38focused on faculty development and
02:40became the director of the teaching,
02:44Optimization and Pediatrics are the top
02:46program, which was really her brainchild.
02:50And then.
02:51This last fall also started the J Fame
02:55course for our junior faculty members,
02:59which is based on the fellows
03:02as medical educators.
03:04And certainly as I said,
03:07Heather has really become
03:10our teachers teacher and so.
03:13Take it away,
03:14Heather.
03:14I couldn't be more proud of what you've
03:17achieved in the last couple of years.
03:19And as I said,
03:20I love reading it because it just reminds
03:22me of the really impressive impact
03:25that you've had in our department.
03:29Thank you, Panina. Well,
03:30you were there every step of the way,
03:32and I couldn't be here without you. So
03:34I really appreciate you too,
03:36Okay. Let me go ahead and share my screen.
03:43Okay. I'm going to be
03:44using men to meter today.
03:45So if any of you have ever
03:47been to one of my talks before,
03:48I love using mentee.com in order to,
03:52you know, be really engage
03:54learners and hybrid teaching.
03:55So if you could go ahead and use
03:57your phone to scan this QR code,
03:59I'll be having some question prompts
04:02throughout the presentation that
04:04you can answer on your phone.
04:05And once you're able to connect,
04:07if you could designate one
04:10of the little emojis.
04:11You can choose a thumbs up or
04:13a heart or a cat if you want,
04:17Just to let me know that you've
04:18been able to join the presentation.
04:20So I'll give you a second to do that
04:25right. And I'm also really
04:28excited to see that a lot of
04:29people in the group I know well,
04:30so it's fun to teach with friends
04:35and colleagues who I respect.
04:37Okay, Great. So my title today is
04:40Lessons Learned from a faculty Peer
04:42Observation of Teaching Program.
04:44And I just before I jump in,
04:46I know a lot of you guys know me,
04:48but Panina gave some of my background.
04:50But this happens to be me and
04:52my husband and our dog Henry.
04:54And we're getting a new puppy
04:56in a few weeks named Cooper.
04:58So this you'll probably be
05:00seeing him on a lot more of my
05:03presentation swimming forward.
05:04Okay objectives today I'm going
05:07to be identifying what existing
05:09resources we have at Yale to support
05:11faculty and effective teaching.
05:13I'll be talking a lot about
05:15teaching selfefficacy.
05:16It's been a particular new interest
05:18of mine and so I've been doing a
05:20lot of reading about it recently
05:21and I think it's really important
05:23in medical education to think about
05:25our educators sense of teaching
05:27selfefficacy and I'll have an
05:29opportunity at the end if you want
05:31to perform your own self analysis.
05:33And then my main point is going
05:35to be reviewing the outcomes and
05:36implications from this faculty peer
05:38observation of teaching program,
05:40the top program that Panina
05:43mentioned earlier.
05:44So before we start,
05:45I would like to know what learner
05:47audience do you typically teach and you
05:50can go ahead and answer on your phone,
05:52you can answer more than one as well,
06:01right? Looks like a little
06:04bit all across the board.
06:05I know a lot of you guys, so I know you
06:07teach a lot of different populations,
06:08but it's just helpful for me to know
06:11where you guys are at. OK, good.
06:14And then one other question.
06:16Have you ever had a peer provide formal
06:18feedback on your teaching? Yes or no?
06:24OK. Looks like a couple people
06:27said yes, one said no. OK, great.
06:35All right. So I'm going to talk a little
06:37bit about the background of clinical
06:39faculty teaching and how I got
06:41interested in this particular program.
06:43So with the current status
06:44is really that you know,
06:46we as clinical faculty faculty are
06:48expected to teach frequently with
06:51minimal education and medical school and
06:52residency on how to teach effectively.
06:55Now I want to give Penina a shout
06:57out because I think she's really
06:58tackled this problem so well within
07:00our pediatric department and now we
07:02have so many opportunities to work on.
07:05Our teaching skills and advance
07:07them and makes us feel more like
07:09better teachers where we have more
07:11opportunities to get skills on all
07:14these things and and so but one thing
07:16that we do is that we get feedback from
07:19learners but not a lot from peers or
07:22trained experts in medical education.
07:24So we're really relying primarily on our
07:27Medhub evaluations of our teaching to
07:29let us know if we're doing a good job.
07:31So some of the existing resources
07:33that I mentioned,
07:33obviously these pelk sessions,
07:35pediatric learning community
07:36sessions are great to discuss
07:38different educational topics on.
07:40The medical education discussion group in
07:43the School of Medicine is wonderful too.
07:45We have our medical education fellowship,
07:47the MHS program that I did,
07:49the top program now which is this
07:52peer observation of teaching.
07:53We have a new junior faculty as
07:55medical educators course and then
07:57the Poor View teaching center which
07:59Janet Haffler heads up is also.
08:01Has lots of resources online
08:02when it comes
08:03not the poor vu oh oh
08:07it's different,
08:08it's different, it's different.
08:09The teaching and learning said OK,
08:10sorry, thanks Janet.
08:13By the way, I was just writing
08:15in the chat not to interrupt
08:16is you gave a shout out to me.
08:18I have to give a shout out to Janet
08:20who oversees many of these programs,
08:22which was the inspiration for many
08:25of our programs and being so. Thanks
08:28Panina. Thank you.
08:30So lots of different resources
08:32when it comes to teaching skills.
08:35So I you know, when doing a little
08:36bit of a literature research,
08:38it's been documented that faculty
08:40teachers believe that student
08:43is feedback is useful in their
08:45clinical development but they can
08:47be uncertain about the honesty or
08:49accuracy of student assessment.
08:51And so I wanted to see if how
08:53much you guys actually agree
08:55with this particular statement.
08:57The one that was published was done
09:00in internal medicine group of faculty
09:02in these semistructured interviews.
09:04So I'm curious to understand how how
09:07much you agree with this statement.
09:18I know that sometimes as a a faculty,
09:21teacher and medical education,
09:22it's difficult for learners
09:24to give you, you know, true,
09:26honest or accurate feedback with
09:27because of the power dynamic that
09:29comes between you and the learner.
09:31So sometimes it can be.
09:34Difficult to get true, honest,
09:35or accurate assessments if there's
09:37any type of implication for
09:40that particular relationship.
09:41All right, so it seems like the majority
09:44of people strongly agree with this one.
09:47And then also in the literature
09:49there they say that faculty teachers
09:51prefer to receive feedback from a
09:53competent clinical teacher who's an
09:55expert in educational performance.
09:57So I'm curious to see if you guys
09:59have the same rating of agreement
10:00with this particular statement.
10:10Okay, great. So about the same,
10:14maybe even a little bit stronger than
10:16the previous statement. Okay, great.
10:18So it's been all documented.
10:20I mean, there's a the published article
10:23that I'm referencing here was from 1983,
10:25but there's been a lot of a lot of data
10:28showing that written teaching feedback.
10:29It increases clinical faculty
10:31teaching evaluation scores over time,
10:34their awareness of teaching behaviors,
10:36their willingness to be evaluated,
10:38it improves their objective teaching skills
10:40and it increases their passion for teaching.
10:43So getting feedback on your teaching
10:45is is really important for us as
10:48clinical faculty and right now we
10:50have this feedback spectrum where
10:51we have what the learners want to
10:54know and how they want to learn it.
10:56And then we have what the teacher
10:58thinks the learners need to know
11:00and how they want to teach it.
11:02And so we have these two ends of
11:04the spectrum and what what's missing
11:06in the middle is something to
11:08kind of join those two together.
11:10And so really it highlights the need for
11:12a multifaceted approach to assessment
11:14of our clinical teaching with an
11:16understanding of how student and feedback,
11:18faculty feedback interrelate
11:20to create the best.
11:22Teaching encounter.
11:23So the the thing that or the
11:27concept that I'm proposing you
11:28know with this top program is to
11:31have this trained peer faculty,
11:32observer to to who's just disconnected
11:35from the teaching encounter and
11:37is observing the learners and
11:40is observing the teacher and is
11:42working to you know bring both of
11:45these pieces of feedback together.
11:48So with a trained peer faculty observer,
11:50the the the benefits is that
11:52they're really not involved in
11:54the teaching encounter at all.
11:55There's no stake in learner
11:57or teacher success.
11:59They can provide some relatively unbiased
12:02feedback without any consequences or
12:05implications to the success of the
12:08learner or the success of the faculty.
12:11And so this is where the the
12:12top program comes in.
12:13So I'm going to go ahead and describe
12:16the program as a whole and go into
12:19some of the outcomes that we have.
12:21So the top program is a pure observation
12:24of teaching for pediatric faculty.
12:26At the beginning of the program,
12:27we recruited 6 faculty observers.
12:30And we were able to get access to
12:33student resident and fellow lectures
12:36at and at this time they were
12:38primarily happening via zoom and
12:39we're mostly hybrid lectures where
12:41some of the learners were in person
12:43and some of them were via Zoom.
12:45But all of the observations
12:47were occurring via Zoom.
12:48And this started back in July 2021 and
12:52ramped up over the the past year and a half.
12:56So with the program we trained
12:58the six observers and using this
13:00top observation tool.
13:01And here is where I will definitely
13:03give a shout out to Janet because
13:05she is the original creator of this
13:07particular tool and with her permission,
13:11Uma Fatak and one other colleague and I,
13:14we made a few changes to this to include
13:17another domain such as the use of media.
13:21And changing some of the wording around
13:24just to highlight our particular population.
13:26So as you can see here, there are 6
13:29domains of the top observation tool,
13:30including the learning environment,
13:33promoting learner engagement,
13:35managing the session content,
13:37verbal and nonverbal behavior,
13:40and then the use of media with several
13:43different types of behaviors under each.
13:45And then in the middle column,
13:46the observer can mark
13:49whether the behavior was.
13:51Exhibited, yes.
13:52It's somewhat or no or not applicable.
13:56They can fill in the comments on
13:57the right and then at the bottom
13:59they can enter one to three
14:01effective teaching skills and one
14:03to three areas for improvement.
14:04And we created this in Microsoft Word
14:07as a form so that they're able to just
14:10type in the responses and click on
14:12the boxes and it marks it with an X.
14:15So this is the tool that we used.
14:18It's not officially validated,
14:19although with this top program
14:21we're actually working on collecting
14:23some validity data for it and are
14:26working toward that, hopefully soon.
14:30The next thing that we did were
14:32was to observe the OR, sorry,
14:34train the observers.
14:35In the use of the ADAPT model for feedback.
14:38So I'm not sure if any of you
14:40have gone to any of Uma Fatak's
14:42lectures on giving feedback at,
14:44but I know she really likes
14:46this format and I do too.
14:48Subsequently,
14:48I think it works really well as a
14:51quick way to train faculty and how to
14:54give feedback by asking for feedback
14:57from the learner discussing it.
14:59You know,
15:00clarifying points and then
15:01planning next steps together.
15:02So the this type of training was given via a
15:07a a short video that the observers watched.
15:11So since the beginning of the
15:14program in 2021,
15:15we have 70 lectures observed in
15:17total with 50 faculty participants,
15:20with 20 faculty kind of completing
15:22the program,
15:23which means that they completed
15:272 observations.
15:28So the faculty themselves.
15:30They receive written and verbal feedback
15:32on two different teaching encounters,
15:35with the hope that they make changes
15:37on their second encounter based on the
15:39feedback they received from the first.
15:42We also had them fill out a
15:44qualtric survey of a pre and post
15:46teaching self efficacy survey score,
15:48which I'll share with you here.
15:51So this is an example of the PTSQ.
15:54We focused on the 1st 11 components
15:57of it because the last components
15:59involved working with patients.
16:02So we didn't feel like it was applicable,
16:04but the PTSQ is a validated instrument
16:07to assess physician's clinical teaching
16:10self efficacy and so it has 11 items.
16:14Where they list certain behaviors
16:17and the faculty can indicate to what
16:20degree the item applies to them.
16:22So you can get a score for each
16:24item from zero to four,
16:26zero where the item does not apply
16:28at all to them to four where the
16:31item fully applies and some of
16:33the items here you can see.
16:35I'll just list a couple examples because
16:36I know it might be a little small,
16:38but if it even if students
16:41ask difficult questions,
16:42I'm able to answer them correctly.
16:44Even if I'm under time strain,
16:46I'm able to concentrate and I'm able to
16:50integrate even the weakest of students.
16:52You know, even when I'm faced
16:54with big student groups,
16:55I can reach every student,
16:56things like that.
16:58So these are different items that
17:01would create a challenge or for for
17:04the teacher and so it reflects at to
17:07like to what degree they feel like.
17:09They can handle that particular
17:11challenge whether it's coming from
17:12within themselves or it's coming
17:14from the relationship between them
17:15and the student and we'll talk a
17:17little bit more about this shortly.
17:21So teaching self efficacy I thought
17:23was an important item to include
17:25in this particular study because
17:27not only do we want to increase
17:29objective teaching behaviors of.
17:31Of of our faculty.
17:32But we want them to feel like they're
17:34doing a good job and that they can
17:37do a good job when they're faced
17:39with different teaching tasks.
17:40So teaching self efficacy in
17:42the Med Ed realm is described as
17:45physician's belief in their capacity
17:47to carry out the actions necessary
17:50to provide high quality teaching.
17:52And it has been well documented that
17:54when teachers have a high self efficacy,
17:57it does influence their teaching performance
17:59and it has an effect on learner success,
18:02which is ultimately why we all teach.
18:05So our learners can succeed.
18:07And self efficacy has been
18:09shown to affect motivation,
18:10so it can affect the our ability
18:13to set goals for ourselves.
18:15It affects how much effort we expend it.
18:18It affects how long we can persevere
18:21in the face of difficulties and
18:23our resilience to failures.
18:25So you know, a lot of us are,
18:28we have,
18:29you know,
18:30stressful jobs working as clinicians
18:32and research and teaching and so
18:35when we're faced with time,
18:38time constraints or difficulties
18:40if we don't have a high sense
18:42of teaching self efficacy.
18:44We may not set the goals or expend that
18:47effort to teach the way that we are as
18:50much as we have a potential to do so.
18:53And like I mentioned,
18:55teaching self efficacy has a positive
18:57impact on students academic achievement,
18:59which again,
19:00is why we're all here,
19:03All right. So when we're talking
19:05about teaching self efficacy, oh,
19:07sorry, I'm just going to go here.
19:08We, we looked the main 2 outcomes really,
19:11of the program we're looking at.
19:13The change in their objective teaching
19:16scores as well as the change in
19:18their teaching self efficacy scores,
19:20again with the hopes that both would
19:23increase with the intervention
19:25that we're providing of this
19:27written and verbal feedback.
19:29So with the top score sheet here,
19:33there are 27 total points
19:36that are applicable.
19:38However, when we did score this,
19:40some of the items weren't,
19:42they were in the not applicable column.
19:44So what we ended up doing was scoring it as
19:47a percent of the total achievable points.
19:50So I will share here that we did
19:55have a significant increase in the
19:57average change in the top score from
20:0088 to 95% out of the achievable
20:03points on the top score.
20:05So an average change in.
20:08Seven points.
20:08So as you can see,
20:10a lot of our faculty are already
20:13pretty good teachers, which is great.
20:15A few outliers,
20:17which we can talk more about later.
20:18But one of the one of the themes
20:22that we really noticed with the top
20:25program was that people who were
20:27potentially already teaching at
20:29a high level were more likely to
20:32accept the invitation to be observed.
20:35Or potentially seek out the
20:38the opportunity to be observed.
20:39So we realized that with the top program,
20:43at least in the initial initial
20:44part of the program,
20:46we were probably reaching the
20:48higher level teachers already who
20:50were teaching a lot.
20:51They were giving a lot of clerkship lectures,
20:53residency lectures because they're
20:54known to be high quality teachers,
20:56so they're just teaching more frequently.
20:58We because of this,
21:00we did pivot a little bit to
21:02include more fellow lectures where
21:04more faculty are just assigned to
21:06teach as part of their job.
21:08They're not being asked specifically.
21:10So we're hopefully see some
21:13changes in that overtime.
21:15So we were happy to see that there was
21:17a significant change in the top score.
21:20After our intervention,
21:22quick question, Heather,
21:22for that one. It's the 20 faculty
21:25who had the pre and post, right?
21:28Yes, that's right, Okay.
21:30And did you go back just cuz I
21:33haven't seen the data and kind of
21:35look at the baseline for everybody
21:37now that you've had a chance to
21:39include some of the fellow lectures.
21:42I'm just curious.
21:43Like your hypothesis is that maybe
21:45now you're reaching some that
21:47weren't as better performing.
21:48And so like any sense of what
21:52the average is these days,
21:55yeah, it's probably more
21:56toward on the 80% side like 75 to 80%.
22:01Also it's you know although we
22:03had initially the six observers,
22:05we're having more observers now.
22:07So we're actually we're having
22:09a little more of a range and.
22:12On in scores as well.
22:13So hopefully it's a little bit more
22:14going to be more accurate instead
22:16of just doing all the observations.
22:17So yeah,
22:18OK, right. OK, thanks.
22:22OK. And then we did do some sub analysis
22:26initially looking at each domain to
22:29see if there were any particular one
22:30of these domains that had a more
22:33significant change than the others.
22:34And so we were able to find that three of
22:38the domains had a significant increase.
22:41And that would be in establishing
22:43the learning environment, promoting
22:45learner engagement and the use of media,
22:48which in my experience certainly reflects
22:51the areas where I I as an observer.
22:55I made a lot of comments,
22:57especially on using cases and different
22:59types of polling software to,
23:01you know, engage learners.
23:03And then also, you know,
23:05I think one thing I emphasized
23:08a lot as an observer was to.
23:11Spend more time and pay more attention
23:13to establishing a learning environment,
23:15especially when you're teaching
23:16in a virtual or hybrid session.
23:18And so I think a lot of the faculty
23:20took that to heart and spent more time,
23:23you know,
23:23establishing some type of a learning
23:25environment and getting to know
23:27their learners and their learner
23:28level before they jumped into their,
23:30their teaching content.
23:32So that was great.
23:36So with the PTSQ score.
23:39Or the teaching self efficacy.
23:42There's eleven items with
23:44a Max total points of 44.
23:47Again each item is ranked zero to four.
23:50So when we looked at this again,
23:53we have a small number.
23:55So we're hoping to get
23:56more significance later on.
23:57But we did see a change,
23:59average change in their self
24:02efficacy score from 27 to 30
24:04with a P value of point O 7.
24:06So we're approaching significance
24:08but not quite there yet.
24:1111 item that was really interesting.
24:13I think Pina you might remember
24:15initially when I was analyzing this
24:17data we were finding that their self
24:20efficacy scores were actually going
24:22down and so I did a little bit of of
24:24research into that and it's fairly,
24:27it's fairly well established that.
24:30You're especially when you started
24:32a high level of of self efficacy,
24:35you're more likely to go down after
24:38feedback because you're you kind of
24:40realized that maybe you weren't kind
24:42of as good as you thought you were,
24:44but that when after the
24:47implementation of feedback,
24:48your self advocacy actually goes
24:50even higher than it was before.
24:53So really the goal of this
24:55program is to have that feedback
24:57implemented with the goal of.
24:59Them,
25:00you know,
25:00faculty teachers feeling even more
25:02efficacious than they did previously.
25:07So what's interesting about this
25:10PTSQ score is that it's broken
25:12up into two different sections.
25:14The first six items are related to
25:18self regulation and the last five
25:20items are related to dyadic regulation.
25:23And I'm going to talk about that because
25:25I did not exactly know what that meant.
25:27So with the selfregulation components,
25:30these behaviors basically reflect
25:32cognitive or emotional threats
25:34to the teacher that require some
25:38type of internal selfregulation.
25:40So you know, it's about,
25:43you know, time management.
25:45It's about, you know, staying calm,
25:47managing your emotions even
25:49when they're triggered,
25:50things like that.
25:51So that's what those first six items
25:54reflect and then the last five items.
25:56Or the dyadic regulation,
25:58where these reflect challenges
26:00that arise between you as a teacher
26:02and the learner that requires some
26:04type of intervention that's really
26:06aimed more towards the learner.
26:07So it's about trying to reach
26:10every student in in your group,
26:13or adapting to different knowledge levels
26:17or potentially you know if you're given.
26:21An assignment to teach on short,
26:23short notice that these are kind of
26:25external things that require regulation,
26:27kind of outside outside of yourself.
26:31So we haven't,
26:32we don't have enough numbers to,
26:33you know,
26:34figure out whether there's a
26:36significant change more in one area
26:39or the other based on the program yet.
26:41But I think this is going to be really
26:44important when it comes to targeting
26:46interventions to support our teachers.
26:49You know whether it comes to teaching
26:51them more self regulation teaching
26:54techniques versus more learn learner
26:56targeted interventions to help improve
26:58their sense of teaching self efficacy.
27:00So more to come there.
27:04So really have some of the
27:04lessons that we learned from the faculty,
27:08teachers themselves where that,
27:09you know faculty really benefit and
27:12appreciate a teaching framework.
27:14I know a lot of us value certain teaching
27:17behaviors more than others, whether it's.
27:20Organization or just trying to think,
27:24you know, doing, making sure we have
27:25objectives and things like that.
27:27But when we can,
27:29when faculty see a list of,
27:31you know, expected behaviors,
27:33I think that really helps them touch
27:35on everything that's expected.
27:38Effective teachers, like I mentioned are,
27:39are more likely to seek and accept feedback.
27:42So certainly finding ways to
27:45reach all teachers is something
27:47that we still have to work on.
27:49Another thing that's really
27:50come up and Panina knows,
27:51I've been talking to the folks that meant
27:53to meter to try to get a a license for us.
27:56But more support in the use of
27:57technology and teaching is needed,
27:59especially in our virtual age now
28:03where we're teaching not only it via
28:06zoom but hybrid is really I think the
28:08number one mode of teaching at this point.
28:11And so trying to engage both,
28:14you know, in person and virtual
28:16learners is really difficult.
28:18And so technology is a way
28:20to bridge some of that gap.
28:21Formal feedback is appreciated.
28:24You know,
28:25I think a lot of the faculty,
28:26teachers, you know,
28:28aren't have never gotten feedback from,
28:30you know,
28:31a trained peer faculty and medical education.
28:33So I think that's really nice.
28:37And then I think like I mentioned before,
28:39that the teaching self advocacy can go down,
28:41but it's important to remember that it can
28:44go back up once the feedback is implemented.
28:48So some of the lessons that we
28:50learned from faculty observers,
28:51so again,
28:52these were the the trained faculty
28:54who were doing the observations was
28:57that observers again appreciate
28:58that rubric to follow.
29:00They benefit from that training
29:03and feedback delivery,
29:05especially when it's peer-to-peer.
29:07Certainly there's no sense of power dynamics.
29:10So you want to give feedback in a way
29:13that maintains that relationship.
29:16I I found that observers are
29:18really pleasantly surprised by what
29:20they learned from one another,
29:21not only from a medical education
29:24technique perspective, but also from content.
29:26I know a lot of faculty, you know,
29:30you know an endocrine like to
29:31go to a lecture on Kawasaki,
29:33and you know it reminds them of you know,
29:36of things that they haven't
29:37learned about in a while.
29:38So that's always kind of fun.
29:39And I think that once observers
29:43see teaching techniques.
29:44That are used successfully,
29:46they're more motivated to
29:47try them themselves.
29:48So whether it is using,
29:51you know,
29:52technology and teaching like pole
29:54everywhere or meant to meter or
29:57using cases or or doing chalk talks
29:59or are doing more hands on teaching,
30:01when they're seeing that you
30:03successfully then they'll be
30:04more motivated to try that too.
30:06And then I think that I think has
30:08been the most difficult is to make
30:09sure that these verbal feedback
30:11sessions are happening.
30:12I think it's certainly easy to fill out
30:14your teaching rubric and pass it along
30:16to the faculty and solicit questions.
30:19But I've found that these verbal
30:20sessions are really helpful just
30:22to review different comments.
30:23And it allows the faculty teacher
30:26to reflect on some of those items
30:28and come up with a, you know,
30:30a plan that they might try differently
30:32for the next teaching encounter.
30:36And then this is a little off topic,
30:38but it was another kind of side project
30:41I have here is looking at some of
30:42the the feedback from the faculty
30:44versus some of some student feedback.
30:47So I've been working with UMA and
30:49Michael comparing some of the Medhub
30:52feedback on clerkship lectures
30:54based on the the faculty feedback.
30:56And so I think I wanted to just highlight
30:59that the number one item that really
31:02overlapped between the two populations was.
31:05Learner engagement.
31:06So really making sure that whatever
31:10type of lecture you're giving to include
31:13not just interactive components,
31:16but activities that promote critical
31:19thinking within within your within
31:21your lecture.
31:22So it doesn't mean that people have
31:24to notice necessarily speak up,
31:26but just promoting that critical
31:28thinking is really important,
31:30another important item to note
31:32in this particular.
31:33Side study was that students focused
31:36a lot more on content of the lecture,
31:39so making sure that it was relevant
31:41to their level of learning,
31:43how much content was presented,
31:45whether it was presented in
31:47an organized fashion.
31:49And they really liked things that were
31:51interactive specifically with cases,
31:54whereas faculty feedback focused a
31:56little bit more on learning environment,
31:59how facile the use of media was,
32:02and PowerPoint.
32:02And the overall management of
32:04the session in terms of pace and
32:07organization and things like that.
32:08So those are just some initial
32:10reflections that I thought
32:11were interesting to include.
32:14So really when we're talking about
32:17implications from the program,
32:18we can think about how how our findings
32:22and and how this particular program
32:25and the tool itself could set teaching
32:28expectations potentially for new hires.
32:30Maybe some teaching
32:31expectations for promotion,
32:33especially with the new academic
32:35clinician track that's coming up soon
32:38and helping faculty who have areas for
32:41improvement and then also appreciation
32:43for faculty who are teaching well
32:46again to get that positive feedback to
32:48continue to have them teach over time.
32:51So hopefully I've highlighted,
32:52you know the need for this multifaceted
32:55approach to assessment of clinical teaching.
32:58Between students and faculty and I wanted
33:01to give you an opportunity to do your own.
33:05I'm teaching self efficacy analysis.
33:07It's a link to a qual trick survey
33:10where you can fill out eleven of the
33:12the components and you can put in your
33:14e-mail and I'll send you a little of your
33:17own analysis if you feel comfortable,
33:19and then we can reflect on
33:21that a little bit together.
33:23So I'm oh, I know I ended quite early,
33:26but I'm available for any questions
33:28or comments about the program
33:29or the study in general.
33:36I'm torn between my question or doing that,
33:42but I I was intrigued because I remember
33:46the earlier data that you had shown in
33:49terms of the PTSQ going down and so.
33:53How did you figure out if they
33:55had incorporated the feedback?
33:56Does it relate to what you said?
33:58Like if you wait longer for them to redo
34:02the survey, that's when it goes up?
34:05So how did you tease that out?
34:07Yeah, honestly, I was looking.
34:09I looked at the PTSQ scores and like who
34:12had the most change versus who didn't.
34:14And then I I I looked at their,
34:16their teaching observations and
34:18I was looking at the feedback
34:20that they received from the first
34:21one compared to the second one.
34:23And I I was just noticing that the the.
34:28Even even the faculty who had
34:30more than six months in between,
34:32like the longer almost the better.
34:34I don't know what it was maybe more
34:36time to reflect on the the feedback
34:38or more time to prepare like a
34:41different type of presentation.
34:42I think, you know,
34:44a lot of us have these canned
34:46lectures that we give all the time
34:48and it takes energy to change
34:49things whether it's the format or
34:52the interactivity that we choose.
34:53And so the the observations that
34:56were within a month or two I could.
34:58Not a lot of changes had been made
35:01probably from time constraints or motivation,
35:04but so yeah,
35:05the longer the better,
35:07which was really interesting.
35:08That's great.
35:09And Heather, cuz I was, I was,
35:11I was just so intrigued by that.
35:13And so in terms of the literature,
35:15cuz you kind of had talked about it,
35:17is there literature to show that
35:20once you've instituted changes or
35:22the duration of time in terms of
35:25the PTSQ you know or perceives?
35:28I always say perceived self advocacy,
35:30but it's really self advocacy is the term.
35:34You know, is there data to say that
35:37the longer you wait or the more things
35:39that you institute that it actually
35:42goes up or is that your hypothesis?
35:44I think that's my hypothesis.
35:45I don't have any data on the
35:49time frame, but you know,
35:51one thing that was interesting too was.
35:53That the folks who had a higher
35:56self efficacy, sense of self
35:58efficacy to begin with often had
36:00less of a change than those prior.
36:06Yeah, you can call on people to unmute
36:08them. I don't have to. But Janet,
36:11sure I have just a bit to add.
36:12Thanks a lot. We've really taken
36:14the program forward and what I did
36:17want everyone to know Zandi Garino,
36:20who's the associate Dean for PA education
36:22now her thesis when she was at Florida,
36:25I I had mentored her through that,
36:26but her whole thesis was on
36:29feedback and how you incorporate.
36:31Certain characteristics of
36:33feedback are incorporated or not,
36:36and it's a wonderful thesis.
36:38She then published it.
36:40Three years ago in Advances
36:42in Health Science Education,
36:44it's worth looking at.
36:45But it really it was a qualitative study,
36:48very well done and she really
36:51talks about learner readiness
36:53to accept the feedback.
36:54So again,
36:55it's it's pretty good,
36:56it's great work.
36:58Thanks Janet. I'm just curious from the
37:01group because I would know I was one
37:03of the first who volunteered to do it.
37:07Is there anybody here? Well,
37:09I don't want to put anybody on the spot,
37:10but reasons for not engaging in this and the
37:14reason why I ask is you wrote as a future,
37:17you know, as a future thought,
37:18which I think is a great one,
37:21which is new faculty,
37:23you know, to get them early
37:25and give them feedback early.
37:28But what would be the barriers
37:31to doing that and why don't some
37:34of our path like I I think?
37:36Most of the people that you reach out,
37:39if they answer you, they agree to do it.
37:41But ideas about those who are not
37:44getting the feedback at this point,
37:50and Heather, you can answer, I'm just,
37:52I was actually going to start with the
37:53group and say who hasn't done this,
37:55why would you not do this?
37:56But I decided I wasn't going
37:58to put anybody on the spot.
37:59So let's go more general as to
38:03reasons why faculty don't do this.
38:08Or don't engage in it, you know,
38:10it's interesting when it's structural
38:12Panina. And if it were just
38:14required that everyone would do it,
38:17because if it's very safe and you might
38:19not even want to use the word evaluation,
38:22I often call it that.
38:23I'll have a dialogue about
38:25what I just observed. So again,
38:27thinking of it being totally safe,
38:30knowing that it's not going to go into
38:32your portfolio or it won't get back to you,
38:34Panino or any other leader.
38:36So the safety issue I think
38:38is really critical.
38:40And also you know,
38:42being very careful saying if you
38:45wanted for it to be more formal
38:47as an observation that would
38:49actually go into their portfolio
38:51when they're coming up for a MP.
38:53So you could have sort of two arms of this.
38:56But again,
38:57really probably if you were saying
38:59University of British Columbia
39:01was one of the first places to
39:04implement the structure of people
39:06having a formal peer review process.
39:09Yeah, I actually. Right.
39:12And so I would, I would just see
39:16the benefit in having all comers.
39:19And since you see the other side, Janet,
39:21I know Heather and I talked about.
39:24You know, kind of how this could
39:26be incorporated into CV Part 2,
39:28especially when there's an improvement
39:30and kind of your thoughts about
39:33sitting at that table, you know,
39:35in terms of promotions and having
39:37this as a piece of somebody's success
39:40as a teacher and self improvement.
39:45So there's a question for you,
39:46Janet, about kind of the,
39:49the potential utility,
39:51I think ideally. We
39:53should be relying on peer
39:56observation and learners.
39:58So if you triangulate the data,
40:00because it's fabulous data,
40:02but we'd also have to be very careful.
40:04So when I do my observations,
40:06I'm very careful if someone's
40:08asking me to also write a letter
40:10to go into their folio versus the
40:13observation as a learning piece.
40:20And I think also what Franz Gellerno did
40:23in OBGYN, she observes all the faculty.
40:27We did a couple of years of training
40:29with her, and then she observes all her
40:31faculty and then writes a letter to them.
40:34And they were so grateful about that.
40:36So again, there are ways that we can
40:39really buff this up and peed so it,
40:41you know, because it's just such a great
40:43program that you're doing. Heather.
40:46We do went for the the 20 faculty
40:48that have completed the observation.
40:50We do send them a completion letter that
40:53it says you know thank you for completing
40:56this top program with two observations.
40:58You know in the state in this state
41:00your you know your teaching score
41:03increased this to this your your
41:05sense of selfefficacy change this
41:07way and then we highlight some of
41:10their their teaching strengths.
41:12And so it is signed by Panina and
41:15myself for them to, you know,
41:17include in their promotional
41:19materials if they desire.
41:23I've done the top program,
41:25so I'm not a person who didn't.
41:27But I will tell you,
41:28I think it's all social, right?
41:30So it's just this weird social
41:32barrier of signing up for something
41:34that you're like, what even is this?
41:36And is this for me?
41:38And by signing up, am I admitting
41:40that I think I'm a bad teacher?
41:42And so I suspect,
41:42and I don't know if this is true,
41:44but I suspect you probably got a lot of
41:45people in the first pass that are like,
41:47I feel okay about my teaching,
41:48but come on in, teach me more, right?
41:50Because you have to have a certain
41:52degree of selfconfidence to
41:54volunteer for this type of feedback.
41:56So if you're meeting people
41:57being a little timid,
41:59that's why I suspect it's a little like
42:01they don't want to be the the nerdy kid.
42:04Who signed up for the program too soon?
42:06But they also were just feeling nervous
42:08about what they might hear and whether
42:10or not they're going to find out what
42:11they thought they were pretty good at.
42:12Maybe they aren't so good at.
42:15That's just my impression of how residents
42:17interact with opportunities like this.
42:19That,
42:20I suspect is also true for junior faculty.
42:23Also, Heather, I think you know,
42:26one of the every time I'm asked
42:29to participate in something,
42:30I'm always like balancing out how
42:31much time is it going to take.
42:33You know,
42:34how is it going to interfere
42:36with my clinic schedule,
42:37all the other things I have going on?
42:39So I I feel like I don't want
42:41to commit to something and not
42:43have the time to really kind of,
42:46you know, foresee the end for it.
42:48So I think my hesitation is always like,
42:52do I have the time and the
42:53day to really get this done?
42:56Kind of, yeah.
42:57Ruchika. Oh, I was just going to say I think.
43:02The the other side of it,
43:03I had signed up and then I did not
43:05have an observer join the sessions.
43:08I'm sure they're more like
43:09happening at your end, Heather,
43:11and how you are kind of.
43:13It's more like a comment and a
43:15question because I I did volunteer,
43:17but I did have my observer
43:19come in for that session.
43:22Yeah, I think Heather,
43:22did you a pilot arm.
43:23So I know I had some feedback. So
43:27your lecture we're actually
43:28using to validate the tool. So
43:30we'll watch your thing over and
43:33over again. Got it. Don't worry.
43:37Yeah. Well, I haven't seen your
43:38name on any upcoming lectures
43:40that we'll get you in. Yeah,
43:42yeah. Yeah. No. So I just,
43:43I feel like there's also,
43:44I guess time constraints on the
43:47observers that are also offering
43:49to sign up for the program, so.
43:51Yeah, it's true because that, you know,
43:53I do receive some protected time for this,
43:55but I'm asking other to faculty to
43:57help me observed, which is, you know,
43:58a time constraint on them that they're
44:00not getting protected time for.
44:01So yeah, it's true.
44:06Yeah. And another thing, yeah,
44:07I think time is a major thing.
44:09You know, when faculty know
44:11that they're being observed,
44:12they really want to take the time to
44:14make this the best luxury that they can,
44:16which again, you know, is time consuming,
44:18you know, to revamp a lecture
44:20that you've already given.
44:21So I think, yeah,
44:23that being a little feeling vulnerable is,
44:26is one major issue and then the other is,
44:29is just time to prepare and time to
44:32participate in the program. I mean,
44:34I certainly try to make it clear that.
44:36You know, we're just there observing.
44:38We're going to send you this sheet.
44:39If you're open to 15 minutes
44:41of feedback via Zoom,
44:42then we do that.
44:43And and that's pretty much
44:45it in terms of your time,
44:47except for filling out those 2P TSQ
44:49surveys at the beginning and the end,
44:52which takes just a couple minutes.
44:53But I certainly recognize that this is
44:57a voluntary program that they don't need
44:59to participate in if they don't have to.
45:01So it's when you're busy.
45:04And it's not a priority,
45:05but probably won't be taken advantage of.
45:08So I have two followup questions for that.
45:10And that'll let everybody go because Janny,
45:13you actually raised something that
45:14I think is really interest, well,
45:16raised many things that are interesting,
45:18but that I was struck by and I'm not sure
45:24we deliberately address, which is 1.
45:27The idea that it's formative and
45:30not summative that it doesn't.
45:32That it doesn't necessarily go anywhere.
45:35And so Heather, do we or do you address that,
45:40I mean specifically to say that you know it,
45:44it won't go anywhere,
45:45it's just for your own personal,
45:49you know, kind of self efficacy.
45:52Yeah, I mean I don't think so.
45:56I I thought it was implied,
45:57but now that you mentioned it,
45:58yeah, now that it was a specific statement
46:00that. This is for your eyes only.
46:03Yeah. So that's a really good point.
46:06And then secondly,
46:07I'm just curious about the group. I mean,
46:09just in terms of your thoughts or feedback,
46:12if this actually just became
46:14standard for all junior faculty,
46:16Like if it was clear that this is formative,
46:20not summative, but it was expected
46:23of all junior faculty when they come,
46:27What do you think?
46:28The what would be your sense
46:30if you were a junior faculty?
46:32How would you perceive this,
46:34Nicole, if you're still on when
46:35you're a junior faculty,
46:37how you would perceive this?
46:41Any any thoughts?
46:45I just have one quick thought and that
46:47I'm I'm giving the fame presentation
46:49at once, so I'm going to sign off.
46:51I I think it's actually really
46:53appreciated if they know that
46:55it's formative and very safe.
46:57And I think you really have to be so
47:00careful with training your feedback givers.
47:04It takes such and to really have a
47:07dialogue versus I'm giving you feedback.
47:10So I would even say, you know,
47:11make sure that you're checking in on the
47:14people who are trained to do this because
47:16if it doesn't feel safe or helpful.
47:20So you know it's interesting where
47:22the three positive and and the three
47:25areas of improvement came from.
47:26Sometimes you just feel that you
47:29know you are giving too many sort of
47:33critical pieces again thinking about
47:36improvement and focusing on just maybe
47:39even one area that is correctable.
47:44So I think it'd be great we
47:46could try it. Yeah, thanks.
47:47I'm here and I I can add something and
47:50sorry Nicole and I have
47:51to run just because
47:54I'll see you at one again Janet, I'll be.
48:00So I was just going to add,
48:02I think it's something that's is
48:04very valuable and I actually had
48:06Heather give me feedback from for
48:09one of my presentation with the
48:12residents and it was very useful
48:14especially for my thesis like.
48:16I developed a curriculum.
48:18I implemented the curriculum,
48:19and as part of that evaluation,
48:21I need to know how well I'm doing.
48:25So it was useful and I really,
48:26really appreciate it.
48:27So, and I didn't even know about this.
48:30I forgot the four letters of the 11 question.
48:34Oh, PTSQ. Yeah. Yeah.
48:37The PTSQI think it's also a great idea.
48:39So I I'm going to do it, Heather.
48:42OK. Thank you.
48:46I was, I was gonna say if we just
48:48implemented it as a standard for
48:50every new faculty to come in,
48:52I think it should go well.
48:53I I don't see a reason why not.
48:56I do think it's a great program
48:59and at least for those who are
49:01really motivated to remain on this
49:03educational kind of track or are
49:05interested in medical education,
49:07I don't see why it they would
49:10not be willing to participate.
49:13Other than the other constraints of time.
49:15And yeah, of course.
49:16Right. Talk about. That's
49:19great. Any other thoughts?
49:23I think it'd be interesting
49:24to talk to not just about the junior faculty,
49:28but some of the faculty who have
49:31been teaching for a long time and
49:33maybe could use some updates.
49:34And in terms of incorporating more
49:37interactive components of their lectures too,
49:40but that could be a different discussion too,
49:42yeah. Don't totally agree, Heather.
49:46Great. OK. Well, thank you so much,
49:48Heather. Thank you for all you've
49:50done for our department and in
49:52terms of faculty development.
49:53And thanks for updating us
49:55about your terrific work.
49:56We really appreciate it. Thanks.
49:58I appreciate everyone's feedback. Thank you.
50:00Great. See everybody in a month.
50:03All right, take care. Bye, Bye.