PELC: "Questionnaires in Medical Research and Practice: Selecting Valid Instruments" with Marney White

January 03, 2024

Information

PELC 12.18.23: "Questionnaires in Medical Research and Practice: Selecting Valid Instruments" with Marney White

ID11145

To CiteDCA Citation Guide

00:00So you don't have to do that.
00:02I it's I'm fine introducing myself.
00:04It's like if you'd like to,
00:06but otherwise I certainly will do a
00:09little bit of a more formal intro.
00:14I've got a lot to get through in a brief
00:16time. People can go to my web page and
00:17probably get anything they need to know.
00:34I like your questions
00:45well as everybody's.
00:46Anybody who's just joining us,
00:49please feel free to start the
00:52survey that Marni with the QR code.
00:55I'm just going to welcome everybody.
00:57It's our last educational
00:59learning community for the year.
01:02We'll be starting fresh
01:04and strong in January.
01:06It is really my pleasure.
01:08And Marni, I'll only take 30 seconds
01:10since I know you have a lot to go
01:12through and I could not possibly get
01:14through your whole CV in 30 seconds.
01:17So I'm actually really thrilled
01:19to have Marni with us.
01:22When I asked the let's see,
01:25I guess it was the MHSMED,
01:27some of our faculty about who they
01:31would recommend as somebody to
01:33talk about surveys and the process
01:36of obtaining validity evidence
01:37and who might be a good teacher.
01:40And resoundingly, Marty Marty's name came up.
01:44And so,
01:45just to give you a little bit of background,
01:48Marty got her PhD in psychology at LSU,
01:51and then her Ms.
01:52and chronic disease epidemiology at
01:54the Yale School of Public Health.
01:56And then she was in the department
01:59of Psychiatry as a post doc
02:01on eating disorders research,
02:03where she did quite a bit of work.
02:05And then she gradually,
02:07well, not gradually,
02:09but became professor of psychiatry as
02:14well as social behavioural sciences
02:17at the Yale School of Public Health.
02:19She's director of online education
02:22and social and behavioral sciences,
02:24core faculty of the National
02:27Clinical Clinician Scholars Program,
02:29Track Director of Critical Topics
02:31and Public Health,
02:32which is the online executive MPH program.
02:35She's been teacher of the year at the
02:37Yale School of Public Health twice,
02:39multiple grants and, let's see,
02:41over 170 publications.
02:42But as I said,
02:44the most salient reason for the
02:46invitation was I asked who was
02:48the best teacher you've ever
02:49had add in this area.
02:51And as I said, Barney, your name came up.
02:53So thanks so much for joining us today.
02:56Thank you so much for having me.
02:57It was a really nice introduction.
02:59Thank you anyway. Thank you.
03:02I appreciate that. Very nice.
03:03So I do teach a course or recently retired
03:06a course actually at the Yale School
03:08of Public Health called Questionnaire
03:10Development and Psychometrics.
03:12And I'll be giving you all today a
03:15very crash course overview in that
03:17that I hope will be very relevant
03:19to your own research endeavours.
03:22I find that, yeah.
03:26So I I am by original training
03:27a clinical psychologist,
03:29secondary training and epidemiologist
03:30and the Yale School of Public Health.
03:32Several years ago,
03:33when I was still a junior faculty
03:35member in the Department of Psychiatry,
03:38I had been cross trained in public health
03:41and epidemiology as part of my key award.
03:43And after I finished that,
03:46YSPH asked me if I would develop
03:48a course on constructing valid
03:49surveys because it was a gap in
03:52the curriculum in that department.
03:54I did do so and then gradually
03:56started to teach more and then about
03:58five years ago shifted my primary
03:59appointment over to the School of
04:01Public Health where I now I'm on
04:03the educator track and I'm really
04:05enjoying the opportunity to teach
04:07scholars at YSPH and Yale College
04:10and the School of Medicine keeps
04:12me very entertained and and and
04:15engaged in a bunch of different
04:17topics and research endeavours
04:19with with academics from all over.
04:22It's pretty cool.
04:22But none of it was by design.
04:25It was just sort of the way this
04:27kind of path happened.
04:28But going back to the roots in psychology,
04:30what a lot of people don't know
04:32about psychologists is that before we
04:34got into this business of treating
04:36people and becoming clinicians,
04:39we were really about evaluation
04:41and assessment.
04:42And so a great deal of the
04:43training in psychology,
04:44even going back to my first
04:46master's degree in psychology,
04:48was around assessment and learning
04:50how to ask questions and try to
04:52and identify sources of of bias.
04:54And I think I've taken somewhere on
04:57the order of 10 or so graduate courses
05:01in assessment or psychometrics.
05:03That's really what we as the field do,
05:06do as our primary foundational
05:10knowledge base.
05:11So to be able to extend that into
05:14medical research and public health
05:16research is a pretty neat opportunity.
05:19But I I so here's what we're going
05:22to try to do today is to teach you
05:25what it means to evaluate self
05:28report questionnaires,
05:28primarily what their their main purposes are,
05:31identify them.
05:32Identify their strengths and weaknesses.
05:35Know what is meant by psychometric
05:38criteria to so that it's a very kind
05:41of cut and dry process and select
05:44the best measures for your research.
05:46Because we are pressed on time,
05:48I am not going to get much
05:50into questionnaire development,
05:51but rather best practices for
05:54using existing surveys.
05:55And the reason for this is because
05:58that is an entire other research area.
06:02It costs money,
06:03It is time consuming.
06:05You'll need a grant just
06:06or you know you'll need
06:08funding for sure,
06:09but you'll probably need about
06:10two years to start from scratch.
06:13So if there's something out there
06:15that you can use that is relevant,
06:18that is what I would highly recommend.
06:20Unless you're wanting psychometrics
06:21to be your research area,
06:23as it has been for me throughout my career,
06:26kind of by accident,
06:28and but it's it's what it is.
06:31So I want to teach you how to find
06:33these well established questionnaires
06:35that I mentioned here and then
06:38and the worst case scenario.
06:40If there's something out there
06:41that looks close to what you need,
06:43but it hasn't been done in your particular
06:46patient population or research focus,
06:47you might how to go about adapting
06:50that survey for use in your own work.
06:53So when we're talking about
06:55selecting A measurement instrument,
06:56and for the most part I'm going to be
06:58talking about self report questionnaires,
07:00the kinds of things that you might
07:01hand over to patients or colleagues
07:03or community members and say,
07:04what is your take on this?
07:06You know you're wanting a group of
07:08individuals to complete this measure.
07:10It could be a screening measure,
07:11it could be an assessment or
07:12a knowledge based survey.
07:13Many times we're actually talking
07:16about something that is unobservable.
07:18You know,
07:19we don't have a lab value
07:21to correspond with this.
07:23There might be some cases where
07:24that that that might happen.
07:26There might be cases where we'd
07:28have a lab value for a blood test,
07:30but it's very, very expensive and
07:31our screening tool which is self report,
07:34might be adequate.
07:35You know we might have a a high correlation
07:38at .8 or .9 between our self report
07:41measure whether it be pain impairment,
07:44subjective interpretation of symptoms,
07:47whatever.
07:47And it's going to be cheaper to ask
07:49people that than it will be to actually
07:52do a full physical examination.
07:54So we find ourselves kind of
07:55weaving the OR finding relevance of
07:57self report measures not only in
07:59research but in clinical practice.
08:01First thing we need to do when we're trying
08:02to figure this out is what is our objective,
08:04what is our research question?
08:05Trying to define that in what we call,
08:09you know, the, the,
08:11the strongest operational definition.
08:13And we're also going to talk about
08:14these things called constructs.
08:16Welcome back to psychology,
08:18these fuzzy things that we don't
08:21have a very clear cut observation,
08:23You know, it's not a cut and dry.
08:26Yes.
08:26No.
08:26But we aim to define it that way so
08:29that we can get consensus as much
08:32as possible when we come figure
08:34out what it is we're measuring.
08:36We're now trying to figure out
08:37if there's something out there
08:38that measures something close,
08:39what's been published.
08:40And I'll show you some tools
08:42for how to figure that out.
08:43Then if they're out there.
08:45Are the second metric properties
08:46of these established measures
08:47or these previously developed
08:49measures ideally published?
08:50Are the second metric
08:53properties good enough and
08:57are they established in your particular
08:59population, whether it be a patient
09:00group or it could be, you know,
09:02a group of physicians really depends on
09:04where your research is going to take you.
09:06So I talk about construct.
09:09We just mean a hypothetical,
09:11A hypothetical variable that
09:13is not directly observable.
09:15We're talking about pain.
09:17We're talking about anxiety.
09:22What are some things outside of
09:23psychology that you all might
09:25be interested in as physicians,
09:33you guys can yell out or put in the chat.
09:36Can this be like a bias?
09:38Yeah, it could. Absolutely.
09:40Do you want to say anything more about that?
09:42Like what you mean about bias? Sure.
09:46Like if you have a bias against or if
09:49you have a bias against like disability
09:52or or or I should say ability,
09:56you may provide, you may look
09:59at people and provide different
10:01counseling based on that. Implicit.
10:07Yeah. Construct, right? Absolutely. That's
10:10exactly it. Perfect.
10:11OK, y'all, Y'all are on the same page
10:12with me in terms of what I'm talking
10:14about with Construct. And by the way,
10:16there's Louisiana creeping in.
10:17Y'all happens from time to time.
10:20OK, so as I mentioned,
10:23I've I had to take so many courses in this.
10:26And the kind of standard thing when
10:28you're teaching about assessment or
10:30questionnaire development or psychometrics
10:32is to give people the task to get into
10:35groups and to define a construct and
10:37come up with a means to measure it.
10:39It's just an experiential learning exercise,
10:42and it's the exercise that I do in the
10:44course that Doctor Goldman has taken with me.
10:46Have students come up with something that
10:49we feel like we know what it is but probably
10:52don't all agree on the same definition?
10:55How are we going to define this and
10:59how importantly will we measure it?
11:01I've always been fascinated
11:02by senses of humor.
11:03I you know,
11:04I I was paired with a couple of of
11:06students and we did not share research
11:09interests or career interests And so I
11:11had to come up with something that we
11:13would all agree would be a worthwhile
11:15endeavour at least entertaining enough to
11:17to get through the semester long project.
11:20And I thought, you know,
11:20how do we define sense of humor?
11:22Let's see a chat thing.
11:23Here we go. Oh, there we go.
11:27Thanks.
11:27And, you know, humor,
11:30as it turns out,
11:31is quite psychologically relevant,
11:33Also medically relevant.
11:34I've I've learned in subsequent years
11:37and thought how how let's come up with a
11:41way to measure people's senses of humor.
11:44Simply OK.
11:45If we look at it at at what we
11:48call face value past people,
11:50do you think something's funny?
11:51Was basically the task we had
11:53to administer something quickly
11:55to a large group of people.
11:56This is before the days of online
11:58surveys or if they did exist,
12:00I didn't know how to use them yet.
12:02It was also the days and it's pre
12:05social media and but people did
12:07have e-mail and I don't know if
12:09anyone remembers when e-mail first
12:11became popular in the early 1990s
12:14but people would send ridiculous
12:16chain emails all the time of
12:18like lists of jokes and things.
12:20So I decided I would go ahead and get
12:22in on that mix and so I sent emails
12:25to everyone I knew and simply said
12:26give me a on a on a scale of 1 to 9.
12:30How funny do you think this joke is?
12:33And this is from my class project,
12:34please help me out.
12:36It was brief 1 liners.
12:37I could score them on what's called
12:40a Likert scale from 1 to 9 and a
12:43priori determined based on face
12:45value only and the convergence of
12:48researchers who determined whether
12:50or not each one liner fit into
12:54particular type of humor category.
12:56And the types of of humor categories
12:59we we we operationally defined.
13:01Again,
13:01it's like 3 sub constructs under
13:04the large construct, right.
13:05So we've got the large construct
13:07being being humor type of humor,
13:09appreciation of humor.
13:10But we saw that as being, you know,
13:12some people like slapstick,
13:13some people only like highbrow.
13:14You know what are these humor types.
13:17And we sort of did this in a way by,
13:19you know, finding a whole bunch
13:21of jokes and then classifying them
13:23according to kind of the types of
13:25comedians or what we thought would
13:27be corresponding with each type.
13:29So we've got witticisms,
13:30the Jerry Seinfeld type of humor,
13:33you know, these sort of clever
13:36little observations about the
13:38actually exposed into the microscope.
13:40This is kind of a little funny,
13:41isn't it?
13:42Then the dry Stephen Wright sort
13:46of humor may be a little bit
13:49dark and then the dark and and
13:53potentially in poor taste.
13:55And then the I I think of it as
13:57like the Lewis Black type of true
14:00but more mocking other people's
14:02shortcomings or seemingly subpar
14:05intelligence or kind of like everyday
14:08gaffes or something like that.
14:11Lewis Black.
14:12Yeah, OK,
14:13What we found highly,
14:15highly reliable subscales and and
14:18reliability refers to the extent
14:20to which items cling together that
14:22theoretically should cling together.
14:24In other words,
14:25if somebody rated one witticism
14:27very high as being humorous,
14:29they would be more likely to evaluate
14:32another witticism as being very humorous,
14:34And conversely, if not humorous,
14:36also not humorous.
14:37So it really is just it's almost like
14:40a large correlation coefficient where
14:42you're actually controlling for the
14:45number of items comprising scale so.
14:48But reliability is one of the first
14:51tenets that we look for in determining
14:54psychometric appropriateness.
14:55So you need reliability,
14:57and I'm going to get into
14:58reliability and validating what
15:00these mean a little bit more.
15:01But that was pretty cool that we found that,
15:04you know,
15:05that these did seem to the the items
15:08that we determined at the face value
15:10should relate to each other did in
15:13fact highly intercorrelate with one another,
15:16did not do what's called a factor analysis.
15:18That was just way too sophisticated for
15:19where I was at that level of training.
15:21And I didn't really have
15:22the table size for it.
15:23I'm sorry.
15:23I'm I'm saying I I I.
15:25Because honestly,
15:25I did all the work on this thing
15:27and then did get a great grade.
15:31Actually,
15:31that professor still remembers me and
15:32wrote to me some decades later because
15:34he very much thought that we should
15:36have tried to submit it for publication.
15:38But I was aiming to get into a doctoral
15:40program to specialize in eating disorders.
15:42And it just seemed I was worried
15:44at the time that it would be,
15:45you know,
15:46not that it wouldn't necessarily
15:48register me as the serious scholar
15:50that I was trying to become.
15:52Ironically, 30 years later,
15:54I'm still not so much.
15:55But you know, whatever,
15:57OK, why are we doing this?
15:59We,
16:01the,
16:01the are are going through
16:03this entire area of
16:04education, the process of defining
16:07constructs, coming upon a consensus
16:09of what of how we are all,
16:11as the researchers and our colleagues,
16:15going to agree on this
16:17operational definition.
16:19Developing questions and these
16:21questionnaires is actually pretty hard.
16:25Doctor Goldman can speak to this.
16:27It's a pretty complicated process.
16:29The numbers end up surprising you.
16:31The method of data collection
16:33ends up surprising you.
16:34And then unfortunately it just it
16:38takes a lot of time and energy.
16:40Subtle little details about the
16:43administration of your questions
16:45can have a significant impact
16:48on on the results of of what
16:50you're actually going to see.
16:51And I'm going to show you more
16:54on that on these subtleties,
16:57what I did in this questionnaire
16:59that you all just completed, right?
17:02Everybody had the QR code and you
17:03answered a couple of questions.
17:05What you didn't know is that I
17:07had embedded a randomizer at the
17:09beginning of the questionnaire so that
17:11people received slightly different
17:13questions in your quick little 6
17:16item survey or whatever it was.
17:18They were subtly different
17:20by just a few little pieces.
17:23So everyone should have seen this meme.
17:25Did everybody see this meme right and
17:28you were simply asked to evaluate?
17:30How funny do you think this is?
17:33Does anybody happen to know which
17:35one they received? Which version?
17:37Do you see the difference in the versions?
17:40OK,
17:45can anyone guess what might happen
17:49here if we were to, let's say,
17:51let's say let's score this at 123 and four,
17:55with four being, you know,
17:56people thought it was funnier.
17:58This let's not score at all
18:00because it's not applicable and
18:02we've still got one 2-3 and four.
18:06Does anybody have any hypotheses?
18:09So let's say for because these people
18:11didn't get the not applicable option,
18:13are people going to skip that question
18:15or what do you think they're going to do?
18:25Well, I could tell you what I did,
18:27which is I got the first one or the
18:30one that's on the left and I didn't
18:32get it and so I rated it as not funny.
18:35So my hypothesis would be that, you know,
18:39that the on the second question,
18:41you know, you would probably have
18:43whoever said not funny would be
18:45further speciated into those two.
18:49Yes, I agree. So the the thought is
18:52OK didn't get it. I'll explain it.
18:54It's a song called Africa by
18:56the band Toto And. Yeah, OK.
18:59So that was like a hit and early.
19:02Oh, now I get it. Oh, thank you,
19:04Marnie. All right. So,
19:08OK, so fun song lyrics,
19:11a whole lot of pop, cultural reference
19:12all mushed together, fun, you know,
19:15I'm amused by these kinds of things.
19:17So, so the hypothesis then being
19:22that's probably the means over here
19:24might be a little bit higher, right?
19:26Because anybody who didn't get
19:28it is just going to be exempted.
19:31And anybody who thinks it's funny or at
19:32least that that's that's my hypothesis.
19:34I thought, I think in terms of what we're at,
19:35we're at the way we're actually
19:37expecting the data to pan out.
19:39So we've got plenty of people thinking
19:41that not applicable, you know,
19:42they're like, I don't get it.
19:44And then some people like know, you know,
19:46very, very few people, whatever.
19:48Not very many people were as
19:49amused by this as I was.
19:51Let's put it that way.
19:53Now for the not applicable, when there's.
19:57When we don't have the not applicable,
19:58we've just got. This is not funny.
20:00Two, it is funny.
20:02We've got.
20:03Now this is consistent with your hypothesis,
20:06right?
20:06Like,
20:07because if we were to add
20:09the not applicable here,
20:11they'd show up here,
20:13right?
20:13Except
20:17what we've got here,
20:19and the way I'm interpreting this is when
20:21we have the not applicable option here.
20:24We should theoretically just
20:28see these two being equivalent.
20:30Instead, we've got a couple of
20:32different factors that might have
20:34influenced the pattern of responses.
20:36One is having that not applicable option
20:40might have told those people who thought
20:42it was a little bit amusing that it's
20:45actually a funnier than it is because oh,
20:48I'm in on this inside joke a little bit.
20:51Other people might not get it.
20:52That actually makes it a little bit funnier.
20:56Or it could be something as simple as
20:59the graphic of the five point scale.
21:02People do look at the number of
21:05response options, 4 versus 5,
21:07and we draw all kinds of inferences,
21:10especially when it comes to Likert scaling
21:12and a five point scale versus a four point.
21:14We look for middle a middle response option.
21:18Obviously the middle response
21:19option is the most popular.
21:21When we've got a 5 point scale,
21:26we see a little bit less of that
21:31when we've only got the four points.
21:36It's there are a lot of different poses.
21:38I don't know which one is correct.
21:39All I know is that including that
21:41non applicable option changes the
21:43pattern of results and this is
21:45highly significant if we actually
21:47put these into means to treat them
21:48not applicables as a missing value,
21:50it's highly significant like point OO1 and
21:52it happens over and over and over again.
21:55All right, shifting gears now,
21:59let's say we've got these.
22:03Now I kind of want to review
22:06some thoughts around real
22:08life research questions and
22:13how we're going to need to determine our
22:15measurement on both sides of the question.
22:17OK. So let's just say we've got
22:20a hypothetical research question
22:21about whether or not our policy or
22:24practice or procedure influences
22:26patient experience, right.
22:27We've got to now think about
22:31what about patient experience?
22:33Do we really care about?
22:34Are we looking at satisfaction?
22:35Are we looking at the competence
22:37of our of our medical staff?
22:39Are we looking at perceptions of
22:41care and nurturance that might be
22:42different from competence and so on.
22:44So there are a lot of different
22:46ways that we might aim to measure
22:48patient experience and there
22:52might be many existing measures
22:55to evaluate each of them.
22:59I don't this three shouldn't
23:01be there. That's an error.
23:05Once we figure out what it is we're
23:07exactly trying to focus on our measure,
23:08we next have to figure out whether or
23:10not there are questionnaires out there,
23:12measures out there that will do it.
23:13There are many different types of
23:15patient experience questionnaires.
23:16It developed in many different
23:19subfields or specializations,
23:20and it is up to you now to go
23:23searching and finding them.
23:25Here's how to do that.
23:27There's actually something called the Health
23:31and Psychosocial Instruments Database.
23:34GAIL has it. It's.
23:35I don't know how to pronounce it.
23:37Happy, I guess.
23:39Health and psychosocial instruments.
23:41Ask your medical librarian.
23:44They're remarkably helpful.
23:46You can also try cited reference searching
23:48and scope as your web of science.
23:50I prefer Google Scholar, even though our
23:52public health librarian doesn't like it.
23:54I'm just.
23:55But anyway,
23:55I feel bad kind of recommending it,
23:59given that librarians have
24:02identified many flaws with it.
24:04This is an incredibly helpful
24:06slide and resource. Go to Ovid.
24:10Psych Test is another option.
24:12Psych Info is another option,
24:14but Happy is really going
24:16to pull your info for you.
24:18The next question are the psychometric
24:21properties of these scales adequate?
24:23All right, now a little crash course
24:25in what psychometric properties are.
24:27Again, you first want to establish
24:30that your properties are
24:35reliable, then valid,
24:39and if you are looking at subscales,
24:41you also need to gauge the
24:44liability and validity of those.
24:51I'm sorry, I'm a little
24:52distracted because I'm.
24:52I'm concerned that I'm showing you
24:54the wrong slide show because I've
24:56got three slides open right now.
24:58Now This is correct.
24:59All right, this is fine.
25:04I I described inter item reliability
25:06a little bit a few slides back.
25:08Inter item liability is almost like a large
25:14correlation coefficient of all of the
25:18items comprising a particular scale.
25:20The by and large,
25:22the more items you have on a scale,
25:26the higher your reliability will be.
25:29But that doesn't mean that's a good thing
25:32because it does control somewhat for the N,
25:36the N being the number of items.
25:37You can artificially drive up an
25:40inter item reliability coefficient
25:42which is called Chromebox Alpha in
25:44most cases just by having extra and
25:46and potentially unnecessary items.
25:48What you always want.
25:49And so sometimes,
25:50like pseudo scientific jargon will say,
25:53oh, our scale is so much better because
25:55we have a coefficient alpha of .93,
25:57and the gold standard that's
25:58been used before this only has
26:00a coefficient alpha of .89.
26:02No, no, no, no,
26:05that's not really that impressive of a leap,
26:08especially if somebody's asking
26:10or measuring something with 35
26:13questions and somebody else can
26:15get it done in six and still
26:17have a good coefficient alpha.
26:19That's the one you want to choose.
26:23So internal consistency is the extent
26:25to which those items interrelate.
26:28You. Also that once you've established
26:31reliability, you can then talk about
26:33various types of validity content,
26:35validity and criteria and validity.
26:37And when you have all of these together,
26:39now you've got evidence of construct
26:43validity, reliability measures.
26:44There are particular different kinds.
26:46There's inter rater reliability which is
26:52if you have you know multiple individuals,
26:56for example evaluating a particular
26:59stimulus or interview or diagnosis.
27:02That's a little bit less relevant
27:05to scale development but you'll
27:07see it in the literature.
27:09Test, retest,
27:10reliability which you generally
27:12want to establish if possible.
27:14That's where when you're when
27:17you're looking at the properties
27:19of your measure you then want to
27:22re administer it to at least a sub
27:25sample of your population and look
27:27at the correlation looked at at how
27:30well these measures align but with
27:33each other from time 1 to time 2.
27:34Now again this is a,
27:35this is really in reference to
27:38questionnaire development and construction.
27:40But when you are choosing your
27:43measures you want those that have
27:45appropriate inter item reliability and
27:47test retest reliability that should
27:49be established if it's going to be
27:52a solid self report questionnaire
27:53that's used out there in the field.
27:55And these you know screen share
27:57this or not screen share just screen
27:59grab this particular slide.
28:00Because these are just the the
28:03reliability coefficients that you'll be
28:05using either alpha split half or ICC,
28:07which is a interclass correlation
28:10coefficient if you're using,
28:11you know, non continuous data.
28:15If you're looking at categorical
28:17outcomes or categorical decisions,
28:18then you might be looking
28:20at what's called the KR 20.
28:22These are just different statistics
28:23that ultimately are going to
28:25tell you the same thing.
28:26And then I also just want to
28:28give you some of the guidelines
28:31for what is considered adequate.
28:33Anything above .7 is going to look
28:35pretty good, especially again,
28:37sometimes you'll see like a three or four
28:40item scale with a .7 reliability coefficient.
28:43That's excellent that I would
28:45get very excited about that.
28:47Of course,
28:48you know you always want to see
28:50something in the point nines,
28:51but you're always wanting to
28:54balance against participant burden.
28:56You'd much rather have full and complete
28:58data than highly reliable data where
29:0120% of your respondents have dropped
29:02off by the end of the of the study.
29:07Validity. OK, I've spoken about
29:08reliability and and you almost
29:10think of reliability as being like
29:12the the likelihood that you can
29:14get the same response every time.
29:16That's the precision.
29:17Validity is whether or not it's true.
29:20So in order for something to be valid,
29:22it must first be established to be reliable.
29:26You can have something be very,
29:27very reliable, but wrong, right?
29:32So the bathroom scale can give
29:35you the same answer every time,
29:37but it might be 10 lbs off, and
29:41it's going to be 10 lbs off for every
29:44single participant that gets on it.
29:45So it's reliable. That reliably gives you
29:48consistent measures, but it's not valid.
29:54And valid validity is established by a
29:57bunch of different types of criteria,
29:59so there are different kinds.
30:00Face validity can generally
30:02get from the research group,
30:03or from focus groups,
30:05or from the Delphi panel.
30:07This is whether or not your measure
30:09is intending what or is measuring.
30:11The it's it's how obvious
30:15it is really does it?
30:17Is it measuring what it
30:18seems like it's measuring?
30:21Does anybody have any ideas of what
30:23this questionnaire is measuring?
30:28It's question one of a of a
30:30widely used validated measure.
30:31What do you think it's measuring?
30:45Any ideas?
30:51Don't be shy, guys. Chat or open. Or unmute.
30:57Yeah, seriously, Jordan, Yell.
30:58I'd like to be interrupted. I'd
31:02like to converse.
31:02I don't like to talk at all.
31:07Right. What do we have?
31:08We've got. This is depression.
31:10Yes. Excellent.
31:11This is the first question of
31:13the Beck Depression Inventory.
31:15Widely used. Excellent.
31:16So face valid? Yes, pretty obvious.
31:20Perhaps even to non specialists,
31:23even to patients.
31:24What this is probably measuring,
31:26right don't need a whole lot of of
31:28education and depression to guess.
31:30This is probably what it's about.
31:31It's at least measuring sadness.
31:36Does anybody know what
31:37this might be measuring?
31:38These are three items on the
31:41same subscale of another widely
31:44used psychiatric instrument.
32:00Any ideas,
32:07Ruchika, You unmuted. There we go.
32:10That's great. Oh, I'm sorry.
32:12I didn't. I didn't realize. Unmuted.
32:14But I was thinking about anxiety.
32:16Good, good, good. Good idea.
32:17So anxiety and depression.
32:19Or anxiety. Or depression.
32:20Maybe a mix of anxiety,
32:22depression or anxiety.
32:23So that right there,
32:26given that we have different ideas
32:30as to what this could be measuring,
32:32suggests that it's maybe not as face valid.
32:35And in fact I would not consider
32:37this to be a face valid measure.
32:39This is the depression scale of the MNPI,
32:42the Minnesota Multi Basic
32:44Personality Inventory.
32:45This measure was what's called
32:47empirically keyed, meaning,
32:49it is not that these items are not
32:51put together based on their value or
32:54their obvious correspondence with the
32:56construct that they're trying to measure.
32:59Rather,
33:00they individuals were classified
33:01according to the construct and then
33:04based on how they responded to these
33:06individual measures or the individual items,
33:08those items then mapped on to the
33:11creation of the scale as it happens.
33:14Appetite disturbance,
33:15sleep disturbance and mood of course
33:17are all prongs that relate to a
33:20clinical diagnosis of depression.
33:22However, not obviously so to everybody,
33:26right? Not face valid one more.
33:32Does anybody have any ideas
33:34what this might be measuring?
33:42Again, 4 items on the same subscale?
34:01Great idea. The question mark already
34:04tells me it's not face valid.
34:12This is the MMPIK scale,
34:15which is actually a correction scale and
34:18what it measures is social desirability.
34:22And when people score very highly on this
34:26scale, it invalidates their responses on the
34:29rest of the of the Large Assessment Battery.
34:33What this? At times I feel like,
34:35and this is all true. False, right?
34:37At times I feel like swearing false.
34:40Come on, criticism or
34:43scolding hurts me terribly.
34:44False. No, I'm good with it.
34:46It really helps me become a better person,
34:48you know, Come on, this is off.
34:49You know, if if people are saying false,
34:52false, false, they're
34:56trying to present themselves
34:57in a better light.
34:58What that means is that their
34:59response is on our side,
35:00something that's very not face valid,
35:04but was able to be determined again
35:06through that empirical keying,
35:08which I think is just fascinating.
35:12Other types of validity criterion, validity.
35:16This is really the meat of a
35:18lot of what we want to do.
35:19We're not usually just going out there
35:21trying to find a measure to measure
35:23something because of the sake of we
35:25want to make sure that we can truly,
35:27you know, know the status of truth
35:29of this particular construct.
35:31We're actually trying to relate
35:33to something meaningful,
35:34which means we want our scores
35:36on our measures to actually tell
35:38us something on down the line,
35:40whether it maps onto a particular
35:43diagnosis or, you know,
35:45recovery button, whatever.
35:47That would be concurrent validity if we're
35:50looking for our measure to relate to a
35:54diagnosis or some other gold standard.
35:56If we're looking for our screening tool
35:59to map onto an actual lengthier battery
36:06or something called predictive living,
36:08which is simply whether or not accurate
36:10knowledge of our questionnaire predicts
36:13something meaningful in the future,
36:15I love the old MCAT score and
36:18performance in medical school.
36:20Predictive validity?
36:22No, not a lot.
36:25We certainly know that's the case
36:27in the PhD sciences that GRE scores
36:29have like I believe it's about,
36:31I think it's at point O2 correlation
36:37coefficient with dissertation
36:38quality of resource, productivity.
36:40So of course we've got other issues
36:42like restriction of range and so on.
36:44But ultimately,
36:45if we're going to go through all the
36:47process of administering a measure,
36:49we want it to relate to something of value,
36:51something meaningful.
36:55These are the measures appropriate
36:57for your target population.
36:58And this is what I had mentioned
37:00previously about, you know,
37:01you might have something that's been
37:04validated for use in adults and
37:07it's not necessarily as translated
37:09to use in adolescence or children.
37:11And so you just want to make sure that
37:13if there's an existing construct out
37:15there that it has been validated for use
37:17in your particular population or that
37:19the population that you're studying.
37:22Hopefully that is the case.
37:23If not, you might have yourself
37:25a revalidation study,
37:28advice, advice, advice, advice.
37:29Once you've picked all of these things,
37:32I want to take you through how to
37:35avoid shortcomings in administration.
37:37And I've done most of these blunders
37:43myself over the past decades.
37:45I've done something like, you know,
37:48it's it's the best way to learn
37:49and it makes you meticulous.
37:50But I I, I collected thousands
37:54of participants of data only to
37:56discover after the fact that I had
37:59somehow left education out of the
38:01out of my demographic battery.
38:03So I had a really flimsy measure of socio
38:07economic status and educational attainment.
38:10And I mean it's just it's sickening,
38:13but it happens anyway.
38:14I want to to draw your attention to some
38:18kind of finely tuned things and and to
38:20help you kind of map out your research.
38:22Think of it,
38:23while you're setting up your questionnaires
38:25in Qualtrics or in Redcap or whatever,
38:28think forward about how you're
38:29going to analyse your data.
38:34I would caution you against using many
38:36open-ended questions if at all possible.
38:39If you do get a qualitative
38:42researcher on board early on,
38:44think about the scale of measurement.
38:46Are you going to be using True,
38:47false, Yes No, Yes, No,
38:50Maybe Yes No does not apply.
38:53Think about whether you're using
38:55multiple choice, rank ordering,
38:58ordinal data, or the good old fashioned
39:01Likert scale continuous data.
39:02I love the Likert scale.
39:05I don't know why.
39:06I find that it allows me
39:09to do a lot more data wise.
39:12That's just my preference and it's almost
39:16like why do I like cats as much as I do?
39:18I can't really explain it, I just know
39:20it's the case open-ended questions.
39:23I would just caution you against
39:26these or use them very sparingly.
39:29They might be appropriate for some things,
39:31especially some knowledge based things.
39:33Some things like you know,
39:34birth year, whatever else you know,
39:38idea of exposure.
39:39You know you might be forced to use
39:41them in particular circumstances,
39:43but the way you're going to be
39:47analyzing the information will
39:49dictate how you'll use them.
39:52If you are using multiple open-ended
39:55questions to ask opinion or experience,
39:58you're definitely going to need
39:59that qualitative researcher,
40:00because you're going to need to
40:02understand how to code open text data.
40:05I don't know how to do it personally.
40:08Some people,
40:09when they're wanting to do
40:10mixed methods research,
40:11they need to get that qualitative person.
40:13I can help on the quantitative
40:15side with questionnaire stuff.
40:16I can't.
40:16Once we start getting into
40:18analyzing open text stuff,
40:19I don't know how to do it that
40:22that's an entirely other parallel
40:25type of research approach.
40:28One option,
40:29if you feel that you must include
40:33open-ended variables is to do
40:35kind of a hybrid approach.
40:36So these are actually questions from
40:38the Food Addictions questionnaire,
40:40the Yale Food Addiction Questionnaire,
40:42where they ask people to indicate
40:44any specific foods that they in which
40:47they experience an addictive like
40:53addictive like qualities when
40:54eating the foods and people feel,
40:56you know out of control or that
40:58they they can't get enough of them.
40:59And so they they ask these very
41:02precise food items and then say, oh,
41:04and are there any others that that we missed.
41:07So that's an option as well seeing is
41:09there a question coming through. OK,
41:15a problem with open-ended,
41:17you run the risk of getting answers that
41:21are different from what you had intended.
41:25So you need to always make sure that
41:29you're providing really good instruction
41:32sets and direction for participants.
41:34Fixed response options when you're
41:36talking about fixed response options,
41:37which I always prefer.
41:38But again, that's just because they they so
41:41nicely lend themselves to data analysis,
41:43to empirical analysis.
41:44But you do have options within that,
41:47so the Likert scale,
41:49which I do love,
41:51can be a really nice one in terms of
41:54using what's called a visual analogue.
41:57Just providing people with two anchors.
41:58Now Qualtrics and Redcap will allow
42:01people to use a draggable scale and
42:03actually get it a range from zero to 10
42:07without having numbers involved at all.
42:09Something you need to consider when
42:11you are administering Likert scales.
42:16You want to make sure that you are
42:20following the response format of the
42:22scale as it was originally validated.
42:25If you're creating your own,
42:26you have to consider whether or
42:27not you're using four points or
42:29five points or seven points or 9,
42:31whether you're going to give
42:32them a middle response option,
42:33and whether or not you're going to
42:35label all those response options.
42:39When you're formatting these things,
42:41you want everything to
42:42be as clear as possible.
42:44Use white space A lot.
42:46Use page breaks a lot. Yeah.
42:50It's probably pretty unlikely
42:52that you're going to be doing many
42:54paper questionnaire administrations
42:55at this point in time. All my,
42:57I can't remember the last time I saw one.
42:59If you're using paper basics are, you know,
43:02only use the front side of the paper.
43:04It's wasteful it,
43:05it kind of kills us in this day of,
43:08you know, trying to be
43:12responsible environmentally.
43:13But people will skip that
43:16second page. They just will.
43:18They don't flip over the pages
43:23when I say code responses.
43:27This has actually been done and I see it a
43:32lot, this whole thing of making people hold
43:35in their head what is the strongly agree,
43:38what is strongly disagree and then undecided.
43:41Now we've got a neon people are having
43:43to like kind of keep on scrolling back
43:45to make sure that they're understanding
43:47you know what each column heading is.
43:50This is also problematic because that
43:53you now looks like a fifth point here,
43:56and so people are assuming that
43:58disagree is a middle point when in
44:00fact this entire column is graphically
44:05telling too much information.
44:08Another bad idea is making
44:10people code and put in the box.
44:13You want to reduce as much burden
44:15as possible from your participant,
44:17from your participants.
44:17That's both in terms of questionnaire length,
44:20but in also what they need to do.
44:25This is from the BDI.
44:27This is the way it's actually
44:29formally administered in the if you,
44:30you know go on to to the psych measures
44:33thing and order a packet of 100 BD is
44:35this is what it's going to look like?
44:37But all the time you see
44:41people put fours in the box.
44:44I think that we can safely
44:46interpret that to mean a three.
44:49But now what do you do for the next question?
44:50When they put a two in the box?
44:55I don't know.
44:56You have to consider that invalid.
44:57So instead, there's nothing wrong with
44:59just asking people to check the box,
45:02to check which one's beside there
45:03to click the particular button,
45:04as opposed to making them do all that work.
45:11Include instructions, but assume
45:12people are not going to read them.
45:15When people do read them,
45:16make sure they're appropriate.
45:18People get very annoyed when
45:20you say check the box and it's
45:22actually circle the number.
45:24Just keep instructions consistent.
45:27Also consider your formatting.
45:32This is so strange. I had a whole animation
45:35for this page. I don't know what's going on.
45:37Anyway, this, I just this is,
45:40this is a real life example from last week
45:43this everything looked great on the desktop
45:45and then I go to check it with my phone.
45:48No one's going to be able this is,
45:50this is so much work for
45:51the participant to try to.
45:52What does that even say?
45:55Constant, constant concerns and no concerns.
46:01This is terrible, right?
46:05It almost flew again,
46:06that comment about the paper.
46:08People aren't going to flip over the
46:13page. Oh, there's my animation. All right.
46:15Check your format every single time.
46:17Pilot it. Make your children take it.
46:20Make your colleagues take it.
46:22When you're or,
46:23think about the ordering of the of the items.
46:25This is a real example as well.
46:29I say use page breaks because
46:31these two were on the same page.
46:34Someone's following along.
46:35What I look like is important.
46:37Yes, I agree. Yes, I agree.
46:39You know, I don't know if I agree that much.
46:41I prefer not to send it.
46:43All of a sudden the response
46:45options have flipped on them.
46:47OK, you can avoid this the the
46:51errors that this will impose,
46:52and it will impose errors because people
46:55are they've gotten into their rhythm.
46:57They're using what's called a
46:59response set now, and to avoid these,
47:03so I'm saying, group themes together.
47:06You know,
47:06if you're asking about particular constructs,
47:07you're administering multiple questionnaires.
47:09Keep them that way.
47:11If you're using a validated questionnaire,
47:13do not shuffle the order of items.
47:17Keep the items administered
47:19in the way in which they are
47:22originally presented and validated,
47:25and then be sure to use page breaks.
47:27Page breaks will allow people to kind of
47:30reset for these flipping around changes
47:33in instructions or themes as well.
47:39When you're looking at at at
47:42question quality, always balance.
47:45You know your reliability and your
47:48validity against the burden on the
47:51participant, people will drop off.
47:56This is all alluding to the
48:01the types of reliability
48:03and the types of validity.
48:04But when you're looking at the
48:05at which measures you're going to
48:07choose and you're likely to find
48:08several that would that would relate.
48:10It's always just kind of a a balance
48:13against participants and and how well
48:15those psychometrics qualities look.
48:18Also, things to kind of ask yourself
48:21or have going in the background more.
48:23The lengthier, the more tedious,
48:25the more cumbersome,
48:26the more confusing.
48:27People are just going to disengage.
48:32I try to remove as many numbers
48:36and instructions as possible.
48:40People don't like numbers.
48:41It's heartbreaking, I know.
48:43All right, tell me some questions.
48:44Some problems with this question.
48:55I feel like some people might
48:56be overly optimistic.
48:59Yes, Assumes consistency.
49:02Assumes exercise.
49:03Assumes people understand what
49:05is meant by exercise. Excellent.
49:08You've got all the main the main things.
49:10Good, good, good.
49:12What does it mean? Guess what?
49:14I asked you all this question,
49:17except I had my little randomizer again.
49:20And what did I ask you?
49:22I asked about typical and I
49:24asked about just this past week
49:31time and time again. This is our pattern.
49:37I never understand what this is,
49:39but that also happens all the time.
49:41But people's the idea of their typical
49:45exercise is often a lot better.
49:48It's not often.
49:49It's again an AP value of like less than
49:52point OO one is better than their actual.
49:55Well, you know, I mean it was cold and rainy.
49:58There was that major storm that came through.
50:00It's the holidays.
50:01I've got to do so much shopping
50:03doesn't matter if I'm asking this
50:05in perfect spring weather in August,
50:07it doesn't matter.
50:07The same phenomenon occurs and we
50:09know this everybody wants to that
50:12social desirability thing of course,
50:13but we generally believe that our
50:15that our typical is a little bit
50:18better than our actual that causes
50:20difference in what we actually observe.
50:24Same thing with the frequency
50:27of our response options.
50:29OK.
50:30So asking on what's called a
50:32high frequency scale versus a low
50:34frequency scale contributes to a
50:36different pattern of responding.
50:38People are drawing inferences about
50:41these anchors and what they actually mean.
50:43This was done in a pain study
50:45or in a pain clinic,
50:47and it's likely that those who had the
50:50high frequency scale interpreted that
50:53to mean like a lower level of pain,
50:56like the meaning of what, you know,
50:58the daily headache or aches and
51:00pains or joint pain or whatever,
51:02versus the debilitating migraine
51:06in bed for the entire day,
51:08right.
51:09So people are drawing things
51:10based on the options you give
51:12them or drawing inferences.
51:16Same thing with something a little bit less,
51:20a little less objective, you know.
51:21So we think about physical pain as
51:23being probably about a, you know,
51:25you've got something that you feel
51:27acutely versus something a little
51:29bit more fuzzy and psychological.
51:31Yet we see here that providing people with
51:36numbers changed the pattern of responding.
51:39This is why I've gotten to the
51:40point that I try to remove all
51:42numbers are my questionnaires.
51:44Unless it's something that's been validly
51:46established as requiring the number.
51:48I asked you all the same thing.
51:50Do you like New Haven
51:549 point scale ranging from
51:57either 1:00 to 9:00 or -4 to 4?
52:00And here's what we see.
52:02It's just wild to me,
52:04that something that subtle.
52:06And again, I do the means analysis
52:08and I translate all of these
52:10to 1:00 to 9:00 with an easy
52:12additive transformation, right?
52:14We see a very different pattern.
52:16For some reason people don't want to
52:21select negative numbers in this context,
52:25which is so interesting.
52:26Anything less in the middle value
52:29should be an insult or whatever,
52:31not just like New Haven,
52:32but you know what I mean?
52:34Wild to me, labeling effects people
52:38also don't want to label themselves.
52:40This is a real story from real
52:43research conducted at Yale.
52:44They had done a whole lot of pilot
52:46research and determined that they had
52:47more than enough people in the community
52:50who drink sugar sweetened beverages.
52:52Then they were bringing them into the
52:53lab study and they just couldn't get
52:55enough people through the screening
52:56And they had found that, you know,
52:58the the plenty of people have,
53:00you know, energy drinks or whatever.
53:03And then people would call
53:04the screening and you know,
53:05the person's like in my office
53:07racking their brain.
53:07Why can't I get enough subjects
53:09for this lab study?
53:10I'm like how are you asking
53:11the question on the screen?
53:12Like we're
53:12asking if they have two or two or
53:14more sugar sweetened beverages?
53:15I'm like ask it open-ended.
53:17Ask them how many they have per week.
53:19Boom. Enrollment's covered.
53:20People don't want to put
53:22themselves in a box of like,
53:24oh, why are you asking that?
53:25I don't want to be in the pathological group.
53:29Now, asking this, do you have at least two?
53:33No, I do not. How many do you have?
53:35open-ended? 35% versus 22%.
53:40Isn't that wild? I
53:42know we're fascinating creatures.
53:44You see why I study psychology.
53:47Love it. All right, similar thing.
53:49First Force choice versus an open.
53:53Again, we've got these.
53:56We've got these irregularly
53:58spaced categories that sort
54:01of impose A Likert continuum,
54:03but they're not.
54:04They're uneven.
54:04I did this on purpose
54:08and we get very different pattern
54:10of responding from the first
54:11forced choice to the open-ended.
54:12Now, I said before,
54:14I I cautioned against using open-ended.
54:16You can do this with Qualtrics or
54:18Redcap by just having it of 0 to 30
54:20drop down or zero to 31 drop down.
54:22That'll cover that.
54:23Or even using a text box and
54:25letting them type it in,
54:27but validate it that the range of
54:28scores can only range from zero to 31.
54:34Food cravings again administered.
54:35This should be very, very straightforward.
54:36I gave some of you check the box
54:39of how many I gave. Some of you,
54:40you know go through and answer yes or no.
54:42How many of these have you?
54:44Have you people take information from
54:48the number of options you give them.
54:51If you give them just eight,
54:52they'll select two or three.
54:53If you give them twenty,
54:54they might select five or six.
54:56If you give them a hundred,
54:57they might select 30.
54:59Forcing people to say yes or no again,
55:02you get a very different
55:05pattern of responding.
55:06These are heuristics.
55:07They're little graphic things that
55:10people unconsciously consider when
55:13generating their answers for you.
55:17What is my point of all of this?
55:18Oh, I'm actually doing it on time this time.
55:22Amazing bias can be introduced accidentally
55:26by any number of subtle things.
55:29Be very careful. Pilot everything.
55:32Look at your raw data too.
55:35So give it to five friends or five
55:38colleagues to complete on their own,
55:40and then pull the spreadsheet and
55:42make sure the coding matches.
55:43Answer it yourself a couple
55:45of times on paper.
55:46You know, like print it out,
55:47do it on paper,
55:48go through and do it,
55:48And then make sure the scores are right.
55:50Because you can have all kinds of
55:52little glitches inside of Qualtrics.
55:53You know you can overcome them
55:55by recoding at the tail end,
55:56but it's just so much you can save
55:58yourself the headaches by getting it taken
56:01care of before you collect your data.
56:03Absolutely administer your questionnaires
56:04in the way they were originally validated,
56:07because these small little modifications
56:09can really mess things up.
56:11Even things like adding a Not applicable.
56:14I know you think that you're like
56:15cleaning up data and you think
56:17that you're coming to the patient
56:19where where they are.
56:20Allow people to skip questions
56:22that'll cover that.
56:23Unless, of course,
56:24the original questionnaire
56:25included or not applicable,
56:29and then power of the questionnaire.
56:32It's glorious.
56:32You can answer your research questions,
56:35You can satisfy your curiosity,
56:37You can screen.
56:38And of course you can win the debate on
56:41how you're going to spell your child's name.
56:43My sweet, sweet spouse knows that I
56:46specialize in psychometrics and yet,
56:48for whatever reason,
56:49has allowed his fate at the fate
56:52of our home to be subject to the
56:55demands of my online questionnaires,
56:57and still hasn't realized that I have
56:59figured out how to introduce bias to
57:01make myself right every single time.
57:03And that is it.
57:04That's
57:04all I wanted to it on time.
57:08Hooray, amazing.
57:09And with a sick child at home by the way,
57:12I might add, so super impressed.
57:14And you know, I feel like I need
57:17to take your course because I have
57:19introduced so much error in retrospect.
57:22But questions from,
57:23you know our audience on here,
57:30I have a question about
57:33the reliability of your respondent
57:36like when you said that
57:39there was that questionnaire
57:40that if if they scored highly it became
57:44basically invalidated their responses.
57:47So you know I think that how do you
57:55control for that? Yeah, I mean basically how,
57:58how could you you really assess and
57:59say you know I'm I've gather all this
58:01data but like how accurately does it?
58:04Absolutely. I think the problem is the,
58:07the power of the, I mean the, you know,
58:09with, I feel like with surveys you only
58:12get like a 10 to 20% response rate.
58:15Yeah, absolutely.
58:18Very complicated. An excellent question,
58:20very important. Michael and I
58:22have fallen victim to an invalid
58:27responses. We were collaborating,
58:28trying to work on a project,
58:30which I think should be
58:31resurrected by the way, Michael,
58:32but let's give it like 6 months.
58:37He was working on a measure
58:39of physician burnout.
58:39Right? Was it burnout?
58:41What were we working on?
58:44Physician trust.
58:45It was, it was a really important construct.
58:47But now I've fed bedside manner.
58:49Bedside manner. Oh my gosh. Yes.
58:53And it was really, really good.
58:55But we were, and we were trying
58:57to measure just by, you know,
58:59we weren't even compensating people,
59:01but we had too many invalid response patterns
59:06and realize that the data were correct.
59:09We can build in flags like if you are
59:12paying it, it's kind of like, you know,
59:13the whole captcha thing does, right?
59:16So you can build in things in between.
59:19You can actually build in Captchas
59:20in the middle of your survey.
59:22But I'll say things like,
59:23you know, to make sure our
59:25survey is functioning properly.
59:27Please select option C for this question.
59:30And that's actually a really good
59:32one to to reset if you've got
59:34different questionnaires from that
59:35strongly agree to agree and then
59:37the next bank might be reversed.
59:39I'll usually build in a page break.
59:41Ask something like that you know to to
59:43make sure that we're doing this correctly.
59:46Please select question three or
59:48to make sure our survey is is,
59:51you know, coding things correctly.
59:53Please select the the question you know.
59:56Please select the correct response
59:58for 2 + 4 or whatever and you'll
01:00:00just build in a couple of these
01:00:03little checks for attention.
01:00:04But they're actually ways to screen out.
01:00:07If you've been hit by, you know,
01:00:09people randomly responding and
01:00:10trying to get the completion code,
01:00:13very frequently,
01:00:13you know people will go on M Turk or one
01:00:17of the other kind of data collection.
01:00:19Services.
01:00:23And then, you know, people are out there
01:00:25just answering questions for money.
01:00:26So you need to make sure that you can build
01:00:29in little checks of attention like that.
01:00:33Probably have time for one more question.
01:00:35Julia, did you want to ask something?
01:00:37I saw you put yourself on on
01:00:41video. I need to see it.
01:00:44I was mostly just playing myself on video.
01:00:46I have I I I guess there's
01:00:49a lot on this topic,
01:00:51so I'm just curious just like if there
01:00:53is something that comes to mind for
01:00:54you to talk about other languages,
01:00:57whether that's like already validated,
01:00:59already gone through that process
01:01:01and translated and using other
01:01:02languages that we might not know.
01:01:04I'm in the process of doing
01:01:05a lot of like transition fact
01:01:06translation with cultural work and
01:01:08there's so many idioms and like
01:01:10the most commonly used things.
01:01:12So just curious overarching
01:01:14thoughts about multilingual.
01:01:15I mean it sounds like you nailed it,
01:01:17translation and back translation.
01:01:19Also consideration of the scaling
01:01:21itself because there are some cultures
01:01:24that the Likert scale or the four
01:01:26point scale really doesn't work.
01:01:28I'm not sure if the visual
01:01:31Analogue would work as well.
01:01:34So we've had kind of, you know,
01:01:37just collaborating with experts
01:01:38who are fluent in that language,
01:01:40ideally as their primary language
01:01:42and culture to weigh in on this
01:01:44and simply defer to them has been
01:01:46their approach for us when I've been
01:01:48involved in this particular process.
01:01:52Thank you. Great.
01:01:53Thank you, Marnie.
01:01:54Thank you again.
01:01:55Really appreciate it.
01:01:56Really appreciate your expertise
01:01:57sharing with us and as I said,
01:01:59doing it with somebody's to get home.
01:02:01So thank you.
01:02:03Thank you. Have a great
01:02:04day everyone. Thanks right.