Presentation at the Career Development Institute for Bipolar Disorder, Boca Raton, FL, April 30 - May 2, 2006

Ellen Frank: A couple of comments, usual reminders, confidentiality, what we say in this room does not leave this room under any circumstances. No comment that's made today should ever go out of this room, and no information about this review should be passed to anyone, regardless of their relationship to the application. I just wanted to remind you about the standard of conflict of interest. Obviously, if you are at the university of the applicant, you'll be asked to step out of the room. If you're a consultant to a project at that university, you'll be asked to step out of the room. And I know how absurd this can become. You're working as a methodologist on someone's grant at the University of Utah. You don't know a single other person at the University of Utah, but we still have to ask you to step out of the room.

But I want to also remind you that - and that's the appearance of conflict of interest part - but the other piece of it is real conflict of interest that is sometimes not apparent to anyone. You've not co authored a chapter with this person. You've not written a grant together with this person. But you went to high school with this person and you've been friends all of your life. So that's the kind of thing that David can't sort out, but we need to know about. So please, if any of you have conflicts on either of these applications, any of these applications, please let me know.

Sue Marcus: I have a question. Suppose I consult for one department at Columbia University and then there's somebody from a totally different department. Is that still a conflict? Because otherwise, I wouldn't really be able to review any, pretty much, because I work for so many different people and -

Ellen Frank: David, what's the rule on this? I honestly don't know the answer, but David, that's the SRA's job is to have the answer. And remind me, how much time do we have for each of these applications? Thirty minutes, total. So 20 - 15 for primary? Ten for primary, five and fifteen for discussion. Okay, fine.

David Kupfer: And it's okay we come short. We've got so many.

Ellen Frank: I know, I know. Okay. So ten for the primary reviewer, five for the secondary reviewer, and about 15 minutes for discussion. And what time are we supposed to finish? Okay.

Key discussion points for each application: significance, approach, innovation, investigators and environment. We'll treat the human subjects issues, minority representation, and the budget separately.

In looking at human subjects, we want to comment on safety risks, plans for monitoring of risk, timetable for subject recruitment, and how patients are going to be recruited in addition to the usual human subjects issues.

Very important issue that we sometimes skip over: does the application meet the criteria for the mechanism? If this is really a K-application. Does it have the appropriate parts? If this is some other mechanism, does it really meet the criteria for the mechanism?

Couple comments about Ks: we want to be giving equal weight to the training plan and the research plan. The training plan needs to be well-articulated and well-integrated into the research plan. The research plan needs to be rigorous, but much more modest in scope than we would expect from an R01.

Is there appropriate mentoring at the applicant's institution? Consider both the primary and the secondary mentors. Does there seem to be appropriate infrastructure available? Can this work actually be done in that environment?

We wanna consider, also, the structure of the application itself. Does it tell a story? I already have A, B, and C; I need to get D, F, and G and here's how I'm going to do it.

Does it have any of the typical pitfalls? No real justification of the measures that are being proposed; no discussion of the pilot work; poor documentation of the institutional support; or, most often, an overly ambitious research plan.

What does the Chair's letter say? Does the Chair's letter convey real enthusiasm for the candidate and do you get a clear statement that there's a faculty position available for this applicant?

This is just an aside; we're stopping out here for a moment. For those of you who are in the K world, we figure that it takes six to twelve months to write a good K application. And assuming you're going to go at least two rounds before you're successful, and then a few months after the Council meeting before you're actually funded. The timeframe from starting to think about a K to being funded is probably two to two and a half years. So that's just an aside.

Ellen Frank: Who's the primary? Lauren?

Lauren Marangell: Yes, I'm primary.

Ellen Frank: So ten minutes.

Lauren Marangell: Do you want temperatures?

Ellen Frank: Yeah, well, let's take a temperature, sure.

Lauren Marangell: 220.

Ellen Frank: 220? Susan?

Susan Marcus: I'm sorry?

Ellen Frank: Temperature on the application. Where - what's your -

Susan Marcus: Oh, you mean -

Ellen Frank: Yeah, where do you think -

Susan Marcus: Oh, it's not, like, 2.2 but we're talking -

Ellen Frank: Yeah, 220, 2.2, whichever way you wanna call it.

Susan Marcus: Oh, okay. Yeah, I'd say 2.2.

Ellen Frank: Okay. I might be a little more generous. Two.

David Kupfer: 2.1

Ellen Frank: Okay? So we're not very far apart.

Lauren Marangell: Okay. This is a revised R21 application that proposes using proton magnetic resonance spectroscopy to identify quantitative differences in myoinositol in a glutamate/glutamine in multiple regions of interest with a focus on prefrontal cortex and anterior cingulate in adolescents with bipolar disorder, ADHD, and healthy controls. Both bipolar and ADHD, alone and in combination, are significant public health concerns. Further elucidation of the neurobiological differences between bipolar disorder and ADHD will provide highly useful information that could be used in treatment development, so I think the significance is reasonably high.

In terms of approach, the application proposes to use a 3T magnet to perform the MRSI data collection of the tabulate concentrations in multiple brain slices. Subjects include 32 patients with narrowly defined bipolar disorder, 32 subjects with ADHD without mood symptoms or learning disabilities, and 32 healthy controls. Subjects are matched on handedness, age, and gender. Subjects with bipolar disorder must have a first-degree relative with bipolar disorder. Comorbid ADHD conduct disorder and oppositional defiant disorder are permitted in the bipolar group. Subjects with only ADHD must not have a first-degree relative with bipolar disorder. All subjects are between the ages of 12 and 16. All are drug- and medication-free. Patients with bipolar disorder are also mood stabilizer na×ve.

Lauren Marangell: In summary, while the application has many strengths, there are several concerns that remain. One significant issue is the variability that's caused between the comorbid bipolar disorder and ADHD. And so since one of the main hypotheses is that there'll be spectrographic differences that will be found bipolar disorder and ADHD, allowing the comorbidity in the patients with bipolar disorder is a confound that I think is difficult to get around.

Now, the application does address that allowing the comorbidity is important because it's a common comorbidity and they're trying to make the sample more generalizable. But nonetheless, it severely interferes with their ability to obtain the data that they're hypothesizing will be important. I think that is one of my significant concerns that I'd like to see addressed.

The other is that it's not clear what mood state the patients with bipolar disorder will be in. This is an important consideration because if the mood state is not controlled, the applicant needs to at least discuss how this variability is gonna impact the data. On the other hand, if mood state is controlled, then I would like the applicant to be more specific about detailing this in the inclusion criteria and considering how this restriction would impact feasibility.

Along the line of feasibility, I'd like to see a clear discussion of feasibility with a separate section. It was difficult for me to ascertain exactly how many patients were gonna be studied and whether or not this would be workable within the R21. And so if you go through the pilot data section, at one point there's mention of a study that screened 300 patients with bipolar disorder over two years to obtain 21 evaluable subjects. And I find that to be very believable. But it also raises the question as to feasibility for the current application because they're proposing recruiting 32 subjects over two years, with what are possibly more restrictive criteria. And so I need more information to make sure that this is actually doable.

I had some concerns about the sample size. I just needed additional data to be able to follow that. As far as a specific parameter, estimates used in the calculations. And I also wasn't sure if, they're powering it for a standard deviation difference of 0.52 and I just didn't know what that meant in terms of what the overall significance of the project was. And so I was hoping that the applicant could discuss that further.

Additional comments that would strengthen the application:

The definition of a mood stabilizer so people will be excluded if they've had prior treatment, but I don't know if this means atypical antipsychotics or if this means Lomotrigine or just the traditional mood stabilizer.

I'd like additional detail on the group with ADHD - it stated they won't have mood symptoms, but I don't see a cutoff on the YMRS or the depression rating scale in the inclusion/exclusion criteria and so I'm not sure how they'll use that as a double-check.

The rationale for setting myoinositol is based on old mechanistic theories of Lithium effects and is less compelling and should be condensed. On the other hand, the rationale for studying glutamate is more novel and I'd like to see that more fully explained.

The CGAS is used to measure functional impairment, and I'd like to see a discussion of the choice of that measure and consideration of other scales that perhaps have better psychometric properties if functioning is gonna be one of the outcomes.

I have a list of kind of more minor, structural things that I may skip for the sake of time and give to the applicant, if you prefer, Madame Chair?

Ellen Frank: You've still got a couple more minutes.

Lauren Marangell: A couple more minutes? Some of this, I think, is just this is a draft. But there are several places where the applicant will refer to the appendices but I don't have enough of information within the body of the grant to know what they're referring to. And although, David, you always do such a good job; I didn't have the appendices. And so it was a little hard for me to sort out. But it's a general rule of thumb to have everything that you're citing kind of clear within the body of the grant. The racial enrollment template doesn't match the number of subjects that are in the proposed study. There's some confusion about the table references. And kind of linking the pilot data with kind of clearly what that's trying to show in terms of support of either feasibility or support of the methods of the grant. I get a little bit lost within the pilot data.

The application, in terms of innovation, I think it is innovative in several respects. I liked the tissue volume correction as a technical advance. The investigator appears to be well-qualified to conduct the proposed study. The environment is well-suited to conduct the proposed study, although I didn't have the resource pages.

And so to sum up, I think that this is a good study by a good investigator, a good junior investigator, but there are relatively moderate methodology considerations that dampen my enthusiasm.

Ellen Frank: Okay. Secondary was David.

David Kupfer: Well, unfortunately, this grant didn't come to this committee before. So I didn't have an opportunity to remind myself about the pink sheets from the previous committee.

Lauren Marangell: Yeah, I had the same problem.

David Kupfer: And so while it's likely that this applicant was responsive to those committee's questions and queries, and we don't know what the score was the first time, it would've been useful for us to have had the summary sheet to see both the responsiveness but also some of the things that you have brought up and I will bring up are concerns that were not necessarily possibly brought up by the first review. So that's the issue sometimes when this grant is not assigned to the same committee that you got the first time.

Now, I don't want to necessarily be redundant in terms of interest of time. Many of the concerns and queries that Lauren has brought up are contained in my written review that you all have.

Ellen Frank: So yes, please just tell us what your additional comments are.

David Kupfer: So I think that there were a couple of concerns that I had which come down to the fact that this is an R21 mechanism, which essentially is not that we should have the same expectation that we would see a complete R01 finished.

But having said that, reminding myself that the primary aim of this grant is contained in terms of the applicant's specific aim which is, the core hypothesis is that bipolar and ADHD are distinguishable even early in their- findings. And that then raises the question about, if that's the case in this preliminary forward, which is an R21, then most of the attention has to be looking at, it seems to me, very sets of subgroups where you're not gonna have too much overlap because you're not gonna be able to deal with the variability that's been raised already.

And so in some ways, I worry less about the generalizability of how many ADHD/bipolar kids are out there versus if the hypothesis is that there are differences, then you've got to work with those groups that are either bipolar or ADHD. And you can't go for something in the middle. And I think that that's a problem that then gets laid out in terms of issues around power, issues around how much variability, issues about even strategy. And the usefulness of normals versus the non-usefulness of normals and how many of an N do you need in all?

One of the things about the n which confused me a little bit is that the table at the end, which is always important, which deals with issues of sex, gender and who is likely to come into the study, shows a great deal of appropriate diversity. But I was confused because it talked about a total planned enrollment of 480 people, and that's not what is specified in the grant. So there was a disconnect in that enrollment -

Ellen Frank: David, you have one minute.

David Kupfer: - which may be another study versus this one. And in the remaining one minute, I would say that the clarity about the methods that are discussed and the very clear set of hypotheses, given a relatively small n, is where it's going to sort of resonate with the review group to say that this is only so-so and has moderate weaknesses. To get it past the moderate weakness level, there has to be more precision about the choice and what your method is, rather than a kind of overall, ‘we're going to measure all of those things'. So it's kind of that issue around focus that concerns me.

Ellen Frank: Susan, comments?

Susan Marcus: Yeah, I have a few statistical issues. Let's see - one is they were planning to exactly match one-to-one on age, gender and handedness. I'm not sure what handedness has to with it. Probably the rest of you would know.

Ellen Frank: It has a lot to do with the brain.

Susan Marcus: But yeah, it's almost impossible to match exactly on three characteristics when you have such a small group. Because for example, let's say you can have a 16-year-old left-handed boy. You might not find a left-handed 16-year-old girl. So I think they need a different strategy for the matching.

I was wondering, also, about the excluding - let's see - they exclude children with ADHD who have learning disabilities or mood disorders, and I think it's almost impossible to find someone with ADHD who doesn't have something additional. Either they're gonna have anxiety or they're gonna have learning disabilities. So I wondered about that.

Ellen Frank: So you have some real questions about recruitment feasibility.

Susan Marcus: Right. And then in the power calculations, it also - there wasn't quite enough information. For example, they said they're going to detect 0.52 standard deviations, which would be about a medium effect, but I didn't see the pilot data exactly correspond to that. And then they said they're going to have enough power to detect a difference of one standard deviation for their dichotomous outcomes, and it's almost impossible to find that big an effect. In this type of study, they might find that effect, but I think with the combination of the ADHD and the bipolar, they probably would be not likely to do that.

If they do use some type of matching, then they would have to account for that in the power calculations. They don't talk about dropouts or anything like that. In the specific aims, they talk about making an adjustment for multiple comparisons using a permutation analysis. I'm not even sure what they're talking about. And also, they didn't make that adjustment in their power analysis. So it's always really important to make sure everything in your aims is consistent with what you say you're going to do. And they say the analyses will take the match design into account, but they don't discuss that in the power analysis. So there are ways of taking the match design into account and I could give them some references on that.

Ellen Frank: Okay. So do you have more to say?

Susan Marcus: No. I just wanted to say though, with a statistical review, sometimes there are fatal flaws that can't really be fixed. And then sometimes if they just get a statistical advisor who helps them then - but most of these can be fixed. I don't wanna make it sound worse than it is.

Ellen Frank: This application is way outside my area of expertise, but what I could intuit about it, I completely agree that this is innovative. She's asking an important question. But I think there are really some methods issues here. My other concern is given all the things that we've found missing, I'm curious as to why this applicant didn't use the full 25 pages available to her.

Lauren Marangell: It's a R21.

Ellen Frank: Is that limited in terms of page numbers? Is it limited to 20 pages?

Lauren Marangell: I think it is a shorter application. We'll have to check that.

Ellen Frank: I don't know. David? Does anyone know? It is. So it's limited to 20 pages. Okay. Okay.

David Kupfer: If I could just raise - do we have a couple of seconds?

Ellen Frank: I think we do.

David Kupfer: All right. Just to push - since unfortunately I didn't have the personnel involved, what is clear is to me that if there is a statistical methodologist or a statistical advisor, then that person probably hasn't weighed in that much and the PI probably had to put this together. One of my concerns is that our study section has been seeing more and more applications using imaging techniques. And I've been impressed with the relative lack of precision of the statistical sections around data analysis and power calculations as compared to some of the clinical intervention grants that we've been looking at.

And the concern is really somehow to let people know that they should get the same methodological rigor. They're not gonna overwhelm a group of us, even if we're not imagers, but if we don't understand the data analysis section and the power analysis, it's going to, in a sense, impact adversely on the final score for these applications.

Ellen Frank: Okay. So are we ready to take a vote on this application?

Lauren Marangell: Well, I was wondering - I mean, I know our job isn't to redesign the applicant's study. But in terms of giving feedback so - you know, I hate it when we give a review back that's contradictory. And so in terms of ADHD comorbidity, would you feel more comfortable if they didn't allow mood disorders but did allow learning disabilities? Would that -

Susan Marcus: I'm concerned that that might be a major weakness that I think it's really hard to separate out the comorbidities in ADHD. But you're -

David Kupfer: But then how can you address the question?

Susan Marcus: I think it's a very - it might be very confounded and a problem.

Ellen Frank: So I think the -

Susan Marcus: But I'm a statistician, not a psychiatrist.

Ellen Frank: Right. Well, okay. So I think -

Susan Marcus: That's just my impression.

Ellen Frank: - that the clinical folks on the committee need to send a clear message. I think we don't want to send this applicant a mixed message about which way she should go with respect to pure groups versus more -

Lauren Marangell: Right.

Ellen Frank: - this is a two-year -

Lauren Marangell: I don't think you can address the -

Ellen Frank: - application, right?

Lauren Marangell: It's a two-year application.

Ellen Frank: So how much - what can be done in two years? That's an important question. How many of these kids can you find? How pure can they be? On the other hand, it's a lot of expensive resources to get no answer. So -

Lauren Marangell: Right. No, I think you have to have a lack of overlap between the ADHD and the mood. But I think there'd have to be -

Ellen Frank: So -

Susan Marcus: To get a signal.

Lauren Marangell: Yeah. Since that's the hypothesis.

Ellen Frank: We want to reinforce her idea that she's going to pick relatively pure bipolar cases in terms of her requirement that the subjects with bipolar disorder have a parent with bipolar disorder, a first degree relative. I can't remember which it was. So that -

Lauren Marangell: You know, I - that's fine. I don't even feel as strongly about that as I do that they don't have ADHD.

Ellen Frank: Okay. So you'd like to see a clear-cut diagnosis of Bipolar I. You're not so concerned about first-degree relative. But you want the pure -

Lauren Marangell: No, I'm more concerned that they don't have ADHD.

David Kupfer: 'Cause you want her to have early-onset bipolar by definition.

Ellen Frank: Right. It -

David Kupfer: In terms of these kids.

Ellen Frank: They're likely to have a first-degree relative, by virtue of their early-onset.

David Kupfer: Right. But I wouldn't make it essential because, again -

Ellen Frank: But you want them with -

David Kupfer: - there's a screening process. You're going through hundreds and hundreds of kids. This budget doesn't have that -

Ellen Frank: Okay.

David Kupfer: - that possibility.

Ellen Frank: You want them pure with respect to ADHD. So the bipolar kids are bipolar -

Lauren Marangell: And not ADHD.

Ellen Frank: - and not ADHD. And the -

David Kupfer: Right. And the ADHD kids -

Susan Marcus: Is it possible to determine that they don't have ADHD and they do have bipolar?

Lauren Marangell: It's a subset. There is. There is.

Ellen Frank: Theoretically?

David Kupfer: Otherwise, there is no hypothesis.

Ellen Frank: Right.

Lauren Marangell: You can't - yeah, exactly.

Susan Marcus: But it'll make it harder to recruit, though.

Ellen Frank: It's gonna make it very hard to recruit.

Lauren Marangell: Absolutely.

Ellen Frank: Right. So and the ADHD kids?

Lauren Marangell: Can't have -

Ellen Frank: Can't have bipolar.

Lauren Marangell: - bipolar disorder.

David Kupfer: But they can have a mood -

Ellen Frank: They can have a mood disorder.

Lauren Marangell: They can have a mood disorder, yeah.

Ellen Frank: Okay.

Susan Marcus: So that gets rid of all the ones with anxiety, is that right?

Ellen Frank: No.

Lauren Marangell: No.

David Kupfer: No.

Lauren Marangell: There's a difference.

Ellen Frank: They can have anxiety. Okay. So where are we on the learning disability?

Lauren Marangell: I don't have strong feelings about the learning disability.

David Kupfer: That's okay.

Ellen Frank: All right. So we ready to vote now? Do we think we can send a clear message?

Lauren Marangell: Yes.

Ellen Frank: David, are you satisfied that you can write this pink sheet? Okay.

Lauren Marangell: We're writing them?

Ellen Frank: We're -

Susan Marcus: We have a range, right? Or no, we have to -

Ellen Frank: This is a vote. This is a vote.

Susan Marcus: But don't you have to vote -

Lauren Marangell: Yeah.

David Kupfer: You still need to get something from -

Ellen Frank: And do we have to do the temperature again? Okay. Okay. All right. So -

Lauren Marangell: I'm still at a 2.2. I haven't heard anything that would change my mind about that score.

Ellen Frank: Okay.

Susan Marcus: Same with me.

David Kupfer: I'm a 2.2.

Ellen Frank: Yeah, I think I'm at the same place.

Susan Marcus: Would anybody else -

Ellen Frank: When I have a -

David Kupfer: Anybody else in this group that's gonna be out of that range?

Lauren Marangell: So normally, if David and I disagreed and I said I was at a 2.8 and he said it's a 1.4, the rest of the committee could vote anywhere in between that range.

David Kupfer: And since we've said 2.2 and nobody has specified, I assume everybody is gonna vote 2.2.

Ellen Frank: Well, I'm probably at a 1.8 on the research plan, but only at about a 2, 2.1 on the training plan. I think there's a real disparity between the training plan and the research plan. So whatever that averages out to. You know, I'm so strong in math. So somewhere around a 2 I guess.

Susan Marcus: Oh, I'm secondary. 2, also 2.

Ellen Frank: Okay.

Lauren Marangell: We weren't assigned, right? So we don't give initial scores? We will, Madame Chair, if you would like.

Ellen Frank: Well, I mean, if you've read the application, you can give us your temperature.

Lauren Marangell: I would do 1.8.

David Kupfer: I'm at 2.2.

Ellen Frank: The training plan is reasonably well described, but not strong. She names as her primary mentor ----, whom she describes as an internationally recognized expert in pediatric bipolar disorder, but was not known to this reviewer. So given that I am fairly familiar with the small cohort of individuals doing pediatric bipolar disorder research, I'm a little concerned that this mentor may be somewhat junior to really fulfill the role. And since I don't have her biographical sketch, it's hard for me to know.

As her third mentor, she has ----, who has I think certainly been one of the people to set the standards for psychosocial research in bipolar disorder and has recently been working in child - really, adolescent bipolar disorder research.

The training plan seems to me to be light on formal coursework. There are only three formal courses named other than the required ethics courses. Big red flag. The applicant plans to spend 25 percent of her time doing clinical work in what sounds like a very, very busy clinical setting and I'm quite worried that she may be pulled to a lot more than 25 percent time in her clinical work.

This is an aside. Since the mentor statements and the Chair statements aren't in the application yet, it's really important how strong these are and the way they're written. Some things that - it's very important because a lot of the application will be evaluated on that issue.

Ellen Frank : Approach: I think we've got clearly stated aims, but I think a bit overstated for the scope of a K. Her aims are to determine the efficacy of X, to determine the efficacy of this treatment with respect to symptoms, to determine the efficacy with respect to functioning. And that's just not possible with the n. You can do feasibility, you can try to understand how the treatment works, but with an n of 25 and 25, you're not going to determine the efficacy of anything.

The background: she reviews the microscopic literature on interventions for child bipolar, but I think the background is notable for the absence of any references to treatment of other mood disorders in children. Is there something to be learned from the literature on treatment of unipolar depression in children? I think the background could make a stronger case for the potential value of psychosocial treatment in an area where pharmacotherapy is such a dicey proposition.

In terms of other work that I think could be included in the background, particularly Marika Kovacs' work on affect regulation and unipolar could be really an important piece.

Preliminary studies make a reasonable - actually, incredible, truly not believable case for the value of child- and family-focused CBT. Effect sizes of - I don't know - three, four, unheard of in the psychosocial literature. So it really makes me wonder what was going on there. What's happening? This is a - it's not credible, so therefore we need some information explaining why we should believe these phenomenal results.

Research design is a relatively straightforward comparison of 25 subjects who will get CFF CBT versus 25 subjects who will get treatment as usual. These subjects are all to be stabilized, but stabilized is never defined. So I don't know - stabilized on medication - I don't know what that means.

The choice of assessment points for the treatment as usual group is unclear. We get one plan in one part of the grant, another plan in another part in the human subjects. The feasibility issues are really not addressed with respect to the treatment as usual group. Will folks who have come to this highly specialized famous clinic be willing to accept referral back out into the community, and there's no evidence that that's gonna be the case.

Measures: measures are not particularly well-justified; they're just listed. But no real information about why these measures are better than other measures.

Data analysis: not as strong as the rest of the application. Pretty na×ve, but no sort of statement that "I really need to learn the skills to do the -" the data analysis section reads kind of like a data analysis section from a 3rd year graduate student in clinical psychology and not using any sophisticated techniques. She's planning to take some courses that may lead her in that direction, but I think a better data analysis section would be one that states that.

Human subjects: I think that the risks of a family intervention are downplayed. And they may not be risks to the child, but risks to family members. Certainly in families where there's bipolar disorder, and a lot of these kids are likely to come from families where there's bipolar disorder, and that's another issue I'll get to in a minute. Family therapy has the potential to be quite an explosive circumstance and I think that some of that needs to be addressed. She says that she's going to exclude severe mental illness - in kids who have parents with severe mental illness, but severe is not defined.

Minority representation seems that the institution has a highly diverse population, but interestingly, in the pilot work, the subjects were much less diverse than the population of the institution. So some statement in the human subject area about how she's going to really keep the study population as diverse as the clinical population.

Just some general comments: the documentation citation was kind of careless. There are some things referenced that don't appear in the citation list; some things in the citation list that don't appear in the- it's not clear whether this is gonna be an individual or a group intervention. Sometimes it's talked about - it seems like she's talking about an individual family intervention; sometimes it sounds like it's a group family intervention.

And it's also not clear who actually developed this treatment. Whether it was her mentor who developed the treatment or whether it was the candidate herself who developed the treatment, and if it's the candidate herself, she oughta be taking credit for that.

So in terms of significance, highly significant. We need treatments for this population; we have none. Approach definitely needs work. I would say highly innovative. The investigator seems highly qualified to do the work. The environment is more of a question, particularly the strength of the primary mentor. So that's my scope on this. Who's the secondary?

Susan Marcus: I'm secondary.

Ellen Frank: Okay.

Susan Marcus: The research plan, with respect to statistics, has two courses which seem appropriate, but there doesn't seem to be a consistency from the research plan to the data analysis and from the training plan to the research plan.

For example, she talks about taking classes using growth mixture models. But in the data analysis section, it's very na×ve and it doesn't really use longitudinal data.

In the introduction to the design and methods, she talks about how the power analysis has to be exploratory because there's little prior work in the area, but I think she could use the pilot data for the intervention group. And there must be something in the literature that would give you some idea of what the treatment as usual would expect, so that would strengthen the power analysis so she could actually find use the pilot data and data from the literature.

In her measures, I don't think she's clearly defining one or two primary measures. And I think that's important to say we're gonna look at a lot of secondary measures, but to actually say here's our primary measure, she has child reports, parent reports and teacher reports, and there's some very good work from Helena Kraemer and David Kupfer about how you combined these reports and what you do when they differ. And you can find some papers from the MTA study of children with ADHD where they do that.

The data analysis plan doesn't really reflect the training and it also is very simplistic, so she doesn't take into account the longitudinal analyses, which will really give her much more information and also more power. And there are references in how you calculate power using longitudinal methods and, as I said, you'll get more power if you can use that.

She mentions doing the primary analyses on the intent to treat sample, which she says are the subjects who agreed to participate in the treatment and attended the first group therapy session. That's not the real definition of intent to treat, so anybody who's randomized, it doesn't matter if they attend or not, would be in the intent to treat.

And then she mentions using ANCOVA model for adjusting her baseline scores, but it'd be much better to use something like mixed effects. She mentions that she's gonna look at the dropout rates, but there's nothing in the power for allowing for dropouts. And there's a reference for calculating power analyses with longitudinal data with dropouts by Hedeker et al. in the Journal of Educational and Behavioral Statistics so I suggest she looks at that. [Sample size estimation for longitudinal designs with attrition: Comparing time-related contrasts between two groups (1999), 24(1), 70-93.]

And then finally, she talks about exploring mediational models with multiple regression, but I suggest that she look at some references from Helena Kraemer about mediation. And I suggest anybody who's interested in mediation to use those references. And I think that's all.

Ellen Frank: Do you really think it's possible to explore mediation with a n of this size?

Susan Marcus: You could explore it -

Ellen Frank: In this small number of time points that she's got?

Susan Marcus: Well, I think you can explore it because mediation requires looking at several associations. And I think she has enough power to at least see if, for example, she would need to see if her mediator is changing over the course of the intervention. She would need to see if the intervention - people with the mediator do better on the intervention etc. So I think she would be able to. Maybe the p-value might not be .05, but she would be able to at least get exploratory information.

Ellen Frank: Okay. Comments?

Lauren Marangell: I wasn't as concerned about the mentor plan. I also was not familiar with the primary mentor, but given the outside mentors, how much of a factor would you make that? I mean, is it a lethal type of thing or -

Ellen Frank: I don't think it's a fatal flaw.

Lauren Marangell: Okay.

Ellen Frank: But I think documentation - it could - I think a first strategy would be not to overstate the case -

Lauren Marangell: Right. No, I agree.

Ellen Frank: - of who the mentor is.

Lauren Marangell: Yeah, I agree.

Ellen Frank: And then to document what this mentor has accomplished. And a very important thing to document would be any prior mentoring that she's done.

Lauren Marangell: Right.

Ellen Frank: Who are the other people that she's mentored?

Lauren Marangell: Right, right. But I was a little bit - that concern was attenuated by the outside mentors - very strong outside mentoring team.

Training plan: I agree with you in terms of adding coursework. Research plan: I share the concerns that you guys both have.

David Kupfer: Just a couple of concerns. Since this is the first time that we've seen this application, I think part of the reason that we're focusing so much on the research plan is there's really a disconnect and that's what I'm not sure that I'm as warm as I was even before I heard the critiques. There's a real total disconnect between the training plan on one side and this R01 on the other side. And it's a major flaw that usually occurs when people first put in their K applications, and they don't get good mentoring about putting in the K application in the first place.

The K application has to be an interaction between the training plan and the research activities that go on over that five-year period. If there isn't that interaction that is perceived by this review group, we are never going to give it a fundable priority score. And so the notion that the applicant is starting a randomized control trial in Year Two means that everything they need to know to run that randomized clinical trial in Year Two, they will pick up either in Year One or they will have picked it up before. So why are we wasting our money giving this person a K if they're ready to do an R01? So those are the kinds of concerns at least that I have with this total lack of connectivity.

So for example, the first two years of this person's research plan should be to define: What are the key outcome measures that one needs to conduct a clinical trial on this particular form of treatment? Outside of the outcome measures, what are the ones that are gonna be clinically significant?

So for example, throwing at me a number of Cohen's ds doesn't give me any assurance that somebody will pick the right outcome measure or have any real convincing, for me, around clinical significance. Ultimately, when they do the clinical trial, they will go for a medium-sized effect as their threshold; not a large-sized effect, because if they go for the large-sized effect and don't achieve it, then the whole study is flawed and they will have no results. So it's that kind of concern that I have with that, even the statistical piece. Which, sometimes, you can't from a statistical course. So I'm less concerned about taking another three or four advanced statistics courses than having somebody who is going to mentor this person ongoing about all the ins and outs of a clinical trial.

Now, again, the clinical trial, as you have all said as the reviewers of this, is not the definitive clinical trial. What I wanna be convinced is that in the first two or three years, Years Two and Three, this applicant is going to put together enough that they will be able to go for an R01 to do a definitive clinical trial to see whether this particular treatment has any evidence. 'Cause right now, an open trial, you know, it's a promissory note. And if that's what this person wants to do, then I'm assuming they're gonna become an expert in clinical trial methodology, the statistics that they need to choose the right outcomes, be able to look at clinical significance and also to figure out - without doing any kind of dismantling - what's so special about this particular treatment approach and has this been used in other mood disorders. Why does one choose it?

The literature review may or may not be there. I haven't had a chance to read the one paper that's in the ___ on it, which looks like it has many authors on it and I don't have very much more information other than it looks like the open trial seemed to be promising.

But the disconnect between the training plan I think is a concern. One can't always pick a mentor who's done a lot of mentoring. The best mentors are people from a group who've also had Ks and have been going on to learn about that. But I would like to know more about the mentoring. Having outside mentors is fine, but you still need this whole notion of frequency of how you're going to be dealing with your mentor on a regular basis.

Ellen Frank: It seems like this mentor is very available to her. And that's definitely a positive. But -

David Kupfer: But then the letters of clear "where is that going to take place," "what else is available," "is there going to be real institutional support to give all the patients that this person's gonna need?" Because the money, $50,000 a year, to do this project is not going to be anywhere enough. And so I would like to see a letter of institutional support to say, "Look, we know this pilot research plan is going to cost $200,000. We as a section are going to put it in and we're not gonna necessarily wait until someone gets a NARSAD."

Ellen Frank: Yeah, but one other issue - thank you for reminding me that it was not addressed at all and needs to be in my critique - is we don't know who the clinicians are going to be. Is it all going to be the candidate herself? If there are going to be other clinicians, who's training the clinicians? to what standard? She talks about manual development, but is the manual gonna be developed before the study starts?

Susan Marcus: You know, I think I'll add one thing. Maybe a better strategy, rather than making this a mini-R01, would be to say it's more of a pilot study to plan an R01. And then for the power analysis, if she used a longitudinal model, she could try to estimate certain parameters. For example, what's the dropout rate? What's the intraclass correlation - that means, like, how correlated are the observations over time? That way, when she finishes this, she would be in a position to know exactly what power she needs.

David Kupfer: I totally agree, and I think when the paper that comes out, which will be soon, which will show that the power analysis cannot be derived from pilot studies, it's going to upset a lot of us who think that that's why we're doing these small pilot studies. So the pilot studies will look at issues around feasibility, choose the right outcome measures, do all the training that's necessary, as well as certain parameters you can estimate, but you may not be able to estimate the primary outcome measure.

Ellen Frank: Just to be clear about what David's saying, there was this old idea that you could - and every K-award said it, and every treatment development grant said it - that you were going to use the pilot data to estimate effect size for a larger clinical trial. And Mark even mentioned that yesterday.

It turns out, at least according to Helena Kraemer and, you know, Helena Kraemer talks to God, but according to Helena Kraemer, you can - and it makes sense - you can only estimate an effect size from an appropriately powered study. And these never appropriately powered. So you can, though, estimate dropout. You can estimate recruitment feasibility. How many screens are you going to have to do to get one subject? How many consents are you going to have to do to get one patient to finish the trial.

David Kupfer: And you're still going to wind up with somewhere between the need for 30 to 60 per cell anyway.

Ellen Frank: Right, right, right. Okay. So I think - are we supposed to be done, Jane? Yeah. And in fact, we haven't voted yet.

David Kupfer: We haven't voted.

Ellen Frank: We haven't voted yet.

Lauren Marangell: You're still primary.

Ellen Frank: I'm still primary. Okay, so I've got to come up with a total score. Yeah, I think, given this discussion, I can't come above a 2.0, or below a 2.0.

Susan Marcus: Let's see - I'm second. I think I'm gonna give it 2.2.

Ellen Frank: Okay.

Lauren Marangell: Yeah, I would agree with 2.2.

David Kupfer: I started at 2.1; that's where I'm at.

David Kupfer: And I think that Susan's point is that you may be better off - most of us dealing with clinical trials push for a medium-sized effect size, which we think is clinically significant in behavioral research to give us enough of a sign. We never put down large effects because that becomes the threshold. That means that that's what we have to show. You hardly ever show that in your controlled trial. So medium seems to be reasonable. You're probably never gonna have the money outside of large contracts to do a small effect size.

Susan Marcus: Right. Sometimes I add things. I say, "A smaller effect size wouldn't be clinically relevant so we're not interested in that." And you might say something - no, I guess that's about all you could say.

Ellen Frank: But I really think in treatment development grants and in Ks, that in some respects, the power oughta be off the table because if they're never going to be powered to really demonstrate a difference. So I think in a certain way, you're shooting yourself in the foot by -

David Kupfer: But on the other hand, review groups like us still look for the power sections. So you have to put something there, but don't listen to them entirely.

Lauren Marangell: Yeah, you can't leave it out.

Susan Marcus: You know, the other thing about power that a lot of people don't think about is that if you're doing a treatment trial such as this, you're probably not gonna have a big effect if you compare it to another active treatment. So it seems treatment in usual versus this new treatment probably will not have a large effect.

However, those of you who are doing imaging studies in biological psychiatry and you're looking at the difference between two very distinct groups, for example, bipolar disorder versus healthy controls, then you might actually see a very large effect. And I know there are a couple of papers in biological psychiatry where they actually calculate - they say in this trial for schizophrenia, the effect size is 1.2.