NYC Midnight : Creative Writing & Screenwriting Homepage
Forum Home Forum Home > GENERAL DISCUSSION > Screenwriting Bar & Lounge
  New Posts New Posts RSS Feed - Go rate your judges
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Go rate your judges

 Post Reply Post Reply
Author
Message
jdadams1 View Drop Down
NYC Midnight Black Belt
NYC Midnight Black Belt
Avatar

Joined: 13 Jul 2018
Status: Offline
Points: 4559
Post Options Post Options   Thanks (2) Thanks(2)   Quote jdadams1 Quote  Post ReplyReply Direct Link To This Post Topic: Go rate your judges
    Posted: 17 Jul 2020 at 12:21pm
Screenwriting Round 2 participants:

Don't forget -- You have the option to rate your judges with a five-point scale and written comments. Now that written feedback for Round 2 is out, I strongly encourage you to take advantage of this option if you have not already done so. I never used to rate my judges, thinking the rating tool was just a place to complain if you didn't like your score. And believe me, I've had plenty of scores I didn't like! In those cases, I just shrugged and moved onto the next comp, with little more thought about the judges or how my own lack of feedback could further enable bad behaviors. 

But with another recent NYCM competition (prose, not screenplay), I had a troubling experience with my judges. I got a low/non-advancing/non-HM score on a story the forum loved. This was hardly the first time that happened -- (I've often said the more the forum likes my story, the worse it does with judges) -- though perhaps this was the first time a story had gotten quite so much forum love and not at least gotten an HM. So, I waited for my written feedback from the judges to see what I could learn and improve. 

The feedback was worthless. Wait, writer friends, is 'less than worthless' a thing? If so, it was that! 

The judges' feedback provided no justification for such a low score. Two of the judges started their "THINGS THAT NEED IMPROVEMENT" sections with statements like "I don't think there's much here to improve..." The third judge called the writing "beautiful." Were there nuggets of constructive criticism peppered within their praise? Sure. But nothing that, in my estimation, would be likely to have a material effect on the story. 

This was very perplexing to me, as I would expect judges' written feedback to sync up with their numeric ranking. If I earned a first-place story, I'd expect a write-up that sounds like it goes with a first-place story. If I earned a 972nd-place story, I'd expect a write-up that sounds like it goes with a 972nd-place story. Instead, I got a whatever-number-comes-after-HM-place story but a write-up that sounds like it goes with a roughly 7th- or 8th-place story. (My estimate numbers, obviously!)

So, I got curious and looked at past feedback to see if I'd had these judges before. Was this a one-off anomaly or a pattern? Two of the judges were new to me; they'd never rated my work before. But the third judge? Yep, I'd had that judge before. Three previous times, in fact. And, yep, that judge did the exact same thing to me at least twice before. In two previous comps (one prose, one screenplay), they provided feedback that did not at all sync up with their score. I vaguely remember rolling my eyes at their previous feedback, as I recognized it as unhelpful, even then. But the main difference was that in the past two instances, this judge was partnered with two other judges who adeptly explained their scores in their written feedback. (This is a common practice known as "meets minimum expectations for the role you were hired to do.") So, in the prior context, it just seemed like one rando judge not doing what they're supposed to do, but overall things were running smoothly. Let me tell you: it's a very different feeling when you're stuck with three judges who all shirk their responsibilities on the same competition.

It was -- and still is -- disheartening to have this happen, particularly when we pay so much for these contests and when so many of us have had so many positive NYCM experiences. I still have no clue why these judges rated my entry as they did. As I said, the forum loved it, and it's already been picked up by a publisher. 

But the experience made me think about all the judging issues I've seen here over the years. I'm sure we all have our stories. The judge who just copied/pasted their same comments across every submission, without bothering to read them. The judge who didn't understand 'bisexuality' was a thing. And now, I can add at least one judge who repeatedly scoffs at doing the bare minimum of justifying their scores.

This is why I will now always rate my judges. 

I'm not saying that rating your judges is a cure-all to every problem you'll ever encounter. But I do think I shoulder at least some of the responsibility for letting a judge fail in their duties not once, not twice, but three times. It's hard, if not impossible, for NYCM to know there's an issue to be addressed if they never hear about it. I finally started using the judge-rating tool after the above debacle, and I used it right away for Screenwriting Round 2. (For the record, I got an HM for Screenwriting Round 2. I personally think this is TOO HIGH for that particular screenplay, which I do not consider a strong entry. But my Screenwriting Round 2 judges did an excellent job explaining why they awarded it such a score. In my rating of them, I told the judges that even though they were perhaps too generous, they at least were able to justify their reasoning, and that was much appreciated! They also provided enough constructive criticism that I feel like I know the key "misses" that could have turned my HM into a Top 5 entry for them.)

So, if you haven't done so, I encourage you to click that little link in your Feedback email and rate your judges. Every time. Bad or good -- not just the judges who aren't pulling their weight! If the only judge feedback NYCM ever gets is negative, there's no model for what good judging behavior looks like. Tell those judges who are doing a good job that they are awesome. Tell those judges who aren't... well, I'll let you come up with those words yourself! Providing feedback only takes a couple of minutes, and you'll at least never be able to say 'Ugh, why didn't I speak up the last two times this judge tried these shenanigans?"

OK, that's it! JOHN'S SOAPBOX IS OFFICIALLY CLOSED FOR THE DAY!!! :) 



Edited by jdadams1 - 17 Jul 2020 at 12:26pm
Back to Top
NilesPerry View Drop Down
NYC Midnight Black Belt
NYC Midnight Black Belt
Avatar

Joined: 10 Dec 2014
Location: Nevada
Status: Offline
Points: 2199
Post Options Post Options   Thanks (1) Thanks(1)   Quote NilesPerry Quote  Post ReplyReply Direct Link To This Post Posted: 18 Jul 2020 at 6:15pm
I couldn't agree with you more, John. In the last 2-years we've seen more and more unprofessional comments coming from judges, and it seems most of the writers let them pass, not wanting to rock the boat, so to say. In the last round alone, I've seen insults and personal attacks on writer's skills. How are these constructive? It needs to stop. Charlie needs to do some serious refresher training with his judges, not to mention retiring a few of the worst offenders. The feedback system will hopefully help him determine where he needs to do the most work.
Back to Top
SuperKarateMonkey View Drop Down
NYC Midnight Addict
NYC Midnight Addict
Avatar

Joined: 18 Jul 2018
Location: Dublin, OH
Status: Offline
Points: 1158
Post Options Post Options   Thanks (1) Thanks(1)   Quote SuperKarateMonkey Quote  Post ReplyReply Direct Link To This Post Posted: 19 Jul 2020 at 3:34pm
Bravo, John. Agree 100%.

And Niles, personal attacks? That's awful. I hope the recipients reported those judges immediately. Completely unprofessional and unnecessary.
Back to Top
dayumjam View Drop Down
NYC Midnight Regular
NYC Midnight Regular
Avatar

Joined: 14 Jun 2020
Location: California
Status: Offline
Points: 409
Post Options Post Options   Thanks (0) Thanks(0)   Quote dayumjam Quote  Post ReplyReply Direct Link To This Post Posted: 27 Jul 2020 at 7:57am
Wait, so the judges keep the same numbers throughout all of the competitions?
Back to Top
jdadams1 View Drop Down
NYC Midnight Black Belt
NYC Midnight Black Belt
Avatar

Joined: 13 Jul 2018
Status: Offline
Points: 4559
Post Options Post Options   Thanks (0) Thanks(0)   Quote jdadams1 Quote  Post ReplyReply Direct Link To This Post Posted: 27 Jul 2020 at 1:36pm
Originally posted by dayumjam dayumjam wrote:

Wait, so the judges keep the same numbers throughout all of the competitions?

Yes! 

For instance, from competition to competition, we have seen Judge 1610 bizarrely focus on title pages, usually to the exclusion of the actual story/screenplay. This is not to say the information on a title page is not important -- having a good logline and a great title are critical! But this judge, competition after competition, whether prose or screenplay, provides feedback like "Don't bold your title" or "Don't write the word 'synopsis' before your story synopsis" as the extent of their "what needs work" feedback, ignoring anything from the second page on.

My personal (least) favorite is Judge 1909, who never seems to sync up low scores with actual constructive criticism. I've seen multiple instances of that judge scoring an entry poorly but giving written feedback that makes it sound like the writer just won the Nobel Prize for literature. 

And, of course, there was Judge 1789, who was caught out by contest participants for posting the same written feedback from participant to participant. (This judge has not been seen for a few months, so my hope is they are gone for good, but I cannot say for sure.)

I do not know if there are ever isolated situations where a judge stays with NYCM but adopts a different number -- for instance, if NYCM knows a judge has a poor reputation and changes their number (versus parting ways with the judge); I assume this does not happen, but there isn't really a way we could know this.

It is a fascinating process overall, and it is always a bit cringe when you open that feedback email and see certain judge #s. 
Back to Top
Lady View Drop Down
Newbie
Newbie
Avatar

Joined: 13 Sep 2019
Location: Australia
Status: Offline
Points: 22
Post Options Post Options   Thanks (1) Thanks(1)   Quote Lady Quote  Post ReplyReply Direct Link To This Post Posted: 14 Aug 2020 at 3:04am
This might be a silly question but are we able to see individual scores from each of the judges? I’ve previously received a low score overall, only to find that two sets of comments are highly positive and the third is highly negative. This has caused me to wonder if the score from one judge could be dragging down my overall. I would love to have access to the scoring criteria and the actual scores from each judge.
Back to Top
jdadams1 View Drop Down
NYC Midnight Black Belt
NYC Midnight Black Belt
Avatar

Joined: 13 Jul 2018
Status: Offline
Points: 4559
Post Options Post Options   Thanks (1) Thanks(1)   Quote jdadams1 Quote  Post ReplyReply Direct Link To This Post Posted: 14 Aug 2020 at 7:18pm
Originally posted by Lady Lady wrote:

This might be a silly question but are we able to see individual scores from each of the judges? I’ve previously received a low score overall, only to find that two sets of comments are highly positive and the third is highly negative. This has caused me to wonder if the score from one judge could be dragging down my overall. I would love to have access to the scoring criteria and the actual scores from each judge.

I would love to have the scoring criteria/rubric and actual scores, too -- and you and I are not the only ones who would like that!

To answer your question, unfortunately, we cannot see individual scores from each of the judges. Reading your feedback, you can make some inferences about who scored you highest vs. lowest, but that will not say their actual scores. 

And keep in mind that some of the judges (particularly lately, as they've gotten worse and worse) do not provide feedback that syncs up to your rating. I've had poor/no-scoring entries that received glowing feedback and high-scoring entries that received poor feedback. To some extent, this can be explained by the overall competition in the heat: Maybe someone is Queen of the Pigs with a horrible story that's still better than everyone else's in the heat. Or maybe they're Worse of the Best with an outstanding story that still doesn't hold a candle to others in the heat. 

But when you see this sort of mismatch of bad-ranking/positive-feedback and good-ranking/negative-feedback getting more and more pronounced overtime, you have to wonder about the judges -- particularly with so many other judge issues.
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.05
Copyright ©2001-2022 Web Wiz Ltd.

This page was generated in 0.172 seconds.