Reffing Discussion

So, reffing is one of the most important activities that goes on in the league, and despite all the changes the league has gone through in the past six years, it's stayed more or less the same, except for how people get paid for doing it. There are some consistent problems with reffing, too: it's hard, so not many people can do it--there are never enough referees to accommodate all the battles that people want to have--and it takes a long time. I'd like to try to change that as much as possible: increase the referee:player ratio and encourage faster reffings, but I'm not really sure how to go about it. So, discussion. Multiple things to talk about, but each one is going to get tl;dr enough of them on their own, so we'll go one at a time.

We'll start with approving referees. The current system is pretty terrible: there's no way that a single round of reffing can tell a great deal about how a person is going to handle all the possible situations that will arise in an actual battle, and once you're approved there's no further oversight. If you mess up and get told to re-apply, all you can do is write up another reffing and wait a few more weeks, hoping that that one goes better. It's obviously not a complete failure, since we *do* have referees, but it both discourages plenty of people who might have stuck around if the system were more responsive and allows through people who don't really know what they're doing.

In general, it would be nice if there were some sort of reffing test that would allow a person to show off their ability to handle reffing a variety of different situations, while getting more immediate feedback on how they were doing. There are a couple of ways that this has been done in the past:

1. Just allow anybody to ref battles as long as the people involved agree, and if they do a good enough job, at the end they'll get approved. This is how I became a referee.

2. Set up some kind of mentorship program, where an approved referee agrees to apprentice a prospective referee. The novice picks up a battle, and as they go through and ref it, the approved referee is there to correct any mistakes they make and give feedback on the process. At the end of the battle, the referee can recommend the novice for approval, or suggest that they try again on a different battle.

3. Applicants are required to ref an extended mock battle rather than a single mock round.

All of these options are more forgiving that the current system, in that if you mess up one round, you get told what you did wrong, but you get to proceed to the next round immediately. As long as it's clear, at the end, that you've gotten the hang of things, then you're good, no matter how much you messed up in the beginning. They also give much more information about how competent a referee someone actually would be, since they have to respond to a much wider range of actual battle situations and also have to actually keep up with battlers' command schedules just as they would once they became full referees.

The problem is that these are all more labor-intensive for the approver(s) than the current system. It's a royal pain to evaluate just one round of mock reffing; these options almost all require the evaluation of many rounds, and ideally quite a bit faster than current ref approvals.

This is somewhat mitigated in the first case, where the reffing is evaluated only at the end of the match. However, it does mean that if someone who really has no clue what they're doing starts reffing something, the battlers are going to have to be constantly correcting them--if they even recognize that something's wrong! This places more of a burden on the trainers involved to police the reffings, and there's the potential for frustration if someone really isn't up to the task and is just kind of blundering along.

On the other hand, if an approved referee is in charge, then there's someone to step in, point out mistakes, and make corrections if it becomes necessary, so the battlers don't have to worry about it. The referee does, on the other hand, which returns to the issue of how much work it is to evaluate battle rounds.

The mock round system is one I'm not really a fan of, since it requires approvers to not only evaluate the prospective referee's work but come up with what they get to work on each step of the way. This does allow for the maximum range of competence testing, since you can make sure there's at least one round where there are too many conditionals, or a pokémon was ordered to use something it doesn't know, etc., which might not happen at all in a real battle. Other than that, I don't see any particular advantages to this option.

Ultimately, I don't really know which of these, if any, might be better than the current system. Thoughts on these, or alternative suggestions? In the end, how do we encourage the maximum number of people to take on the challenge of becoming referees, and see to it that as many of them as possible are successful and good at what they do? That's what I'm really getting at here.

I think I'd sort of like option #2. Maybe give the already-approved referee pay for that, as a motivator?

Can be done, sure.

Out of the options there, I'm liking the second one the most. Since there's an approved referee overseeing the reffings, the battle should be running smoothly. Perhaps the novice sends the mentor the round over PM, the mentor critiques/corrects it, and then the round is posted? In addition, it'll help with the too many battles, too few referees problem to an extent.

However, it'll take a while, and I don't know how many referees would be willing to take on the responsibility of mentorship... I, for one, am willing to try it out (only one novice, though, any more and I might keel over).

Of course there'll have to be some sort of incentive - probably money. I dunno how much work goes into evaluating rounds, though, so I'm not going to suggest how much a mentor should get. If a mentor is too busy to evaluate rounds, though, maybe we could have an emergency-mentor post, or something? Similar to calling for an e-ref.

Finally, I think it'd be best if novices stuck to short battles - 1vs1, or battles with limited rounds (e.g. Outside the Battle Arena, I think it was?), so they can potentially be approved faster and thus take more battles, and so a full 3vs3 doesn't drag on for ages.

Perhaps a written test, updated monthly(?), the answers of which are sent (to one of a number of reimbursed elite referees) over PM. Questions would be in the vein of the amount of clones a mareep would make with -2 speed, who goes first in x situation, etc. Negrek (or whoever writes the test that month, I guess) would send the correct answer(s) to the approvers, who would check answers sent in against those. There would be a certain amount of responsibility on the approvers to interpret the sent-in answers and see if they fall within the parameters of the correct response, and it might be too much of a work load for active referees, but it's an idea.

Yeah, I think Option 2 is great too but who the hell would pick me as a mentor, mostly because it spreads out the evaluation over many different people, so that Negrek doesn't have to sift through every single application even if half are submitted by people who are obviously not ready. bulbasaur's paying idea sounds good as well.

Of course, I'd only do it if I could call my apprentice "Padawan".

I rather like Blastoise's idea, but that's only for the mechanics - we also need to see if the aspiring referee's writing is up to par. Maybe we could give them the test first, then those who pass get to pick up a battle which is overseen by an approved referee? Therefore, I think the approved referee will have less to worry about in terms of mechanics.

I'd be willing to take on a Padawan. I've already been admittedly half-assedly helping Mewtini a bit so I could just step up my evaluating game and keep on chugging.

So the second option for sure, pay or no.

I like the mentor system, but I agree that you would definitely need some sort of cash incentive to keep it from being only a few generous people who approve.

I'd do the mentor thing pay or no. Preferably pay now that ref pay has been somewhat nerfed, but either way is fine with me.

I particularly like the first two options. While the second one is kind of just a modified version of the first, I can see advantages to both.

The "just go for it" method heralds the chance for a potential referee to learn from their own mistakes the most I feel, as it forces them to be more attentive to their own reffings and actually might discourage sloppy work, because they want to get a good review in the end. However, this unfortunately results in the detriment of the actual battle, because sometimes a person won't recognize their mistakes, or if they're going about something wrong. And on the subject of battlers policing round reffings, I feel like a potential ref might not listen to the battlers (depending on who both parties are), unless on of the battlers is a ref.

On the other hand, you have the mentor program, which would be more labor intensive on the already-active refs, which might discourage them to not volunteer. This said, I do think this method might actually be more productive, because it would breed more effective refs. Actually, something that occurs to me now is that we could have the mentor referee be one of the battlers, so that they're checking the thread regularly anyhow. Something else we could do with this, if the mentor isn't a battler, is have the trainee PM their mentor the round before they post it so that the comments and criticism could be more personal, and potentially less embarrassing. (though the latter shouldn't be much of an issue)

If I may, as someone who is an aspiring referee, option #2 sounds like it would be the best way to help my sub-par skills while not being too gruelingly painful.

Yeah, Omskivar's been assisting me.

As a prospective referee, I'd go for option two; I'm already doing it to an extent.

Also, I wouldn't mind reffing an extended battle. I think.

-only skimmed first post-

edit: option one sounds awesome too

#1 seems like it'll end up all over the place and cause more trouble than it's worth, though it's a good idea in theory. This might work if the person trying to become a referee passes a short test or something beforehand to prove they know what they're doing, though? Simple questions like 'how much damage and energy will X cost if X' or something.

#2 sounds good! Though the number of referees who'd be willing to take (or are able to take) novices might be low, which kinda defeats the purpose :V I'd take a novice, but given the number of mistakes I make on average in a battle that might not be the best idea.

#3 would probably only work for Novice and maaaybe Advanced battles, though. I doubt there's anyone aside from Negrek who'd be willing to go through that amount of work for Elite battles (and probably not even Negrek..?)

also ninja'd by Blastoise okay fine. Maybe you could add a short writing sample portion to the test?

I like a mix of one and two. Have the novice ref an actual battle, and a more advanced ref give them advice; at the end, Negrek decides whether the novice was actually good enough.

I like #2, personally. Partly because mentoring someone sounds fun, and partly for the reasons discussed above.

For the record I'll be back being an active referee in december.

Negrek said:

2. Set up some kind of mentorship program, where an approved referee agrees to apprentice a prospective referee. The novice picks up a battle, and as they go through and ref it, the approved referee is there to correct any mistakes they make and give feedback on the process. At the end of the battle, the referee can recommend the novice for approval, or suggest that they try again on a different battle.

Click to expand...

While I'm not a referee, I think it would be a great idea to hold short, 1vs1 battles for novice referees - this basically amounts to fast EXP to people who don't ref. EXP is something everybody wants but is also short of. I know I wouldn't mind participating in a battle that could be slightly off if it means I get EXP!

If a blend of #1 and #3 was made - not a mock battle per se, but the prospective ref would take a short battle in which one or both battlers are advanced/elite referees - that could also be an option. They'd battle as normal and would be primed to pick up on mistakes, but it's not as intensive since they're not in armed-with-red-pens mode.

On the whole, I think a mentorship program is probably best especially if you're looking for quality referees. A system where the apprentice PMs their tutor with the round reffed, along with notes and calculations that would not normally show up in a posted round, would simplify things. ("Meowth used zap cannon, but missed (8E). Slugma used flamethrower (11D/4E), Meowth is burned. [blah] Meowth's HP: 43-11-11-11-3 = 13% (capped), Slugma's HP: 94-4-6 = 84%" and so on, not that much effort. Extra notes for anomalies, or the mentor shooting off a question would clear it up.) It should probably only be advanced/elite referees acting as mentors, and if the rate remains the same as the applications at Ref HQ, it doesn't seem like it'd be that big a deal. In this case, though, battles should be of moderate length, probably at least 2v2.

A pre-test sounds like it'd be useful but ruin your nefarious plot to delegate all the work you'd have to create a slightly different one for everyone. Referees willing to take on an apprentice could probably devise their own tests, though, depending on how stringently they want to screen applicants and such; since they should still be looking over calculations throughout the application, by the end of it, the referee will know whether they're satisfied or not.

Option #2 sounds best to me. The burden of evaluating is inevitable, so, spreading it out among a several willing and at least marginally qualified people seems like the best idea.

I'm a bit doubtful on whether the mentor should get pay for it, though. On one hand it encourages people to do it, but on the other it potentially invites some people to do it half-assedly.

Metallica Fanboy said:

Option #2 sounds best to me. The burden of evaluating is inevitable, so, spreading it out among a several willing and at least marginally qualified people seems like the best idea.

I'm a bit doubtful on whether the mentor should get pay for it, though. On one hand it encourages people to do it, but on the other it potentially invites some people to do it half-assedly.

Click to expand...

the same applies to refs, too. I don't think all that many reffings are half-assed, so.