Power Level of Commander decks: Empirical Data

[Morganator 2.0]

"What is the power level of your deck?" It's a common phrase you hear. A question asked by both casual and competitive players alike. No rational person wants to walk into a commander game with a deck that isn't on the same level as everyone else. The common metric is to give the deck a rating from 1 to 10 on how powerful it is. The problem; everyone's opinion is different. I've seen people who think their deck is a 10, because they've never seen a Flash Hulk or storm deck before. I've seen people give a 7 to decks that are much more powerful than they think. How well does this rating system actually work? And is there a way to unify the rating system?



You know what my favorite thing about Deckstats is? The stats.

Over the last month or two, I've been collecting data from the people in my playgroup. The task was simple; I gave them a sheet of paper that listed all of the active decks in our meta, and asked them to give each deck a rating from 1 to 10. If they didn't know the deck well enough, they would leave the entry blank.

Here are the decks, ordered from highest average rating to lowest. Names have been changed in order to keep identities a secret. Commanders with (c) next to them denote a deck at competitive commander power.

Commander	Player	Self-rate	Rating	st.dev	Number of votes
Urza, Lord High Artificer (c)	Gohan	10	8.89	0.928	9
Rashmi, Eternities Crafter (c)	Krillin	9	8.79	0.699	7
Krenko, Mob Boss	Morganator 2.0	8	8.75	0.849	14
Edric, Spymaster of Trest (c)	Morganator 2.0	10	8.61	1.147	14
Brago, King Eternal	Piccolo	9	8.33	0.816	6
Brago, King Eternal (c)	Android 18	8	8.14	1.464	7
Marath, Will of the Wild	Piccolo	9	8.00	0.000	3
Marwyn, the Nurturer (c)	Gohan	10	8.00	0.866	9
Oona, Queen of the Fae (c)	Frieza	9	7.83	0.764	3
Vilis, Broker of Blood	Krillin	7	7.80	0.837	5
Atla Palani, Nest Tender (c)	Morganator 2.0	9	7.78	1.856	9
K'rrik, Son of Yawgmoth (c)	Cell	8	7.75	1.500	4
Kruphix, God of Horizons	Krillin	8	7.63	0.479	4
Selvala, Heart of the Wilds (c)	Goku	8	7.58	1.114	6
Prime Speaker Vannifar (c)	Gohan	8	7.57	1.134	7
Jodah, Archmage Eternal	Bulma	8	7.46	1.030	13
Najeela, the Blade-Blossom (c)	Gohan	7	7.45	1.066	10
The Scarab God	Morganator 2.0	7	7.43	1.072	14
Alesha, Who Smiles at Death	Piccolo	6.5	7.33	0.577	3
Jhoira, Weatherlight Captain	Tien	7	7.25	0.957	4
Meren of Clan Nel Toth	Goku	7	7.10	0.894	5
Gaddock Teeg	Goku	7	7.08	1.114	6
Ayula, Queen Among Bears	Bulma	7	7.07	1.158	14
Karlov of the Ghost Council	Frieza	7	7.00	0.816	4
The First Sliver	Frieza	8	7.00	1.785	9
Marrow-Gnawer	Yamcha	6	7.00	2.160	4
Elsha of the Infinite (c)	Gohan	8	6.94	1.321	8
Tasigur, the Golden Fang	Piccolo	8	6.83	0.764	3
Marchesa, the Black Rose	Goku	7	6.79	1.075	7
Elenda, the Dusk Rose	Piccolo	7	6.75	0.500	4
Garna, the Bloodflame	Trunks	8	6.75	1.500	4
Yarok, the Desecrated	Gohan	8	6.73	0.876	11
Tuvasa the Sunlit	Gohan	7	6.63	1.188	8
Gonti, Lord of Luxury	Piccolo	6	6.60	0.548	5
Rhys the Redeemed	Android 17	7.5	6.60	1.673	5
Grenzo, Dungeon Warden	Goku	5	6.58	1.497	6
Kadena, Slinking Sorcerer	Gohan	6	6.57	0.787	7
Gargos, Vicious Watcher	Gohan	7	6.50	1.000	4
Marchesa, the Black Rose	Yamcha	7.5	6.50	1.291	4
Hapatra, Vizier of Poisons	Android 18	7	6.50	1.323	7
Marwyn, the Nurturer	Trunks	7	6.50	1.378	6
Krav+Regna	Chi-Chi	5	6.44	1.116	8
Nekusar, the Mindrazer	Android 17	6	6.40	0.894	5
Golos, Tireless Pilgrim	Chi-Chi	4	6.36	1.180	7
Breya, Etherium Shaper	Piccolo	9.5	6.33	0.577	3
The Scorpion God	Piccolo	5.5	6.33	0.577	3
Greven, Predator Captain	Gohan	6	6.29	1.380	7
Arcades, the Strategist	Chi-Chi	5	6.25	1.541	6
Edgar Markov	Vegeta	4	6.14	0.690	7
Anje Falkenrath (c)	Cell	9	6.00	2.449	4
Gahiji, Honored One	Yamcha	7.5	5.75	0.500	4
Momir Vig, Simic Visionary	Trunks	4	5.33	1.528	3
Meren of Clan Nel Toth	Tien	5	5.00	2.000	3
Gisela, Blade of Goldnight	Chiaotzu	6.67	4.92	1.379	12
Golos, Tireless Pilgrim	Piccolo	2	4.40	2.074	5
Yargle, Glutton of Urborg	Bulma	2	4.09	2.023	11

Pre-analysis

I did some asking around about why some people voted the way they did. A lot of these outcomes conflicted with how I perceived the strength of decks. In particular, Krenko, Mob Boss versus Edric, Spymaster of Trest. As the creator of both these decks, I am certain that Edric is more powerful, no contest. Edric can take on powerful cEDH decks, while Krenko is limited to high-power decks. So this prompted me to ask about how people rate decks. Two in particular caught my attention. Chi-Chi would rate decks entirely based on how fast they could win. She claims that she used to be in a cEDH league, so she's seen much faster decks before. The next one was Android 17. He rated decks based on how much of a threat they were to him. For example, he considered Yamcha's Marrow-Gnawer deck to be a 10, because his Rhys the Redeemed deck can't deal with an army of Rat Colony with fear.

The difference of opinions is something I've noticed. The two Brago, King Eternal decks both made it to the top 10, but only one of them is a cEDH deck... and it was rated lower than the non-cEDH deck. Now if you know how standard deviations work, you'll know that there isn't a statistically significant difference between the two Brago decks, and there also isn't a significant difference between my Edric and Krenko decks. But what is important is that some people did rate Krenko higher than Edric, and some people did rate the cEDH Brago lower than the casual Brago.

I'm not the only one that noticed this. When I told Gohan that his Urza, Lord High Artificer was the strongest deck, he quite comically said "What? No it isn't."

Error ratings
I made sure that people would rate their own decks, for two reasons. One, it gave them a baseline to compare deck powers to. Two, I wanted to see how well they could guess the power of their decks compared to everyone else. Here's the data for that. Names with (c) next to them denote players who have experience with cEDH deck-building.

Player	Number of Decks	Average deck power	Self-rate avg	Highest power	Lowest Power	Deck Range	Self-guess error	Other deck guess error	# of ratings
Android 17	2	6.50	6.75	6.60	6.40	0.20	10%	16%	10
Android 18 (c)	4	7.32	7.50	8.14	6.50	1.64	5%	11%	28
Bulma	3	6.21	5.67	7.46	4.09	3.37	20%	22%	10
Cell (c)	4	6.88	8.50	7.75	6.00	1.75	27%	16%	4
Chiaotzu	1	4.92	6.67	4.92	4.92	0.00	36%	19%	32
Chi-Chi	3	6.35	4.67	6.44	6.25	0.19	26%	24%	24
Frieza (c)	4	7.28	8.00	7.83	7.00	0.83	10%	15%	11
Gohan (c)	10	7.16	7.70	8.89	6.29	2.60	11%	18%	47
Goku (c)	8	7.03	6.80	7.58	6.58	1.00	7%	17%	29
Krillin (c)	3	8.07	8.00	8.79	7.63	1.16	6%	10%	19
Piccolo	10	6.77	6.94	8.33	4.40	3.93	20%	8%	10
Tien	3	6.13	6.00	7.25	5.00	2.25	2%	13%	26
Morganator 2.0 (c)	4	8.14	8.50	8.75	7.43	1.32	12%	16%	48
Trunks	4	6.19	6.33	6.75	5.33	1.42	17%	8%	10
Vegeta	1	6.14	4.00	6.14	6.14	0.00	35%	22%	2
Yamcha	3	6.42	7.00	7.00	5.75	1.25	20%	16%	12

Damn. I'm getting close to Deckstat's character limit. It's a little much to ask you guys to draw conclusions from this, so instead ask me questions. Ask for details about the group, and also some tests that I could do on this data. I know for sure that I'm going to test the predictive power of cEDH players versus casual players.

What I do know right now; the 1 to 10 system for rating commander decks is very inaccurate. It is a metric based entirely on personal experience, and who am I to say that I know better than everyone else? I really want to find a better way for people to rate their commander decks, similar to what Judaspriester and Dexflux were doing a while back.
https://deckstats.net/forum/index.php/topic,49777.0.html

[Slyvester12]

I think asking people to rate decks with no guidelines is mostly a waste of time (which is why most people are unhappy with the 1-10 system).
Judaspriester and Dexflux have a good idea with establishing metrics to rate, like mana base and individual card power, but they're also basing it on arbitrary levels within those metrics.
The easiest way to measure power would be to check clearly definable categories like "by what turn does the deck have a 90% chance of producing mana equal to its average CMC in all of its colors?" This would be an easy number to calculate, and it wouldn't be biased against decks the way demanding a certain amount of mana or other questions would be. Fast, low CMC decks would have low numbers, and shambling rainbow monstrosities with an average CMC of 6 would have huge numbers.
Taking a 5-10 categories like this, then playing in a league style format that makes every deck play every other deck, then checking the results to see which categories had the most predictive power and repeating this process 10 or so times should give a reasonably reliable system, in my opinion.
But that sounds like a crazy amount of work that no one wants to do. Sorry for the long post.

[robort]

All right I will ask questions. First Gohan said even comically "No it isn't" but yet why did he rated it a 10. So why the small hint of surprise that it actually was rated the strongest?

Yet the Urza also got less votes then Krenko and Edric. That brings me to Marath deck that has 3 votes with an avg of 8 and a self rating of 9. So 2 others have played against this deck and 1 rated it a 7 while the other rated it an 8. Another vote or 2 can significantly change the rating in either direction. Is it because that deck isn't brought out enough for others to give it a rating? or?

Then I wonder how many 9-10's were gave out and how many 1-3's were gave out. I am wondering how much a deck is either overrated or underrated. Why I say this is because Yargos and Golos's deck owners both gave out 2's but yet both avg over 4.

My Last question for now also pertains to there is nothing really rated under a 4. Just 3 decks are rated under 5 and 3 are rated under 6. Is there something about having a deck lower then 6 so bad?? If you were to go play against say 3 random other players and you have nothing lower than a 6 but yet none of these players have anything higher than 4 or 5 then why isn't there many lower end decks to combat this if everyone is aiming to get the majority of their decks well over 6?

[Morganator 2.0]

I like all of these questions. I'll address them one at a time.

All right I will ask questions. First Gohan said even comically "No it isn't" but yet why did he rated it a 10. So why the small hint of surprise that it actually was rated the strongest?

Gohan gave two other decks a rating of 10; his own Marwyn, the Nurturer deck and my Edric, Spymaster of Trest deck. So while he knew Urza was strong and deserved a 10 rating, seeing it as #1 overall confused him. He's confident that both Marwyn and Edric are stronger decks, a sentiment I agree with.

Yet the Urza also got less votes then Krenko and Edric. That brings me to Marath deck that has 3 votes with an avg of 8 and a self rating of 9. So 2 others have played against this deck and 1 rated it a 7 while the other rated it an 8. Another vote or 2 can significantly change the rating in either direction. Is it because that deck isn't brought out enough for others to give it a rating? or?

Damn, I forgot to bring up two important details. First off, the self-rate doesn't count towards the deck's overall rating. I did this initially because I wanted to compare how well people guessed the strength of their deck compared to what everyone else (excluding them) thought.

For that Marath deck all three people gave the deck a rating of 8, which is why it has no standard deviation. When you see a deck with a low number of votes, it's hard to draw anything conclusive from it, because your absolutely right: one or two votes could swing this deck away from the average of 8. Be wary about the decks with a low number of votes (it's why I included that part of the table). I eliminated all decks that only has 1 or 2 votes.

Then I wonder how many 9-10's were gave out and how many 1-3's were gave out. I am wondering how much a deck is either overrated or underrated. Why I say this is because Yargos and Golos's deck owners both gave out 2's but yet both avg over 4.

Here are the counts for each number.
1: 2
2: 3
3: 4
4: 17
5: 31
6: 67
7: 111
8: 72
9: 45
10: 11

As you can see, the most common vote was 7. Most decks (even the ones near the top) got at least one vote of 7. People weren't too inclinde to give out votes of less than 4, although I'm not sure why. I certainly don't think any of these decks are deserving of 3 or less, because none of them are all that bad. Even I didn't give out anything less than a 6.

This could be because people think of 7 as being a pretty average deck. If you have a strong mana base, included some interaction, and have a solid game-plan, you pretty much have a deck at a rating of 7.

Of course, saying you have an average deck is a pretty vague answer, because as we've seen, everyone has a different opinion of average. Same thing if you say your deck is pretty strong. That could mean anything.

My Last question for now also pertains to there is nothing really rated under a 4. Just 3 decks are rated under 5 and 3 are rated under 6. Is there something about having a deck lower then 6 so bad?? If you were to go play against say 3 random other players and you have nothing lower than a 6 but yet none of these players have anything higher than 4 or 5 then why isn't there many lower end decks to combat this if everyone is aiming to get the majority of their decks well over 6?

This could be something exclusive to my meta. Compared to some of the other play-group's I've been to, it's actually pretty strong for a non-cEDH group. Me and a friend (Bulma) went to a recent commander tournament hosted by an outside group. We weren't using cEDH decks, and we still won every game we played. Oops.

This also has to do with the way averaging works. You'll also notice that no deck is rated above a 9. 9 to 10 is supposed to be the score for a competitive deck, so are none of these decks competitive? Certainly not.

[Morganator 2.0]

I'm still trying to answer this question. I'm going to work on a new way of rating decks, but before that, I want to see what the differences between cEDH players and casual payers are when it comes to rating decks. I'm not going to be doing any actual stats this time around, because the sample sizes are very low so nothing conclusive can be drawn (also I'm lazy). Instead this will be purely visual.

Step 1, I cut down the number of decks. The table only includes decks that had at least 5 votes, so I removed about half of the decks that were originally here. Step 2, make two columns, one for the average score cEDH players gave to the deck, and one for the casual players. Here's the table, ordered from highest cEDH score to lowest.

Commander	Player	cEDH	Casual	cEDH votes	Casual votes	Self-rate	Original Rating
Edric, Spymaster of Trest (c)	Morganator 2.0	9.33	8.06	6	8	10	8.61
Brago, King Eternal (c)	Android 18	9.00	7.80	2	5	8	8.14
Rashmi, Eternities Crafter (c)	Krillin	8.80	8.75	5	2	9	8.79
Urza, Lord High Artificer (c)	Gohan	8.67	9.33	6	3	10	8.89
Krenko, Mob Boss	Morganator 2.0	8.33	9.06	6	8	8	8.75
Brago, King Eternal	Piccolo	8.33	8.33	3	3	9	8.33
Marwyn, the Nurturer (c)	Gohan	8.20	7.75	5	4	10	8.00
Najeela, the Blade-Blossom (c)	Gohan	8.00	7.08	4	6	7	7.45
Selvala, Heart of the Wilds (c)	Goku	8.00	7.17	3	3	8	7.58
Vilis, Broker of Blood (c)	Krillin	8.00	7.00	4	1	7	7.80
Prime Speaker Vannifar (c)	Gohan	7.75	7.33	4	3	8	7.57
The Scarab God	Morganator 2.0	7.58	7.31	6	8	7	7.43
Hapatra, Vizier of Poisons	Android 18	7.50	6.10	2	5	7	6.50
Atla Palani, Nest Tender (c)	Morganator 2.0	7.40	8.25	5	4	9	7.78
Gaddock Teeg	Goku	7.33	6.83	3	3	7	7.08
Marwyn, the Nurturer	Trunks	7.33	5.67	3	3	7	6.50
Jodah, Archmage Eternal	Bulma	7.17	7.71	6	7	8	7.46
Yarok, the Desecrated	Gohan	7.08	6.30	6	5	8	6.73
Rhys the Redeemed	Andriod 17	7.00	6.00	3	2	7.5	6.60
Meren of Clan Nel Toth	Goku	7.00	7.25	3	2	7	7.10
Elsha of the Infinite (c)	Gohan	6.70	7.33	5	3	8	6.94
Grenzo, Dungeon Warden	Goku	6.67	6.50	3	3	5	6.58
The First Sliver	Frieza	6.58	7.83	6	3	8	7.00
Ayula, Queen Among Bears	Bulma	6.57	7.57	7	7	7	7.07
Gonti, Lord of Luxury	Piccolo	6.50	7.00	4	1	6	6.60
Tuvasa the Sunlit	Gohan	6.40	7.00	5	3	7	6.63
Edgar Markov	Vegeta	6.33	6.00	3	4	4	6.14
Kadena, Slinking Sorcerer	Gohan	6.25	7.00	4	3	6	6.57
Krav+Regna	Chi-Chi	6.20	6.83	5	3	5	6.44
Arcades, the Strategist	Chi-Chi	6.13	6.50	4	2	5	6.25
Golos, Tireless Pilgrim	Chi-Chi	6.10	7.00	5	2	4	6.36
Greven, Predator Captain	Gohan	6.00	6.67	4	3	6	6.29
Marchesa, the Black Rose	Goku	6.00	7.38	3	4	7	6.79
Nekusar, the Mindrazer	Andriod 17	6.00	7.00	3	2	6	6.40
Golos, Tireless Pilgrim	Piccolo	5.67	2.50	3	2	2	4.40
Gisela, Blade of Goldnight	Chiaotzu	5.17	4.67	6	6	6.67	4.92
Yargle, Glutton of Urborg	Bulma	4.00	4.20	6	5	2	4.09

I'm a little more happy with how this table looks than the last one, and not just because it was my deck that made the top spot. The competitive Brago deck is ranked higher than the non-competitive. I'm still not sure about some things though. Like why is Elsha ranked so low?

But here's the real question, and it's one that I still can't answer:

Who is right about the strength of decks?

Are cEDH players better at rating decks? Are casual players better? Does it even matter if everyone has a different view?

I still hate that this 1 to 10 rating system is based entirely on past experience. There has got to be a way to measure the strength of a deck where it isn't based on opinion. I'd really like "What turn can you win on?" to be a good way, but it just isn't. From my experience people tend to exaggerate how fast their deck can win, and not all decks are based on speed. There is also the issue that you don't really count how many turns go by in a casual game. After 6 you kinda stop counting.

Alright, brainstorming time. I'm determined to find a better rating system.

Who's with me?

[dexflux]

Rating complex structures is quite interesting.

As you already now, I am a fan of rating decks according to metrics. As Slyvester12 said, we used arbitrary levels for those metrics (which is mainly because the people we play with can't be bothered to calculate relevant numbers for their decks and because it's fairly simple), but we could do better.

The easiest way to measure power would be to check clearly definable categories like "by what turn does the deck have a 90% chance of producing mana equal to its average CMC in all of its colors?" This would be an easy number to calculate, and it wouldn't be biased against decks the way demanding a certain amount of mana or other questions would be. Fast, low CMC decks would have low numbers, and shambling rainbow monstrosities with an average CMC of 6 would have huge numbers.

That is what I would try. The first problem is obviously to define the metrics properly and map the numbers that the calculations spit out to a more readable format.

Essentially, have a function that takes metrics data and spits out a number that then can be mapped to human-readable [1..5].

"by what turn does the deck have a 90% chance of producing mana equal to its average CMC in all of its colors?"

I'd take the data the common cEDH decklists produce with such a calculation as a ceiling, hence mapping it to 5 and going down from there.