Breaking down The Command Zone Stats

[WWolfe]

You're right. I was just saying that particular episode wasn't what I was thinking of. I could swear I remember him using a Demonic Tutor recently.

[robort]

LOL I said if I remember correctly but however he did say something about tutors. The exact quote of what Josh says I am not fully sure of

[Red_Wyrm]

LOL I said if I remember correctly but however he did say something about tutors. The exact quote of what Josh says I am not fully sure of

It sounds like something he would say, to not use tutors, but perhaps he was giving advice on how to power down your deck because some people have the problem of their decks being too powerful for their meta. So one way to stop that would be less tutors to make the deck more random.

[Morganator 2.0]

Now to talk about my favorite thing Deckstats has to offer: the stats!

I would like to see the stats that The Command Zone has (I refuse to dig through YouTube videos looking for it). What was the value of the correlation coefficient (>0.70 is a fairly good correlation), what was their sample size, what were their error margins, and how did they measure which decks did better?

Ok, so this link leads to the analyst's own page: https://public.tableau.com/profile/andrew.greene#!/vizhome/CommanderStatistics/Title
Andrew Greene is a statistician the Command Zone hired for this purpose. They manually sifted through 300 commander games and looked at various variables and correlations in the data. The only way to get all the conclusions is to actually watch the videos (sorry), but the raw data dump is linked on Andrew's page. Here's the link: https://docs.google.com/spreadsheets/d/10c7mflt6FJ253rtKeFAbQhPT282JDzJ6BcwrOV5MIzo/edit#gid=0

Just by looking at the data dump you can see they tracked a lot of things. Their sample is obviously a little skewed because all the games as far as I understood were recorded games, which means they were played by a small group of people. This is not a cumulative archive of all games played since the beginning of time but the good thing is that we can go back and verify every data point from the recordings.

The way they did the conclusions were mostly in terms of win percentages. This means that in four-way games their base line was always 25% and they looked at things like land count in the deck, land count at the end of the game, T1 Sol Ring, colors, deck cost and stuff like that. Then the charts would display an increase of decrease in percentage points compared to the 25% base line.

TL;DR n=316

I'm moving the conversation back here.

First off, n actually equals 304. Some of the games had incomplete data, and some of the games didn't have a winner. I excluded those games because if we're looking at win percentages, then the games should have a clear winner.

Second, I did download the data set and analysed it myself (next time I'm getting paid). The issue wasn't that the data was shown incorrectly; all of the graphs were accurate, the issue is that they didn't present the stats. In the parts of the video that I did watch, they would say something like "You can see the correlation; an early game Sol Ring makes you less likely to win." This isn't true though. I actually did the correlation and it wasn't strong enough to conclude that an early game Sol Ring hurts your chances of winning (correlation coefficient= -0.019).

So while they presented the data, they didn't present the statistics.

We cannot reject any of the null hypotheses, so we must accept them.

[Aetherium Slinky]

I'm moving the conversation back here.

Uhh... I have no idea what happened. I just went through threads that had "new" replies (i.e. ones I hadn't seen yet). I'm pretty sure I've never seen this thread and yet I'm somehow quoted in the first message.

TL;DR: Seems like you've addressed everything already and now I just look dumb for not noticing there's a thread I'm apparently a participant in?

[Red_Wyrm]

Now to talk about my favorite thing Deckstats has to offer: the stats!

I would like to see the stats that The Command Zone has (I refuse to dig through YouTube videos looking for it). What was the value of the correlation coefficient (>0.70 is a fairly good correlation), what was their sample size, what were their error margins, and how did they measure which decks did better?

Ok, so this link leads to the analyst's own page: https://public.tableau.com/profile/andrew.greene#!/vizhome/CommanderStatistics/Title
Andrew Greene is a statistician the Command Zone hired for this purpose. They manually sifted through 300 commander games and looked at various variables and correlations in the data. The only way to get all the conclusions is to actually watch the videos (sorry), but the raw data dump is linked on Andrew's page. Here's the link: https://docs.google.com/spreadsheets/d/10c7mflt6FJ253rtKeFAbQhPT282JDzJ6BcwrOV5MIzo/edit#gid=0

Just by looking at the data dump you can see they tracked a lot of things. Their sample is obviously a little skewed because all the games as far as I understood were recorded games, which means they were played by a small group of people. This is not a cumulative archive of all games played since the beginning of time but the good thing is that we can go back and verify every data point from the recordings.

The way they did the conclusions were mostly in terms of win percentages. This means that in four-way games their base line was always 25% and they looked at things like land count in the deck, land count at the end of the game, T1 Sol Ring, colors, deck cost and stuff like that. Then the charts would display an increase of decrease in percentage points compared to the 25% base line.

TL;DR n=316

I'm moving the conversation back here.

First off, n actually equals 304. Some of the games had incomplete data, and some of the games didn't have a winner. I excluded those games because if we're looking at win percentages, then the games should have a clear winner.

Second, I did download the data set and analysed it myself (next time I'm getting paid). The issue wasn't that the data was shown incorrectly; all of the graphs were accurate, the issue is that they didn't present the stats. In the parts of the video that I did watch, they would say something like "You can see the correlation; an early game Sol Ring makes you less likely to win." This isn't true though. I actually did the correlation and it wasn't strong enough to conclude that an early game Sol Ring hurts your chances of winning (correlation coefficient= -0.019).

So while they presented the data, they didn't present the statistics.

We cannot reject any of the null hypotheses, so we must accept them.

Forgive me. One quote says n=316 and you said its 304. I didnt go look for myself. What happened to those other 12 games? You said there was no clear winner? But why would you exclude them? If I play 1 game with a deck and nobody wins, I lose, right? So my win percentage is 0% with that deck surely this is the case for those 12 games with no winner?

[Morganator 2.0]

I don't actually know why those games didn't have a clear winner. If I had to guess it would be because some kind of time limit was being enforced. I really didn't want to re-watch those videos if it wasn't necessary. I've been to finance meetings that are more entertaining than those videos.

The reason why I excluded them is because we're looking at win rates and which factors cause decks to win. When this is the case, we want there to be a clear winner. Also, it messes with the Chi-squared tests. You see normally, Chi-squared assumes equal variance of outcomes: the null is that you are just as likely to flip heads as you are tails. The chi-squared tests I used assumed that there were two outcomes: win, which is one person, or lose, which is three. Not 50-50. If there is a possibility of draw (all three losing), which adds an extra factor to the Chi-squared distribution, which changes the null that we're testing. If we wanted to test the odds of a deck winning/losing/everyone losing, I would include them. But getting rid of 12 samples, while still keeping 304 good ones, means that the sample size is still strong.

What I should have maybe done is actually remove some more for the color analysis. There were 7 colorless decks, with 3 wins. That's not a good enough sample size to justify that colorless is one of the most powerful color combinations.

[Red_Wyrm]

I don't actually know why those games didn't have a clear winner. If I had to guess it would be because some kind of time limit was being enforced. I really didn't want to re-watch those videos if it wasn't necessary. I've been to finance meetings that are more entertaining than those videos.

The reason why I excluded them is because we're looking at win rates and which factors cause decks to win. When this is the case, we want there to be a clear winner. Also, it messes with the Chi-squared tests. You see normally, Chi-squared assumes equal variance of outcomes: the null is that you are just as likely to flip heads as you are tails. The chi-squared tests I used assumed that there were two outcomes: win, which is one person, or lose, which is three. Not 50-50. If there is a possibility of draw (all three losing), which adds an extra factor to the Chi-squared distribution, which changes the null that we're testing. If we wanted to test the odds of a deck winning/losing/everyone losing, I would include them. But getting rid of 12 samples, while still keeping 304 good ones, means that the sample size is still strong.

What I should have maybe done is actually remove some more for the color analysis. There were 7 colorless decks, with 3 wins. That's not a good enough sample size to justify that colorless is one of the most powerful color combinations.

Colorless has a 43% win %. It is clearly the most powerful.

One other question (which may foster into several more, we'll see.)
How do you account for the same color being played by multiple people? I mean if we have 1 five color deck, one Boros, one Jund and one Grixis, red is guaranteed to win that match, but we all know having red in your deck d oesn't guarantee you will win.

[Morganator 2.0]

I was worried someone might ask this. A little while back I figured out how they might have determined that white decreases your win chance by 1%, red increases it by 3%, and so on.

Question for those that watched the video. At any point, did someone say "Multiple linear regression"?

A multiple linear regression is an equation with multiple variables, that gives a weight to each variable. As a real-world example, this is one of the ways to determine wage gaps. You make "yearly gross earnings" the variable that the equation solves for, and then you can add the variables that would affect your pay. Some of these variables could be "time spent with the company", "level of education", "position in job", and so on. The cool thing about a multiple linear regression, is that you can tack on anything else you think might affect your pay. "Race", "gender", "sexual orientation". Each variable is then given a weight (how much it affects the pay), an error, and a significance value. So if "race" has a high weight, and has significance, that means that your employer does descriminate between race and pays employees differently because of it.

If we were to apply this to our stats, we would make "win chance" the dependent variable that we're solving for, and add each color as one of the variables. The equation might look something like this:

Win chance= 0.25+0.08x_u+0.08x_b+0.08x_g+0.03x_r-0.01x_w+error

Each instance of "x" represents one of the colors (denoted by the subscipt), and it would either be 1 (if the deck has the color) or 0 (if it doesn't).

So I could do this... but I kinda don't want to. As you can imagine, this would take a lot of work.

What I want to know first is if The Command Zone did this. I only ever saw that bar graph, which is not a good substitute for a multiple linear regression. Did the stats guy that they hired do anything like this?

[Red_Wyrm]

I was worried someone might ask this. A little while back I figured out how they might have determined that white decreases your win chance by 1%, red increases it by 3%, and so on.

Question for those that watched the video. At any point, did someone say "Multiple linear regression"?

A multiple linear regression is an equation with multiple variables, that gives a weight to each variable. As a real-world example, this is one of the ways to determine wage gaps. You make "yearly gross earnings" the variable that the equation solves for, and then you can add the variables that would affect your pay. Some of these variables could be "time spent with the company", "level of education", "position in job", and so on. The cool thing about a multiple linear regression, is that you can tack on anything else you think might affect your pay. "Race", "gender", "sexual orientation". Each variable is then given a weight (how much it affects the pay), an error, and a significance value. So if "race" has a high weight, and has significance, that means that your employer does descriminate between race and pays employees differently because of it.

If we were to apply this to our stats, we would make "win chance" the dependent variable that we're solving for, and add each color as one of the variables. The equation might look something like this:

Win chance= 0.25+0.08x_u+0.08x_b+0.08x_g+0.03x_r-0.01x_w+error

Each instance of "x" represents one of the colors (denoted by the subscipt), and it would either be 1 (if the deck has the color) or 0 (if it doesn't).

So I could do this... but I kinda don't want to. As you can imagine, this would take a lot of work.

What I want to know first is if The Command Zone did this. I only ever saw that bar graph, which is not a good substitute for a multiple linear regression. Did the stats guy that they hired do anything like this?

I don't quite remember. I watched the videos upon release which was what? October or something. Most of the videos were their theorizing why the data looks like this. For instance, they concluded that sol ring in the first 3 turns decreases your chances to win (We know this isn't accurate now), but then they spent like 8 minutes talking about how playgroups may be self correcting for the early advantage of sol ring by attacking that player. They would give examples like this, "Hey, you have a soul ring, I'll attack you with my little 1/1. Any of the data analysis they talk about is probably done in the first 10 minutes of the first video as they explain everything, and I don't remember hearing multiple linear regression. I doubt they mentioned it because the majority of their audience has no clue what it means, while some have only a faint idea, and there is the .5% that are statisticians like apparently you, Morganator .

[Aetherium Slinky]

This isn't really scientific but I've watched a few of the games and I have a gut feeling like some of the players play very differently compared to many play groups. They focus on politics as if they were role playing: hold grudges and punish people. I haven't experienced this in our play group (insignificant sample size ) but I can't also imagine that to be unique. Command Zone speculated on the idea that having a Sol Ring early would make them a generic "threat" that people respond to. This isn't true unless, of course, you play on YouTube and your ad revenue depends on the amount of drama and tension you can generate. Thus it would only make sense that they're role playing their games.

Josh sometimes does point out that some of his moves were just 'the best play' but the Professor for example often makes decision based on his idea of justice and his gut feeling. The Professor has stated multiple times that his objective is not to win, but to play an interesting game. There could be an argument made here that his games should be excluded from the sample how is something going to affect win rate is an insane question to ask if some people aren't even necessarily trying to win in the sense that they're willing to sacrifice the best possible play for more interesting and engaging play.

***

I thought of something we haven't discussed yet: ratios of colours in a deck.

Instead of looking at decks containing colours X, Y or Z we could look at the cards' colour identity. I don't know if it would be best to exclude lands or not.

Here's an extreme example: Samut, Voice of Dissent is a Gruul card with white tacked onto it. You could, in theory, play a Gruul deck with no way to produce white mana and never use the ability. This wouldn't be evident from the colour identity but it would be very evident from the individual cards being played; the deck would show up as a Naya deck even though it's closer to a Gruul deck.

This would also yield a larger sample size for colourless since every Sol Ring, every Burnished Hart, every Scour from Existence and every Lightning Greaves would actually count towards the colourless sample. You could then compile the individual cards into pools of 6 colours or 32 colour combinations.

I think this would better reflect the power of each colour, in a commander deck.

Most of the matches do contain the link to the deck list so in theory this could be done. I'm not actually asking anyone to do it, just toying around with the idea.

[Morganator 2.0]

This is also something that the multiple linear regression could handle. In the example I gave, the examples were categorical (the presence/absence of a colour) but I could make them discreet, by changing the variable to be "# of red cards" or "# of colorless cards", and so on.

The only other issue is that for the multiple linear regression to work, it assumes that each variable is independent. This means that having white in the deck doesn't affect the performance of the red cards. But sometimes it does, especially with Boros. Anyone ever notice that there are more good mono-red commanders than Boros commanders? To account for this, I would have to add a variable for color pairs as well, which turns a 5 variable equation into a 32 variable equation.

In any case, this is a lot of work that I don't really feel like doing, but all of this is possible to do.

[Aetherium Slinky]

In any case, this is a lot of work that I don't really feel like doing, but all of this is possible to do.

I'm actually more interested in your opinion on the matter: do you think this approach would be more appropriate than the one Command Zone took?

The only other issue is that for the multiple linear regression to work, it assumes that each variable is independent. This means that having white in the deck doesn't affect the performance of the red cards. But sometimes it does, especially with Boros. Anyone ever notice that there are more good mono-red commanders than Boros commanders?

This is a very interesting point. Many colours struggle to draw cards or ramp efficiently, which means that cards like Mind Stone are probably more powerful in a blue artifact synergy or Boros deck that struggles to generate extra mana while for example green could easily substitute it with Rampant Growth or Llanowar Elves. Sometimes you may not have a choice if Mind's Eye is the only draw engine that is available, in which case it's not powerful, it's irreplaceable.

Perhaps the question itself is insane. X is the most powerful colour...in a vacuum? I'm pretty sure a generic mono green control deck would be pretty bad. Or a mono white artifact combo. Either way, the answer we're looking for is probably not very descriptive of the actual problem we're trying to solve. For single cards, sure. You can probably measure the impact of Sol Ring and maybe roughly say that it's most effective in artifact combo and least effective in green decks that ramp fast anyway - and that we can generate some numbers that mean something. But a part of the insanity of the question is that Boros shouldn't be worse than mono red, because the latter is a subset of the former.

[Morganator 2.0]

I am going to regret this. I do not want to revive this thread. I am going to regret this. Why am I bringing back this trauma? I am going to regret this.

*Deep inhale*

I'm looking back through this thread to re-gather some of The Command Zone's conclusions. I'm re-doing one of the analyses, that I'm pretty sure the data set will help me solve.

But I also never answered this question.

In any case, this is a lot of work that I don't really feel like doing, but all of this is possible to do.

I'm actually more interested in your opinion on the matter: do you think this approach would be more appropriate than the one Command Zone took?

I think that any approach would have been more appropriate than what The Command Zone did.

They hired a guy (with money) to make some graphs in Microsoft Excel. Anyone with a high school education could have done that. They made the graphs, but didn't do the stats. While the multiple linear regression would take a long time, a correlation test takes a matter of seconds assuming the data is organised properly. A chi-squared test takes 5 minutes, and you can do it on the back of a napkin with a calculator nearby if you really need to. Both of these tests I did several times. Where's my money?

[robort]

I am going to regret this. I do not want to revive this thread. I am going to regret this. Why am I bringing back this trauma? I am going to regret this.

*Deep inhale*

I'm looking back through this thread to re-gather some of The Command Zone's conclusions. I'm re-doing one of the analyses, that I'm pretty sure the data set will help me solve.

But I also never answered this question.

In any case, this is a lot of work that I don't really feel like doing, but all of this is possible to do.

I'm actually more interested in your opinion on the matter: do you think this approach would be more appropriate than the one Command Zone took?

I think that any approach would have been more appropriate than what The Command Zone did.

They hired a guy (with money) to make some graphs in Microsoft Excel. Anyone with a high school education could have done that. They made the graphs, but didn't do the stats. While the multiple linear regression would take a long time, a correlation test takes a matter of seconds assuming the data is organised properly. A chi-squared test takes 5 minutes, and you can do it on the back of a napkin with a calculator nearby if you really need to. Both of these tests I did several times. Where's my money?

Check is in the mail