Breaking down The Command Zone Stats

[crimsonking]

I am going to regret this. I do not want to revive this thread. I am going to regret this. Why am I bringing back this trauma? I am going to regret this.

*Deep inhale*

I'm looking back through this thread to re-gather some of The Command Zone's conclusions. I'm re-doing one of the analyses, that I'm pretty sure the data set will help me solve.

But I also never answered this question.

In any case, this is a lot of work that I don't really feel like doing, but all of this is possible to do.

I'm actually more interested in your opinion on the matter: do you think this approach would be more appropriate than the one Command Zone took?

I think that any approach would have been more appropriate than what The Command Zone did.

They hired a guy (with money) to make some graphs in Microsoft Excel. Anyone with a high school education could have done that. They made the graphs, but didn't do the stats. While the multiple linear regression would take a long time, a correlation test takes a matter of seconds assuming the data is organised properly. A chi-squared test takes 5 minutes, and you can do it on the back of a napkin with a calculator nearby if you really need to. Both of these tests I did several times. Where's my money?

While I think it's positive to question the results published by the Command Zone (when having enough knowledge to do so), I think they made clear from the beginning that, while the data base they put together was as wide as possible, it still wasn't enough to be mathematically accurate.
So I don't really see the point in reiterating the same statement.
Also, I don't see the point in claiming one can do more accurate stats on the same data base that we all kind of agreed isn't accurate itself.

[Morganator 2.0]

You're completely missing the point. Aside from a few of the games, there is nothing wrong with the data set. It's enormous, its well recorded, and it represents a good sample of games played by YouTubers.

The data is fine, the stats part is not. You can't just make a bar graph and conclude that an early game Sol Ring causes you to lose more often than not. That is not good statistics. That just spreads misinformation.

[crimsonking]

I don't want to sound arrogant, but if I understand correctly we're talking about something more than 300 games.
I mean, EDHREC has more than 200,000 decks, this site has 10,000 decks, what do you pretend 300 games could represent, compared to these numbers?

[Morganator 2.0]

300 games gives us actual data on how games of commander operate. A decklist tells us nothing about what the strongest opening moves are, the importance of interaction, or which colors are the strongest in commander. It doesn't matter if you have 200 000 decks at your disposal, you need to see them in action. They need to be played against other deck, and data needs to be gathered on how they performed.

That's what the data from The Command Zone is good for. It's actual recorded games, with a wide variety of decks, playstyles, and players. Considering that for most statistical analyses a sample size of 30 or 50 is enough, 300 is an excellent sample to work with.

The downside is that it is not a sample of all possible commander games, it is just the population of commander games posted on YouTube. But there are still inferences that can be made from this. I can say with confidence that an early game Sol Ring/Mana Crypt does not correlate with losing.

Next I'm checking turn order. Does going first make you more likely to win?

[crimsonking]

I insist.
Assuming each of the 300 or so games were among 4 players, we are representing at most 1200 decks.
To give an idea, this is 1 order of magnitude less than the decks in this website; add the fact that the decks involved in said videos are often quite random (like, all commanders from the newcoming set, all tribal decks, all budget decks etc.), this by no means can be considered to represent the metagame.
I mean, you don't need statistics to know that going first or playing an early Sol Ring helps you win the game.
The very fact that they had a Sol Ring's win-rate slightly negative (which is at the very least counter-intuitive) only confirms my point: the data base were insufficient.
Their "fault" only has been to set aside this handicap and try to explain the outcome by saying people try to team up against the player who plays an early Sol Ring, which by the way ain't that out of this world as a sentence...
Moreover, I'm not undermining your calculations, I'm just saying you probably went too far with them. Only that.

[Aetherium Slinky]

I don't want to sound rude, but you're wrong, crimsonking. 300 games is a whole lot more data than 10 000 decks.

Consider this: most studies in psychology have way smaller sample sizes with far fewer variables. They don't usually struggle to achieve p<0.05, which is considered 'significant enough'. Like Morganator said this data set is very representative of a typical Youtuber game.

300 doesn't sound like a lot but many relationships between variables start to show up already at n=20. I'm very surprised if the data set is somehow ambiguous and number crunching ends up saying "we have no clue".

TL;DR A data set of n=300 is plenty.

EDIT:
Also pretty sure they did a shitty job with the stats. Sol Ring should come out on top if you look at it game-by-game instead of comparing averages to averages. You'd be surprised how weird results one can conjure up with Bad Math (™).

[WWolfe]

I could see a case to be made for a t1 Sol Ring making you an early target due to a fast start. The same could probably be said about Mana Crypt and the like. That is, assuming how fast a start it gives you and whether or not the table can catch up.

[crimsonking]

I could see a case to be made for a t1 Sol Ring making you an early target due to a fast start. The same could probably be said about Mana Crypt and the like. That is, assuming how fast a start it gives you and whether or not the table can catch up.

They also said if you don't capitalize by turn 3, turn 1 Sol Ring ain't that effective as expected, which in my opinion is totally true.
In fact, if you're going to get advantage from it on turn 4 or later, I'd say Cultivate could be a better option.
This makes sense in a casual game, but it's obviously bulshit in cEDH.
What I'm trying to explain all along is that statistics is a nice tool as long as you have a sample that represents your population.
I'm a senior software engineer and you can mark my word when I say you can statistically prove whatever you want by purposefully selecting your sample data (I'm not saying the Command Zone did that, though), especially in psychology! 

[Morganator 2.0]

I'm a senior software engineer and you can mark my word when I say you can statistically prove whatever you want by purposefully selecting your sample data (I'm not saying the Command Zone did that, though), especially in psychology!

But we're not purposefully selecting sample data, and I have been fully open on what the population is. I have data from 300+ games based on YouTubers. This sample represents the population of possible games among YouTubers.

I'm not biased one way or the other, and I'm not manipulating statistics to suit my needs/desires.

[Morganator 2.0]

Do you know what my favorite thing about Deckstats is? The stats.

That joke will never get old.

Null hypothesis: The turn order has no significant effect on the outcome of the game. Everyone has equal probability of winning no matter what turn they start on.
Alternate hypothesis: The turn order does have an effect on the outcome of the game. If the alternate turns out to be true, I will test to see which turn is the best and which is the worst.

The first part was organizing the data so it makes sense. It's divided into two categories. Observed is what actually happened. Expected is what we assume would happen if the null hypothesis is true. If what we observed does not match the null, we accept our alternate hypothesis.

|	First  |	Second  |	Third  |	Fourth  |	Total
Observed	92	70	73	68	303
Expected	75.75	75.75	75.75	75.75

Just so its clear, the 92 means that 92 of the winners went first, 70 of the winners went second, and so on. Now its very easy to jump to conclusions and say that going first is best (this is what The Command Zone did by the way). After all, the people that went first won more games than anyone else. But you always do the math, just to make sure. Chi-squared time!

I keep talking about this test, so I should probably go into more detail about it. Chi-squared checks to see if the result you got is not what you expect. If your observed values are out of the ordinary, you reject the null hypothesis. It uses this formula.



That really big letter "E" looking thing means "the sum of all". O_i is a given observed value and E_i is it's respective expected value. So in our case:



This gives us our chi-squared statistic of 4.81, which means nothing without context. Next you compare it to a Chi-squared table, and see if this chi-squared statistic exceeds the value on the table, with our given degrees of freedom and significance.

Shoot, forgot to mention that. Degrees of freedom is always n-1 when dealing with a sample. Because we have four outcomes for who is going to win (first player, second player, third player, and fourth player) we have 3 degrees of freedom. Our significance value is 0.05, which more or less means "There is a 5% chance that we got this result through luck". Here's the table.



With 3 degrees if freedom and a significance of 0.05, the value we are trying to exceed is 7.81... which we did not exceed. We only barely got past the 0.25 significance value, which is not enough to reject the null.

Conclusion: going first does not give a significant advantage over the other players. There are just way too many other factors that come into a game for turn order to be the deciding factor.

[Soren841]

You have to eliminate as many other variables as you can to determine if turn order does anything.. hence my suggestion to play games where all the decks are the same

[robort]

Conclusion: going first does not give a significant advantage over the other players. There are just way too many other factors that come into a game for turn order to be the deciding factor.

I said this to begin with awhile back 

[Morganator 2.0]

I said this to begin with awhile back 

Yep, you were right.

You have to eliminate as many other variables as you can to determine if turn order does anything.. hence my suggestion to play games where all the decks are the same

I would really like to eliminate all other variables, but I can't see any way of doing this where it represents a good population of commander games. What the analysis of The Command Zone's data shows is that turn order is not a strong factor, so we shouldn't worry too much about it.

[Soren841]

Their data definitely didn't isolate any variables

[Red_Wyrm]

I'm a senior software engineer and you can mark my word when I say you can statistically prove whatever you want by purposefully selecting your sample data (I'm not saying the Command Zone did that, though), especially in psychology!

But we're not purposefully selecting sample data, and I have been fully open on what the population is. I have data from 300+ games based on YouTubers. This sample represents the population of possible games among YouTubers.

I'm not biased one way or the other, and I'm not manipulating statistics to suit my needs/desires.

Didn't you point out at somewhere in this thread that the games were all meant to be entertaining? And as you mentioned, watching people combo out isn't exactly fun or entertaining to watch. This is why combat damage was the deciding factor in most games. Isn't that some bias in the sample as they are all youtubers?

These are YouTube videos, they are being done for entertainment. That's why combat damage was the most common win condition; watching someone win with a combo is boring, but with combat damage, it's more entertaining to see someone pull ahead. That's why Helix Pinnacle is a more common win condition than Approach of the Second Sun; it's more entertaining to watch. Those 304 games aren't a sample of all possible commander games, they are only a sample of the possible games made by YouTubers.

I mean, I know it isn't purposefully selecting the sample as it is the only decently sized sample we have. Perhaps we could all keep track of our stats as Wwolfe and Morganator are so good at doing and compile our own, non youtuber sample set.



Okay, this isn't really stats, but I have been wondering about this. Some dude or collection of dudes (and dudetts) was able to figure out exactly how many chess games were possible. They set the parameter that each player is always trying to make the best possible in order to win, so to exclude games where each player moves their knights back and forth between the same two spots, creating infinitely many games. Here is the video. Very interesting. I liked it.

https://www.youtube.com/watch?v=Km024eldY1A

Is it possible to do this with MTG? I mean we would need more parameters. For instance you can't have an infinite number of players with infinite deck sizes. Perhaps for 1v1 60 card decks and for 4 person 100 (well 99) card commander decks. I am no mathematician, so I am not 100% positive, but would it be reasonable to assume that since no actual parameter has infinite options, that there is a finite, quantifiable amount of possible games of Magic: The Gathering?