Modern polling with Civis Analytics...and how the Republican primary has changed since August

What better way to gauge where you are in any situation, especially politics. I have been hammered in the last few weeks by presenting polling data. The argument is always the same.

This poll is from a place you cannot trust
This poll is from a left/right wing think tank
This poll does not have enough people in one demographic so it is all garbage
All they talk to is people on the phone who are probably lying
It is wrong
I don't know anyone who fits this poll
Phone polls are garbage


I understand. A persons first reaction when they read something they do not agree with is to pick apart where it came from. The issue is that even if the data or poll is 95% accurate, they will hammer the 5% and say they debunked the poll.

civisanalytics.com...

The above link will take you to a very nice gif that will show the landscape of the GOP and how it has changed. It also explains how they have come up with their data.

Before delving into the results, it’s important to understand why Civis polls are unlike the surveys sponsored by news organizations or universities. Nearly all public polls try to interview adults by randomly calling telephone numbers, a technique known as random digit dialing. They adjust the responses to match the demographic characteristics of the adult population, then remove those people who say they’re not registered to vote.

This is how this certain poll has collected its data.

What we do is a bit different than what you’ve been reading:

1. We collect a lot more data. We’ve gathered an enormous amount of survey data on the GOP primary so far. Back in August, we released data from an initial poll of 757 self-identified Republicans from a total sample of 3,007 Americans (you can read more about the findings in the New York Times). Since that initial survey starting August 10, 2015, we’ve collected over 10,000 more survey responses to the GOP Primary horserace question on our ongoing weekly national tracking survey of 2,000+ respondents (if you’d like to add your own question, learn more here).

2. We’re using math that isn’t typically used in election analytics. To build the maps you’re looking at, we’re running tens of thousands of simulations using proprietary Bayesian algorithms that leverage all of that data to make estimates of survey responses in small geographies or demographic subgroups (if you’re interested in learning more, check out multilevel regression and post-stratification).

Using these methods we’re able to confidently generate estimates within 8.7 percentage points at the Congressional level which is 5.2 times better than what we could do with surveys alone.

My final question is, if polls are so worthless, why have the GOP and Democrats spent over 500 million dollars so far on this election to sway public opinion?

a reply to: matafuchs

I personally don't like the way they don't source their data. They webcrawl for data. But it's ok, they use Bayesian algorithms. Nothing can go wrong with that.

a reply to: matafuchs
Your source:

Civis Analytics conducted 40,050 live telephone interviews of adults in the United States contacted on telephones from August 10, 2015 to December 27, 2015. Among respondents of these surveys there were 11,441 self-identified Republican or lean Republican adults. These respondents were asked their candidate preference in the GOP primary. Undecided respondents are not considered as part of the analysis, map, or trend lines.

civisanalytics.com...

72% of those who were called did not self-identify as being Republican or Republican leaning. Interesting. How would you interpret this statistic?

originally posted by: matafuchs
I have been hammered in the last few weeks by presenting polling data.

You've been hammered because you're misinterpreting the polling you're citing and because you cited one obviously bogus poll.

The poll you've linked to here seems consistent with most of the other polling. But I would say the gif is a little visually deceptive in that it colors states where Trump is leading completely his color when he is only polling in the thirty some percent range. That indicates 60 to 70 percent of Republicans are supporting someone other than Trump.

Here's a more accurate visual representation of the polling over time (note the Y axis stops at 35% not 100%.)

www.realclearpolitics.com...

a reply to: DelMarvel



To address the gif it is showing who is leading in each state. That would be Trump. How else could it be interpreted?

Second, If we you use your logic, it would mean the second runner Cruz has 80% who back someone else, with 35% of them Trump supporters. This leave 50% of the voters tipping the hat to someone else. Right?

After Iowa, those 50% are going to start to back either Trump or Cruz and it will be interesting to see where the GOP voter goes. Throw in the other information about independents(that I have read could be the decision makers this election) and the Dems who will not want HIllary since she is sure to get the nod and it gets very very interesting.

These polls will all change in 30 days. It will be interesting to see as the GOP field will be thinning very quickly in January.

Thanks for your input.

a reply to: matafuchs

I wish there were a way to stop that gif from looping. They may have a unique idea, but I don't know who these 40k people are. Apparently, most aren't Republican. Were they given the choice of "If you were republican.....?"

In my entire life I have only received 2 political polling calls and have had a registered phone number since my first apartment at 20 years old. I am also on various political mailing lists, phone number updated. Have donated to various campaigns since I was 18. Still, not getting calls.

a reply to: reldra

Were they given the choice of "If you were republican.....?"

The selected group was said to self-identify as Republican or Republican leaning. So...yes.

And 28% of those called did so self-identify. They are the ones who were asked which Republican candidate they favored.

originally posted by: matafuchs
a reply to: DelMarvel



To address the gif it is showing who is leading in each state. That would be Trump. How else could it be interpreted?

Well, sure. But it is visually deceptive in that it shows a huge swath of nothing but yellow for Trump when he really only has a third of the vote according the polling. The RealClearPolitics graphic I linked to is much more accurate and easier to read. Of course, it's not broken down by states.

originally posted by: matafuchsSecond, If we you use your logic, it would mean the second runner Cruz has 80% who back someone else, with 35% of them Trump supporters. This leave 50% of the voters tipping the hat to someone else. Right?

Well, 45% actually.

I agree with your basic sentiment, but maybe not the specifics. Most people who dismiss statistics do so out of ignorance. They've never been exposed to an academic course on statistics and are not familiar with even the basic issues of statistical analysis. They dismiss statistics with a wave of their hands using one of the excuses you cited and believe they have sufficiently addressed the subject.

They have not, of course, because statistics is the single most powerful tool of modern science. It forms the basis of modern physics. Without it quantum mechanics would not exist because quantum mechanics is at its heart statistical in nature. And so is everything else, from National League baseball batting averages to political campaign polling. A well designed statistical poll reflects reality. That's why you can predict the results of a national election by polling a little over 1,000 people. The more people you poll, the tighter your margin of error becomes.

AND the more people you poll, the more expensive your poll becomes. THAT'S why political campaigns and everybody else are usually content with a poll that generates a number that is "accurate" to a 95% level instead of "accurate" to a 99% level. Put another way, a "95% accurate" poll means that there is a 1 in 20 chance (5%) that the results you get are entirely attributable to chance. Put yet another way, out of 20 polls you conduct, one of those polls is going to show a correlation where none actually exists. It will be inaccurate. In "statistics speak" your "confidence interval" or "confidence level" is .05 (5%).

To me a confidence level of 5% is pretty big, and accepting it can lead to disaster. For example, remember the Iranian "Hostage Crisis" when Carter tried to rescue the hostages from the embassy? It was an audacious plan in its specifics and when they put it together they ran a statistical analysis on what an extra helicopter would do to their chances for success. At a .05 confidence level it made no difference, so they went ahead with the mission and it failed completely. Had they used a .01 confidence level it would have become apparent that the extra helicopter was vital to their success, but they used .05.

But all the formulas used to produce these results, although they look very formidable at first glance, are usually pretty straightforward. As my old statistics professor used to tell us, "Never be afraid of a formula. The longer it is, the more work it does for you. Plug in the numbers and do the math."

And that's a problem with these guys you cited. They are telling us they use different math than most, but they're not specifying exactly what they are doing, so we can't check their results. The graph looks very spiffy, indeed, and I'm sure there's a lot of work that went into it, but any professional statistician should be asking, "Exactly what formulas are you using and exactly what do you think you are telling us?"

But they appear to be treating their data and methods as proprietary.

And If you're trying to create trust around your conclusions, this isn't helping.

a reply to: schuyler
They do seem to utilize some interesting math:
www.abovetopsecret.com...

a reply to: matafuchs

Polls are, today, entirely worthless. Period.

The concept of being able to weigh the nation's feelings on anything, has long been lost.
Today, it is all preconceived, preordained and preconcluded.

The drive is to make those who live to blend... blend according to what they see as everyone else's approval. They really don't give a flying rat's patootie what is involved, what is at risk, what might be lost or gained. It's all about being contemporary.

The last poll I gave value to was in 1974 when the question was asked: "Do you feel that Jimmy Carter should run for president?"
It ran in the Atlanta J-C. The majority - 49%, said NO.

He ran anyway. He won anyway. He failed anyway.
Yeah... now you can smell Obama, as well.

...

a reply to: schuyler

Polls can easily be influenced without raw data. Looking into Civis, they webcrawl several sources, along with random dialing to achieve results. They do source their webcrawling (I'll let you guys find that), but they don't source their dialing data. They conducted 40,500 phone interviews. How many phone calls total did they make? They cited of 40,500 calls of interviewing, 11,441 were self-identified republicans or lean republicans. What is the raw data of that? How many of the rest fell in what category? What is the age group? Where did they call? What about the other 29,059 called. What is their specific and/or preferred political stance?

These numbers do not exist anywhere in their studies. Without raw data, and other raw data to compare it to I can't count it valid. It is way too easy to make a biased poll. I hate polls, they completely contradict the scientific method.

As my old statistics professor used to tell us, "Never be afraid of a formula. The longer it is, the more work it does for you. Plug in the numbers and do the math."

You can get whatever results you want with that theory, as long as you put in the numbers you want.

originally posted by: Vector99
a reply to: schuyler

As my old statistics professor used to tell us, "Never be afraid of a formula. The longer it is, the more work it does for you. Plug in the numbers and do the math."

You can get whatever results you want with that theory, as long as you put in the numbers you want.

It's not a theory. I did not present it as a theory. As I clearly stated prior to this quote, the idea the professor was trying to get across was that statistics is not difficult and that students of statistics should not be frightened of large formulas even though they look formidable. Really, they're mostly regression equations. Somehow you have twisted that statement to suggest that you can put in numbers you want, which I never said, never implied, and never meant.

If you do that, if you do as you just suggested, then you're doing it wrong. You don't plug in the numbers you want; you plug in the numbers you have. And the numbers you have must have been obtained legitimately, i.e.: With true randomness according to the generally accepted best practices of data gathering. If you don't do that, you won't have legitimate results. Further, you won't be in business long.

You cannot dismiss statistics with a mere wave of the hand, at least not legitimately. That's an ignorant thing to do.

a reply to: schuyler

What is your professors algorithm? Is it open source? What numbers get plugged in?

originally posted by: Vector99
a reply to: schuyler

What is your professors algorithm? Is it open source? What numbers get plugged in?

An algorithm usually means a method of computation done in a computer language. A statistical formula is not, strictly speaking, an "algorithm" at all until it is is embedded and translated into a program. Statistical measurements depend on what you are trying to do. Showing a correlation between two sets of data is much different than attempting to find, say, the standard deviation of a single data set. The formulas are entirely different.

The numbers that get plugged in depend on what you are measuring. The formulas will tell you what the most common average is, the range where 68% of the averages fall, etc. If you're measuring scores on Stanford-Binet IQ test, your "answer" might be that the "standard deviation" on this particular test is 16 so that 68% of the population has an IQ between 84 and 116 and 97% of the population has an IQ between 68 and 132 in a "normal distribution" with a "confidence level" of .05, which means 19 out of 20 times this formula will work and 1 out of 20 the answers you get will be attributable to chance. That's an example of measuring variance in a population.

But if you're doing something a lot more strange, like comparing the price of housing in an area to the amount of clay in the soil you'd measure both on their own scales and see if there is a "correlation" between the two with an entirely different kind of formula. If you got a very high correlation between the two, say .9, then you could go around predicting the price of housing based on the amount of clay in the soil or conversely, predict the amount of clay in the soil based on the price of housing. That's an example of measuring the correlation between two sets of data.

The formulas themselves can be found in any statistics textbook and are therefore entirely "open source." There's nothing at all proprietary about them. You can't print those formulas here on ATS because we are limited by the ASCII character set, but here are a few for your viewing pleasure.

a reply to: schuyler

An algorithm usually means a method of computation done in a computer language. A statistical formula is not, strictly speaking, an "algorithm" at all until it is is embedded and translated into a program

So when do you start measuring data, and what do you input it to?

The numbers that get plugged in depend on what you are measuring. The formulas will tell you what the most common average is, the range where 68% of the averages fall, etc. If you're measuring scores on Stanford-Binet IQ test, your "answer" might be that the "standard deviation" on this particular test is 16 so that 68% of the population has an IQ between 84 and 116 and 97% of the population has an IQ between 68 and 132 in a "normal distribution" with a "confidence level" of .05, which means 19 out of 20 times this formula will work and 1 out of 20 the answers you get will be attributable to chance. That's an example of measuring variance in a population.

I completely understand how statistics work.

But if you're doing something a lot more strange, like comparing the price of housing in an area to the amount of clay in the soil you'd measure both on their own scales and see if there is a "correlation" between the two with an entirely different kind of formula.

Yep with ABSOLUTE data.

The formulas themselves can be found in any statistics textbook and are therefore entirely "open source."

Political polling falls out of the range of your professor I guess, these things called variables and such.

originally posted by: Vector99
a reply to: schuyler

Political polling falls out of the range of your professor I guess, these things called variables and such.

The "professor" I quoted was 45 years ago. I'm sure he is quite dead by now. Do you have a point to your questions then? If you "entirely understand" how statistics works why did you ask how statistics works? Or are you just arguing for the sake of arguing?

a reply to: schuyler

I didn't ask how they work, I asked for the algorithm used.

OP, the polls you are posting are being dismissed for two reasons.

1) Your interpretation of them has been wrong on at least a couple of threads I have seen
2) You are providing cherry picked polls - its best to take a sample of polls to get a read on the overall picture so as not to fall victim of outliers or bad sources.

Also, and this is a wider point, polls have been very unreliable in the past. They make a good talking point and can give a candidate some momentum and bragging rights, but they are often dismissed no matter who presents them. Look at what happened in Iowa for example in 2012. Santorum won and he was polling 4-5% across a number of polls just a month before the election!