Astrology Project Discussion - "Solutions"

Data (35 points): A brief description of your experiment with the inclusion of all requested data was worth 30. To get more points, you needed to clearly mention that you ensured the subjects hadn't seen these horoscopes before, or discuss ways in which you masked which sign was which. A significant improvement on the experiment as outlined was required for 35 points (components of improvement might be: choosing subjects carefully, not letting subjects know what other subjects, testing several different horoscopes...)
Analysis and Conclusions (30 points): To get full marks here, you needed to draw sensible conclusions, notice and discuss any trends, and remark on the limitations of this experiment - in particular the need for a larger sample of people! You can't conclude that astrology is completely bogus based on 5 people's guesses on a single day! You can conclude that your own experiment doesn't support astrology. Some of the limitations: not enough people tested, only testing one set of horoscope predictions, only testing a short period of time, horoscope predictions are frequently vague and difficult to assess, only testing generalized horoscopes and not ones made for a specific person, not testing the "personality profile" aspects of horoscopes.
Discussion and Further Analysis (10 points): You needed to clearly answer the questions I asked.
Relating to Astronomy (15 points): Here, a reasonably correct answer was required!
- Taurus is high overhead in late November. This doesn't match the Nov 25 sign of Sagittarius, but I wouldn't expect it to, since I defined a "star sign" as the constellation in which the sun is located on your birth date (which will NOT be overhead at night on that date!)
- The Sun is in Scorpio/Libra at the end of November. I would have expected Sagittarius, based on the definition of a star sign, but it isn't.
- Precession explains this discrepancy. The earth's axis is tilted relative to the orbit, but circles like a top every 26,000 years due to gravitational effects of the Sun and Moon. The astrological system was outlined 2000 years ago, and so your "star signs" are based on what constellation the sun would have been on your birthday 2000 years ago, which is not the same as it is today.
- The stars in the constellation of the zodiac are not physically related to one another. They aren't closer to one another than other stars, since even within a constellation the stars can be at vastly differing distances - it is only their direction that is similar. They aren't closer to us than other stars. They just happen to be the background stars through which the sun and planets APPEAR to move!
- The gravitational force that the stars and planets exert on you is TINY because they are so far away! In fact, a doctor present at your birth exerted a larger gravitational force on you than did Mars. Remember from our scale of the solar system exercise that the Sun is closer than Jupiter, and much more massive. But you don't feel your weight change when the sun is overhead compared to at midnight! Although the moon does exert a tidal force on the Earth, the tidal force on you (due to the Moon) is incredibly tiny. This is because you are not very tall, and so the difference in the moon's gravity at your head and at your feet is not very different.
- In the simplest view (not the only one), horoscopes (predictions) that were cast before the discovery of Uranus (1781) didn't include the effects of these 3 planets. If these planets are supposed to affect horoscopes now, they should have affected them then. So the horoscopes that were cast before their discovery must have been invalid by today's standards.
Extending Results (10 points): You did NOT need to do the second experiment - I just wanted you to predict the results! Below I discuss the results of a similar experiment. To get full credit you needed to justify your non-science/pseudo-science choice (no one chose science). A pseudo-science isn't just something that is non-scientific, it is something that in some way claims to be or at first appears to be scientific, but doesn't measure up under careful scientific testing.

Further Comments on the Projects

This project attempted to test the accuracy of daily "prediction-type" horoscopes. I was trying to get you to think about how to perform a scientific study, and the sorts of questions scientists must ask themselves when they test a hypothesis. What are the possible errors and sources of bias?

Many of you pointed out that there are several aspects of astrology that could be tested, and this experiment only tested the "general daily horoscope" aspect. This is true. Below I discuss the results of other experiments that have been done to test astrology.

What were the results of YOUR experiments?

Many of you chose to test the same subjects for several days, or with more than one set of horoscopes. Together, you made 225 separate tests of astrology (225 people-horoscope combinations), and the correct match was chosen 41 times. If horoscopes were accurate, you'd expect most people to guess correctly most of the time. In random chance, you'd expect 1/12, or just under 10% to guess correctly. Here, about 18% chose correctly. This is far closer to the expectation of random chance than the expectation of accurate horoscopes! Possible reasons for the matches to be more frequent than expected by random chance: order of horoscopes not always randomized, so guesses can be based on typical location within sequence; some of you included only the signs relevant to your group of subjects, thereby increasing the likelihood that they would randomly choose the correct one; those familiar with astrology can often pick out their own horoscopes based on key phrases.

The main limitation on the individual experiment is the number of people you could test. Just as flipping a coin 5 times and getting 5 heads might simply be due to random chance, so might the results of the choices of 5 people. Some of you attempted to improve your statistics by including more people, and testing for more days. In many cases, a person who chose correctly one day, failed to do so on subsequent days!

Several of you pointed out that these generalized horoscopes are extremely vague, so that almost any sign would fit. The horoscopes also tended to give advice, rather than predictions (eg. "Be careful of money matters" and not "You will come into a large sum of money"). Some of you correctly noted that advice like this is very hard to prove or disprove in a scientific manner.

Some of you noticed trends in your data. People living together had similar weeks and chose the same sign. Or your subjects were all students, and most of the horoscopes had language and advice that was more relevant to people in the working world, so the students gravitated to the horoscope without such language. If one horoscope was more "attractive" than the others, many people might be drawn to it (regardless of sign).

Western astrologers don't take precession into account. Last year it was pointed out to me that in India, astrologers DO include precession in their calculations. And that there are many many styles of casting horoscopes. You could argue that if astrologers can't agree with each other, they can't be right! This experiment only tested (and apparently invalidated) typical newspaper and online horoscopes. To scientifically rule out all of astrology, you need to perform many more experiments!

Improving the Experiment

The experiment as outlined has several problems. The most obvious is that it includes so few subjects that random chance can skew the results. Another potential problem is that of BIAS. One person has pointed out that a person familiar with how horoscopes are done might be able to pick theirs out just based on the way things are phrased. The choice would not be based on how well it matched their day (what we're trying to test), but rather on their knowledge of how the horoscope was created. A more subtle version of this is that subjects familiar with the personality types of assigned to their astrological sign might try to choose the horoscope that included the traits of their personality type. Scientists must always be careful to design experiments without biases such as this. Sometimes they fail, and realize the bias that pollutes their results only AFTER they have finished the experiment.

On Homework #1, I asked you to design your own test of astrology. I received several suggestions. One person said they'd tell their subjects they were testing how the colour of their shirt affects their day, and have them write down information about their day. This is one way of removing bias. If the subjects don't even know astrology is being tested, they won't lie in their daily reports to make them fit the newspaper's horoscopes. Sometimes in science, REMOVING BIAS can mean lying to your subjects!

Some students have commented that professional astrologers would never claim that these general weekly or daily columns are accurate, but that horoscopes should be cast for individuals based on the times of their births. One went so far as to find an online source for such horoscopes, and to perform the experiment using these. They didn't appear to be any more accurate than the more general horoscopes, but the test didn't include enough people to be sure! Other tests have, and I discuss one of those below.

Extended Astrology Discussion

I promised that I would tell you about the results of some other experiments. Here's some examples (descriptions and references taken from Studies on Astrology (http://www.skepsis.no/english/subject/astrology/studies.html) and Your Astrology Defense Kit (http://www.astrosociety.org/education/astro/act3/astrologyprint.html)). There are many others.

116 adults filled out California Personality Index surveys and provided natal data. One set of natal data and the results of three personality surveys (one of which was for the same person as the natal data) were given to an astrologer who was to interpret the natal data and determine which of the three CPI results belonged to the same subject as the natal data. The San Francisco chapter of the National Council for Geocosmic Research recommended the 28 astrologers who took part. They approved the procedure in advance and predicted that they would select the correct CPI profiles in more that 50 per cent of the trials. Out of 116 trials, the astrologers chose the correct CPI 34 per cent of the time. This agrees with the random chance prediction of 1 of 3 trails producing a correct choice. (Shawn Carlson. A Double-blind Test of Astrology. Nature, 318, 419, 1985)

In another study in which astrologers attempted to match astrological birth charts to case files, astrologers did no better than chance or than a nonastrologer control subject at matching the birth charts to the personal data; this result was independent of astrologers' confidence ratings for their predicted matches. Astrologers also failed to agree with one another's predictions. (McGrew, John H., McFall, Richard M. A Scientific Inquiry Into the Validity of Astrology. Journal of Scientific Exploration, 4, 75-83, 1990)

Found no correlation between occupation, medical problems, height, longevity, and the zodiac signs containing Mercury, Venus, Mars, and Jupiter at the time of birth. (Barth, J., Bennet, J. Leonardo, 7, 235, 1974)

Psychologist Bernard Silverman of Michigan State University looked at the birth dates of 2,978 couples who were getting married and 478 who were getting divorced in the state of Michigan. Most astrologers claim they can at least predict which astrological signs will be compatible or incompatible when it comes to personal relationships. Silverman compared such predictions to the actual records and found no correlations. For example "incompatibly signed" men and women got married as frequently as "compatibly signed" ones.

Out of 3,000 specific predictions made by professional astrologers that appeared in publications over a 5 year period, only about 10% came to pass.

Don't be convinced that a "sophisticated" brand of astrology that takes ALL of the motions into account is more "scientific" than generic astrology. As you can see, more sophisticated tests of more sophisticated brands of astrology also match the expectations of random chance.

As for the test of giving many people the same horoscope, and asking them if it is theirs, it has been done. Researcher Michel Gauquelin sent 150 people a horoscope that was created for a French mass murderer, and 94% said that the horoscope pertained to them! Most people will believe it is their own horoscope, whether it is or not. Psychology can play a significant role here. People want to believe in astrology, and so stretch the horoscopes or personality profiles to match their own experience. It helps that these are frequently written in a vague manner!

Another study also showed how much people want guidance from astrology. Astrological readings were done for a groups of subjects. The content of some of the readings were reversed (changed phrases describing the subject to their opposites). Subjects reported that both the reversed and normal readings applied 95 per cent of the time. (Dean, Geoffrey, trying to find reference)

My goal here is not to force you to believe that astrology is invalid, but rather to point out that there is no scientific evidence for it! It is not a science! And to point out that although it is "based" on stars and planets, there are several things in astrology as it is usually practiced that don't match up with the expectations of astronomy.

In my mind, astrology is a pseudo-science, because it claims to be based on complex calculations and observations of the motions of heavenly bodies, but it doesn't stand up to rigorous scientific testing. While I personally don't believe in astrology, I do believe that there are other kinds of truth than factual/scientific truth - intuition, story, myth, and faith can all teach us something. Astrology is probably telling us more about our culture and psychology than the details of our day-to-day lives.

To read more on pseudoscience, "bad" science, and websites describing what science is and is not, go to http://homepages.wmich.edu/~korista/sci-pseudo-rel.html.