Overview | Rating System Development | The Rating System

Sorry, your browser doesn't support Java(tm). A. OVERVIEW.

One of the goals of the TIPA is to develop a structure which can be used across the nation to encourage candidates and campaigns to run ads that say something meaningful, to encourage them to say it honestly, to create incentives for them to say it accurately, and to get them to say it intelligibly.

The TIPA Rating System will evolve over time and several versions are anticipated to be released over time as we revise it based on the input we receive from our Advisory Panel, advertising and political professionals, and academics. While it attempts to be simple and direct, we realize the development of such a rating system is an extraordinarily difficult and complex task. To make it easy to understand and to make it visually attractive for news organizations, the Project will deploy a three-part rating system – part one for factual accuracy, part two for fairness, and part three for relevance.

One of the biggest obstacles to any Rating System, of course, is the role personal biases and prejudices play in such a process. The TIPA is attempting to address this challenge head-on.

The ratings will be depicted in a graphically-attractive manner, using an “Accuracy Meter” and a “Fairness Meter” as well as a separate “Relevance Scale.” These will be used as the primary images conveyed to media organizations and in the Project’s own communications. The Rating Systems will be applied on an individual advertisement-by-advertisement basis as well as for an averaging approach which then can be used to characterize each campaign.

The Accuracy Rating is based on whether the advertisement delivers factual information about the candidates and real issues, or whether it distorts the facts and takes events and other information out of context to paint a negative, inaccurate, and even intentionally misleading image of a candidate. Its basis is a traditional fact-checking approach.

The Fairness Rating is based on whether the advertisement paints a fair picture of the candidates and their positions on the issues, or whether it is a smear designed to unfairly assault the character and/or position of one or more candidates. While more subjective than the Accuracy Rating, the TIPA is confident the moral compass most people have will allow them to reasonably and accurately gauge the fairness of an advertisement.

The Relevance Scale is based on a straightforward subjective evaluation by the participant who simply decides whether or not the advertisement includes information he or she believes is relevant to the current contest. While one could argue only certain issues – e.g., financial, economic, environmental, security, educational – may be “relevant,” actual relevance definitely is in the mind of the beholder. What may be relevant to one person easily can be irrelevant to another. Rating relevance, therefore, is an opportunity for the evaluator to say what is important to him or her at that particular point in time.

The role of political partisanship looms large in any rating of relevance. One could argue, “Why rate relevance if it likely will be a totally partisan consideration?” The answer lies in the information provided by those participants who take their assignment seriously and are willing to be critical of advertisements in support of candidates from their own party – as opposed to those who simply will respond with a knee-jerk rating paralleling their own partisanship. By analyzing this subgroup, the TIPA may be able to explore issues of how fair or objective partisan participants can be (assuming the TIPA can determine what objective ratings of relevance should be for each advertisement – which may be difficult to achieve).

Combined with the TIPA's diverse panel of academic, political, media, and community experts, the rating system is designed to provide the most powerful and easy-to-understand profile of each individual advertisement as well as the overall advertising character of each campaign.


Developing a Rating System for political advertising is no easy task. The TIPA’s offering today is little more than a suggested starting point. Here is how the TIPA is addressing the challenge.

  • MULTIPLE RATINGS. Rather than using a single rating, the TIPA has elected to divide the ratings into three categories. The first category addresses the factual accuracy of the advertisement. The second category rates its fairness. Obviously, the latter is more subjective than the former. The third category is relevance and is very subjective. Relevance can refer to the relationship the ad content has to the candidates, their respective campaigns, or any other factor. One will expect supporters of a candidate to always rate their candidate’s advertisements as relevant so the revealing information will be the rare occasion when the opposite happens.
  • DETAILED DESCRIPTIONS. Rather than simply using a scale with minor explanations for the numerical ratings, the TIPA’s approach is to use a highly detailed description for each number or range. Also, rather than having a relatively short scale (e.g., 1 to 3), the TIPA deploys a 10 position scale. A half negative/half positive scale was considered (e.g., -5 to +5) but was not deployed due to arithmetic preference for simple averaging as well as the preference to avoid a potential rating (“0”) which appeared to grant content neutrality to an advertisement.
  • NONPARTISANSHIP. The TIPA knows a more traditional approach seeking accuracy in a rating system is to exclude the involvement of participants with prejudice in the actual evaluation segment of a research project. Normally, one seeks to avoid investigators or participants at similarly influential levels with prejudices which could affect the way any study is conducted (as opposed to the subjects of a study, who are expected to have biases and prejudices -- which often, in fact, are what is being studied). The TIPA believes the content of political advertising is so emotionally charged and is so difficult to study objectively that the best way to guarantee nonpartisan results is to include absolutely partisan people in the evaluation phases of the effort.
  • PROTECTIVE MECHANISMS. One of the ways the integrity of the Project will be ensured is that there always will be one of the following conditions imposed on and occurring with every analysis: (a) there will be an equal number of partisan participants on each political side of an evaluation or (b) where there is not an equal number, opinions will be weighted proportionately to guarantee a “level playing field.” The primary protective mechanism, however, is to balance the sample for each evaluation so there is a “wash” between those supporting one candidate and the other. A secondary mechanism will be used to add an additional layer of protection in the event gamesmanship occurs and individual participant are taking extreme positions so as to skew group averages. This can be statistically addressed in a number of ways which any expert in robust analysis can deploy.


The Truth In Political Advertising Project depends most of all on the integrity of its special Advisory Panel participants. The Executive Director, Aaron Harber, personally knows the vast majority of Panel members and is confident they will evaluate each advertisement as honestly and as fairly as possible. As a rule, these are extraordinarily honest people with great integrity who care about the health of the political process and sincerely want to contribute to its improvement.

By including partisan participants yet seeking balance, the TIPA’s Rating System is designed to incorporate -- rather than forego or even attempt to avoid -- the expertise political operatives, candidates, party officials, and similarly experienced people have. The TIPA understands these people, more than any others, pay more attention to political advertising than anyone else and have a deeper understanding of it. These people also have grave concerns about the political system in which they operate and most, if not all, at times sincerely wish the political battles in which they are engaged were fought fairly and on principles and about relevant issues, rather than via the pure gamesmanship which dominates politics today.

The Accuracy Rating is based on each participating evaluator’s sense of the factual correctness of an advertisement. The TIPA will strive to consistently provide information about each advertisement which addresses the factual statements in the ad and their accuracy but separately is interested in every participant’s perception of factual accuracy.

Each evaluator will bring his or her own expertise, knowledge, information resources, experience, perspective, and sentiments to that evaluation. The TIPA will use the ratings of each Advisory Panel member and contrast those ratings with its own assessment of the factual accuracy of each advertisement. By using this parallel tracking mechanism, the TIPA will be able to develop its own internal model for determining the degree of impartiality each Advisory Panel member presents.

The Accuracy Rating is on a one to ten scale per the following legend:

01 = Grossly inaccurate with total disregard for the truth; an advertisement which intentionally misstates facts to definitively create an impression which is the opposite of the truth and which omits key facts which otherwise would lead the observer to the opposite conclusion; viewed as truly despicable by anyone with a conscience or a modicum of decency.

02 = Extremely inaccurate with “facts” fabricated and/or distorted so as to give a false impression of the truth; a damning indictment of misleading advertising.

03 = Generally inaccurate with a plurality of key facts misstated and/or intentionally omitted, giving the observer the impression something false is true or vice versa.

04 = A mix of accurate and inaccurate statements (e.g., 50- 50) with more than one significant fact misstated and/or with key, known or well-established facts omitted; a factually misleading advertisement.

05 = Superficially accurate (approximately 70% of the time) with one or more significant facts intentionally misstated or erroneous; clearly not meeting any kind of minimal standards for overall accuracy but appearing to do so as part of an effort to mislead the observer.

06 = Nominally accurate (approximately 80%) but with one to three key facts subject to misinterpretation or misunderstanding and such misinterpretation or misunderstandings likely to have been intentional as part of a “smokescreen of truth” used to pursue a falsified argument.

07 = Generally accurate (approximately +90%) with one or two key facts subject to misinterpretation or misunderstanding – and they clearly should have been corrected before the advertisement was used.

08 = Very accurate (approximately +95%) with one or two facts unintentionally unclear or subject to misinterpretation or misunderstanding.

09 = Almost 100% accurate, with some minor unintentional misinterpretation or misunderstanding of one or two facts possible but unlikely; an advertisement obviously created with the pursuit of truth in mind.

10 = Absolutely and totally accurate from a factual perspective.

The TIPA staff contemplated simpler 1-to-3 and 1-to-5 scales but felt such limited choices would force evaluators to select options which were more extreme than their true feelings. By using a 1-to-10 scale, the TIPA allows each participant to more closely express the degree of his or her true feelings. With the broader scale, there is more opportunity for nuance and differentiation as opposed to being forced into a “harder” category which may not accurately express the true sentiment of the evaluator (e.g., where hesitance or uncertainty may exist).

A 1-to-10 scale also gives the TIPA the option of later combining evaluation categories (e.g., adding the 1’s and 2’s and comparing them to the 9’s and 10’s) for cross-tabulation and other analytical purposes. The larger numerical scale also should be more meaningful arithmetically when group averages are calculated. Furthermore, by having the explanation for each rating be so well-defined, Advisory Panel members and public participants can refer to the legend when performing their ratings for ease of use and for consistency in their own application of the Rating System as well as in the application of the ratings by all other participants.

* * * * * * * * * * * * * * *

The Fairness Rating:

01 = So terribly mean-spirited, evil, despicable, and/or unfair that the candidate should be ashamed off him- or herself and resign from the election; this rating is so low that voters supporting the candidate sponsoring the advertisement or intended to benefit from the advertisement should cast their vote for his or her opponent; contains multiple, horribly vicious personal attacks, clearly intended to ruin the personal life of an opponent or opposed candidate; the ultimate in scummy.

02 = Extremely unfair and/or inaccurate with “facts” fabricated and/or so extraordinarily distorted to vilify or otherwise severely harm the opposing candidate; containing false or highly personal yet irrelevant material which could unfairly damage the targeted candidate for life; contains one or more vicious personal attacks; truly reprehensible.

03 = Grossly misleading and unfair effort to stain the reputation of an opponent, meant to severely harm him or her.

04 = Scurrilous personal attack intentionally distorting the truth to give a false impression of an opponent, yet mixed with enough relevant or reasonable claims to soften what otherwise would be a brutal attack.

05 = Contains an unfounded or unjustified personal attack on a candidate which is patently unfair and which is not fair game.

06 = Hard-hitting but fair and reasonable; hard-edged and perhaps a bit too tough yet just within the bounds of what should be considered acceptable.

07 = Very fair; portrayal of the opponent might be negative but is not exploitative or unreasonable in any way; shows a contrast which is accurately and fairly represented.

08 = Extremely well-balanced and reasonable; does nothing to unfairly portray the opponent in a negative light but still may use one very relevant negative to contrast the candidates or ballot issues yet does so in an acceptable manner.

09 = Bends over backwards to be fair and kind to an opponent; intentionally portrays the opponent in a good light; makes no statements which could be unfairly construed.

10 = Extraordinarily fair to the point of being magnanimous to the other side; if every candidate were this generous, negative advertising would not exist; one might think the opposition developed and sponsored this advertisement.

There is no question that the Fairness Rating is the most complex of the three components of the TIPA rating System. Fairness is very subjective and a number of different tests can be used when considering what is “fair.” Is it “the right thing to do?” Is it “how we would want to be treated?” (Think of the Golden Rule as a constant in the evaluation process.) Given the world of politics and the rules by which it is played, is the advertisement being evaluated n the bounds of fair play, good sportsmanship and/or good taste? How would the sponsoring or benefiting candidate or organization feel if the tables were turned and the advertisement’s approach were directed back at him, her or them? Would a principled person consider the advertisement fair? Is the advertisement honest and not deceptive in any way? These are the kinds of questions Panelists and members of the public will have to ask each and every time when rating an advertisement.

In addition to reflecting on all of these issues and variables, each evaluator will need to stay cognizant of his or her personal biases. These prejudices often work subconsciously to excuse bad behavior. This is why a partisan person in favor of a candidate or ballot issue is likely to excuse or not even recognize behavior or actions which, if they had been taken or deployed by the opposition, would be seen as unfair, improper, and even scurrilous. Trying to maintain objectivity during the peak of bias in an emotional political setting – especially as Election Day draws near – is the difficult but quite doable assignment of TIPA’s participants.

Furthermore, it is common for attacks against ballot issues to be even more mean-spirited and disingenuous. This is because such attacks are not “personal” in nature although they can be even more vicious than any other. That is, such attacks typically are not directed against a particular individual but, instead, are directed against a more amorphous or difficult-to-identifyt group. The “issue” rather than a “person” usually is the subject of such attacks and, as a result, there is greater opportunity to be extreme in such instances without being held responsible for any personal attack.

* * * * * * * * * * * * * * *

The Relevance Rating is a corollary measure of fairness and accuracy in that the use of irrelevant material is indicative of a candidate and/or campaign interested in misdirecting voters. There is an inherent dishonesty in candidates from any party or group who seek to focus the public’s attention on non-issues – transforming them into issues so as to gain favor and discreetly submerge the true and key issues of the day below the surface of political waters. The misdirection play is common in politics and offers great convenience to those on the “wrong side” of an issue as well as those who find great discomfort at the prospect of otherwise being forced to confront difficult issues. This rating also is on a one to ten scale per the following legend:

01 = So totally irrelevant that the sponsoring candidate or the person or group for whom the ad is intended to benefit should be tarred and feathered (figuratively speaking) for his, her or their attempts to use these issues in the campaign; the transgression is so egregious that voting for the opponent of the sponsoring or candidate or organization or other beneficiary is a must so the target of the irrelevant ads ultimately benefits.

02 = Extremely irrelevant with issues or “facts” selected, fabricated, deployed and/or otherwise used in an intentional manner to redirect voters away from the real issues of the day but done so in a manner which serves to attack the opposition via distortions, vilification, et cetera, often with the simultaneous intent to severely harm the opposing candidate; containing misleading or false material or one or more vicious personal attacks which are designed to suppress discussion of what truly are the relevant issues of the day.

03 = Grossly misleading effort to redirect voters, possibly including an attempt to stain the reputation of an opponent, meant to severely harm him or her; scurrilous personal attack intentionally distorting the truth to give a false impression of an opponent and avoid addressing the issues voters see as key in a race.

04 = Contains subject matter which has little to do with the campaign or contest at hand; obviously created to focus on a subject which casts the sponsoring or benefiting candidate in a positive light although the subject matter has little or no relationship to the political race.

05 = A slight majority of the material is not relevant (e.g., +50%) and, instead, unintentionally obscures those issues which are on the minds of most voters; this advertisement may be sloppier in its construction than intentionally devious (although the ultimate ramification may involve voter confusion).

06 = More relevant than not (i.e., +65%) but containing a confusing mix of topics, subjects, and themes (in terms of their relevance) -- some of which apply to the contest at hand and others which simply do not apply at all (hence the confusion).

07 = Almost all (80%) understandably applicable and relevant subject matter and material which is related to one or more key issues of the day for a plurality of voters and which is articulated clearly.

08 = Quite relevant and appropriate (+90%); the subject matter of the advertisement involves one or more topics which are critically important at this time and are on the minds of most voters but something minor and irrelevant may still have been included in the advertisement.

09 = Very relevant material (i.e., approaching 100% relevance); the subject matter of the advertisement is right on target and highlights the issues which clearly are important to most people today; this advertisement “hits the nail on the head” when it comes to topic selection.

10 = Extremely relevant (definitely 100%)); could not have selected more appropriate and fitting topics, information, and claims; this is what this campaign should be all about; a model for this and other political campaigns; this candidate or campaign should be lauded and supported for the willingness to address what truly are the key issues of the day.

Relevance is important but, again, it is so subjective that differences may be hard to ascertain in the data once core partisanship is factored out of any analysis. What “fits” a particular situation in a political campaign at a particular point in that campaign is likely to be seen differently by opposing partisans. What is “appropriate” or what truly “applies” to a particular political race or a specific point of time – i.e., what is truly relevant? -- are questions that can be honestly and reasonable answered quite differently.

With the above factors in mind, the most interesting finding may be the degree to which evaluators can or cannot separate their personal perspectives from an honest evaluation of the true issues of the day when evaluating relevance. Similarly, examining how their answers correspond with responses for the other ratings may generate fascinating results. Despite these complexities, the TIPA Project staff is confident that, because of the integrity of those involved on the Advisory Panel, they consistently will make fair, reasonable, accurate, and objective evaluations of the advertisements put before them.

* * * * * * * * * * * * * * *

An Overall Rating or Score will be considered by the TIPA. It likely will be weighted so as to emphasize accuracy. In this manner, the most subjective elements of the Rating System are somewhat ameliorated. By intentionally using the same 1 (most negative) to 10 (most positive) scale, the demonstration project Rating System lends itself to a single scoring system (i.e., this makes the most logical Overall Rating system compatible with scores in the 1 to 10 range, with similar connotations related to each scoring level). The initial Overall Rating scoring system would take an average of the three ratings and incorporate them as follows:

Accuracy Rating: 45% of the Overall Score.

Fairness Rating: 35% of the Overall Score.

Relevance Rating: 20% of the Overall Score.

Other distributions will be considered and may be a function of how well each component of the Rating Systems works as the Rating System actually is used. At a 20% weight, for example, one might argue the Relevance Rating could become irrelevant itself (i.e., only an extremely positive or negative rating would affect and Overall Rating or score). With this in mind, would partisan evaluators then tend to score Relevance in an extreme way (i.e., simply to have their opinions “register” when they know those opinions otherwise would not)? The range of weighting originally and still being considered was and is as follows, with any final combination (as detailed above) obviously having to add up to 100%:

Accuracy Rating: 33% to 60% of the Overall Score.

Fairness Rating: 33% to 40% of the Overall Score.

Relevance Rating: 10% to 33% of the Overall Score.

The TIPA admits that the development of any Rating System is a significant challenge. We have been struggling with this for some time and recognize it is a work-in-progress. Your comments regarding how the Rating System can be improved as well as complete alternates to the TIPA Rating System would be warmly welcomed. Your criticism can only make the TIPA Project more successful in the long run. Please tell us what we are doing wrong, where our Rating System does not make sense, how to improve and better differentiate each scoring group, level, and description, and how to make the Rating System more accurate, more relevant, more understandable, and easier to use. The TIPA staff sees this initial Rating System only as a starting point and is depending on your critique to help us improve it.

The TIPA staff is aware and is concerned that the structure of the ratings themselves may drive respondents towards certain answers not just in terms of the aforementioned potential gamesmanship but due to our own failures to make adequate descriptions or distinctions. For example, in some cases, the descriptive difference between two adjacent ratings is minimal. This may result in slight biases moving whole numbers when any analysis is done (i.e., almost everyone rates an advertisement a “2” rather than a “3” because there is no discernible difference or it is too small).

A separate phenomenon based on distinction failures also may be that campaigns consider the ratings when developing ads and tone down the negative aspects of some of them. Hence, the less-than-desired variation in ratings may promote campaigns to change their tactics so as to avoid a low rating. While the reality of campaigns makes this phenomenon unlikely, this could become a long-term impact of the TIPA Project.

There remains the separate issue but critically important issue of accountability and how it is assigned by the electorate. With the plethora of independent expenditure committees (most notably the approximately 15,000 527 committees which have been registered in the United States), there already is a great need to address the issue of how voters should respond electorally when a third party entity inaccurately, unfairly, and/or irrelevantly attacks the opposition.

When one candidate attacks another in such a manner, it is easy to hold the attacking candidate responsible simply by voting against him or her (i.e., and in favor of the candidate suffering the attack). When a third party makes that attack, however, while the understandable electoral response initially may be the same, the additional question becomes, “Is it reasonable for voters to hold responsible a candidate who may otherwise benefit from the attack but who did not make the attack?”

This then moves the discussion into the intrigue of third party participants, their actual relationships with the campaigns they seek to support, et cetera. It even can involve devious and complex third party efforts to defame a candidate the third party supports in an effort to generate sympathetic support for him or her. These are issues the TIPA and voters ultimately will need to address although in Colorado, where “clean” campaigning has been prevalent for decades and where voters tend to punish bad campaign behavior, these have not yet come to the forefront. In time, they will (and that time may be short.)

* * * * * * * * * * * * * * *

Members of the TIPA Advisory Panel are asked to simultaneously e-mail their ratings and comments to Zachary@TIPAP.org and to Aaron@TIPAP.org as soon as they are notified there is a new advertisement to review or as soon as they see a new advertisement on the TIPA Web site. Please note the advertisement being reviewed as well as your three 1-to10 numerical ratings for Accuracy, Fairness, and Relevance. Everyone is welcome to send in their ratings – including members of the public. Advisory Panel member ratings will be calculated separately from the general public and, when appropriate, the two sets of ratings will be compared. A rating submission should appear as follows. Use the following format via a cut-and-paste approach or use the form ( click here) available for Advisory Panel members and members of the public.

Advertisement Title: ________________________________

Accuracy Rating: _________

Fairness Rating: _________

Relevance Rating: _________

Comments (optional):

That’s all there is to it. If you want to add information such as where and when you saw the advertisement, that can be included in the Comment section. This would be helpful as some advertisements are broadcast in a highly targeted and/or limited manner. While most campaigns have agreed to cooperate with the TIPA, vigilant supporters of our Project will help us stay current. Thanks for your help doing this.

If any evaluator – whether an Advisory Panel member or a member of the general public -- sees an advertisement not on our Web site, please click here and send an e-mail message notifying us immediately of the existence of the advertisement. Again, while participating campaigns have agreed to provide us with every advertisement as soon as it goes into rotation anywhere in the State of Colorado but it is entirely possible that, from time to time, one may be missed or just slips through the cracks – either at a campaign or at the TIPA Project.

Finally, the ratings of the Advisory Panel as well as those made by individual members of the public always will be kept confidential. Your name never will be associated with your own rating of any advertisement. Your responses will be reported only as part of the total set of responses received. And your e-mail address and any other contact information will never be shared with any other organization. Anonymity is crucial to our success and we need you to always have confidence your individual, identifiable responses will not be disclosed.

Thank you very much for participating in our pilot project and for helping begin the process of developing mechanisms to bring integrity to the political process. The TIPA is just the beginning of a long-term effort and every contribution made of time, expertise, and other assets is greatly appreciated by the entire TIPA staff. With your help, we will change politics in America.

Home | About TIPA | Advisory Panel | TIPA on TV | The TIPA Plan | Press Releases | Information for Participating Campaigns |Participating Organizations | TIPA-USA: The Future | Our Sponsors | About The Executive Director | The Political Ads | Rate The Ads |
The TIPA Rating System
| The TIPA Rating Legend | Links | Voting Websites |
Election Websites
| Political Commentary Websites | Political Party Websites |
Political Blog Websites
Mass Media Websites | PBS Station Websites | Academic Websites | American Documents | Articles & Resources | Relevant Articles - National Media |
Relevant Articles - Colorado Media
| Books & Journal Articles | FAQ | Contribute | Latest Ad | Automatic E-mail Reports | Contact