This outline focuses on some of the strategic and tactical issues which face the trademark litigator in conducting or attacking a consumer survey. It should be read in conjunction with Dr. Dupont’s paper, which addresses the practical issues involved in designing and executing a survey. Some effort has been made to avoid duplication. In addition, other parts of the Practical Tips curriculum address subjects which have direct applicability to litigation surveys, particularly the Discovery, Pre-Trial and Trial sections. Accordingly, those subjects are treated only lightly in this outline, if at all.

1. Should you have a survey?
   
   
   1. The first question to address is whether a survey is appropriate for your case. That involves the weighing of a number of factors, including:
      
      
      1. Whether there are issues presented in your case which might be amenable to survey evidence. As a general matter, surveys can be useful in attempting to measure
         
         
         1. likelihood of confusion
            
            
         2. secondary meaning
            
            
         3. genericness, and
            
            
         4. in the context of dilution claims, fame and likelihood of dilution.
            
            
      2. How expensive the survey is likely to be, and how that compares to the amounts at stake and the client’s budget. The expense of conducting a survey can vary widely, from a low of $15-20,000, to a high well into six figures. The primary factors affecting cost are:
         
         
         1. The nature of the survey. Face-to-face interviews are more expensive than telephone, for obvious reasons.
            
            
         2. The number of respondents required. Generally, surveys rarely require more than 200 respondents or so to produce results with an acceptable confidence level. The cost can escalate quickly, however, if multiple cells are required. If you want to test mark and packaging separately, for example, and need a control for each, suddenly you are looking at 800 respondents.
            
            
         3. Incidence, which is the frequency with which qualified respondents are found among the population from which candidates are drawn. This, in turn, is driven by the screening criteria which you and the survey director establish. For example, if your universe is limited to female smokers who have purchased breath mints, your interviewers will need to do a lot more screening, and your survey will therefore be more expensive, than if the universe is recent purchasers of soft drinks.
            
            
      3. The strength of other evidence going to the same issues.
         1. If you have a strong senior mark with significant consumer awareness, relatively high similarity to the junior mark, solid evidence of actual confusion, and proof of bad faith intent to confuse by the defendant, you may not need a survey. At the same time, if you do choose to conduct a survey in such a situation, your chances of obtaining results showing substantial confusion are probably high.
            
            
      4. The likelihood that the survey will produce the desired results.
         1. This is by definition a guess, given the fact that a survey has not yet been conducted. An experienced survey director can be of help in making this assessment, but it is remarkable how often results can surprise even the most seasoned experts.
            
            
         2. If time and budget allow, the best approach to this question is through the use of a pre-test, discussed below.
            
            
      5. The likelihood that failing to conduct a survey will result in an inference adverse to your client.
         1. Some courts have suggested that when a party has the financial means to conduct a survey and fails to do so, it justifies an inference that the results of a survey would have been adverse to that party. See, e.g., Eagle Snacks, Inc. v. Nabisco Brands, Inc., 625 F.Supp. 571 (D.N.J. 1985).
            
            
         2. Other courts have recognized the unfairness of expecting every trademark plaintiff to incur the significant expense of a consumer survey, and have emphasized that a survey is not a prerequisite to relief. See, e.g., Charles Jacquin Et Cie, Inc. v. Destileria Serralles, Inc., 921 F.2d 467 (3rd Cir 1990) (proper to refuse to instruct jury that it may draw adverse inference from failure to conduct survey; relief in absence of survey affirmed).
            
            
         3. Notwithstanding that the majority of cases now recognize that a survey is not required, if the case is one which might normally lead the court to expect a survey and one does not materialize, there is still the potential for compromising the court’s view of the case, even if only informally and subjectively.
            
            
   2. The above assessment is one which can only be conducted on a case-by-case basis, because evaluation of each of the above factors, as well as their interplay, varies dramatically with the specific circumstances. Nonetheless, a few rules of thumb can be articulated:
      
      
      1. As a plaintiff trying to prove likelihood of confusion, if you lack solid evidence of either actual consumer confusion in the marketplace or bad faith intent to confuse by the defendant, you will almost certainly need a good survey.
         
         
      2. On the other end of the spectrum, if you have a well-known senior mark coupled with strong evidence of actual confusion, a survey may be gilding the lily. See, e.g., Sara Lee Corp. v. Kayser-Roth Corp., 81 F.3d 455 (4^th Cir. 1996) (strong senior mark coupled with widespread actual confusion may be sufficient standing alone to establish likelihood of confusion, without reference to other factors). In such situations, a survey may even detract from the case, by introducing possible questions and complications.
         
         
      3. If you know or suspect that the other side will conduct a survey, you would be well advised to do your own. Criticism of your opponent’s survey methodology and interpretation, however well founded, will rarely carry the day when the survey you are attacking is the only survey in the case.
         
         
2. Selecting a survey director
   
   
   1. A good survey director for litigation is a rare breed indeed. The expert needs a combination of skills, temperament and experience which are not often found in a single individual.
      
      
   2. In terms of academic background and credentials, a solid grounding in statistics and quantitative methods is essential. In addition, however, an understanding of marketing and advertising is important, and some of the best survey experts also have backgrounds in psychology, enabling them to opine credibly on the often-complex mental processes in consumers which surveys attempt to measure.
      
      
   3. Beyond the above substantive qualifications, substantial experience in conducting surveys specifically for litigation is an absolute must, for several reasons.
      
      
      1. In litigation, mastery of a field is of little use if that mastery cannot be conveyed forcefully from the witness stand. There is no way to know in advance how a person will perform under cross-examination except by actual experience.
         
         
      2. Litigation surveys have their own unique requirements. Despite the theory that expert testimony should be admissible so long as it is based on principles which are generally accepted and relied on in the field, in the realm of surveys, at least, courts have taken it upon themselves to establish standards for admissibility and persuasiveness which do not always mirror accepted practice among market researchers generally. For example, for business purposes most market researchers will recommend 10 to 15 percent validation (that is, calling back 10 to 15 percent of respondents to verify that they did in fact participate in the survey), but for a litigation survey experienced directors will do 100% validation, because many courts expect it.
         
         
      3. Most importantly, the survey director needs to have an understanding of the ground rules in working with the attorney, which requires the drawing of a careful balance. The attorney has only one interest, of course: to win the case. The good survey director understands that his or her priorities must be slightly different. The survey director must conduct the survey properly and according to his or her own professional standards. If the expert compromises professional judgment to obtain a result in a particular case, then in the long run (and often even in the short run), the expert’s credibility will be crippled.
         
         
      4. Of course, good attorneys appreciate this as well, and know that the credibility of the director is vastly more important to the case than another five points of net confusion. Experienced counsel and experts, then, both understand that their interests are in fact identical: to produce a sound and defensible survey which will persuasively measure a factor or factors important to the success of the case. To produce this effective working relationship, the expert must have sufficient experience to know when to be receptive to advice from the attorney about what the issues are and how they must be addressed, and when to draw the line on matters which could compromise the integrity of the survey.
         
         
3. Involvement by the court or the adversary in the design and execution of the survey
   
   
   1. Some authorities, out of a well-intentioned desire to remove some of the perceived gamesmanship and uncertainty from the process, have recommended that parties contemplating a survey for litigation submit the survey design in advance to the court and/or the adversary, so that methodological issues can be resolved before the survey is executed. See, e.g., Federal Judicial Center, Manual for Complex Litigation § 21.493 at p. 102-03 (3d ed.1995). It has even been suggested that ideally a survey should be jointly commissioned by the parties. American League Works, Inc. v. United States Trunk Co., 158 F.Supp. 50 (D.C. Mass. 1957), aff’d 259 F.2d 69 (1st Cir. 1958).
      
      
   2. With respect to all such suggestions, only one word of advice is needed: don't. You must retain absolute control of your survey until the moment you have decided that it is appropriate to submit it to your adversary and the court. There are any number of tactical decisions the attorney must make during the design and execution of the survey (not least of which is the decision whether to kill the survey altogether at any point), which decisions would be impossible to make if a third party was involved. Much better to make all those decisions without interference, and then present the court with a fait accompli which it will likely accept despite what it may view as a few flaws, than to allow others to meddle in the design and execution and thereby endanger the results altogether.
      
      
   3. Besides, despite its trappings, survey research is much more art than science. Even if you have very good survey experts on both sides, and they were to act in the best of good faith (neither of which premises can be taken for granted), trying to involve them both in the survey process is a recipe for disaster. What sort of paintings would have resulted if Monet had been forced to collaborate with Picasso?
      
      
4. Working with the survey director

   As discussed above, the working relationship between the attorney and the survey director is an unusual one, due to the interplay between the need for the director to exercise independent professional judgment and the need for guidance from the attorney as to the legal framework and the issues to be addressed. For the attorney, a few ground rules can help keep this relationship on the right track.

   1. Do not dictate. Understand that the director has certain standards and principles that he or she cannot deviate from. You are entitled to (and should) discuss every aspect of the survey with the director, but the director will feel much freer to do so if there is no fear of you trying to "overrule" something. Your role is to test and probe, to insure that the director has thought through each issue and can defend the survey against attack. Ultimately, however, the final decision on how the survey is conducted is the director’s, not yours. After all, the director is the one who will be under oath.
      
      
   2. Assume that everything you say to or give to the director is subject to discovery. That means that, as a general rule, you should not send anything in writing unless it is or will be of record in the case, or is otherwise something which you are comfortable having fall into the hands of your opponent. Similarly, you should only discuss your strategy of the case with the survey expert to the extent that you are confident in his or her ability to respond to cross-examination questions about it.
      
      
   3. Communication back to you from the director should similarly be limited, particularly in written form. There are really only four documents you should get from the director through the entire process (until the opposition requests all of the workpapers and interviewer notes in discovery, of course):
      
      
      1. an initial proposal and engagement letter;
         
         
      2. the survey design, including draft interviewer instructions, screener and questionnaire;
         
         
      3. the final report of results; and
         
         
      4. the expert’s Rule 26 report, which will include both the final survey report and the expert’s resume.
         

   It is possible to receive these documents in draft form from the expert for your comment and, perhaps, revision before they are sent to you in final form, but bear in mind that drafts are discoverable as well, and any changes made at your suggestion will be scrutinized closely.

5. Designing the survey
   
   
   1. In designing a survey, it is important to be realistic. The goal, of course, is to make the survey as impervious to criticism as possible, but the very nature of the beast makes a "bulletproof" survey impossible. As Professor McCarthy has cautioned in a widely-quoted passage,
      
      One must keep in mind that there is no such thing as a "perfect" survey. . . . Like any scientific method related to statistics in the social sciences, every survey, no matter how carefully constructed and conducted, has some potential flaws somewhere. The proper approach is to view such evidence with some understanding of the difficulty of devising and running a survey and to use any technical defects only to lessen evidentiary weight, not to reject the results out-of-hand.
      
      5 J.T. McCarthy, McCarthy on Trademarks and Unfair Competition § 178 (4th ed. 2000). Most often, survey design is a process of making trade-offs, and choosing the most defensible or least objectionable of various options, all of which are usually subject to attack on one ground or another.
      
      
   2. As the attorney, do not hesitate to become involved in the design process. Some cases have suggested that such involvement is improper, e.g., Boehringer Ingelheim GmbH v. Pharmadyne Laboratories, 552 F.Supp. 1040 (D.N.J. 1980), but these are based on a misunderstanding of the attorney’s role, and often on a misreading of language which appeared in early versions of the Manual for Complex Litigation that the survey should be "conducted independently of the attorneys." That language was removed in 1981, see Federal Judicial Center, Manual for Complex Litigation 116 (5th ed. 1981). Even before then, most courts recognized that the language was intended to prevent attorneys from actually conducting the interviews, but not to curtail their appropriate and necessary involvement in the design of the survey so as to insure that it addresses the correct issues and conforms to the requirements imposed by the case law. See Pittsburgh Press Club v. United States, 579 F.2d 751 (3d cir. 1978) (recognizing that only attorney involvement in conducting the survey is proscribed).
      
      
   3. For discussion of specific survey methodologies, see Dr. Dupont's accompanying paper. The case law concerning various methodologies and techniques is summarized comprehensively at 5 J.T. McCarthy, McCarthy on Trademarks and Unfair Competition §§ 172 - 177 (4th ed. 2000).
      
      
   4. As a general rule of thumb, less is more in litigation surveys. Every additional question you ask introduces the potential for additional uncertainty, and creates a possible basis for an attack by your adversary. In marketing surveys generally, the tendency is to "tack on" additional questions at the end of a survey, since there is virtually no incremental cost involved, but in litigation such a temptation should be resisted as much as possible.
      
      
6. Some specific design and execution issues
   
   
   1. Pilot tests and pre-tests
      
      
      1. As a matter of standard survey technique, it is common to conduct a "pilot" test in advance of the full survey. This consists of a limited number of interviews, usually less than 25, and is intended solely to help the survey director determine whether respondents properly understand the questions, whether the interviewer instructions have any holes, and whether there are any other unanticipated problems encountered in executing the survey. A pilot is not intended to produce representative results, and is intentionally done with few enough respondents that one can tell little or nothing about what the substantive results of the full survey will be.
         
         
      2. The standard "pilot" test described above should not be confused with a pre-test, which, while expensive, is often helpful in litigation surveys. A pre-test essentially consists of an initial execution of the survey itself, perhaps with fewer than the total number of interviews that will be done for the full survey, but with enough so that meaningful trends in responses can be detected. It can be very useful when it is uncertain whether one methodology or another will produce the best results, or when it is not clear whether the results will be usable at all.
         
         
      3. To the extent that your survey director is involved in conducting the pre-test, of course, its results will be fully discoverable. Accordingly, if the case is substantial enough to warrant it, the best practice is often to conduct the pre-test with a non-testifying expert in order to preserve work product protection, making sure that the survey director knows nothing about it.
         
         
      4. As an alternative to conducting a separate pre-test, you can ask the director to give you "top line" results from the full test as it is being fielded. You should be able to get results of the first fifty interviews, the next fifty, and so on. This might slow things down some, but it gives you and the client the opportunity to evaluate where the survey is headed. If it becomes clear that the results are not going to help the case, you can at least pull the plug before the full cost is incurred. In this way, if the survey goes well it will proceed normally and the cost of a separate pre-test is avoided, but if problems are encountered the survey can be aborted and treated as a (rather expensive) pre-test. You then have the option of turning to a different methodology in a new test, though you will at that point need to use a different survey director in order to protect the first aborted test from discovery.
         
         
   2. Probability vs. non-probability samples
      
      
      1. A "probability sample" is one which has been drawn using techniques which insure that it is a truly random representation of the entire universe to be surveyed. The power of the probability sample is that the odds that it represents the entire population can be quantified mathematically, resulting in an expression such as "range of error of +/-5% at the 90% confidence level." This means that if ten different samples were randomly drawn from the same population, the results would be within 5% of this survey’s results in nine of those ten samples. Put another way, more colloquially and slightly less accurately, there is a 90% chance that the attitudes of the population as a whole are within 5% of the measures produced in the survey. Clearly, this provides grist for a compelling argument to the fact-finder.
         
         
      2. Unfortunately, a true probability sample can be very expensive to generate. It requires having complete access to every member of the relevant population, and then being able to select candidates to interview entirely at random. As a practical matter, the only realistic method of obtaining such a probability sample is through telephone interviews, and even there many logistical challenges arise. Moreover, any execution which requires the respondent to view a product or label in person obviously cannot be conducted by telephone.
         
         
      3. As a result, the vast majority of surveys, both litigation and non-litigation, are not probability samples. The most common method of execution is the "mall intercept" technique, which uses survey facilities located at shopping malls, who "intercept" passing shoppers and ask them to participate in the survey.
         
         
      4. While such non-probability samples do not have mathematical rigor, they are nonetheless frequently employed in business, and are widely accepted. See Jacoby & Handlin, "Non-Probability Sampling Designs for Litigation Surveys," 81 Trademark Reporter 169 (1991) (97% of in-person commercial surveys are non-probability samples, and 95% of the empirical literature relies upon them). Because they are so widely used, the courts "have readily accepted mall intercept surveys in litigation," Tyco Industries v. Lego Systems, Inc., 5 USPQ2d 1023 (D.N.J. 1987), aff'd 853 F2d 921 (3d Cir.), cert. denied 488 U.S. 955 (1988), though some have given non-probability surveys less weight. See, e.g., Frank Brunckhorst Co. v. G. Heileman Brewing Co., 875 F.Supp. 966 (E.D.N.Y. 1994) (probability sample survey is "entitled to more weight than" a non-probability mall intercept, but both are admissible).
         
         
   3. Controls and "noise"
      
      
      1. In survey methodology, a "control" is a separate duplicate survey, conducted simultaneously with the test survey and using exactly the same methodology, but substituting a stimulus which would not elicit a positive response from a reasonable respondent. The purpose is to try to measure the extent to which responses in the test survey are attributable to generalized confusion, failure to understand the question, unjustified or irrational assumptions by the respondents, guessing, or other phenomena unrelated to that which the survey is intended to measure, all grouped by survey researchers under the general term "noise." See, e.g., S.S. Kresge Co. v. United Factory Outlet, Inc., 598 F.2d 694 (1st Cir 1979) (test showed 7.2% confusion between THE MART and K-MART, but control showed 5.7% confusion between THE MART and KING'S DEPARTMENT STORE, indicating net confusion of only 1.5%).
         
         
      2. There are many situations where use of a separate control is essential. For example, you and your expert may not be able to come up with a way to address a certain issue except through a question which may arguably be leading to some degree. Leading questions are generally improper, of course, but if you have conducted a control panel which uses the same form of questioning with an innocuous stimulus, then you have a strong argument that, to the extent there is any leading effect, the effect was also present in the control survey and is therefore corrected when the control figure is subtracted from the test result to arrive at "net confusion."
         
         
      3. A control can in fact be a "safety net" to deal with many methodological issues, even those which you and the director may not have anticipated. However convincing your argument is about why a particular methodological attack has no merit, it cannot hurt your position to be able to follow it up with, "And even if there were any bias introduced by this factor, it would have appeared equally in the control panel, and is therefore removed from the net confusion result entirely."
         
      4. The downside of controls is two-fold: their use can substantially increase the cost of a survey, and they can also introduce additional opportunities for second-guessing by your adversary and the court, since selecting an appropriate control can sometimes be tricky. This is an area where it is wise to place heavy reliance on your survey director.
         
         
7. Discovery issues
   

   Discovery in trademark litigation is treated elsewhere in the program, but there are a few issues unique to surveys that merit mention here.

   1. With regard to your own survey, be prepared to produce everything written which is generated in connection with the survey, including questionnaires, interviewer instructions, coding sheets, workpapers, correspondence with the director, etc., and surviving preliminary drafts of such documents (to the extent the director has kept them in accordance with usual practice).
      
      
   2. The only information relating to the survey which you can (and must) withhold is the name and identifying information of the individual respondents. These often appear on, and must be redacted from, the original questionnaires.
      
      
      1. It is a central tenet of survey research that the identity of individual respondents must be kept confidential and may not be disclosed to anyone. This is necessary both in order to be able to obtain willing participants, as well as to maintain the integrity of the survey process.
         
         
      2. Fortunately, the courts have generally appreciated the need for respondent anonymity, as well as the chaos that could result if aggressive opposing counsel were allowed to cross-examine individual respondents and induce them to second-guess their responses. See, e.g., Jewel Cos. v. GranJewel Jewelers & Distributors, Inc., 185 USPQ 504 (M.D. Fla. 1975) (interviewee identities are considered work product, unless proponent of survey intends to call them as witnesses at trial).
         
         
      3. In opposing a survey, you should insist that the original questionnaires be produced, albeit with respondents’ identifying information redacted, so that you can assess the survey director’s coding of responses, and perhaps have another expert re-code the responses entirely.
         
         
   3. The survey director must testify in order to lay a foundation for admission of the survey, but testimony of the actual interviewers is not required. See La Maur, Inc. v. Alberto-Culver Co., 179 USPQ 607 (D.Minn. 1973), aff’d 496 F.2d 618 (8th Cir.), cert. denied 419 U.S. 902 (1974). As a result, the survey director will almost certainly be deposed in discovery, since there is much that will not be apparent from his report. Aggressive opposing counsel may seek to depose others involved in the conduct of the survey, and whether to allow it is within the court’s discretion. At least a few courts have declined to permit examination of individual interviewers. See, e.g., Piper Aircraft Corp. v. Wag-Aero, Inc., 741 F.2d 925 (7th Cir. 1984) (no need to cross-examine interviewers; testimony of survey director is sufficient).
      
      
   4. In discovery, do not overlook pre-existing in-house market research which may have been conducted by the opposing party in the ordinary course of business. In-house research conducted by one’s own client rarely will satisfy the requirements of independence, objectivity and sound methodology required for admissibility, but an opposing party can hardly complain of such defects when it was willing to rely on the research, however flawed, in conducting its own business.