Analytical Procedures

Questions are designed to collect information from respondents. We refer to a set of questions put together to measure some outcome(s) as a TOOL. Examining the effectiveness of such tools is of interested to both developers and users of measurement tools. Both developers and users can find out more about assessment tools of various kinds in the following pages.

Focusing on what is measured

A tool can be designed to measure one or more of the following four classifications of outcomes.

Measurement of QUANTITY: This type of measurement focuses on characteristics that are physical and therefore measured with high reliability. For example, age, weight, cholesterol, job title, salary, income, or other specific empirical details. These outcomes can be utilized in two types of analysis.

	Description frequently focuses on statistics that summarize aggregated cases in terms of averages, and distributions.
	Classification and categorization frequently focuses on statistics that identify how sets of cases are substantially different on measured characteristics.

Measurement of Competence: This type of measurement frequently focuses on cognitive information such as specific recall or knowledge, learned abilities such as mathematics, or abilities such as forms of intelligence, IQ. Physical capacities or abilities may also be included such as skills in sports or arts when such measurement can be shown to be consistent (reliable) and valid.

Evaluating competence or ranking levels of knowledge and abilities.

Measurement of Opinions: This type of measurement focuses on beliefs, choices and opinions that are expressed by individuals. This type of public polling is frequently used in marketing, politics, and social science research. The most frequent forms are questionnaires and surveys.

Measurement of Psychological characteristics: This type of measurement focuses on broad-spectrum characteristics that are frequently not accurately perceived by an individual. Fore example, clusters of values, broad patterns of behaviors, or personality traits. These broad patterns are usually identified through a self-report of indicators that are more observable behaviors, or preferences. For example, questions may be used to build a scale that identifies conservative v. liberal social values. Such a scale may be more usable than asking a respondent if they are liberal or conservative because their response may be more about their self-concept and social perceptions than about the inherent trait.

Psychological and Social Scale construction using information collected from respondents that identifies traits, values, beliefs that may not be available through direct questioning.

Examining the Effectiveness of Tools

A major part of developing a good tool is examining how well it does the job a developer designed it for. One way of doing this is through trial and error, and anecdotal experience. Another is to examine the outcomes from a tool using statistics. There are a wide range of statistical procedures, which can be applied to assess a tools effectiveness. Of course these statistical tools vary depending on the expected outcomes from a tool.

For each of the measurement outcomes listed above there are a range of statistical tools available to address two crucial issues. The most frequently mentioned issues are reliability and validity. Each of these issues break out into multiple specific applications. However, as a starting point reliability refers to consistency in measurements, and validity refers to representative measurement. Reliability asks the question -- "Are our measurements accurate and consistent?" Validity asks the question -- "Is the measurement a solid report of what one thinks is being measured?"

	Statistics used to evaluate how well tools differentiate or discriminate among classifications of cases.
	Statistics used to evaluate reliability of a scale
	Statistics used to evaluate the validity of a tool
	Issues of control, and cause and effect.