Psychology as a Science: Concepts, Research and Ethics · Lesson 2

From Impressions to Observation and Measurement

8 min read

Lesson objectives

Write an operational definition specifying the behavior, recording unit, time window and exclusions.
Choose observation or self-report to fit a question and explain what each method does not reveal.
Calculate a frequency, rate and proportion from small datasets while keeping denominators and units explicit.
Distinguish reliability from validity and a description of particular performance from a judgment about a person.

A facilitator in a fictional learning circle asks two people to record “good participation” in a written discussion. One counts every message; the other counts only messages related to the task. They obtain different numbers. This need not mean that either was careless: they used different meanings of the same term. Before collecting numbers, they need to agree on what counts, over which period and in what unit.

From a concept to an operational definition

A concept is the idea of interest, such as participation or understanding instructions. An operational definition states how a particular study will observe or measure an aspect of it. It does not reduce the entire concept to one number. Counting relevant contributions measures one aspect of visible participation, not the quality of listening or everything a person has learned. Keeping that distinction clear prevents the conclusion from exceeding the measure.

A useful definition specifies the recorded action, the inclusion rule, the recording unit, the period and exclusions. To calculate a proportion, it also specifies which opportunities form the denominator. Try the rule on clear examples and borderline cases before using it. Do not add a rule after seeing results to favor a participant or support a prior impression; when a rule is repaired, apply it consistently to all affected data.

Message in the fictional discussion	Count it?	Reason
Hello, everyone.	No	Greeting only.
Does the second example meet the condition in the task?	Yes	Question about task content.
We could compare the two options by how long each takes.	Yes	An idea that helps address the task.
I repeat my earlier suggestion without adding anything.	No	Unchanged repetition is excluded by the definition.
My audio is not working.	No	Technical issue rather than task content.
Here is a new example, with a reason for my choice.	Yes; once	One message even though it contains two elements.

Which method answers which question?

Structured observation records actions using an explicit rule; self-report records what someone says about their experience or judgment. To ask “How many messages met the definition?”, use the message record. To ask “How did participants describe the clarity of the instructions?”, obtain their reports. Such a report matters, but it remains what the person reported and may be affected by recall, interpretation of the question or the impression they wish to give. Observation alone likewise cannot tell us what silence means to someone.

We might examine one case in depth, survey several people or analyze an existing record. The choice depends on the question, not on one method having a more scientific-sounding name. A detailed case supplies context but does not automatically represent everyone. An existing record may omit a needed variable, such as connection availability during discussion. Agreement between observation and self-report may strengthen understanding; disagreement may reveal that they capture different aspects. Neither should be discarded automatically.

A count, a rate and a proportion are different

Write the unit beside a result, and when calculating a proportion check that the numerator is part of the denominator. With no opportunities, do not calculate 0 ÷ 0 or record a zero percentage; report that the proportion cannot be calculated from these data. When periods differ, a time rate may be more useful than a raw count, but does not itself resolve differences between tasks. An absent record should not mean “did not occur” unless recording actually covered the required period.

A consistent measure can miss its intended target

Reliability concerns measurement consistency under conditions suited to comparison; one example is agreement between people recording events with the same rules. Validity concerns whether the interpretation made from the measurement is appropriate for its intended purpose. If two people agree completely on the message count, that supports consistency for this record; it does not establish message count as a measure of understanding. Behavior can also genuinely change between days, so a difference should not automatically be called unreliability without examining conditions.

Exercise

Exercise 1 — In a fictional ten-minute discussion, count “requests to clarify the meaning of instructions,” not requests for task answers. The five messages are: “Should we select one answer or two?”, “Hello”, “I think the answer is the second option”, “Does compare mean giving one difference?”, and “How do I send the file?” Write a definition with a recording unit and exclusions, then calculate the matching count and its proportion of all five messages. Explain why the number alone does not represent every participant’s understanding.

Check your answer

Guided solution: Count a message once if it asks about the meaning of an instruction or the required number of selections during the ten minutes. Exclude greetings, proposed answers and technical questions about submission. The first and fourth messages count, giving 2 and a proportion of 2 ÷ 5 × 100 = 40%. The denominator is messages, not people. One person could ask twice and another could understand without writing, so do not conclude that 40% of participants failed to understand. Investigating submission difficulty would need a different definition that includes the fifth message.

Exercise

Exercise 2 — In fictional data, two counters accurately record Mazen pressing “Done” 8 times and Hala pressing it 4 times. Each has 8 opportunities. The system does not check whether answers are correct. The organizer wants to say Mazen learned twice as much as Hala. Calculate each pressing proportion, identify evidence of reliability and the gap in the learning interpretation, and propose appropriate additional evidence. What should be recorded if Hala had no opportunity at all?

Check your answer

Guided solution: Mazen has 8 ÷ 8 × 100 = 100%, and Hala 4 ÷ 8 × 100 = 50% of pressing opportunities. Agreement and accuracy of the counters concern recording presses, not validity of treating them as learning. Evidence should include a task assessing the content, such as solving a new example with explicit scoring criteria, rather than another button. More pressing can occur with or without understanding, so no relative amount of learning follows. In the alternative with no opportunities, the proportion is not calculable rather than zero; explain the missing opportunities instead of classifying ability.

Reference guide: “Analyzing Findings” in OpenStax’s Psychology 2e explains operational definitions, reliability and validity; “Approaches to Research” compares information-gathering methods. All messages and numbers here are fictional practice data. When applying this reasoning outside the exercise, start with nonsensitive information available with permission rather than collecting data about others merely to test a definition.

Lesson summary

Good measurement begins not with a number but with what that number represents. Define the behavior and counting rules, then choose a method that fits the question. Compare counts with clear opportunities and time windows. Reliability is not validity: people can agree perfectly when counting something that does not answer the intended question. Describe data in context rather than turning them into permanent judgments of people.