Core data concepts · Lesson 1

Representing data: structure is not a file extension

8 min read

Lesson objectives

  • Classify a record by its actual structure and identify what is needed to extract its fields without confusing file type and data store
Reading tools

The core data concepts domain asks you to distinguish structured, semi-structured, and unstructured data. Start with the record the application must process, rather than the name of a cloud service. Structured data follows a relatively stable schema: known fields, a type for each value, and rows whose constraints can be checked. Appointment records with appointment_id, patient_id, slot, and status fit that description if every row follows those fields, even when they arrive as a CSV file. A comma does not make content structured by itself. A CSV cell can contain free text or nested JSON, and files in one feed can have different columns. Ask which fields software may rely on and which constraints can be enforced before selecting a database.

Semi-structured data carries tags or keys identifying its parts without requiring every record to share the same set of columns. One JSON visit note might have visit_id and symptoms, while another also includes attachments or triage_score. An analyst can extract a key when it exists and define what happens when another document lacks it. Semi-structured does not mean disorganized: keys and nested structure are meaningful. A scanned form or an audio recording, by contrast, does not expose ready-made fields for querying the diagnosis or date inside its content. Its filename, upload time, and media type may be structured metadata. The content remains unstructured until an extraction step is performed, and a returned transcription is not automatically accurate enough for a decision.

For a practical classification, separate the content before processing from the questions you intend to ask. To count confirmed appointments per day, consistent status and slot fields and an appointment identifier make aggregation and validation possible. To count documents mentioning a drug allergy, the number of PDF files is insufficient: define a text extraction method, measure errors, and add appropriate human review. A JSON feed with variable note details permits queries for selected keys if missing fields have an explicit policy. These are data decisions before they are service decisions. A structured file can reside in object storage; metadata about an image can reside in a table; and the image content can still be unstructured. Avoid classifying all three as the same thing because they share a container.

Before turning classification into a storage decision, test four properties of each input. First, schema stability: can a record lacking required fields be rejected, or is a missing field expected? Second, query level: does the user ask about a whole row, a key inside a document, or meaning hidden in an image or recording? Third, transformation: what step must occur before a measure can be computed, and can that step introduce error or loss? Fourth, counting unit: a file, appointment, person, and mention of a term are different units. In the example, 72/120 gives a 60% confirmed-appointment share if the 120 appointments are valid and distinct. It does not mean 60% of patients confirmed, because one patient could hold several appointments. Likewise, 12/20 means 60% of note documents contain the key, not that the average triage score is 60%; the score values are unknown. Keeping the denominator explicit prevents a simple exam distinction from becoming a misleading workplace report. If the JSON shape changes, define a policy for absent fields: exclude a document from a value-dependent calculation or label it unrecorded, rather than silently filling zero. If software extracts text from an image, review a sample before counting sensitive mentions and document error at a level appropriate to the decision. Classification identifies the next work; it does not guarantee extraction accuracy.

An invoice image arrives with a table produced by a reading tool. The output has a structured shape, but its values still need a separate accuracy check. Compare five cells with the image, including the name, amount and date. A decimal separator confused with a thousands separator can produce a valid numeric value that is nevertheless wrong. Record the source value, extracted value and correction reason. Keep the image link so a colleague can repeat the check. Establish the row grain before counting. An invoice containing three items can produce three rows while remaining one document. Count distinct document identifiers for invoices and validated rows for items. Output structure, extraction accuracy and counting meaning are separate questions.

Lesson summary

Classify a record by its actual structure and identify what is needed to extract its fields without confusing file type and data store.