Transforming raw MIDI metadata into a clean, consistent, human‑editable dataset

After extracting the raw metadata from your MIDI library, the next step is to convert that unfiltered snapshot into a normalized, machine‑clean, human‑friendly spreadsheet. The Metadata Normalizer script takes the raw dump and produces a structured, consistent, encoding‑safe metadata file that is ready for human review.

This step is crucial because raw metadata often contains:

• mojibake (UTF‑8/CP1252 corruption)
• inconsistent punctuation
• multiple composers or ambiguous composer fields
• missing or duplicated titles
• stray comments or text events
• inconsistent capitalization
• whitespace artifacts

The normalizer script does not attempt to interpret the music — that’s the human’s job in Step 3. Instead, it ensures that every row is clean, consistent, and ready for curation.

What This Script Does

✔ Repairs encoding issues

The script attempts to reverse common CP1252/UTF‑8 mojibake patterns so titles and composers become readable again.

✔ Normalizes whitespace and punctuation

Extra spaces, stray underscores, and odd punctuation are cleaned up.

✔ Extracts the primary composer

If multiple composers are listed (e.g., “Smith / Jones”), the script selects the first one as the canonical composer.

✔ Ensures consistent column structure

Every row contains the same fields, even if some are empty.

✔ Writes a clean, normalized XLSX and CSV

The output file is named something like:

normalized_metadata.xlsx
normalized_metadata.csv

This file becomes the authoritative source for human review in Step 3.

Why This Step Matters

• It separates mechanical cleanup from human interpretation.
• It ensures the renamer script receives clean, predictable input.
• It prevents encoding issues from propagating into filenames.
• It makes the human review step dramatically easier.
• It creates a stable, reproducible metadata foundation for the entire library.

✅ FULL METADATA NORMALIZER SCRIPT

import csv
import re
from pathlib import Path
from openpyxl import Workbook, load_workbook

RAW_METADATA_FILE = "raw_metadata"   # base name only; script auto-detects CSV or XLSX
OUTPUT_XLSX = "normalized_metadata.xlsx"
OUTPUT_CSV = "normalized_metadata.csv"


# ------------------------------------------------------------
# Encoding cleanup
# ------------------------------------------------------------

def fix_encoding(s):
    """Attempt to repair common CP1252/UTF‑8 mojibake."""
    if not isinstance(s, str):
        return ""
    try:
        return s.encode("cp1252").decode("utf-8")
    except:
        return s


def clean_text(s):
    """Normalize whitespace and punctuation."""
    s = fix_encoding(s)
    s = s.replace("\u2018", "'").replace("\u2019", "'")
    s = s.replace("\u201C", '"').replace("\u201D", '"')
    s = s.replace("\u2013", "-").replace("\u2014", "-")
    s = s.replace("\u2026", "...")
    s = re.sub(r"\s+", " ", s)
    return s.strip()


# ------------------------------------------------------------
# Composer normalization
# ------------------------------------------------------------

def extract_primary_composer(composer):
    """Use only the first composer if multiple are listed."""
    composer = clean_text(composer)
    parts = re.split(r"[\/,&]| and ", composer, flags=re.IGNORECASE)
    return parts[0].strip()


# ------------------------------------------------------------
# Load raw metadata
# ------------------------------------------------------------

def load_raw_metadata(base_name):
    csv_path = Path(base_name + ".csv")
    xlsx_path = Path(base_name + ".xlsx")

    if csv_path.exists():
        with open(csv_path, newline="", encoding="utf-8") as f:
            return list(csv.DictReader(f))

    if xlsx_path.exists():
        wb = load_workbook(xlsx_path)
        ws = wb.active
        rows = list(ws.iter_rows(values_only=True))
        headers = rows[0]
        return [dict(zip(headers, row)) for row in rows[1:]]

    raise FileNotFoundError("No raw metadata file found.")


# ------------------------------------------------------------
# Write output files
# ------------------------------------------------------------

def write_xlsx(rows, path):
    wb = Workbook()
    ws = wb.active
    ws.append(list(rows[0].keys()))
    for row in rows:
        ws.append(list(row.values()))
    wb.save(path)


def write_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)


# ------------------------------------------------------------
# Main normalization logic
# ------------------------------------------------------------

def main():
    rows = load_raw_metadata(RAW_METADATA_FILE)
    normalized = []

    for row in rows:
        filename = row.get("filename", "").strip()
        title_raw = row.get("title_raw", "")
        composer_raw = row.get("composer_raw", "")
        comments_raw = row.get("comments_raw", "")
        tempo = row.get("tempo", "")
        time_sig = row.get("time_signature", "")
        key_sig = row.get("key_signature", "")

        title = clean_text(title_raw)
        composer = clean_text(composer_raw)
        comments = clean_text(comments_raw)
        primary_composer = extract_primary_composer(composer)

        normalized.append({
            "filename": filename,
            "title": title,
            "composer": composer,
            "primary_composer": primary_composer,
            "comments": comments,
            "tempo": tempo,
            "time_signature": time_sig,
            "key_signature": key_sig,
        })

    write_xlsx(normalized, OUTPUT_XLSX)
    write_csv(normalized, OUTPUT_CSV)

    print("\nNormalized metadata written to:")
    print(f"  {OUTPUT_XLSX}")
    print(f"  {OUTPUT_CSV}")
    print("\nNormalization complete.")


if __name__ == "__main__":
    main()

Summary

The Metadata Normalizer is the bridge between raw extraction and human curation.
It produces a clean, consistent, encoding‑safe metadata file that is easy to review and ready for the renaming stage. This step ensures that the entire workflow remains deterministic, reversible, and archivally sound.