Creating a complete, unfiltered snapshot of your MIDI library

The first step in normalizing a MIDI library is to capture everything we can possibly know about each file before any cleanup or renaming occurs. This is the “archaeological dig” phase: we gather the raw material that all later steps depend on.

A dedicated metadata‑dump script scans the target folder and extracts:

• the current filename
• any embedded title information
• any embedded composer information
• comments or text events
• tempo markings
• time signatures
• key signatures
• other recoverable metadata from Track 0 or additional tracks

The script writes all of this into a single raw metadata spreadsheet (XLSX).
This file is intentionally unedited and uncorrected — it reflects the true state of the library, including inconsistencies, encoding issues, and missing data.

Why this step matters

1. It preserves the original state of the library

Before any normalization or renaming happens, this spreadsheet becomes a permanent record of:

• what the filenames were
• what metadata was present
• what metadata was missing
• how the files were structured internally

If anything ever needs to be undone, re‑checked, or re‑interpreted, this file is the fallback.

2. It provides the raw material for normalization

Later scripts rely on this dump to:

• repair encoding issues
• standardize title and composer fields
• extract primary composers
• detect malformed metadata
• build a consistent metadata table

Without this raw snapshot, normalization would be guesswork.

3. It separates extraction from interpretation

This is a key design principle of the workflow.

Extraction is mechanical.
Interpretation is human.
Normalization is algorithmic.
Renaming is deterministic.

By keeping these phases separate, the pipeline stays transparent and reversible.

What the output looks like

The raw metadata file is an XLSX spreadsheet with one row per MIDI file and columns such as:

• filename
• title_raw
• composer_raw
• comments_raw
• tempo
• time_signature
• key_signature

The exact column names may vary depending on the script version, but the goal is always the same: capture everything without altering it.

How this file is used later

The raw metadata file is:

• a reference archive
• a safety net
• the input to the metadata normalizer
• a historical snapshot of the library before cleanup

The normalized metadata file (created in the next step) uses this raw dump as its source, but the raw file itself is never overwritten. It remains a trustworthy record of the original state.

Summary

Extracting the raw metadata is the foundation of the entire normalization workflow.
It creates a complete, unfiltered, archival‑grade snapshot of the MIDI library before any cleanup occurs. This ensures that every later step — normalization, human review, and renaming — is based on accurate, preserved information and can be repeated or reversed at any time.

This script performs a complete, unfiltered scan of every MIDI file in the folder and extracts all recoverable metadata. The output is saved as both raw_metadata.xlsx and raw_metadata.csv.

This file is never edited and never overwritten — it serves as the archival snapshot of the library before normalization.

import csv
import re
from pathlib import Path
from openpyxl import Workbook
import mido

# Folder containing the MIDI files to scan
MIDI_FOLDER = Path(".")

# Output file (raw, unnormalized metadata)
OUTPUT_XLSX = "raw_metadata.xlsx"
OUTPUT_CSV = "raw_metadata.csv"


# ------------------------------------------------------------
# Encoding helpers
# ------------------------------------------------------------

def fix_encoding(s):
    """Attempt to repair common CP1252/UTF‑8 mojibake."""
    if not isinstance(s, str):
        return ""
    try:
        return s.encode("cp1252").decode("utf-8")
    except:
        return s


# ------------------------------------------------------------
# Metadata extraction helpers
# ------------------------------------------------------------

def extract_meta_from_track(track):
    """Extract title, composer, comments, tempo, key, and time signature."""
    title = ""
    composer = ""
    comments = []
    tempo = ""
    time_sig = ""
    key_sig = ""

    for msg in track:
        if msg.type == "track_name" and not title:
            title = msg.name
        elif msg.type == "text":
            comments.append(msg.text)
        elif msg.type == "copyright" and not composer:
            composer = msg.text
        elif msg.type == "set_tempo" and not tempo:
            tempo = str(mido.tempo2bpm(msg.tempo))
        elif msg.type == "time_signature" and not time_sig:
            time_sig = f"{msg.numerator}/{msg.denominator}"
        elif msg.type == "key_signature" and not key_sig:
            key_sig = msg.key

    return (
        fix_encoding(title),
        fix_encoding(composer),
        fix_encoding(" | ".join(comments)),
        tempo,
        time_sig,
        key_sig,
    )


def extract_metadata(path):
    """Extract metadata from a single MIDI file."""
    try:
        mid = mido.MidiFile(path)
    except Exception as e:
        return {
            "filename": path.name,
            "title_raw": "",
            "composer_raw": "",
            "comments_raw": f"[ERROR READING FILE: {e}]",
            "tempo": "",
            "time_signature": "",
            "key_signature": "",
        }

    # Prefer Track 0, but fall back if needed
    track = mid.tracks[0] if mid.tracks else []

    title, composer, comments, tempo, time_sig, key_sig = extract_meta_from_track(track)

    return {
        "filename": path.name,
        "title_raw": title,
        "composer_raw": composer,
        "comments_raw": comments,
        "tempo": tempo,
        "time_signature": time_sig,
        "key_signature": key_sig,
    }


# ------------------------------------------------------------
# Write XLSX and CSV
# ------------------------------------------------------------

def write_xlsx(rows, path):
    wb = Workbook()
    ws = wb.active
    ws.append(list(rows[0].keys()))
    for row in rows:
        ws.append(list(row.values()))
    wb.save(path)


def write_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)


# ------------------------------------------------------------
# Main
# ------------------------------------------------------------

def main():
    midi_files = sorted(MIDI_FOLDER.glob("*.mid"))
    rows = []

    print(f"Scanning {len(midi_files)} MIDI files...")

    for path in midi_files:
        print(f"Extracting: {path.name}")
        rows.append(extract_metadata(path))

    if rows:
        write_xlsx(rows, OUTPUT_XLSX)
        write_csv(rows, OUTPUT_CSV)

    print("\nRaw metadata written to:")
    print(f"  {OUTPUT_XLSX}")
    print(f"  {OUTPUT_CSV}")
    print("\nExtraction complete.")


if __name__ == "__main__":
    main()

Transforming raw MIDI metadata into a clean, consistent, human‑editable dataset

After extracting the raw metadata from your MIDI library, the next step is to convert that unfiltered snapshot into a normalized, machine‑clean, human‑friendly spreadsheet. The Metadata Normalizer script takes the raw dump and produces a structured, consistent, encoding‑safe metadata file that is ready for human review.

This step is crucial because raw metadata often contains:

• mojibake (UTF‑8/CP1252 corruption)
• inconsistent punctuation
• multiple composers or ambiguous composer fields
• missing or duplicated titles
• stray comments or text events
• inconsistent capitalization
• whitespace artifacts

The normalizer script does not attempt to interpret the music — that’s the human’s job in Step 3. Instead, it ensures that every row is clean, consistent, and ready for curation.

What This Script Does

✔ Repairs encoding issues

The script attempts to reverse common CP1252/UTF‑8 mojibake patterns so titles and composers become readable again.

✔ Normalizes whitespace and punctuation

Extra spaces, stray underscores, and odd punctuation are cleaned up.

✔ Extracts the primary composer

If multiple composers are listed (e.g., “Smith / Jones”), the script selects the first one as the canonical composer.

✔ Ensures consistent column structure

Every row contains the same fields, even if some are empty.

✔ Writes a clean, normalized XLSX and CSV

The output file is named something like:

normalized_metadata.xlsx
normalized_metadata.csv

This file becomes the authoritative source for human review in Step 3.

Why This Step Matters

• It separates mechanical cleanup from human interpretation.
• It ensures the renamer script receives clean, predictable input.
• It prevents encoding issues from propagating into filenames.
• It makes the human review step dramatically easier.
• It creates a stable, reproducible metadata foundation for the entire library.

✅ FULL METADATA NORMALIZER SCRIPT

import csv
import re
from pathlib import Path
from openpyxl import Workbook, load_workbook

RAW_METADATA_FILE = "raw_metadata"   # base name only; script auto-detects CSV or XLSX
OUTPUT_XLSX = "normalized_metadata.xlsx"
OUTPUT_CSV = "normalized_metadata.csv"


# ------------------------------------------------------------
# Encoding cleanup
# ------------------------------------------------------------

def fix_encoding(s):
    """Attempt to repair common CP1252/UTF‑8 mojibake."""
    if not isinstance(s, str):
        return ""
    try:
        return s.encode("cp1252").decode("utf-8")
    except:
        return s


def clean_text(s):
    """Normalize whitespace and punctuation."""
    s = fix_encoding(s)
    s = s.replace("\u2018", "'").replace("\u2019", "'")
    s = s.replace("\u201C", '"').replace("\u201D", '"')
    s = s.replace("\u2013", "-").replace("\u2014", "-")
    s = s.replace("\u2026", "...")
    s = re.sub(r"\s+", " ", s)
    return s.strip()


# ------------------------------------------------------------
# Composer normalization
# ------------------------------------------------------------

def extract_primary_composer(composer):
    """Use only the first composer if multiple are listed."""
    composer = clean_text(composer)
    parts = re.split(r"[\/,&]| and ", composer, flags=re.IGNORECASE)
    return parts[0].strip()


# ------------------------------------------------------------
# Load raw metadata
# ------------------------------------------------------------

def load_raw_metadata(base_name):
    csv_path = Path(base_name + ".csv")
    xlsx_path = Path(base_name + ".xlsx")

    if csv_path.exists():
        with open(csv_path, newline="", encoding="utf-8") as f:
            return list(csv.DictReader(f))

    if xlsx_path.exists():
        wb = load_workbook(xlsx_path)
        ws = wb.active
        rows = list(ws.iter_rows(values_only=True))
        headers = rows[0]
        return [dict(zip(headers, row)) for row in rows[1:]]

    raise FileNotFoundError("No raw metadata file found.")


# ------------------------------------------------------------
# Write output files
# ------------------------------------------------------------

def write_xlsx(rows, path):
    wb = Workbook()
    ws = wb.active
    ws.append(list(rows[0].keys()))
    for row in rows:
        ws.append(list(row.values()))
    wb.save(path)


def write_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)


# ------------------------------------------------------------
# Main normalization logic
# ------------------------------------------------------------

def main():
    rows = load_raw_metadata(RAW_METADATA_FILE)
    normalized = []

    for row in rows:
        filename = row.get("filename", "").strip()
        title_raw = row.get("title_raw", "")
        composer_raw = row.get("composer_raw", "")
        comments_raw = row.get("comments_raw", "")
        tempo = row.get("tempo", "")
        time_sig = row.get("time_signature", "")
        key_sig = row.get("key_signature", "")

        title = clean_text(title_raw)
        composer = clean_text(composer_raw)
        comments = clean_text(comments_raw)
        primary_composer = extract_primary_composer(composer)

        normalized.append({
            "filename": filename,
            "title": title,
            "composer": composer,
            "primary_composer": primary_composer,
            "comments": comments,
            "tempo": tempo,
            "time_signature": time_sig,
            "key_signature": key_sig,
        })

    write_xlsx(normalized, OUTPUT_XLSX)
    write_csv(normalized, OUTPUT_CSV)

    print("\nNormalized metadata written to:")
    print(f"  {OUTPUT_XLSX}")
    print(f"  {OUTPUT_CSV}")
    print("\nNormalization complete.")


if __name__ == "__main__":
    main()

Summary

The Metadata Normalizer is the bridge between raw extraction and human curation.
It produces a clean, consistent, encoding‑safe metadata file that is easy to review and ready for the renaming stage. This step ensures that the entire workflow remains deterministic, reversible, and archivally sound.

The curatorial step that transforms machine‑clean metadata into library‑grade truth

Once the metadata has been extracted (Step 1) and normalized (Step 2), the next phase is the most important one in the entire workflow: human review. This is where the machine’s best guesses are checked, corrected, clarified, and elevated into a clean, authoritative dataset.

The scripts can repair encoding, normalize punctuation, and extract primary composers — but they cannot interpret ambiguous titles, resolve conflicting metadata, or understand historical context. That’s where human judgment comes in.

This step produces the final, curated metadata file that the renamer script will use to generate deterministic filenames and rewrite internal MIDI metadata.

What You’re Working With

You will be editing:

normalized_metadata.xlsx

This file contains:

• filename
• title
• composer
• primary_composer
• comments
• tempo
• time_signature
• key_signature

Every row corresponds to one MIDI file.

Your job is to turn this into the canonical metadata for the entire library.

What to Look For During Review

Below are the most common issues you’ll encounter and how to fix them.

1. Incorrect or incomplete titles

Examples:

• “Long Way Tipperary” → “It’s a Long Way to Tipperary”
• “SabreDance” → “Sabre Dance”
• “Unknown Title” → look up the correct name if possible

Best practice: 
Use the most widely recognized title, not the filename’s guess.

2. Multiple or ambiguous composers

Examples:

• “Smith / Jones” → choose the primary composer
• “Traditional / Arr. X” → composer = “Traditional”, comments = “Arr. X”
• “Unknown” → leave blank or mark as “Traditional” if appropriate

Best practice: 
Composer should reflect authorship, not arrangement.

3. Medleys and multi‑part works

Examples:

• “Disney Medley”
• “Beatles Medley (Part 1)”
• “Star Wars Suite”

Best practice: 
Keep the title exactly as you want it to appear in the final filename.
If the medley has a known canonical name, use it.

4. Encoding artifacts the normalizer couldn’t fix

Examples:

• “François” → “François”
• “It’s” → “It’s”

Best practice: 
Correct these manually — the renamer will preserve your edits.

5. Comments that belong elsewhere

Sometimes comments contain:

• arranger names
• performance notes
• copyright info
• leftover text events

Best practice: 
Move arranger names to comments, not composer.
Remove junk text entirely.

6. Missing metadata

If a row is missing:

• composer
• title
• comments

…fill in what you can.

Best practice: 
If you cannot determine the composer, leave it blank — the renamer handles this gracefully.

General Best Practices

Be consistent

If you choose “Traditional” for folk tunes, use it everywhere.
If you prefer “arr. X” in comments, use that format consistently.

Use proper capitalization

The renamer does not change capitalization — what you write is what you get.

Avoid punctuation that normalizes poorly

Characters like /, ?, :, and * will be stripped or replaced.
Prefer simple ASCII punctuation.

Keep titles human‑readable

The renamer will normalize filenames, not titles.
Titles should look good in a catalog.

Don’t worry about filename formatting

The renamer handles:

• underscores
• double dashes
• normalization
• collision avoidance

You only need to provide clean metadata.

When You’re Done

When the spreadsheet is fully reviewed and corrected, save it as:

normalized_metadata_edit.xlsx

or export as:

normalized_metadata_edit.csv

This curated file becomes the authoritative source of truth for Step 4: the renaming pass.

Summary

The Human Review & Cleanup step is where the library becomes yours.
It’s the moment where machine‑generated guesses are replaced with curated, historically accurate, musically meaningful metadata. This curated file ensures that the final filenames — and the internal MIDI metadata — reflect the true identity of each piece.

Once this step is complete, the renamer script can safely and deterministically transform the entire library into a clean, consistent, future‑proof collection.

Deterministic filename generation and internal metadata repair

Once the metadata has been extracted (Step 1) and normalized + human‑curated (Steps 2–3), the final stage of the workflow is to apply the new metadata to the actual MIDI files. This renamer script is the engine that transforms a chaotic library into a clean, consistent, future‑proof collection.

The script uses the curated metadata spreadsheet as the single source of truth and performs two major tasks:

1. Generate clean, deterministic filenames
2. Rewrite internal MIDI metadata safely and consistently

This step is fully automated, but intentionally includes a dry‑run mode so you can preview all changes before committing them.

What the Renamer Does

✔ Builds filenames using a consistent pattern

The script generates filenames in the form:

composer--title.mid

Composer and title are normalized separately, ensuring the double dash (--) remains a semantic separator rather than being collapsed by normalization.

✔ Ensures collision‑safe naming

If two files normalize to the same base name, the script automatically appends:

_filename_1.mid
_filename_2.mid

This prevents accidental overwrites.

✔ Rewrites internal MIDI metadata

The script updates:

• Track name
• Text meta events
• Copyright
• Comments

All text is passed through a Latin‑1‑safe filter so Mido can write it without errors.

✔ Produces a rename report

Every run generates:

rename_report.csv

mapping:

old_filename,new_filename

This makes the process reversible and auditable.

✔ Supports dry‑run mode

Before making any changes, you can run:

DRY_RUN = True

to preview all renames without touching any files.

When to Use This Script

Run the renamer only after:

1. Raw metadata has been extracted
2. Metadata has been normalized
3. You have manually reviewed and corrected the metadata

4. You have saved the curated file as:

   normalized_metadata_edit_fixed.csv
   

This curated file becomes the authoritative source for renaming.

✅ FULL RENAMER SCRIPT

(Copy/paste directly into your Contraptions page)

import csv
import re
import mido
from mido import MetaMessage
from pathlib import Path
from openpyxl import load_workbook

# Base name only — script auto-detects .csv or .xlsx
METADATA_FILE = "normalized_metadata_edit_fixed"

MIDI_FOLDER = Path(".")
REPORT_PATH = "rename_report.csv"

DRY_RUN = True   # ← Set to False to actually rename + modify files


# ------------------------------------------------------------
# Encoding cleanup
# ------------------------------------------------------------

def fix_encoding(s):
    if not isinstance(s, str):
        return ""
    try:
        return s.encode("cp1252").decode("utf-8")
    except:
        return s


def latin1_safe(s):
    """Convert text to something Mido can encode (Latin-1)."""
    if not isinstance(s, str):
        return ""

    replacements = {
        "\u2018": "'",   # left single quote
        "\u2019": "'",   # right single quote
        "\u201C": '"',   # left double quote
        "\u201D": '"',   # right double quote
        "\u2013": "-",   # en dash
        "\u2014": "-",   # em dash
        "\u2026": "...", # ellipsis
    }

    for bad, good in replacements.items():
        s = s.replace(bad, good)

    # Final fallback: replace anything Latin-1 can't encode
    return s.encode("latin-1", "replace").decode("latin-1")


# ------------------------------------------------------------
# Normalization helpers
# ------------------------------------------------------------

def normalize_filename(s):
    s = fix_encoding(s)
    s = re.sub(r"[^\w]+", "_", s)
    s = re.sub(r"_+", "_", s)
    s = s.strip("_")
    return s.lower()


def extract_primary_composer(composer):
    """Use only the first composer name."""
    composer = fix_encoding(composer)
    parts = re.split(r"[\/,&]| and ", composer, flags=re.IGNORECASE)
    primary = parts[0].strip()
    return primary


def ensure_unique_name(base_name, used_names):
    name = base_name
    counter = 1
    while name in used_names:
        name = f"{base_name}_{counter}"
        counter += 1
    used_names.add(name)
    return name


# ------------------------------------------------------------
# MIDI metadata helpers
# ------------------------------------------------------------

def replace_or_insert(track, meta_type, text):
    """Replace or insert a meta message of a given type, using correct attribute."""
    uses_name = meta_type in ("track_name", "instrument_name", "device_name")

    for msg in track:
        if msg.type == meta_type:
            if uses_name:
                msg.name = text
            else:
                msg.text = text
            return

    # Insert new message at top
    if uses_name:
        new_msg = MetaMessage(meta_type, name=text, time=0)
    else:
        new_msg = MetaMessage(meta_type, text=text, time=0)

    track.insert(0, new_msg)


def append_comment_safely(track, comments):
    if comments:
        replace_or_insert(track, "text", comments)


# ------------------------------------------------------------
# File reader (auto-detect CSV vs XLSX)
# ------------------------------------------------------------

def load_metadata_rows(base_name):
    csv_path = Path(base_name + ".csv")
    xlsx_path = Path(base_name + ".xlsx")

    if csv_path.exists():
        # FIX: read as UTF-8
        with open(csv_path, newline="", encoding="utf-8") as f:
            return list(csv.DictReader(f))

    if xlsx_path.exists():
        wb = load_workbook(xlsx_path)
        ws = wb.active
        rows = list(ws.iter_rows(values_only=True))
        headers = rows[0]
        return [dict(zip(headers, row)) for row in rows[1:]]

    raise FileNotFoundError("Neither CSV nor XLSX found.")


# ------------------------------------------------------------
# Main processing
# ------------------------------------------------------------

def process_midi_file(old_path, title, composer, comments, used_names):
    title = fix_encoding(title)
    composer = fix_encoding(composer)
    comments = fix_encoding(comments)

    primary_composer = extract_primary_composer(composer)

    # Normalize separately
    norm_title = normalize_filename(title)
    norm_composer = normalize_filename(primary_composer)

    # Prevent composer == title duplication
    if norm_composer == norm_title or norm_composer == "unknown":
        base = norm_title
    else:
        base = f"{norm_composer}--{norm_title}"

    base = ensure_unique_name(base, used_names)
    new_name = base + ".mid"
    new_path = old_path.with_name(new_name)

    if DRY_RUN:
        return new_name, False

    mid = mido.MidiFile(old_path)
    track = mid.tracks[0]

    safe_title = latin1_safe(title)
    safe_composer = latin1_safe(primary_composer)
    safe_comments = latin1_safe(comments)

    replace_or_insert(track, "track_name", safe_title)
    replace_or_insert(track, "text", safe_title)
    replace_or_insert(track, "copyright", safe_composer)
    append_comment_safely(track, safe_comments)

    mid.save(new_path)

    if new_path != old_path:
        old_path.unlink()

    return new_name, True


def main():
    used_names = set()
    report_rows = []

    rows = load_metadata_rows(METADATA_FILE)

    for row in rows:
        filename = str(row.get("filename", "")).strip()
        title = str(row.get("title", "")).strip()
        composer = str(row.get("composer", "")).strip()
        comments = str(row.get("comments", "")).strip()

        midi_path = MIDI_FOLDER / filename

        if not midi_path.exists():
            report_rows.append([filename, "MISSING"])
            print(f"Missing file: {filename}")
            continue

        new_name, changed = process_midi_file(
            midi_path, title, composer, comments, used_names
        )

        report_rows.append([filename, new_name])
        print(f"{'(dry-run)' if DRY_RUN else ''} {filename} → {new_name}")

    with open(REPORT_PATH, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["old_filename", "new_filename"])
        writer.writerows(report_rows)

    print("\nReport written to:", REPORT_PATH)
    print("Dry-run mode is", DRY_RUN)
    print("\nAll operations completed successfully.")


if __name__ == "__main__":
    main()

Summary

The renamer script is the final, transformative step in the normalization workflow.
It takes your curated metadata and produces a clean, deterministic, collision‑safe set of filenames while repairing internal MIDI metadata. With dry‑run mode and a detailed rename report, the process is transparent, reversible, and safe.

Once this step is complete, your MIDI library is fully normalized and ready for archival, playback, or further processing.