CSV Data Cleaner

Clean a messy CSV file into a tidy, analysis-ready table using a dependency-free Python script (standard library only, so it runs anywhere without installing pandas).

When to use

• The user provides a .csv (or Excel-exported table) that has trailing spaces, blank rows, repeated headers, or duplicate records.
• The user asks to "清洗", "去重", "整理", "规整", or "预处理" a data file.
• The user wants a quick report of what was changed before trusting the data.

How to run

Use the bundled script scripts/clean_csv.py:

python scripts/clean_csv.py <input.csv> \
  -o cleaned.csv \
  --report clean_report.json \
  [--dropna-col "客户ID"]   # 删除该列为空的行
  [--no-dedup]              # 关闭去重

Arguments:

• input.csv — source file (required).
• -o / --output — cleaned output path (default cleaned.csv).
• --report — JSON change report path (default clean_report.json).
• --dropna-col — drop rows where this named column is empty.
• --no-dedup — keep duplicate rows (off by default; dedup is on).

What it does (in order)

1. Read the file as UTF-8 (with BOM tolerant).
2. Trim whitespace in every cell.
3. Drop fully empty rows.
4. Normalize headers — collapse internal spaces, strip surrounding spaces.
5. Drop fully empty columns.
6. Deduplicate rows (keeps first occurrence).
7. Optionally drop rows missing a key column (--dropna-col).
8. Write the cleaned CSV and a JSON report with counts of each action performed.

Example report output

{
  "input_rows": 1042,
  "empty_rows_dropped": 37,
  "duplicates_dropped": 12,
  "empty_cols_dropped": 1,
  "cells_trimmed": 880,
  "headers_normalized": 3,
  "output_rows": 993
}

Tips

• Keep the script as the single source of truth; do not re-implement cleaning inline.
• For very large files, this streams row-by-row and is memory friendly.
• If the user wants different rules (e.g., case-insensitive dedup, numeric coercion), extend scripts/clean_csv.py rather than describing steps in prose.

Boundaries

• Only the standard library is used; do not assume pandas is installed.
• It does not infer or invent cell values — it only removes/normalizes what is already there.
• Cover and icon live in assets/ (csv-cover.png / csv-icon.png) for SkillHub display.