Duplicate Line Finder by langbot-team

# Duplicate Line Finder

Find / count / remove duplicate lines. Pure stdlib.

```
python scripts/run.py list.txt find
python scripts/run.py list.txt count
python scripts/run.py list.txt remove
python scripts/run.py list.txt only
cat list.txt | python scripts/run.py - remove -i -w
```

Flags: `-i` ignore case, `-w` ignore whitespace, `-s` sort.


---
name: duplicate-line-finder
display_name: Duplicate Line Finder
description: "Find, count, and remove duplicate lines in text with case/whitespace-insensitive options and order-preserving dedupe. Triggers: find duplicate lines, remove duplicates, dedupe, unique lines, count repeats."
---

# Duplicate Line Finder

Detect and handle duplicate lines in pasted text or a file.

## When to use
The user has a list or log and wants to find which lines repeat, count occurrences, or remove duplicates (keeping order).

## How to use
Run `scripts/run.py file [mode]`:
- `python scripts/run.py list.txt find` — show duplicated lines with their counts (default).
- `python scripts/run.py list.txt count` — show every line's occurrence count.
- `python scripts/run.py list.txt remove` — print the text deduped, original order preserved.
- `python scripts/run.py list.txt only` — print just the lines that are duplicated, once each.
- `cat list.txt | python scripts/run.py - remove -i -w` — dedupe via stdin, ignoring case and whitespace.

Flags: `-i` case-insensitive, `-w` ignore surrounding whitespace, `-s` sort output.

## Pitfalls
- Comparison can ignore case/whitespace, but the original first-seen line is what's printed.
- `find`/`count` prefix each line with a tab-separated count for easy parsing.
- Trailing newlines are normalized away (lines compared without line endings).