Deduplicate text without losing records: a six-line worked example

01Decide which lines are allowed to count as equal

For this exercise, duplicates are identical whole lines and the first occurrence keeps its position. Do not trim nonblank lines, fold case or sort. These are editorial sample labels, not four identified people. Include the rule in the handoff so a recipient understands why similar-looking entries remain.

1. 1
   Keep a read-only original and separate 01-original, 02-no-blank and 03-exact-dedup copies. Their names identify stages; never overwrite the source.
2. 2
   If the list represents attendees or customers, request IDs and use an identity-review workflow. Two occurrences of Ada do not establish that they are one person.
3. 3
   If the business rule also ignores surrounding spaces, document that as a separate change. Leave it out of this main exercise so each reduction remains explainable.

02Follow the path: blank removal, then exact deduplication

The input has six lines. Line 1 starts with one ordinary space before Ada; line 4 is empty. The block preserves that space, so do not tidy the pasted input by hand.

1. 1
   Open Blank cleaner, paste the original and choose Process text. Save the result as 02-no-blank and count five nonblank lines. Three lines would mean extra normalization already occurred.
2. 2
   Open Text deduplication, paste that result and run it. Compare all four reference lines, especially the two Ada variants and ada, rather than checking only the count.
3. 3
   Save 03-exact-dedup before copying or downloading it. If space normalization is required, branch from the saved 02 copy and document its rule and new count separately.

 Ada
Ada
Bo

Bo
ada

 Ada
Ada
Bo
ada

If you get three or two lines, check manual trimming, case folding or an assumed identity match. Do not recreate deleted records from memory.

03Explain every deletion with a row ledger

Only the blank line explains 6→5; only the second identical Bo explains 5→4. Blank removal uses trim to decide whether a line is empty, but retains the original nonblank line. Exact deduplication does not determine spelling equivalence, identity or the latest version.

• For 5→3, check whether someone removed the leading space. Review that as an independent normalization rule.
• For 4→2, check whether trimming and case folding were both applied. They may suit some labels but damage case-sensitive identifiers.
• A count of four can still hide one wrong deletion and one missed duplicate. Compare the actual lines with the reference.

04Replacing cat also changes the cat inside catalog

The current find-and-replace entry replaces all literal, case-sensitive occurrences; it does not interpret regular expressions. With invented input cat catalog Cat, replacing cat with dog yields dog dogalog Cat. It does not identify whole words or match Cat automatically. Test word-boundary expressions in a tool that supports regex rather than assuming that \bcat\b has that meaning here.

When only one field’s fixed value should change, isolate that field or review each occurrence. A short global replacement across contracts, code or brand names can affect unintended text. Test a wanted match, an occurrence inside a longer word and a different-case example before processing the full input.

05Many displayed differences need not mean many rewritten sentences

This text-diff entry compares lines at the same position rather than finding a minimum-edit paragraph alignment. If the old text has A, B and C and the new text inserts X at the top, the shifted lines also appear different. A minus sign in that output does not by itself establish that the content was deleted.

For a short list, use numbered originals to explain moved rows; longer documents may need a comparison method that aligns paragraphs. Do not sort away meaningful business order merely to make positions agree. Keep original order and a cleaned copy so actual rewrites, moves and deletions remain distinguishable.

06Optional tools for a different input or next step

• Remove blank lines: Check whether blank lines mark separate groups before removing them.
• Deduplicate text entries: Use exact duplicate removal only after deciding whether whitespace and case differences matter.
• Sort a text list: Keep a numbered original when input order carries priority; inspect numeric-looking labels after sorting.
• Change letter case: Apply to the intended fields only; brand capitalization, acronyms and case-sensitive identifiers may need exceptions.
• Replace a known pattern: Test a term inside and outside a longer word before replacing all matches, then inspect the difference.
• Review text differences: Compare the original with the final copy and separate intended cleanup from accidental content changes.
• Extract readable HTML text: Inspect headings, paragraph breaks and table relationships after removing tags; a plain string may lose structure.
• Convert simplified and traditional Chinese: Choose the intended direction and manually review names, regional terms and fixed product wording.
• Mask or encrypt sensitive text: Review the entire visible result after masking; for password encryption, keep the password separately and test decryption on a non-sensitive sample.
• Reverse a test string: Use for a deliberate text exercise, with a multilingual sample; reversing characters does not reverse the linguistic meaning.
• Generate dummy identifiers: Set length and allowed characters for dummy records, then label them as synthetic data.
• Create layout placeholder text: Choose enough paragraphs to test wrapping, then replace placeholders before publishing or collecting feedback on meaning.

07Hand off an explained change with a recovery path

Also try cat catalog Cat with literal find cat and replacement dog. The expected output is dog dogalog Cat. The current tool is case-sensitive and replaces text inside words too; it does not enforce word boundaries. This example follows MDN’s literal replacement semantics and the implementation, without requiring a regular expression.

MDN explains regular-expression patterns and flags; use boundary cases when a cleaning rule depends on pattern matching.

Use stable IDs when names collide. If the result will be Markdown, check the rendered structure too.