Mode
Lines
Data-cleaning workflow
How to deduplicate a single column of survey IDs locally
Create a distinct list from an already exported one-ID-per-line survey column. This workflow does not merge full responses, choose which record survives, or parse a structured CSV table.
Use this workflow only for one exported column
Decide the ID rules before deduplicating
The source system or data owner must answer these questions; the duplicate remover cannot infer them from appearance.
- Are IDs case-sensitive?
- Are leading and trailing spaces accidental?
- Are leading zeros significant?
- Is an empty line a missing value or export noise?
- Does first or last occurrence carry a defined meaning?
- Do repeated IDs represent accidental duplication, repeat attempts, or legitimate longitudinal records?
Synthetic example
Assume the survey system documents IDs as case-insensitive and outer whitespace as insignificant. Use the visible settings below.
Show
Result
Keep
First
Ignore case
On only if documented
Trim whitespace
On for outer export noise
Remove empty lines
On
Sort lines
Off
SURV-1042
SURV-1048
SURV-1042
SURV-1051
surv-1051
SURV-1063
SURV-1063SURV-1042
SURV-1048
SURV-1051
SURV-10637 non-empty occurrences; 4 distinct values; 3 repeated value kinds; 1 once-only value; 3 extra occurrences removed.
2 SURV-1042
2 SURV-1051
2 SURV-1063SURV-1048If the source system treats IDs as case-sensitive, leave Ignore case off. For a simple non-tabular comma or tab list, use Remove Duplicates From a List.
Step-by-step workflow
- 1
Preserve the raw export
Keep an unchanged copy of the original survey export. Record its source and export date when the cleaned list will support analysis, evaluation, or participant follow-up.
- 2
Export only the ID column
Select the stable response or respondent identifier in the source system. Remove the header from the working copy and export one ID per line.
Format the source column as text when IDs contain leading zeros; a spreadsheet may otherwise convert 000184 to 184 before the tool sees it.
- 3
Open the line-specific page
Use Remove Duplicate Lines for one-ID-per-line data. A simple comma, tab, or custom-delimited list can use Remove Duplicates From a List, but a structured CSV table cannot.
- 4
Set the matching rules
Enable Trim whitespace only when spaces around IDs are export artifacts. Enable Ignore case only when the source system explicitly treats capitalization as equivalent.
Keep sorting off to preserve first-seen order and choose Keep First or Keep Last only from a documented retention rule.
- 5
Inspect duplicates before exporting
Check total, distinct, repeated, once-only, and removed counts. Repeated plus Copy with counts creates a review list, but a repeated ID does not explain why the underlying records repeat.
- 6
Export the distinct ID list
Copy the complete Result or download deduplicated.txt. The output contains IDs only; associated timestamps, responses, consent fields, and source-row numbers are not carried along.
- 7
Reconcile records in a structured system
Take the reviewed duplicate-ID list back to the spreadsheet, database, or survey platform. Decide which complete records to retain using timestamps, completion status, response quality, consent rules, and the study protocol.
Never use the plain ID output by itself to delete full records automatically.
Structured CSV boundary
This workflow is safe only after the intended column has been extracted.
respondent_id,wave,comment
SURV-1042,1,"First visit"
SURV-1042,2,"Follow-up, completed"The two rows can be legitimate records from separate waves, and the quoted comment contains a comma.
- The tool does not select columns or interpret a header row.
- Quoted delimiters, escaped quotes, and multiline fields are not parsed.
- Lines mode compares complete CSV rows, not the value in one column.
- Comma Delimiter mode splits raw text even when a comma appears inside quotes.
- Use a spreadsheet, database query, statistical package, or quote-aware parser to extract the intended column and retain complete records.
Privacy and research-data boundary
- Survey IDs may be pseudonymous without being anonymous and may remain personal or regulated data.
- Minimize the input: exclude names, email addresses, free-text answers, invitation links, participant tokens, and unrelated columns.
- Local browser processing does not override an IRB protocol, data-processing agreement, institutional rule, or legal requirement.
- When organizational policy prohibits using a web page for the dataset, use an approved locally installed or controlled environment.
Text and selected file contents are processed locally for deduplication, while limited Production telemetry is a separate boundary. Read the Privacy Policy and use only systems authorized by the study or organization.
Frequently asked questions
Can a survey CSV be uploaded and the ID column selected?
No. The tool does not select or parse CSV columns. Export the required column first as one plain-text ID per line.
Will it remove duplicate survey responses?
No. It returns a distinct list of text IDs. It does not preserve associated columns, compare answers, rank responses, or delete source records.
What happens to a header row?
A header is treated like any other line. Keep it outside the input and add it back later only if the destination requires one.
Are differently capitalized IDs the same?
Only when the source system says capitalization is insignificant and Ignore case is enabled. Do not infer that rule from appearance alone.
Are leading zeros preserved?
The tool compares and outputs text, so 000184 remains 000184. A spreadsheet may still coerce it during export or re-import, so format the source and destination columns as text.
Can duplicates be reviewed without deleting them?
Yes. Switch Show to Repeated. Copy with counts creates one repeated ID per line with a tab-separated occurrence count.
Is local browser processing sufficient for sensitive survey data?
Not automatically. Policy, contractual, ethical, telemetry, endpoint-security, and re-identification risks still apply. Use only data and systems authorized by the study or organization.
What are the input limits?
Typed or pasted text is limited to 300,000 UTF-8 bytes. Local text files are limited to 10,000,000 bytes per file and for the complete accepted batch.