Mode
Lines
Security workflow
How to deduplicate IOC and log lines locally in your browser
Remove exact repeated lines from an already extracted IOC, URL, domain, or log list without sending the contents to a deduplication server. This workflow does not parse, enrich, or normalize indicators.
Before you paste: protect secrets and evidence
What counts as a duplicate?
Lines mode treats each CRLF, LF, or CR-separated line as one item. Under the settings below, matching is exact and input order is preserved.
Show
Result
Keep
First
Ignore case
Off
Trim whitespace
Off
Remove empty lines
On
Sort lines
Off
203.0.113.42
198.51.100.7
203.0.113.42
suspicious.example
Suspicious.example
https://portal.example.test/login
https://portal.example.test/login/
https://portal.example.test/loginThe IP addresses and example domains are reserved documentation values, not live indicators.
203.0.113.42
198.51.100.7
suspicious.example
Suspicious.example
https://portal.example.test/login
https://portal.example.test/login/8 total occurrences; 6 distinct values; 2 repeated value kinds; 4 once-only values; 2 extra occurrences removed.
203.0.113.42
https://portal.example.test/login2 203.0.113.42
2 https://portal.example.test/loginThe report format is count, one tab, then the retained value.
198.51.100.7
suspicious.example
Suspicious.example
https://portal.example.test/login/Step-by-step workflow
- 1
Create a safe working copy
Keep the original feed, log, or evidence export unchanged. Preserve its source, timestamps, and hash in the appropriate evidence system before creating a working copy.
- 2
Put one comparison unit on each line
Use one IP address, domain, URL, hash, or complete log record per line. Newline-delimited JSON is compared as whole text lines; fields and key ordering are not understood.
- 3
Open the line-specific page
Use Remove Duplicate Lines so the workflow starts in Lines mode with Keep First and sorting off instead of inheriting unrelated homepage preferences.
- 4
Choose matching rules deliberately
Start with Ignore case and Trim whitespace off. Enable them only when the artifact format says those differences are noise. Leave sorting off when source order matters.
Keep First preserves the first matching spelling and position. Keep Last retains the final spelling and its later position.
- 5
Paste text or select a local text file
Typed or pasted text is limited to 300,000 UTF-8 bytes. Each accepted file and the complete accepted batch are limited to 10,000,000 bytes.
Several files are joined with a newline and processed as one collection; filenames and file boundaries are not included in the output.
- 6
Review Result, Repeated, and Once only
Result keeps one value for each distinct match. Repeated isolates values seen at least twice; Copy with counts creates a tab-separated count report. Once only shows values seen exactly once, which is not a verdict about validity or risk.
- 7
Export and validate
Copy the complete result or download deduplicated.txt. Verify counts, encoding, destination format, and required provenance before importing into a SIEM, case system, or detection platform.
Important URL and log boundaries
The tool performs exact-text comparison after the visible options you select. Semantically related values can remain different text lines.
- No scheme, hostname, port, path, query, fragment, trailing-slash, redirect, or tracking-parameter normalization.
- No defang/refang conversion or canonicalization of IPv6, internationalized domains, percent encoding, or punycode.
- No timestamp, log-level, JSON-field, CSV-column, or key-value parsing.
- Differently formatted but semantically related events remain different text lines.
https://portal.example.test/login
https://portal.example.test/login/Use a typed, reviewed normalizer before exact-line deduplication when semantic URL equivalence is required.
Local processing and policy boundaries
Pasted text and selected file contents are processed locally and are not uploaded for deduplication. Limited page and interaction telemetry may still operate in Production. Review the Privacy Policy and your organization's handling rules before working with confidential or regulated data.
- Redact passwords, API keys, cookies, bearer tokens, private keys, authentication headers, and active credentials before using any web page.
- Deduplication changes occurrence counts and may remove evidence needed for chronology, incident reconstruction, or chain of custody.
- A credential found in a log must still be revoked or rotated; deduplication does not make it safe.
- Browser-local processing is not a secrets manager, evidence platform, or substitute for an organization-approved workflow.
Frequently asked questions
Does the tool normalize URLs before removing duplicates?
No. URLs are plain text. Schemes, ports, paths, queries, fragments, encodings, capitalization, and trailing slashes remain different unless a visible text option happens to collapse them.
Can it extract IOCs from a log file?
No. It compares complete lines or delimiter-separated text items. It does not extract IPs, domains, hashes, or URLs from prose, JSON fields, or structured log records.
Should Ignore case be enabled for domains?
Only for a homogeneous list whose rules you understand. Ignore case affects the entire line and should not be applied indiscriminately to URL paths or query values.
Can live credentials or authentication logs be pasted?
Redact credentials and access tokens first. Local processing is not a substitute for an approved secrets-handling or evidence workflow.
Does it preserve input order?
Yes, while sorting is off. Keep First preserves the first matching value and position; Keep Last preserves the final matching value and later position.
Can several log files be processed together?
Yes, within the 10 MB per-file and complete-batch limits. Accepted files are joined with a newline. Output does not preserve filenames, source-file boundaries, or metadata outside line text.
Is a CSV log parsed into fields?
No. CSV is handled as plain text. Headers, columns, quoted delimiters, escaped quotes, and multiline fields are not parsed.