Deduplication is not a node you drop in. It is one decision: what makes two rows the same person. Here is the decision, the Code node that implements it, and the real run it did on a messy export — 12 rows in, 7 usable leads out, with the other five named and explained.
Take one export from a CRM, a form tool and a spreadsheet and the same address arrives in four shapes. A dedupe that compares raw strings sees four people:
| Raw value in the file | Normalised key |
|---|---|
Marko.Petrovic@PetroCo.rs | marko.petrovic@petroco.rs |
marko.petrovic@petroco.rs | marko.petrovic@petroco.rs |
ana@nova.rs (trailing space from the export) | ana@nova.rs |
Ana Jovanovic <ana@nova.rs> | ana@nova.rs |
Four steps, in this order: drop a leading mailto:, take the address out of Name <addr>, remove all whitespace, lowercase. Lowercasing is safe for the domain (DNS is case-insensitive) and in practice safe for the local part on every provider that will take your money. Whitespace removal is what catches the copy-paste duplicates.
Code node, Run Once for All Items, mode JavaScript. It returns every input row annotated — _status is keepduplicate or rejected — so a Filter node sends the keeps to your sheet and everything else to a report. Nothing leaves the node silently.
const norm = (v) => String(v == null ? '' : v)
.replace(/^\s*mailto:/i, '') // mailto:ana@nova.rs
.replace(/^[^<]*<([^>]+)>\s*$/, '$1') // Ana Jovanovic <ana@nova.rs>
.replace(/\s+/g, '') // " ana@nova.rs "
.toLowerCase(); // Ana@Nova.RS
const EMAIL = /^[^@\s]+@[^@\s.]+\.[^@\s]{2,}$/;
const filled = (o) => Object.keys(o)
.filter(k => !k.startsWith('_') && String(o[k] == null ? '' : o[k]).trim() !== '')
.length;
const wrappers = items.map(i => ({ json: { ...i.json } }));
const seen = new Map();
for (const w of wrappers) {
const row = w.json;
const key = norm(row.email);
if (!EMAIL.test(key)) { // stefan(at)dokic.rs, mila.saric@
row._status = 'rejected';
row._reason = key ? 'no usable email: "' + row.email + '"' : 'empty row';
continue;
}
row.email = key;
const prev = seen.get(key);
if (!prev) { // first time we see this address
row._status = 'keep'; row._reason = '';
seen.set(key, w);
continue;
}
// same address twice: keep the more complete row, fill its blanks from the other
const [keepW, dropW] = filled(row) > filled(prev.json) ? [w, prev] : [prev, w];
for (const k of Object.keys(dropW.json)) {
if (k.startsWith('_')) continue;
if (String(keepW.json[k] ?? '').trim() === '' && String(dropW.json[k] ?? '').trim() !== '')
keepW.json[k] = dropW.json[k];
}
keepW.json._status = 'keep';
keepW.json._reason = '';
keepW.json._merged = (keepW.json._merged || 0) + 1 + (dropW.json._merged || 0);
dropW.json._status = 'duplicate';
dropW.json._reason = 'duplicate of ' + key;
seen.set(key, keepW);
}
return wrappers; // every input row comes back, annotated - nothing is dropped
Copy it from the gist if the page is easier to send than a link: n8n dedupe leads gist — the same node, plus what it printed on the 12-row sample. Same code, no ads, nothing to install.
Why keep the rejects at all: a row with stefan(at)dokic.rs is a real lead with a mangled address. Dropping it quietly is how a pipeline loses 10% of a list and nobody notices for a quarter. Marking it rejected with the original value quoted lets you fix the source and re-run.
The sample is a 12-row export built the way exports actually arrive: mixed case, a trailing space, one quoted name with a comma in it, one address written as name(at)domain, one truncated address, and three people entered twice.
| Outcome | Rows | Which ones |
|---|---|---|
| keep | 7 | marko.petrovic@petroco.rs, ana@nova.rs, nikola@ilic.rs, petar@firma.rs, ivana@maric.rs, luka@stankovic.rs, jelena@popovic.rs — three of them carry _merged: 1 |
| duplicate | 3 | “marko petrovic”, “Ana Jovanovic”, “ivana maric” — each merged into the row above, blanks filled from whichever copy had more fields |
| rejected | 2 | “stefan(at)dokic.rs” and “mila.saric@”, both named with the original value in the reason |
12 in, 12 out, 7 worth emailing. The merge rule is one line and it matters: keep the row with more filled fields, then copy any field the other row has and this one does not. Without that step a dedupe can keep the sparse copy and lose the phone number. see the input · the cleaned output · the full report.
The same logic ships as a free workflow: import the JSON, press Execute, read the annotated output. No credentials, no HTTP request, nothing leaves your instance. The paid pack adds the parts that make it a weekly pipeline instead of a one-off — the webhook intake, the dedupe-then-draft chain and the report a colleague can read.
The free workflow stops at a clean, deduped file. The pack ($49 launch, 14-day refund on request) continues: webhook intake into the same sheet, an outreach draft per surviving lead with a review queue instead of a send button, and a report naming every rejected row and every merge. Four workflows, importable JSON, a first-run guide and a troubleshooting page written from runs that failed.
The receipt page hands you the ZIP immediately — no email round-trip and no account to create. The charge comes from dragana.karans@gmail.com; questions, files and refunds go to perun.intelligence@gmail.com.
Normalise the email to one key, keep a Map of the first row per key, merge later rows into it. The node above is the whole implementation; a Merge node set to “remove duplicates” does the same thing only when the values are already identical strings.
Names collide across different people and split across spellings. In the sample, the three duplicates share an address but two of the three would not match by name.
No — they come back marked rejected with the original value quoted, so they can go to a report and be fixed at the source.
No. It reads the items it is given, writes nothing, makes no network call, needs no credential. Sending stays a separate step you build deliberately.
The node is linear and holds one Map, so it runs in memory — fine for tens of thousands of rows on a normal n8n instance. Past that, dedupe in the database with a unique index, not in a workflow.
Honest limits: matching is exact after normalisation — no fuzzy name matching, no address verification, no role-address filtering. Nothing here scrapes, buys lists or sends mail. Formatting is cleaned; deliverability is not checked. The counts above come from the shipped sample and reproduce with the node as printed.
Free workflows live at github.com/fejzadenis/n8n-lead-toolkit · more free fixes: "Invalid Record Length: columns length is 7, got 8" · names arriving as ???? in Google Sheets · all three free workflows · LeadFlow home