Skip to content
Coverton Blog
In this article
Tables and data

Deduplicate CSV with matching columns and retention rules

Two records sharing an order code can still differ in their other values. Define the matching columns and which occurrence to keep before evaluating the rows removed by deduplication.

Open the toolHow to use this tool ↓
Remove duplicate CSV rows
This illustration explains the tool's workflow. It is not the processed result of an uploaded file.

How to use this tool

  1. Choose files in CSV format. Maximum files per batch: 1.
  2. Review Key columns (empty for full row), Duplicate row to keep, Ignore surrounding whitespace, then start processing.
  3. Download the CSV result and check its content and quality.

Choose the right settings

Enter key-column names separated by commas or leave them empty to compare entire rows. Choose first or last retention and the trimming or case-insensitive options according to the intended matching rule.

Supported input
  • CSV
Output
  • CSV
Available settings
  • Key columns (empty for full row)
  • Duplicate row to keep
  • Ignore surrounding whitespace
  • Ignore letter case

Before you start

Establish the meaning of the input

Before converting or arranging data, identify what a row or value represents and keep a source copy. In tabular files, review headers, separators and text encoding; in structured files, inspect the required object or array shape. Identifiers, dates and amounts should not be interpreted solely from how a spreadsheet happens to display them.

Where the tool offers matching columns, type rules or output fields, choose them according to the destination’s requirements. Test a sample with a blank, a repeated value and a value containing punctuation. For labels, charts or personalized documents, check which column supplies each displayed value and whether the source units and wording are correct.

Define a few checks before processing: an identifier that must retain its spelling, a value with a known interpretation and an expected relationship between records. If the tool offers type conversion or matching rules, distinguish a source value from the program’s display of it. Test ambiguous dates or differently written keys before accepting a broad rule. Keep the chosen rules and destination requirements with the result; this makes a discrepancy traceable to input, interpretation or the requested transformation.

Verify values, structure and exceptions

Reopen or import the result using the destination’s settings. Compare headers, record counts and several values with the source, allowing for the operation you requested. A summary, split or duplicate-removal task can legitimately change the count; verify why it changed rather than expecting every transformation to keep identical rows.

Inspect any exception or validation report before continuing. If columns shift, characters are garbled or a key is not found, check delimiters, encoding and exact header spelling first. Encoding data does not encrypt it, and formatting a value does not confirm its meaning. For printed codes or labels, test the actual output before making the full set.

Example input and output

repeated order IDs → CSV with retained records

Examples describe the workflow. Results depend on the source file and settings.

Check the saved result before using it

For each repeated key, review the retained date and values. Inspect whitespace or case variants that become equivalent under broader matching, and check the saved row order before using the spreadsheet.

If the result is different from what you expected

Unknown header names or invalid quoting can prevent processing. A trimming option used for comparison does not mean that whitespace in every saved cell has also been cleaned.

Limits to keep in mind

  • Trimming whitespace and ignoring case can merge values that you intended to keep distinct.

Continue with a related tool

Convert CSV to JSON

The first CSV row supplies the keys for each data row's JSON object. Cell values are stored as strings, so inspect their types before passing the result to a system that expects numbers or boolean values.

Choose the source encoding and use automatic delimiter detection or select the actual delimiter. Resolve empty or duplicate headers and ensure each row has the same number of columns as the header.

Open the tool

Convert JSON to CSV

This spreadsheet conversion requires an array whose entries are objects. Decide whether nested objects should become separate columns or remain JSON strings inside cells, because the resulting spreadsheet is read differently in each case.

Enable flattening to turn nested object properties into headers joined with dots. Arrays are not expanded into multiple rows, so review those cells separately if you intend to analyze each array item.

Open the tool
CovertonRemove duplicate CSV rows
Open the tool

This guide explains the public browser tool. Keep a copy of the original and inspect the result before sharing it.