Find repeated image URLs in a product CSV

Duplicate image URLs in product data

What a duplicate image URL is

A duplicate image URL is a single image address that appears on more than one row of a product file. If two rows both list https://cdn.example/img/blue-shirt.jpg in the image column, those rows share an image URL. ImageDup groups rows by that address and shows you every group where the same address is used more than once.

This is a text comparison. ImageDup reads the URL string, applies a few safe adjustments — trim spaces, lowercase the scheme and domain, drop the #fragment — and then checks whether the result is identical. It does not open the image, download it, or compare pixels. Two rows only match when their image addresses match.

Why catalogs end up with repeated image URLs

Most repeated image URLs are a side effect of how product data is produced and exported, not deliberate choices:

• Variant rows. Many platforms export one row per size or color. A shirt in five sizes is five rows, and if the size variants were never given their own photos, all five point at the same image.
• Copy-paste product setup. A new product is duplicated from an existing one to save time, and the image field is never changed.
• Placeholder images. A "coming soon" or generic category image is assigned to every product without a real photo yet. Hundreds of rows can share one placeholder.
• Feed joins and re-exports. When a feed is built by joining several tables, or exported repeatedly from a system that appends rather than replaces, the same product can appear on multiple rows with the same image.
• Migration artifacts. Moving between platforms often maps several old fields onto one new image column, producing repeats.

When a repeated image URL is fine

A repeated image URL is not automatically a problem. There are ordinary reasons for it:

• A product and its variants genuinely share one photo because the variants look identical in the image — a ring in three ring sizes, a book in hardcover and paperback with the same cover.
• A bundle or gift set reuses a component's photo on purpose.
• Two listings for the same physical item, for example a standard and a promotional SKU, intentionally use one image.
• A brand uses a single lifestyle image across a small coordinated range.

ImageDup does not decide these cases for you. It shows you where an image is reused and how many distinct product IDs are involved, and you judge whether that reuse is intended.

DUPLICATE and REVIEW

ImageDup sorts every repeated image URL into one of two groups:

• DUPLICATE — the same image URL is attached to two or more different, non-blank product IDs. This is the case most likely to need a fix, because separate products are sharing one picture.
• REVIEW — the same image URL repeats, but the rows do not establish two or more different product IDs. That happens when one product ID is listed on several rows, when rows are exact duplicates, or when the product ID is blank. These are often harmless variant rows, but they can also hide a data problem, so they are listed rather than ignored.

Rows with no product ID never create a DUPLICATE on their own, because ImageDup cannot tell whether they belong to one product or several.

Product CSV files

A product CSV is a plain-text table exported from an ecommerce platform, PIM, marketplace, or feed tool. Each row is a product or a product variant; each column is a field such as SKU, title, price, or image URL. ImageDup reads the first row as headers, then lets you map your columns to the two fields it needs — a product identifier and an image URL — plus two optional fields, a title to make the results readable and an image hash.

Column names vary between systems — image_link, Image Src, main_image, photo_url — so ImageDup guesses the mapping from the header text and marks a guess you should confirm. If a guess is wrong, pick the right column from the dropdown.

Catalog cleanup and feed auditing

Checking for repeated image URLs is one step in a broader catalog review. A typical pass also looks at missing images, broken image links, missing titles or descriptions, inconsistent pricing, and duplicate SKUs. ImageDup focuses on the image-URL question and gives you a downloadable CSV of every flagged row, tagged with a group number and a reason, so you can hand the list to whoever maintains the source data.

Running the check after each export, or before each feed submission, catches repeats introduced by new products or a changed export setting before they reach a storefront or a shopping channel.

The limits of exact URL matching

Exact matching is precise but literal. It will miss:

• The same image at two addresses. The URLs https://cdn.example/a.jpg and https://images.example/a.jpg, or a CDN URL and its origin URL, serve one picture but are different strings. ImageDup treats them as different.
• The same image with a cache-busting parameter. The URLs photo.jpg?v=1 and photo.jpg?v=2 are kept separate, because a query string can legitimately change the response.
• Resized or re-encoded copies. A thumbnail and a full-size version of one photo are different files at different URLs.

It also, by design, does not merge http with https, or www with the bare domain, or a trailing slash, because those can point at different resources.

To catch the same picture served from different URLs, add a column of image hashes to your export — an MD5 of the file, or a perceptual hash such as pHash or dHash, in hex — and map it. ImageDup then matches on the hash for any row that has one and falls back to the URL for rows that do not. It still does not compute hashes itself or compare images visually; it trusts the hash values you provide.