How to Measure File Conversion Quality: Key Metrics

How to Measure Conversion Quality

Conversion quality is measurable, not a matter of taste. Four metrics cover most cases: character error rate for text and OCR, structural similarity for anything that must look the same, layout fidelity for structure, and the output-to-input size ratio for efficiency. Run them against a fixed test set and you can compare tools objectively and catch regressions early.

Start With a Fixed Test Corpus

You cannot compare numbers that come from different inputs. Before measuring anything, assemble a corpus you will reuse for every test: 50 to 100 documents that mirror your real workload. Mix contracts, reports, spreadsheets, scanned pages, and presentations so the scores reflect the files you actually convert.

Keep the corpus under version control and never edit it once you start benchmarking. When you re-run the tests after a tool update, the only variable that changed is the tool, so any score movement is real signal.

Character Error Rate for Text and OCR

For anything that involves OCR or text extraction, character-level accuracy is the metric that matters. Character Error Rate (CER) compares the extracted text against a known-correct transcript:

CER = (substitutions + insertions + deletions) / total characters in the reference

You compute it with an edit-distance (Levenshtein) function. Python's jiwer library reports CER and Word Error Rate directly if you feed it the reference and the hypothesis strings.

As a working rule of thumb, teams often treat a CER under 1% as strong for clean printed text and expect higher rates on skewed scans, handwriting, or noisy backgrounds. Report the number per document type rather than as a single blended average, because a good score on clean PDFs can hide a poor one on photographed receipts.

Structural Similarity for Visual Fidelity

Some conversions have to look identical to the source: PDF to image for previews, or a document to PDF for print. Pixel comparison quantifies the difference. The Structural Similarity Index (SSIM) returns a value from 0 to 1, where 1 is a perfect match.

To measure it, render the source and the output to images at the same resolution, then compare them:

• Python: skimage.metrics.structural_similarity from scikit-image.
• Command line: magick compare -metric SSIM original.png converted.png diff.png from ImageMagick.

An SSIM above 0.95 usually reads as near-identical to the human eye. Below that, open the diff image the tools produce and look at where the mismatch concentrates, since a low score is often caused by one shifted element rather than uniform degradation.

Layout Fidelity for Structure

When the output format keeps structure (PDF to Word, HTML to PDF), check that the structure survived. This is a scripted comparison, not a visual one:

• Headings: same count of H1, H2, and H3 elements, in the same order.
• Tables: row and column counts match, and merged cells are handled the way you expect.
• Lists: ordered and unordered lists keep their nesting and numbering.

Parse both files, extract these counts, and diff them. A conversion that turns a structured report into a flat wall of text will fail this check even when it looks acceptable at a glance.

Output-to-Input Size Ratio

Divide the output file size by the input size and track the result. For a format upgrade such as BMP to WebP you should see real compression, so a ratio well under 1 is the goal. For same-fidelity conversions such as DOCX to PDF the ratio should stay close to 1. A conversion that doubles file size is quietly wasting storage and bandwidth, and a sudden change in the ratio after a tool update is worth investigating.

Latency and Throughput

Speed is easy to measure yourself: record wall-clock time from the moment you submit a file to the moment the output is ready, and log it per file. Report percentiles, not just the average, because the p95 (the slowest five percent) is what your users feel on a bad day. Run the same corpus a few times to smooth out network noise before you trust the numbers.

Turn the Metrics Into a Dashboard

Aggregate the scores so degradation is visible before anyone complains:

• Daily averages for CER, SSIM, and layout fidelity per document type.
• Outlier alerts when any metric drops below a threshold you set.
• Trend lines to catch slow drift, such as a library update that gradually worsens table handling.

Benchmark Novitora Against Your Own Corpus

The most honest number is the one you produce on your own files. Novitora is free and needs no signup, accepts files up to 100 MB, handles up to 10 files per batch, and includes OCR for PDFs. Run your corpus through it, score the output with the methods above, and compare the results against whatever you use today. Your measurements will tell you more than any headline claim.