fix: report undetectable duplicates and uniqueness as unknown instead of clean (#414)

* fix(quality): report undetectable duplicates as unknown instead of clean

_score_debt counted remaining duplicates with cleaned.duplicated() and,
when that raised (TypeError for object columns of lists or dicts, and
since #412 NotImplementedError for nested Arrow list/struct/map
columns), set dup_remaining = 0. "Cannot check" was therefore scored as
"no duplicates": the dimension read 0.0 with "0 duplicate row(s)
detected", counted as clean evidence toward the gate, and was written
to the ledger as a real 0.0, so a later measured run could be compared
against a score that was never measured.

DebtItem gains assessed (default True). When detection is impossible
and nothing was removed, the duplicates dimension is unassessed: it is
never over threshold or worsening, adds nothing to total_score, never
drives the gate status, is skipped when writing the ledger (escalation
compares against the last run that measured it), serialises with score
and over_threshold as None, and is listed in QualityDebtGate.unassessed
and in a "? duplicates: not assessed" summary line. The detail names the
columns that block detection. If rows were already removed through a
hashable duplicate_subset, that count stays a known lower bound.

Frames where duplicates can be detected take the same code path and
produce identical scores, dicts, frames and summaries.

* fix(enterprise): report undetectable uniqueness as unknown in trust scores

compute_trust_score computed uniqueness from frame.duplicated() and,
when that raised (TypeError for object columns of lists or dicts,
NotImplementedError for nested Arrow list/struct/map columns, caught
since #413), set uniqueness = 100.0. "Cannot check" was scored as "no
duplicates": a perfect dimension blended into overall at its full
weight, lifting the trust score of frames whose duplicates were never
checked. It also disagreed with the quality-debt duplicates dimension,
which now reports the same situation as not assessed.

When duplicate rows cannot be checked, uniqueness is now NaN
(TrustScore.uniqueness_assessed is False) and overall blends
completeness, validity and consistency with their weights renormalised
to sum to 1 (equal weights if all of the weight was on uniqueness).
to_dict() gives uniqueness None, str() and both Markdown tables show
n/a, and each column that blocks the check carries an "unhashable
values: duplicate rows not checked" issue.

Frames where duplicates can be detected keep the original four-way
expression and produce identical scores, dicts and renderings.

* fix(render): show unknown trust uniqueness as n/a in the report summary

TrustScore.uniqueness is NaN when duplicate rows could not be checked. The report summary formatted it with :.0f and printed 'uniqueness nan'; it now prints 'uniqueness n/a', matching TrustScore's own text and Markdown output.

* fix(quality): compare debt against each dimension's last recorded score

Unassessed dimensions are no longer written to the ledger, but _previous_scores read only the latest run's items. A run after an unassessed one therefore lost its previous score and could not be marked worsening. Read each dimension's score from the last run that recorded it; when every run records every dimension the result is unchanged.

Author: kevincostner17

CHANGELOG.md

@@ -243,6 +243,21 @@ adheres to [Semantic Versioning](https://semver.org/).
   no column, instead of masking nothing and exiting 0 (#251).
 
 ### Fixed
+- `fd.evaluate_quality_debt` reports the `duplicates` dimension as not assessed
+  when duplicate rows cannot be checked (object columns of lists or dicts,
+  nested Arrow list/struct/map columns), instead of scoring it a clean 0.0 with
+  "0 duplicate row(s) detected". The item serialises with `score` and
+  `over_threshold` as `None`, its detail names the unhashable columns, it is
+  listed in `QualityDebtGate.unassessed`, and it never counts toward the total,
+  the gate status or the ledger history. Frames where duplicates can be
+  detected score exactly as before.
+- `compute_trust_score` reports uniqueness as unknown when duplicate rows cannot
+  be checked (the same unhashable columns), instead of a perfect 100.
+  `TrustScore.uniqueness` is `NaN` with `uniqueness_assessed` False, `to_dict()`
+  gives `None`, `str()` and the Markdown tables show `n/a`, the blocking columns
+  carry an "unhashable values: duplicate rows not checked" issue, and `overall`
+  blends the other three dimensions with their weights renormalised. Frames
+  where duplicates can be detected score exactly as before.
 - Trust-gate integrations now validate `on_low_score` policies at configuration
   boundaries, rejecting typos instead of silently skipping failure handling (#345).
 - `QualityReport.to_markdown()` now escapes every cell in the Actions table, so

docs/decision-workflow.md

@@ -75,6 +75,16 @@ instability, outlier spikes, PII risk, category churn, failed repairs, human-
 review backlog), persists the history to SQLite, and **escalates warn→fail when
 an issue repeats or worsens** across runs.
 
+The duplicates dimension counts duplicate rows left in the cleaned output (or
+the rows removed, if more). When duplicates cannot be checked because a column
+holds unhashable values (lists, dicts, or nested Arrow list/struct/map columns),
+the dimension is **not assessed** rather than scored clean: `to_dict()` gives
+`score` and `over_threshold` as `None`, the detail names the columns,
+`gate.unassessed` lists it, and `summary()` prints a `? duplicates: not
+assessed` line. An unassessed dimension adds nothing to the total, never
+changes the gate status on its own, and is not written to the ledger, so
+escalation compares against the last run that measured it.
+
 ## Dirty-join assistant
 
 Reviewable fuzzy joins for messy keys — never a silent low-confidence join.

docs/feature-overview.md

@@ -31,7 +31,7 @@ dependencies stay lazy, so a plain `import freshdata` is unaffected.
 | Capability | API |
 |---|---|
 | Full enterprise pipeline | `clean_enterprise(df, *, enterprise=…)` → `EnterpriseResult` |
-| Data Trust Score (0–100) | `compute_trust_score(df)` → completeness / validity / uniqueness / consistency |
+| Data Trust Score (0–100) | `compute_trust_score(df)` → completeness / validity / uniqueness / consistency (uniqueness is unknown and left out of the overall when a column holds unhashable values such as lists) |
 | Fuzzy value clustering | `merge_clusters(df, cols)` / `cluster_column(df, col)` |
 | PII masking | `mask_dataframe(df, rules)` — hash / redact / partial / regex-scrub / drop |
 | Semantic validation | `run_semantic_validation(df, configs)` — reference / regex / API checks |

src/freshdata/enterprise/metrics.py

@@ -6,7 +6,11 @@
 - **validity** — share of *present* cells already in a clean, parseable form
   (no stray whitespace, no sentinel-as-value, parses to its inferred dtype, not
   a statistical outlier),
-- **uniqueness** — share of rows that are not exact duplicates,
+- **uniqueness** — share of rows that are not exact duplicates. When duplicate
+  rows cannot be checked (a column holds lists, dicts or nested Arrow
+  list/struct/map values) it is *unknown* (``NaN``, ``None`` in
+  :meth:`TrustScore.to_dict`) and the overall score is blended from the other
+  three dimensions with their weights renormalised,
 - **consistency** — share of columns free of structural defects that
   corruption can introduce (mixed types, duplicate labels). Constant columns
   are surfaced as per-column issues instead of lowering this dimension:
@@ -21,7 +25,8 @@
 from __future__ import annotations
 
 import json
-from dataclasses import dataclass, field
+import math
+from dataclasses import dataclass, field, replace
 from datetime import datetime, timezone
 from typing import TYPE_CHECKING, Any
 
@@ -42,6 +47,9 @@
 #: Minimum non-null count before outliers are scored (matches profile.py).
 _MIN_FOR_OUTLIERS = 20
 
+#: Per-column issue for values that make duplicate rows impossible to check.
+_UNHASHABLE_ISSUE = "unhashable values: duplicate rows not checked"
+
 
 @dataclass(frozen=True)
 class ColumnTrust:
@@ -60,6 +68,9 @@ class TrustScore:
 
     Render with ``print(score)``, export with :meth:`to_dict` /
     :meth:`to_markdown`. ``grade`` maps the overall score to an A-F letter.
+
+    ``uniqueness`` is ``NaN`` when duplicate rows could not be checked (see
+    :attr:`uniqueness_assessed`); ``overall`` then excludes it.
     """
 
     overall: float
@@ -71,6 +82,14 @@ class TrustScore:
     n_cols: int
     columns: tuple[ColumnTrust, ...] = ()
 
+    @property
+    def uniqueness_assessed(self) -> bool:
+        """False when duplicate rows could not be checked (unhashable values)."""
+        return not math.isnan(self.uniqueness)
+
+    def _uniqueness_text(self, fmt: str) -> str:
+        return format(self.uniqueness, fmt) if self.uniqueness_assessed else "n/a"
+
     @property
     def grade(self) -> str:
         """Letter grade: A (>=90), B (>=80), C (>=70), D (>=60), else F."""
@@ -86,7 +105,7 @@ def to_dict(self) -> dict[str, Any]:
             "dimensions": {
                 "completeness": round(self.completeness, 2),
                 "validity": round(self.validity, 2),
-                "uniqueness": round(self.uniqueness, 2),
+                "uniqueness": round(self.uniqueness, 2) if self.uniqueness_assessed else None,
                 "consistency": round(self.consistency, 2),
             },
             "n_rows": self.n_rows,
@@ -106,12 +125,12 @@ def to_dict(self) -> dict[str, Any]:
     def to_markdown(self) -> str:
         """Markdown table of the score and its dimensions."""
         rows = [
-            ("Completeness", self.completeness),
-            ("Validity", self.validity),
-            ("Uniqueness", self.uniqueness),
-            ("Consistency", self.consistency),
+            ("Completeness", f"{self.completeness:.1f}"),
+            ("Validity", f"{self.validity:.1f}"),
+            ("Uniqueness", self._uniqueness_text(".1f")),
+            ("Consistency", f"{self.consistency:.1f}"),
         ]
-        body = [_md_table_row((dim, f"{val:.1f}")) for dim, val in rows]
+        body = [_md_table_row(row) for row in rows]
         lines = [
             f"### Data Trust Score: **{self.overall:.1f} / 100**  (grade {self.grade})",
             "",
@@ -126,7 +145,7 @@ def __str__(self) -> str:
         return (
             f"Data Trust Score {self.overall:.1f}/100 (grade {self.grade}) — "
             f"completeness {self.completeness:.0f}, validity {self.validity:.0f}, "
-            f"uniqueness {self.uniqueness:.0f}, consistency {self.consistency:.0f}"
+            f"uniqueness {self._uniqueness_text('.0f')}, consistency {self.consistency:.0f}"
         )
 
     def __repr__(self) -> str:
@@ -211,6 +230,21 @@ def _is_constant(s: pd.Series, n_rows: int) -> bool:
         return False
 
 
+def _unhashable_positions(frame: pd.DataFrame) -> list[int]:
+    """Positions of columns that make ``DataFrame.duplicated`` impossible.
+
+    ``duplicated`` factorizes every column, which is what raises; a lone
+    ``Series.duplicated()`` on a list column does not.
+    """
+    positions = []
+    for i in range(frame.shape[1]):
+        try:
+            pd.factorize(frame.iloc[:, i])
+        except (TypeError, NotImplementedError):
+            positions.append(i)
+    return positions
+
+
 def compute_trust_score(
     df: pd.DataFrame,
     *,
@@ -219,7 +253,10 @@ def compute_trust_score(
 ) -> TrustScore:
     """Profile *df* (pandas or polars) and compute its Data Trust Score.
 
-    Read-only: the input is never modified.
+    Read-only: the input is never modified. If duplicate rows cannot be
+    checked, uniqueness is ``NaN`` and the overall score uses the remaining
+    dimensions with their weights renormalised (equal weights if all of the
+    weight was on uniqueness).
     """
     frame = to_pandas(df)
     cfg = config or CleanConfig()
@@ -274,18 +311,33 @@ def compute_trust_score(
         dup_rows = int(frame.duplicated().sum())
         uniqueness = 100.0 * (1.0 - dup_rows / n_rows) if n_rows else 100.0
     except (TypeError, NotImplementedError):  # lists/dicts, nested Arrow: undetectable
-        uniqueness = 100.0
+        # "Cannot check" is not "no duplicates": leave uniqueness unknown and
+        # name the columns that block the check.
+        uniqueness = math.nan
+        for i in _unhashable_positions(frame):
+            col_trust[i] = replace(col_trust[i], issues=(*col_trust[i].issues, _UNHASHABLE_ISSUE))
 
     dup_labels = int(frame.columns.duplicated().sum())
     flagged = min(n_cols, inconsistent_cols + dup_labels)
     consistency = 100.0 * (1.0 - flagged / n_cols) if n_cols else 100.0
 
-    overall = (
-        w["completeness"] * completeness
-        + w["validity"] * validity
-        + w["uniqueness"] * uniqueness
-        + w["consistency"] * consistency
-    )
+    if not math.isnan(uniqueness):
+        overall = (
+            w["completeness"] * completeness
+            + w["validity"] * validity
+            + w["uniqueness"] * uniqueness
+            + w["consistency"] * consistency
+        )
+    else:
+        measured = w["completeness"] + w["validity"] + w["consistency"]
+        if measured > 0:
+            overall = (
+                w["completeness"] * completeness
+                + w["validity"] * validity
+                + w["consistency"] * consistency
+            ) / measured
+        else:  # all weight was on uniqueness: fall back to equal weights
+            overall = (completeness + validity + consistency) / 3.0
     return TrustScore(
         overall=overall,
         completeness=completeness,
@@ -358,8 +410,8 @@ def to_markdown(self) -> str:
             _md_table_row(("Completeness", f"{before.completeness:.1f}",
                            f"{after.completeness:.1f}")),
             _md_table_row(("Validity", f"{before.validity:.1f}", f"{after.validity:.1f}")),
-            _md_table_row(("Uniqueness", f"{before.uniqueness:.1f}",
-                           f"{after.uniqueness:.1f}")),
+            _md_table_row(("Uniqueness", before._uniqueness_text(".1f"),
+                           after._uniqueness_text(".1f"))),
             _md_table_row(("Consistency", f"{before.consistency:.1f}",
                            f"{after.consistency:.1f}")),
             _md_table_row(("**Overall**", f"**{before.overall:.1f}**",