Repository navigation
Commit 6f53878
fix(strings): a declared vocabulary outranks a generic null marker (#473)
fieldcheck has applied this rule since the TestAllowedValuesBeatNullMarkers
regression -- "'NA' may be Namibia: when the schema literally allows a value,
it is a value, not a missing marker" (fieldcheck.py:466). fd.clean did not.
normalize_sentinels applied DEFAULT_SENTINELS unconditionally, so a caller who
had explicitly declared NA as permitted still lost it:
fd.validate_fields(df, {"country": FieldSpec(allowed_values=[..., "NA"])})
# -> no issue; NA is a value
fd.clean(df, context="Allowed country values are US, ..., NA.")
# policy compiles cleanly: [('country','allowed_values','soft')], 0 issues
# -> country is NaN
The same declared fact was respected by one public API and ignored by another.
'NA' is Namibia in an ISO-3166 column and 'None' is a real brand; both were
destroyed. The only escapes were protecting the column outright, which disables
every other repair, or normalize_sentinels=False, which is global and would
also stop nulling genuine "N/A" markers in every other column.
A column's declared allowed_values -- whether passed through semantic_context
or compiled from a context= policy, which lower to the same place -- now
removes those tokens from that column's sentinel set, matched casefolded and
trimmed for consistency with fieldcheck and extra_sentinels.
Deliberately narrow:
- No declaration means no change. "NA" with no vocabulary is still missing,
which is the gauntlet 'sentinel_collision' gold label, and that fixture
spells its countries out ("Namibia"), so its labels are untouched.
- A vocabulary that excludes NA still nulls it, mirroring fieldcheck's
test_na_outside_vocabulary_is_still_a_null_marker.
- The exemption is scoped to the declaring column.
- A malformed hint is ignored rather than fatal, and a bare string is rejected
so that "NA" cannot be iterated into the letters N and A.
7 of the 14 new tests fail on main; the other 7 are the must-not-change cases
and pass on both. Full suite 6588 passed / 0 failed, coverage 93.91%, and the
gauntlet gate passes with 0 corruption and 0 false positives.
Co-authored-by: Kevin Costner <kiran.gangalakunta@gmail.com>1 parent 2dbd4bd commit 6f53878
3 files changed
Lines changed: 199 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
27 | 42 | | |
28 | 43 | | |
29 | 44 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
165 | 191 | | |
166 | 192 | | |
167 | 193 | | |
168 | 194 | | |
169 | 195 | | |
170 | 196 | | |
171 | 197 | | |
172 | | - | |
173 | | - | |
174 | | - | |
| 198 | + | |
175 | 199 | | |
176 | 200 | | |
177 | 201 | | |
| |||
182 | 206 | | |
183 | 207 | | |
184 | 208 | | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
185 | 212 | | |
186 | 213 | | |
187 | 214 | | |
188 | | - | |
| 215 | + | |
189 | 216 | | |
190 | | - | |
191 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
192 | 223 | | |
193 | | - | |
194 | | - | |
195 | | - | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
196 | 230 | | |
197 | | - | |
198 | | - | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
199 | 237 | | |
200 | 238 | | |
201 | 239 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
0 commit comments