-
Notifications
You must be signed in to change notification settings - Fork 51
Expand file tree
/
Copy pathduration_did.py
More file actions
1313 lines (1190 loc) · 55.2 KB
/
Copy pathduration_did.py
File metadata and controls
1313 lines (1190 loc) · 55.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""Duration Difference-in-Differences (Deaner & Ku 2026).
Two-group, common-timing causal duration analysis for a binary ABSORBING
outcome (``Y_it = 1`` once the spell has ended). Under a restriction on the
UNTREATED hazards of the two groups — a constant additive gap (common
dynamics, ``method="cd"``) or a constant ratio (proportional hazards,
``method="ph"``) — the treated group's counterfactual survival is imputed
from the control group's cumulative hazard and the treated baseline, and
the absorption ATT ``E[Y_it - Y_it(0) | treated]`` is reported for every
post-treatment date (Theorem 1; Equations 3.1-3.4 with the mean-of-ratios PH
estimator). Inference is the whole-individual pooled bootstrap of Appendix B
Algorithm 1 (centered absolute-deviation pointwise and simultaneous bands);
the Algorithm 2 fixed-anchor pre-treatment specification test is reported
separately. See ``docs/methodology/REGISTRY.md`` (DurationDiD) and
``docs/methodology/papers/deaner-ku-2026-review.md``.
Notation (review lines 94-108): ``S_kt`` group survival, ``R_kt = -log S_kt``,
``D_kt = R_kt - R_k1``, ``H_kt = D_kt / e_t`` with the actual elapsed time
``e_t = time_t - time_1``; ``tstar`` is the last untreated date.
"""
from __future__ import annotations
import math
import warnings
from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
import numpy as np
import pandas as pd
from diff_diff._base import BaseEstimator
from diff_diff.bootstrap_chunking import compute_block_size
from diff_diff.duration_did_results import (
DurationDiDPretestResults,
DurationDiDResults,
_native_time_label,
invalid_curve_message,
)
from diff_diff.utils import (
safe_inference,
safe_inference_batch,
validate_binary,
validate_n_bootstrap,
)
_VALID_METHODS = ("cd", "ph")
#: A used log-survival moment backed by fewer than this many survivors (but at
#: least one) is reported as weak numerical support — a warning only, no
#: behavior changes (review lines 917-918). A count rule: a survival
#: PROPORTION is never below ``1/n_group``, so a proportion threshold would be
#: unreachable at any realistic sample size.
_WEAK_SUPPORT_MIN_SURVIVORS = 5
#: Relative tolerance of the equal-spacing check on the time grid.
_SPACING_RTOL = 1e-8
#: Baseline + last pre-treatment date + one post-treatment date.
_MIN_PERIODS = 3
#: Cap on bootstrap rows per chunk (the count matrix is ``(rows, n)`` float64).
_MAX_CHUNK_ROWS = 256
def _errstate() -> Any:
"""Silence every floating-point warning class (CONTRIBUTING: protect all arithmetic)."""
return np.errstate(divide="ignore", invalid="ignore", over="ignore", under="ignore")
def _time_scalar(value: Any, name: str) -> Union[int, float]:
"""Validate a real date without rounding integer identity or floating labels."""
if isinstance(value, (bool, np.bool_)) or not isinstance(
value, (int, float, np.integer, np.floating)
):
raise ValueError(f"{name} must be a numeric real time value, got {value!r}")
if isinstance(value, (int, np.integer)):
# Selectors can be arbitrarily large absent integers. Never send them
# through float() or isfinite(), even just to validate them.
return int(value)
if not np.isfinite(value):
raise ValueError(f"{name} must be a finite numeric time value, got {value!r}")
with _errstate():
native = float(value)
if not math.isfinite(native) or type(value)(native) != value:
raise ValueError(f"{name} floating time values must be exactly representable as float64")
return native
# Per-family draw-failure reasons, in FIRST-MATCH precedence order.
_POST_FAILURE_ORDER = (
"group_empty",
"zero_survival_baseline",
"zero_survival_last_pre",
"zero_control_increment",
"control_survival_zero",
"nonfinite_counterfactual",
"nonfinite_effect",
)
_PRETEST_FAILURE_ORDER = (
"group_empty",
"zero_survival_baseline",
"zero_survival_last_pre",
"zero_control_increment",
"nonfinite_contrast",
)
# =============================================================================
# Constructor validation
# =============================================================================
def _validate_method(method: Any) -> None:
if not isinstance(method, str) or method not in _VALID_METHODS:
raise ValueError(f"method must be 'cd' or 'ph', got {method!r}")
def _validate_alpha(alpha: Any) -> None:
if isinstance(alpha, bool) or not isinstance(alpha, (int, float, np.floating)):
raise ValueError(f"alpha must be a float strictly between 0 and 1, got {alpha!r}")
if not (0.0 < float(alpha) < 1.0):
raise ValueError(f"alpha must be a float strictly between 0 and 1, got {alpha!r}")
def _validate_seed(seed: Any) -> None:
if seed is None:
return
if isinstance(seed, bool) or not isinstance(seed, (int, np.integer)) or seed < 0:
raise ValueError(f"seed must be None or a non-negative integer, got {seed!r}")
def _validate_draws(n_bootstrap: Any) -> None:
validate_n_bootstrap(n_bootstrap)
if int(n_bootstrap) == 1:
raise ValueError(
"n_bootstrap must be 0 (point estimates only, no inference) or at "
"least 2 (a bootstrap SD needs two draws); got 1"
)
def _validate_all_params(params: Dict[str, Any]) -> None:
"""Validate the full hyperparameter dict (used by ``__init__`` and ``fit``).
``fit()`` re-runs this on ``get_params()`` before touching the data, so a
direct attribute mutation after construction (``est.method = "typo"``)
raises instead of silently selecting an estimation branch.
"""
_validate_method(params["method"])
_validate_draws(params["n_bootstrap"])
_validate_alpha(params["alpha"])
_validate_seed(params["seed"])
# =============================================================================
# Numerical core (pure numpy; a leading draw axis where noted)
# =============================================================================
def _validate_and_arrange(
data: pd.DataFrame,
outcome: str,
unit: str,
time: str,
treatment: str,
last_pre_period: Any,
) -> Dict[str, Any]:
"""Validate the balanced absorbing panel and arrange it as arrays.
Returns ``Y`` (n x T float 0/1, treated units first), ``n_treated``,
``grid`` (the sorted common dates in their native numeric dtype),
``elapsed`` (float, ``grid - grid[0]``) and ``tstar_idx``.
"""
if not isinstance(data, pd.DataFrame):
raise ValueError(f"data must be a pandas DataFrame, got {type(data).__name__}")
for name, col in (
("outcome", outcome),
("unit", unit),
("time", time),
("treatment", treatment),
):
if col not in data.columns:
raise ValueError(f"{name} column {col!r} not found in data")
# Identifier checks BEFORE any grouping/pivot (no silent groupby drop, no
# phantom NaN unit).
if data[unit].isna().any():
raise ValueError(
f"unit column {unit!r} contains missing values; every row needs a "
"unit identifier (no silent groupby drop)"
)
t_col = data[time]
if pd.api.types.is_datetime64_any_dtype(t_col) or pd.api.types.is_timedelta64_dtype(t_col):
raise ValueError(
f"time column {time!r} must be numeric; convert datetime/timedelta "
"values to a numeric elapsed scale (e.g. days since the spell "
"start) before fitting"
)
if (
pd.api.types.is_bool_dtype(t_col)
or pd.api.types.is_complex_dtype(t_col)
or not pd.api.types.is_numeric_dtype(t_col)
):
raise ValueError(f"time column {time!r} must be numeric (got dtype {t_col.dtype})")
if t_col.isna().any():
raise ValueError(f"time column {time!r} contains missing or non-finite values")
grid = np.unique(t_col.to_numpy())
labels = [_time_scalar(p, f"time column {time!r}") for p in grid]
if len(data) == 0 or len(grid) < _MIN_PERIODS:
raise ValueError(
"DurationDiD requires at least three distinct time periods (baseline, "
f"last_pre_period, and one post-period); found {len(grid)}"
)
if data.duplicated(subset=[unit, time]).any():
raise ValueError(
f"DurationDiD requires exactly one row per (unit, period); found duplicate "
f"({unit!r}, {time!r}) combinations"
)
counts = data.groupby(unit, sort=True, observed=True)[time].size()
n_periods = len(grid)
incomplete = counts[counts != n_periods]
if len(incomplete) > 0:
bad = incomplete.index.tolist()[:5]
raise ValueError(
"Unbalanced panel: every individual must be observed at every date of the "
f"common time grid ({n_periods} periods); {len(incomplete)} unit(s) are not "
f"(e.g. {bad}). Late entry, dropout, and missing cells are not supported; "
"an administrative end of a complete window is fine."
)
# Subtract native scalars BEFORE conversion: machine integers can wrap,
# while casting absolute dates to float64 can erase entire time steps.
diffs = [right - left for left, right in zip(labels, labels[1:])]
step = diffs[0]
if not all(math.isfinite(d) and d > 0 for d in diffs):
raise ValueError("time grid must have a positive finite common spacing")
if any(abs(d - step) > _SPACING_RTOL * abs(step) for d in diffs):
raise ValueError(
"DurationDiD requires an equally spaced time grid (relative tolerance "
f"{_SPACING_RTOL:g}); found spacings {sorted(set(diffs))[:5]}"
)
elapsed = np.asarray([p - labels[0] for p in labels], dtype=float)
if not np.all(np.isfinite(elapsed)) or np.any(elapsed[1:] <= elapsed[:-1]):
raise ValueError(
"time grid must give finite, strictly increasing float64 elapsed durations"
)
# Binary columns: explicit float coercion, then missing/non-finite, then
# the 0/1 domain (validate_binary strips NaN before its membership test).
coerced: Dict[str, np.ndarray] = {}
for name, col in (("outcome", outcome), ("treatment", treatment)):
if pd.api.types.is_complex_dtype(data[col]) or not (
pd.api.types.is_numeric_dtype(data[col]) or pd.api.types.is_bool_dtype(data[col])
):
bad_vals = pd.unique(data[col].astype(object))[:5].tolist()
raise ValueError(
f"{name} column {col!r} must be a numeric 0/1 column (got dtype "
f"{data[col].dtype}; values such as {bad_vals}); numeric strings are not "
"coerced"
)
try:
arr = data[col].to_numpy(dtype=float)
except (ValueError, TypeError) as exc:
bad_vals = pd.unique(data[col].astype(object))[:5].tolist()
raise ValueError(
f"{name} column {col!r} must be numeric 0/1; could not convert values "
f"such as {bad_vals} ({exc})"
) from None
nonfinite = ~np.isfinite(arr)
if nonfinite.any():
rows = data.loc[nonfinite, [unit, time]].head(5).values.tolist()
raise ValueError(
f"{name} column {col!r} contains {int(nonfinite.sum())} missing or "
f"non-finite value(s); first offending (unit, period) pairs: {rows}"
)
validate_binary(arr, name)
coerced[name] = arr
# Internal frame with fixed names, so a user column named like a temporary
# (or a role column named "unit"/"time") can never collide.
# Pandas indexes need float64 for some floating dtypes (e.g. float16).
# This conversion is lossless after _time_scalar validation; retain the
# original grid dtype in results. Integer labels never enter this path.
floating_clock = pd.api.types.is_float_dtype(t_col)
index_grid = grid.astype(float) if floating_clock else grid
frame = pd.DataFrame(
{
"unit": data[unit].to_numpy(),
"time": t_col.to_numpy(dtype=float) if floating_clock else t_col.to_numpy(),
"y": coerced["outcome"],
"g": coerced["treatment"],
}
)
g_nunique = frame.groupby("unit")["g"].nunique()
if (g_nunique > 1).any():
bad = g_nunique[g_nunique > 1].index.tolist()[:5]
raise ValueError(
f"treatment column {treatment!r} must be a fixed 0/1 group indicator "
f"(constant within unit), not a time-varying received-treatment variable; "
f"units with varying values include {bad}"
)
y_wide = frame.pivot(index="unit", columns="time", values="y").reindex(columns=index_grid)
g_units = frame.groupby("unit")["g"].first().reindex(y_wide.index)
Y = y_wide.to_numpy(dtype=float)
G = g_units.to_numpy(dtype=float)
n_treated = int(np.sum(G == 1.0))
n_control = int(np.sum(G == 0.0))
if n_treated == 0 or n_control == 0:
raise ValueError(
"both groups are required: found "
f"{n_treated} treated and {n_control} control individual(s)"
)
reversal = np.diff(Y, axis=1) < 0
if reversal.any():
bad_units = y_wide.index[reversal.any(axis=1)].tolist()[:5]
raise ValueError(
"outcome must be absorbing (once 1, always 1 within each individual); "
f"found reversals (1 -> 0) for {int(reversal.any(axis=1).sum())} unit(s), "
f"e.g. {bad_units}"
)
tstar_val = _time_scalar(last_pre_period, "last_pre_period")
if tstar_val not in labels:
raise ValueError(
f"last_pre_period {last_pre_period!r} is not a value of the time column "
f"(grid: {grid.tolist()[:8]}{'...' if len(grid) > 8 else ''})"
)
tstar_idx = labels.index(tstar_val)
if tstar_idx == 0:
raise ValueError(
"last_pre_period equals the first date; at least two pre-treatment dates "
"(the baseline and last_pre_period) are required"
)
if tstar_idx == n_periods - 1:
raise ValueError(
"last_pre_period equals the last date; at least one post-treatment date is required"
)
order = np.argsort(-G, kind="stable") # treated first, stable within group
return {
"Y": np.ascontiguousarray(Y[order]),
"n_treated": n_treated,
"grid": grid,
"elapsed": elapsed,
"tstar_idx": tstar_idx,
}
def _group_survival(
Y: np.ndarray, n_treated: int, W: np.ndarray
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""Group survival curves for count-weight rows ``W`` (C x n).
Returns ``S`` (C x 2 x T; treated row 0, control row 1), plus the
resampled group sizes ``n1``, ``n2`` (C,). Integer-valued sums below
``2**53`` make the GEMM bit-identical to a per-draw loop. An empty group
yields NaN survival (caught by the draw-failure predicates).
"""
W = np.asarray(W, dtype=float)
W1, W2 = W[:, :n_treated], W[:, n_treated:]
n1, n2 = W1.sum(axis=1), W2.sum(axis=1)
with _errstate():
S1 = 1.0 - (W1 @ Y[:n_treated]) / n1[:, None]
S2 = 1.0 - (W2 @ Y[n_treated:]) / n2[:, None]
return np.stack([S1, S2], axis=1), n1, n2
def _log_survival_moments(
S: np.ndarray, elapsed: np.ndarray
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""``R = -log S``, ``D = R - R[..., :1]``, ``H = D / elapsed`` (baseline NaN)."""
with _errstate():
R = -np.log(S)
D = R - R[..., :1]
H = D / elapsed
return R, D, H
def _estimate_from_survival(
S: np.ndarray,
R: np.ndarray,
D: np.ndarray,
elapsed: np.ndarray,
fit_idx: np.ndarray,
fit_weights: np.ndarray,
method: str,
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""Fitted coefficient and imputed counterfactual, vectorized over draws.
Returns ``c`` (C,), ``R0`` (C x T), ``S0`` (C x T) and ``tau = S0 - S1``
(C x T, every date; the caller slices the post dates). Under
``method="cd"``: ``c = sum_t alpha_t (H_1t - H_2t)`` and
``R0 = R_11 + D_2 + e*c`` (Equations 3.3-3.4); under ``method="ph"``:
``c = sum_t alpha_t D_1t / D_2t`` (mean of ratios, Theorem 1) and
``R0 = R_11 + c * D_2`` (Equation 2.16, the treated baseline outside the
exponent).
"""
S1 = S[:, 0, :]
R1, D1, D2 = R[:, 0, :], D[:, 0, :], D[:, 1, :]
w = np.asarray(fit_weights, dtype=float)
with _errstate():
if method == "cd":
e_fit = elapsed[fit_idx]
H1 = D1[:, fit_idx] / e_fit
H2 = D2[:, fit_idx] / e_fit
c = (H1 - H2) @ w
R0 = R1[:, :1] + D2 + elapsed[None, :] * c[:, None]
elif method == "ph":
ratio = D1[:, fit_idx] / D2[:, fit_idx]
c = ratio @ w
R0 = R1[:, :1] + c[:, None] * D2
else: # pragma: no cover - fit() validates method first
raise ValueError(f"method must be 'cd' or 'ph', got {method!r}")
S0 = np.exp(-R0)
tau = S0 - S1
return c, R0, S0, tau
#: Relative roundoff tolerance of the curve-validity gate (scaled by the
#: magnitudes actually summed into ``R0``).
_CURVE_GATE_RTOL = 1e-12
def _curve_tolerance(
R: np.ndarray, D: np.ndarray, elapsed: np.ndarray, c: np.ndarray, method: str
) -> np.ndarray:
"""Per-date absolute tolerance for the curve-validity gate (leading draw axis).
``R0`` is a sum of ``R_11``, ``D_2t`` and ``e_t c`` (CD) or ``c D_2t``
(PH); at a mathematical boundary (e.g. an exactly-zero imputed cumulative
hazard when the treated group has no pre-treatment exits) the sum can
land a few ulps on either side of the exact value, and which side
depends on the time labelling. The tolerance is ``1e-12`` times the
largest magnitude summed, floored at ``1e-12``.
"""
with _errstate():
term = elapsed[None, :] * c[:, None] if method == "cd" else c[:, None] * D[:, 1, :]
scale = np.maximum.reduce(
[
np.ones_like(term),
np.abs(np.broadcast_to(R[:, 0, :1], term.shape)),
np.abs(D[:, 1, :]),
np.abs(term),
]
)
return _CURVE_GATE_RTOL * scale
def _curve_status(
R0: np.ndarray, S2: np.ndarray, tstar_idx: int, tol: Optional[np.ndarray] = None
) -> List[str]:
"""Validity of the imputed curve at every date (first match wins).
Review lines 920-924: negative or decreasing imputed cumulative hazards
and survival outside [0, 1] are invalid. The gate is on the imputed
curve itself over the WHOLE path (fitted pre-dates included), never on
the observed treated hazard at ``tstar`` (that residual is Algorithm 2's
object). ``tol`` (per date, from :func:`_curve_tolerance`) keeps an
exact mathematical boundary — ``R0 == 0`` or a zero step — from being
flagged on roundoff; ``None`` means exact comparisons.
"""
n_periods = len(R0)
tol_arr = np.zeros(n_periods) if tol is None else np.asarray(tol, dtype=float)
status = ["ok"] * n_periods
for t in range(1, n_periods):
if t > tstar_idx and S2[t] <= 0:
status[t] = "control_survival_zero"
elif not np.isfinite(R0[t]):
status[t] = "counterfactual_nonfinite"
elif R0[t] < -tol_arr[t]:
status[t] = "counterfactual_survival_above_one"
elif np.isfinite(R0[t - 1]) and R0[t] < R0[t - 1] - tol_arr[t]:
status[t] = "counterfactual_nonmonotone"
return status
def _resolve_fit_periods(
S: np.ndarray,
D: np.ndarray,
method: str,
grid: np.ndarray,
tstar_idx: int,
pre_periods: Optional[Sequence[Any]],
pre_period_weights: Optional[Sequence[float]],
) -> Tuple[np.ndarray, np.ndarray, Dict[Any, str], List[str]]:
"""Validate the fitting selectors and resolve the eligible fitting set.
Returns the sorted fitting date indices ``F``, their normalized weights,
an exclusion dict ``{period: reason}`` and warning messages. Zero-weight
candidates are excluded first, then eligibility (positive survival in
both groups; under PH a positive control increment); surviving weights
are renormalized to sum to one. ``F`` and the weights are frozen for
every bootstrap draw.
"""
labels = [_native_time_label(p) for p in grid]
if pre_period_weights is not None and pre_periods is None:
raise ValueError("pre_period_weights requires pre_periods (the dates the weights refer to)")
if pre_periods is None:
cand_idx = np.arange(1, tstar_idx + 1)
requested = False
else:
req = _selector_array(pre_periods, "pre_periods", "time values")
if req.size == 0:
raise ValueError("pre_periods must name at least one pre-treatment date")
idx_list: List[int] = []
for p in req:
if p not in labels:
raise ValueError(f"Pre-period '{_fmt(p)}' not found in time column")
k = labels.index(p)
if k < 1 or k > tstar_idx:
raise ValueError(
f"pre_periods value {_fmt(p)} must lie strictly after the baseline "
f"date {_label(grid, 0)!r} and at or before last_pre_period {_label(grid, tstar_idx)!r}"
)
idx_list.append(k)
if len(set(idx_list)) != len(idx_list):
raise ValueError("pre_periods contains duplicate dates")
cand_idx = np.asarray(idx_list, dtype=int)
requested = True
if pre_period_weights is None:
cand_w = np.ones(len(cand_idx), dtype=float)
else:
cand_w = _selector_array(pre_period_weights, "pre_period_weights", "nonnegative weights")
if cand_w.shape != (len(cand_idx),):
raise ValueError(
f"pre_period_weights must have one entry per pre_periods date "
f"({len(cand_idx)}), got {cand_w.shape[0]}"
)
if not np.all(np.isfinite(cand_w)):
raise ValueError("pre_period_weights must be finite")
if np.any(cand_w < 0):
raise ValueError("pre_period_weights must be nonnegative")
if np.all(cand_w == 0):
raise ValueError("pre_period_weights must not all be zero")
excluded: Dict[Any, str] = {}
keep_idx: List[int] = []
keep_w: List[float] = []
S1, S2, D2 = S[0], S[1], D[1]
for k, w in zip(cand_idx.tolist(), cand_w.tolist()):
label = _label(grid, k)
if w == 0:
excluded[label] = "zero_weight"
elif S1[k] <= 0:
excluded[label] = "zero_treated_survival"
elif S2[k] <= 0:
excluded[label] = "zero_control_survival"
elif method == "ph" and D2[k] <= 0:
excluded[label] = "zero_control_increment"
else:
keep_idx.append(k)
keep_w.append(w)
if not keep_idx:
raise ValueError(
"no eligible fitting period: every candidate pre-treatment date was "
f"excluded ({excluded}). PH needs a positive control cumulative-hazard "
"increment at some pre-date after the baseline; both groups need positive "
"survival at the fitting dates."
)
order = np.argsort(keep_idx)
fit_idx = np.asarray(keep_idx, dtype=int)[order]
weights = np.asarray(keep_w, dtype=float)[order]
# Scale-invariant normalization: the kept weights are finite and strictly
# positive, so dividing by the maximum first keeps the sum finite even for
# weights near the float64 limit (a bare sum could overflow to inf and
# silently normalize to zeros).
weights = weights / weights.max()
weights = weights / weights.sum()
messages: List[str] = []
elig_excluded = {k: v for k, v in excluded.items() if v != "zero_weight"}
if elig_excluded:
who = "requested" if requested else "default"
messages.append(
f"Excluded {len(elig_excluded)} {who} fitting period(s) as ineligible "
f"{elig_excluded}; the remaining fitting weights were renormalized to sum to one."
)
if tstar_idx not in set(fit_idx.tolist()):
messages.append(
f"The fitting periods omit last_pre_period {_label(grid, tstar_idx)!r}; the "
"pre-treatment diagnostic keeps that date as its fixed anchor regardless."
)
return fit_idx, weights, excluded, messages
def _selector_array(value: Any, name: str, kind: str) -> np.ndarray:
"""Flatten ordered inputs, preserving date identity separately from weights.
Scalars, strings and bytes are rejected explicitly: ``list("34")`` would
otherwise split a numeric string into two different dates and silently
change the fitting set. Sets are rejected because ``pre_periods`` and
``pre_period_weights`` are paired by position.
"""
if isinstance(value, (str, bytes)) or np.isscalar(value) or not hasattr(value, "__iter__"):
raise ValueError(f"{name} must be a list of {kind} (e.g. [...]), got {value!r}")
if isinstance(value, (set, frozenset)):
# Unordered: pre_periods and pre_period_weights are paired positionally.
raise ValueError(
f"{name} must be an ordered list of {kind} (a set has no positional order "
f"to align with its companion selector), got {value!r}"
)
try:
raw = np.asarray(list(value), dtype=object).ravel()
if name == "pre_periods":
arr = np.asarray([_time_scalar(p, name) for p in raw], dtype=object)
else:
if any(isinstance(p, (complex, np.complexfloating)) for p in raw):
raise ValueError("weights must be real")
with _errstate():
arr = np.asarray(raw, dtype=float)
except (TypeError, ValueError, OverflowError) as exc:
raise ValueError(f"{name} must be a list of numeric {kind}, got {value!r}: {exc}") from None
return arr
def _label(grid: np.ndarray, k: int) -> Any:
"""Native Python scalar for a grid date (never a numpy repr in messages)."""
return _native_time_label(grid[k])
def _fmt(p: Union[int, float]) -> Union[int, float]:
return int(p) if isinstance(p, float) and p.is_integer() else p
def _pretest_contrasts(
D: np.ndarray, H: np.ndarray, J: np.ndarray, tstar_idx: int, method: str
) -> np.ndarray:
"""Algorithm 2 fixed-anchor contrasts over ``J`` (leading draw axis)."""
with _errstate():
if method == "cd":
gap = H[:, 0, J] - H[:, 1, J]
anchor = H[:, 0, tstar_idx] - H[:, 1, tstar_idx]
else:
gap = D[:, 0, J] / D[:, 1, J]
anchor = D[:, 0, tstar_idx] / D[:, 1, tstar_idx]
return gap - anchor[:, None]
def _quantile_inverted_cdf(x: np.ndarray, p: float) -> float:
"""Inverse empirical CDF quantile: the ``ceil(p*B)``-th order statistic.
``p*B`` is evaluated with a ``1e-9`` tie guard (``ceil(p*B - 1e-9)``) so a
product that lands within floating-point noise of an integer (e.g.
``0.95 * 20 = 19.000000000000004``) selects that integer's order statistic,
matching the documented rule for the alphas users actually pass (the same
guard magnitude as ``utils._frac_gt``).
"""
xs = np.sort(np.asarray(x, dtype=float))
n = xs.shape[0]
if n == 0:
return float("nan")
k = int(math.ceil(p * n - 1e-9))
k = min(max(k, 1), n)
return float(xs[k - 1])
def _centered_bootstrap_summary(
point: np.ndarray, draws: np.ndarray, alpha: float
) -> Dict[str, Any]:
"""Centered absolute-deviation bootstrap summary on COMPLETE draws.
``se`` is derived from the diagonal of the covariance (never a separate
``np.std``) so ``se == sqrt(diag(vcov))`` holds exactly. Pointwise
and simultaneous critical values are inverse-empirical-CDF quantiles of
``|draw - point| / se`` and of its per-draw maximum; p-values are the
empirical tail fractions (equality counted). ``|point/se| > crit`` is
exactly ``p <= alpha``.
"""
point = np.asarray(point, dtype=float).ravel()
draws = np.asarray(draws, dtype=float).reshape(draws.shape[0], -1)
with _errstate():
vcov = np.atleast_2d(np.cov(draws, rowvar=False, ddof=1))
# Summing identical floats can round their mean away from the stored
# value, creating spurious positive variance. Enforce exact degeneracy
# without treating legitimate small variation as zero.
constant = np.all(draws == draws[0], axis=0)
vcov[constant, :] = 0.0
vcov[:, constant] = 0.0
se = np.sqrt(np.diag(vcov))
crit = np.full(point.shape, np.nan)
p = np.full(point.shape, np.nan)
crit_sim = p_joint = statistic = float("nan")
if np.all(np.isfinite(se)) and np.all(se > 0):
z = np.abs(draws - point[None, :]) / se[None, :]
crit = np.array(
[_quantile_inverted_cdf(z[:, k], 1.0 - alpha) for k in range(z.shape[1])]
)
t_abs = np.abs(point / se)
p = np.mean(z >= t_abs[None, :], axis=0)
m = np.max(z, axis=1)
crit_sim = _quantile_inverted_cdf(m, 1.0 - alpha)
p_joint = float(np.mean(m >= np.max(t_abs)))
statistic = float(np.max(t_abs))
return {
"vcov": vcov,
"se": se,
"crit": crit,
"ci_lower": point - crit * se,
"ci_upper": point + crit * se,
"p": p,
"crit_sim": crit_sim,
"band_lower": point - crit_sim * se,
"band_upper": point + crit_sim * se,
"p_joint": p_joint,
"statistic": statistic,
}
def _draw_indices(rng: np.random.Generator, n: int, size: int) -> np.ndarray:
"""Pooled whole-individual resample indices (``size`` x ``n``)."""
return rng.integers(0, n, size=(size, n))
def _first_match(masks: Dict[str, np.ndarray], order: Tuple[str, ...], n: int) -> np.ndarray:
"""Per-draw reason string: the first predicate in ``order`` that fires."""
reason = np.full(n, "", dtype=object)
for name in order:
reason = np.where((reason == "") & masks[name], name, reason)
return reason
def _run_bootstrap(
Y: np.ndarray,
n_treated: int,
elapsed: np.ndarray,
tstar_idx: int,
fit_idx: np.ndarray,
fit_weights: np.ndarray,
method: str,
J: np.ndarray,
n_bootstrap: int,
rng: np.random.Generator,
) -> Dict[str, Any]:
"""Algorithm 1: ``n_bootstrap`` pooled whole-history resamples.
Every draw recomputes survival, moments, the coefficient, the
counterfactual, the post effects, the headline and the pretest
contrasts with the frozen fitting set. Failures are recorded per family
(first-match reason); failed draws are NaN rows.
"""
n, n_periods = Y.shape
post_idx = np.arange(tstar_idx + 1, n_periods)
n_post = len(post_idx)
tau_star = np.full((n_bootstrap, n_post), np.nan)
head_star = np.full(n_bootstrap, np.nan)
delta_star = np.full((n_bootstrap, len(J)), np.nan)
ok_post = np.zeros(n_bootstrap, dtype=bool)
ok_pretest = np.zeros(n_bootstrap, dtype=bool)
reason_post = np.full(n_bootstrap, "", dtype=object)
reason_pretest = np.full(n_bootstrap, "", dtype=object)
n_invalid_curve = 0
chunk = int(min(compute_block_size(n, n_bootstrap), _MAX_CHUNK_ROWS))
pretest_dates = np.concatenate([J, [tstar_idx]]).astype(int)
for start in range(0, n_bootstrap, chunk):
size = min(chunk, n_bootstrap - start)
idx = _draw_indices(rng, n, size)
W = np.zeros((size, n), dtype=float)
for r in range(size):
W[r] = np.bincount(idx[r], minlength=n)
S, n1, n2 = _group_survival(Y, n_treated, W)
R, D, H = _log_survival_moments(S, elapsed)
c, R0, S0, tau = _estimate_from_survival(S, R, D, elapsed, fit_idx, fit_weights, method)
tau_post = tau[:, post_idx]
with _errstate():
head = tau_post.mean(axis=1)
delta = (
_pretest_contrasts(D, H, J, tstar_idx, method) if len(J) else np.zeros((size, 0))
)
S1, S2 = S[:, 0, :], S[:, 1, :]
group_empty = (n1 == 0) | (n2 == 0)
base_zero = ~(np.nan_to_num(S1[:, 0], nan=0.0) > 0) | ~(
np.nan_to_num(S2[:, 0], nan=0.0) > 0
)
tstar_zero = ~(np.nan_to_num(S1[:, tstar_idx], nan=0.0) > 0) | ~(
np.nan_to_num(S2[:, tstar_idx], nan=0.0) > 0
)
with _errstate():
if method == "ph":
zero_inc_post = ~np.all(D[:, 1, fit_idx] > 0, axis=1)
zero_inc_pre = ~np.all(D[:, 1, pretest_dates] > 0, axis=1)
else:
zero_inc_post = np.zeros(size, dtype=bool)
zero_inc_pre = np.zeros(size, dtype=bool)
ctrl_zero = ~np.all(np.nan_to_num(S2[:, post_idx], nan=0.0) > 0, axis=1)
nonfinite_cf = (
~np.isfinite(c)
| ~np.all(np.isfinite(R0[:, post_idx]), axis=1)
| ~np.all(np.isfinite(S0[:, post_idx]), axis=1)
)
nonfinite_eff = ~np.all(np.isfinite(tau_post), axis=1)
if method == "cd":
inputs_ok = np.all(np.isfinite(H[:, :, pretest_dates]), axis=(1, 2))
else:
inputs_ok = np.all(np.isfinite(D[:, :, pretest_dates]), axis=(1, 2))
nonfinite_delta = ~(inputs_ok & np.all(np.isfinite(delta), axis=1))
r_post = _first_match(
{
"group_empty": group_empty,
"zero_survival_baseline": base_zero,
"zero_survival_last_pre": tstar_zero,
"zero_control_increment": zero_inc_post,
"control_survival_zero": ctrl_zero,
"nonfinite_counterfactual": nonfinite_cf,
"nonfinite_effect": nonfinite_eff,
},
_POST_FAILURE_ORDER,
size,
)
r_pre = _first_match(
{
"group_empty": group_empty,
"zero_survival_baseline": base_zero,
"zero_survival_last_pre": tstar_zero,
"zero_control_increment": zero_inc_pre,
"nonfinite_contrast": nonfinite_delta,
},
_PRETEST_FAILURE_ORDER,
size,
)
okp = r_post == ""
okq = r_pre == ""
sl = slice(start, start + size)
ok_post[sl] = okp
ok_pretest[sl] = okq
reason_post[sl] = r_post
reason_pretest[sl] = r_pre
tau_star[sl][okp] = tau_post[okp]
head_star[sl][okp] = head[okp]
if len(J):
delta_star[sl][okq] = delta[okq]
# Diagnostic count: complete draws whose imputed curve leaves the
# domain (finite S0 > 1 or a decreasing step) — not failures.
if okp.any():
R0_ok = R0[okp]
tol_ok = _curve_tolerance(R, D, elapsed, c, method)[okp]
with _errstate():
bad = (R0_ok < -tol_ok).any(axis=1) | (np.diff(R0_ok, axis=1) < -tol_ok[:, 1:]).any(
axis=1
)
n_invalid_curve += int(bad.sum())
return {
"tau_star": tau_star,
"head_star": head_star,
"delta_star": delta_star,
"ok_post": ok_post,
"ok_pretest": ok_pretest,
"reason_post": reason_post,
"reason_pretest": reason_pretest,
"n_draws_invalid_counterfactual": n_invalid_curve,
}
def _count_reasons(reasons: np.ndarray) -> Dict[str, int]:
out: Dict[str, int] = {}
for r in reasons.tolist():
if r:
out[r] = out.get(r, 0) + 1
return out
# =============================================================================
# Estimator
# =============================================================================
class DurationDiD(BaseEstimator):
"""Duration difference-in-differences (Deaner & Ku 2026) for absorbing outcomes.
Two-group, common-timing design: ``treatment`` is a FIXED 0/1 group
indicator, ``outcome`` is a binary absorbing spell-ended indicator on a
balanced, equally spaced numeric time grid, and ``last_pre_period`` is
the last untreated date. The counterfactual treated survival is imputed
from the control group under a restriction on the untreated hazards:
- ``method="cd"`` (common dynamics): the untreated hazards differ by a
constant additive gap ``c`` (Equation 2.3); fitted as the weighted mean
of the pre-treatment average-hazard gaps (Equations 3.2-3.4).
- ``method="ph"`` (proportional hazards): the untreated hazards are
proportional, ratio ``c`` (Equation 2.4); fitted as the weighted mean of
the pre-treatment cumulative-increment ratios (Theorem 1).
The reported effect at each post-treatment date is the absorption ATT
``E[Y_it - Y_it(0) | treated]`` (positive = more cumulative exit); the
headline ``att`` is its uniform average over the post-treatment dates.
Inference is the Appendix B whole-individual pooled bootstrap with
centered absolute-deviation pointwise and simultaneous (``max-|t|``) bands,
plus the Algorithm 2 fixed-anchor pre-treatment specification test.
Parameters
----------
method : {"cd", "ph"}, default="cd"
Untreated-hazard restriction.
n_bootstrap : int, default=1000
Whole-individual bootstrap draws. ``0`` returns point estimates with
NaN inference; otherwise at least ``2``.
alpha : float, default=0.05
Significance level for every band and the pretest.
seed : int, optional
Seed for ``numpy.random.default_rng``.
Notes
-----
Identification requires binary absorbing outcomes, a fixed population,
no anticipation before the common intervention, unaffected controls, and
the chosen hazard restriction on the UNTREATED hazards (not on outcome
levels). Bootstrap validity additionally assumes independence across
individuals with arbitrary serial dependence within each history.
Covariates, staggered adoption, censoring, survey weights and cluster
dependence are not supported in this version. Fitting dates default to
every eligible pre-treatment date after the baseline with equal weights;
``fit(pre_periods=..., pre_period_weights=...)`` selects a subset and
nonnegative weights (see :meth:`fit`).
"""
def __init__(
self,
method: str = "cd",
n_bootstrap: int = 1000,
alpha: float = 0.05,
seed: Optional[int] = None,
):
_validate_all_params(
{"method": method, "n_bootstrap": n_bootstrap, "alpha": alpha, "seed": seed}
)
self.method = method
self.n_bootstrap = n_bootstrap
self.alpha = alpha
self.seed = seed
self.is_fitted_ = False
self.results_: Optional[DurationDiDResults] = None
# get_params/set_params come from BaseEstimator.
def fit(
self,
data: pd.DataFrame,
outcome: str,
unit: str,
time: str,
treatment: str,
*,
last_pre_period: Any,
pre_periods: Optional[Sequence[Any]] = None,
pre_period_weights: Optional[Sequence[float]] = None,
) -> DurationDiDResults:
"""Fit the estimator on a balanced long individual panel.
Parameters
----------
data : pd.DataFrame
Long panel with exactly one row per (individual, date).
outcome : str
Real numeric or boolean absorbing spell-ended indicator (0/1; once 1,
always 1 within an individual). Baseline absorption is allowed.
unit : str
Individual identifier column.
time : str
Real numeric calendar or elapsed-duration column; every individual
must be observed at the same equally spaced dates (relative
tolerance 1e-8, zero absolute tolerance). Signed/unsigned integer
labels, including nullable integer dtypes without missing values,
retain exact identity across their dtype's range. Differences are
computed before conversion to float64 elapsed durations, which
must remain finite and strictly increasing. Floating labels must
be finite and exactly representable as float64; this includes
float32 and exactly representable longdouble values. Object,
string, boolean and complex time columns are not supported.
Precision already lost in caller-created floats cannot be recovered.
treatment : str
Fixed real numeric or boolean 0/1 group indicator (constant
within individual).
last_pre_period : value of ``time``
The last untreated date (``tstar``); the intervention occurs
strictly afterwards. Matched by exact numeric identity, never
inferred from the data. Boolean, string, complex and nonfinite
date selectors are rejected.
pre_periods : list of ``time`` values, optional
Pre-treatment dates used to fit the hazard relationship (strictly
after the baseline, at or before ``last_pre_period``). Default:
every eligible pre-treatment date after the baseline. Dates use
the same exact numeric matching as ``last_pre_period``.
pre_period_weights : array-like, optional
Finite real nonnegative weights aligned with ``pre_periods`` (normalized to
sum to one; a zero weight drops that date). Requires
``pre_periods``. Default: equal weights over the eligible set.
Returns
-------
DurationDiDResults
"""
# Re-validate the configuration BEFORE any data work: attributes can
# be mutated directly after construction (bypassing set_params).
_validate_all_params(self.get_params())