Metrics

causeinfer metrics provide statistical impressions of model performance.

Functions

• causeinfer.evaluation.get_cum_effect()

• causeinfer.evaluation.get_cum_gain()

• causeinfer.evaluation.get_qini()

• causeinfer.evaluation.auuc_score()

• causeinfer.evaluation.qini_score()

• causeinfer.evaluation.get_batch_metrics()

• causeinfer.evaluation.signal_to_noise()

causeinfer.evaluation.get_cum_effect(df, models=None, outcome_col='y', treatment_col='w', treatment_effect_col='tau', normalize=False, random_seed=None)

Gets average causal effects of model estimates in cumulative population.

Parameters:

dfpandas.DataFrame

A data frame with model estimates and actual data as columns.

modelslist

A list of models corresponding to estimated treatment effect columns.

outcome_colstroptional (default=y)

The column name for the actual outcome.

treatment_colstroptional (default=w)

The column name for the treatment indicator (0 or 1).

treatment_effect_colstroptional (default=tau)

The column name for the true treatment effect.

normalizeboolnot implemented (default=False)

For consitency with gain and qini.

random_seedint, optional (default=None)

Random seed for numpy.random.rand().

Returns:

effectspandas.DataFrame

Average causal effects of model estimates in cumulative population.

causeinfer.evaluation.get_cum_gain(df, models=None, outcome_col='y', treatment_col='w', treatment_effect_col='tau', normalize=False, random_seed=None)

Gets cumulative gains of model estimates in population.

Parameters:

dfpandas.DataFrame

A data frame with model estimates and actual data as columns.

modelslist

A list of models corresponding to estimated treatment effect columns.

outcome_colstroptional (default=y)

The column name for the actual outcome.

treatment_colstroptional (default=w)

The column name for the treatment indicator (0 or 1).

treatment_effect_colstroptional (default=tau)

The column name for the true treatment effect.

normalizebooloptional (default=False)

Whether to normalize the y-axis to 1 or not.

random_seedint, optional (default=None)

Random seed for numpy.random.rand().

Returns:

gainspandas.DataFrame

Cumulative gains of model estimates in population.

causeinfer.evaluation.get_qini(df, models=None, outcome_col='y', treatment_col='w', treatment_effect_col='tau', normalize=False, random_seed=None)

Gets Qini of model estimates in population.

Parameters:

dfpandas.DataFrame

A data frame with model estimates and actual data as columns.

modelslist

A list of models corresponding to estimated treatment effect columns.

outcome_colstroptional (default=y)

The column name for the actual outcome.

treatment_colstroptional (default=w)

The column name for the treatment indicator (0 or 1).

treatment_effect_colstroptional (default=tau)

The column name for the true treatment effect.

normalizebooloptional (default=False)

Whether to normalize the y-axis to 1 or not.

random_seedint, optional (default=None)

Random seed for numpy.random.rand().

Returns:

qinispandas.DataFrame

Qini of model estimates in population.

causeinfer.evaluation.auuc_score(df, models=None, outcome_col='y', treatment_col='w', treatment_effect_col='tau', normalize=False, random_seed=None)

Calculates the AUUC score (Gini): the Area Under the Uplift Curve.

Parameters:

dfpandas.DataFrame

A data frame with model estimates and actual data as columns.

modelslist

A list of models corresponding to estimated treatment effect columns.

outcome_colstroptional (default=y)

The column name for the actual outcome.

treatment_colstroptional (default=w)

The column name for the treatment indicator (0 or 1).

treatment_effect_colstroptional (default=tau)

The column name for the true treatment effect.

normalizebooloptional (default=False)

Whether to normalize the y-axis to 1 or not.

random_seedint, for inheritance (default=None)

Random seed for numpy.random.rand().

Returns:

AUUC scorefloat

causeinfer.evaluation.qini_score(df, models=None, outcome_col='y', treatment_col='w', treatment_effect_col='tau', normalize=False, random_seed=None)

Calculates the Qini score: the area between the Qini curve of a model and random assignment.

Parameters:

dfpandas.DataFrame)

A data frame with model estimates and actual data as columns

modelslist

A list of models corresponding to estimated treatment effect columns.

outcome_colstroptional (default=y)

The column name for the actual outcome.

treatment_colstroptional (default=w)

The column name for the treatment indicator (0 or 1).

treatment_effect_colstroptional (default=tau)

The column name for the true treatment effect.

normalizebooloptional (default=False)

Whether to normalize the y-axis to 1 or not.

random_seedint, for inheritance (default=None)

Random seed for numpy.random.rand().

Returns:

Qini scorefloat

causeinfer.evaluation.signal_to_noise(y, w)

Computes the signal to noise ratio of a dataset to derive the potential for causal inference efficacy.

Parameters:

ynumpy.ndarray(num_units,)int, float

Vector of unit responses.

wnumpy.ndarray(num_units,)int, float

Vector of original treatment allocations across units.

Returns:

sn_ratiofloat

Notes

• The signal to noise ratio is the difference in treatment and control response to the control response.

• Values close to 0 imply that CI would have little benefit over predictive modeling.