Utils

This is the documentation for util functions.

dataframe_functions

sam.utils.sum_grouped_columns(df: DataFrame, sep: str = '#', skipna: bool = True) → DataFrame

Utility function to sum columns together based on groups. The column names are assumed to look like groupname#suffix. For example: GROUP1#lag_1_day, or GROUP2#sum_1_week. In these examples, the groups are GROUP1 and GROUP2 respectively. This function will find all the groups, and sum all the columns in the same group together. If a column does not contain the separator sep, the entire column name is assumed to be the groupname. This means columns like GROUP2 and GROUP2#lag_0 will be assumed to be in the same group.

This function is mainly useful when dealing with a dataframe filled with shapley values. In this case, when many features are in the same group, it may be useful to sum these shapley values, to calculate a combined contribution that the entire group has. Keep in mind that this has the potential to ‘wipe out’ shapley values: if GROUP1#lag_0 has a large positive contribution, and GROUP1#lag_1 has a large negative contribution, then the group GROUP1 as a whole will have a contribution near 0. This is mathematically correct, and does indeed mean that GROUP1 as a whole had a very small effect on the prediction.

Parameters:

df: pd.DataFrame: The dataframe whose columns will be added together
sep: str, optional (default=’#’): The seperator character. The group of a column is defined as everything before the first occurrence of this character
skipna: boolean, optional (default=True): Whether or not to ignore missing values in columns. If true, missing values are treated as 0. Else, missing values are not ignored and the sum for that particular group/row combination will be missing as well.

Returns:

summed_df: dataframe: A dataframe with the same row-index as df, but with less columns. All the columns in the same group have been summed together.

Examples

>>> import pandas as pd
>>> df = pd.DataFrame({
...    'X#lag_0': [1, 2, 3],
...    'X#lag_1': [1, 2, 3],
...    'Y': [5, 5, 5]
... })
>>> sum_grouped_columns(df)
   X  Y
0  2  5
1  4  5
2  6  5

sam.utils.has_strictly_increasing_index(df: DataFrame, linear: bool = True) → bool

Utility function to validate the index of a dataframe: - Check if it is a DatetimeIndex - Check if it is strictly increasing - Optional: check if increases are constant over the index

Parameters:

df: dataframe: The dataframe whose index needs to be checked
linear: bool, optional (default=True): True to also check that the index steps are constant over the whole index length

Returns:

bool: Whether the index of the dataframe is a DatetimeIndex and is strictly monotonic increasing

sam.utils.contains_nans(df: DataFrame) → DataFrame

Utility function to check if a dataframe contains any NaN values.

Parameters:

df: dataframe: The dataframe whose index needs to be checked

Returns:

bool: Whether the dataframe contains any NaN values

sam.utils.assert_contains_nans(df: DataFrame, msg: str = 'Data cannot contain nans') → None

Utility function to check if a dataframe contains any NaN values.

Parameters:

df: dataframe: The dataframe whose index needs to be checked
msg: str, optional (default=”Data cannot contain nans”): The error message to raise if the dataframe contains nans

sklearn helpers

class sam.utils.FunctionTransformerWithNames(func: Callable = None, inverse_func: Callable = None, validate: bool = False, accept_sparse: bool = False, check_inverse: bool = True, kw_args: dict = None, inv_kw_args: dict = None)

Bases: FunctionTransformer

Helper class that extends FunctionTransformer by adding ‘get_feature_names’ method. This method returns the column names of the last dataframe that was returned by this transformer.

Importantly, the function that is given to this transformer must have an output x with attributes ‘x.columns.values’, such as a pandas dataframe. If the underlying function outputs a numpy array, this transformer will crash. In this case, it is reccommended to use the regular FunctionTransformer instead.

Everything is unchanged in the FunctionTransformer, except that the default value of the ‘validate’ parameter is changed to False. This mimics the behavior of sklearn from 0.22 onwards. This is particularly important here, because validation turns pandas dataframes into numpy arrays, which means that the column names are removed.

Parameters:

funccallable, optional default=None: The callable to use for the transformation. This will be passed the same arguments as transform, with args and kwargs forwarded. If func is None, then func will be the identity function.
inverse_funccallable, optional default=None: The callable to use for the inverse transformation. This will be passed the same arguments as inverse transform, with args and kwargs forwarded. If inverse_func is None, then inverse_func will be the identity function.
validatebool, optional default=False: Indicate that the input X array should be checked before calling func. The possibilities are: - If False, there is no input validation. - If True, then X will be converted to a 2-dimensional NumPy array or

sparse matrix. If the conversion is not possible an exception is raised.
accept_sparsebool, optional: Indicate that func accepts a sparse matrix as input. If validate is False, this has no effect. Otherwise, if accept_sparse is false, sparse matrix inputs will cause an exception to be raised.
check_inversebool, default=True: Whether to check that or func followed by inverse_func leads to the original inputs. It can be used for a sanity check, raising a warning when the condition is not fulfilled.
kw_argsdict, optional: Dictionary of additional keyword arguments to pass to func.
inv_kw_argsdict, optional: Dictionary of additional keyword arguments to pass to inverse_func.

Methods

`fit`(X[, y])	Fit transformer by checking X.
`fit_transform`(X[, y])	Fit to data, then transform it.
`get_feature_names_out`([input_features])	Returns the feature names saved during transform
`get_metadata_routing`()	Get metadata routing of this object.
`get_params`([deep])	Get parameters for this estimator.
`inverse_transform`(X)	Transform X using the inverse function.
`set_output`(*[, transform])	Set output container.
`set_params`(**params)	Set the parameters of this estimator.
`transform`(X[, y])	Applies the function, and saves the output feature names

Examples

>>> import pandas as pd
>>> from sam.utils import FunctionTransformerWithNames
>>> from sam.feature_engineering import decompose_datetime
>>> data = pd.DataFrame({'TIME': [1, 2, 3], 'VALUE': [4,5,6]})
>>> data['TIME'] = pd.to_datetime(data['TIME'])
>>> transformer = FunctionTransformerWithNames(decompose_datetime,
...                                            kw_args={'components': ['hour', 'minute']})
>>> transformer.fit_transform(data)
                           TIME  VALUE  TIME_hour  TIME_minute
0 1970-01-01 00:00:00.000000001      4          0            0
1 1970-01-01 00:00:00.000000002      5          0            0
2 1970-01-01 00:00:00.000000003      6          0            0
>>> transformer.get_feature_names_out()
['TIME', 'VALUE', 'TIME_hour', 'TIME_minute']

get_feature_names_out(input_features=None) → List[str]: Returns the feature names saved during transform

transform(X, y=None): Applies the function, and saves the output feature names