Experimentation

fdsim provides an easy and flexible experimentation interface that takes care of using the appropriate statistical tests and number of simulation runs, so that you don’t have to.

There are three usable experiment setups available: ABTest to compare two alternative scenarios, MultiScenarioExperiment to compare more than two alternatives, and the MultiFactorExperiment for experiments with multiple independent decision variables.

All you have to do is choose the right experiment for your question, set up the corresponding experiment class and run it.

Setting up the experiments differs somewhat between the three cases, because they are of different complexity.

class experiments.ABTest(sim_A=None, sim_B=None, evaluator=None, **kwargs)

Class that performs A/B test to evaluate two scenarios against each other (usually one is the current situation (A) and one is a considered alternative (B)).

Parameters:

• sim_B (sim_A,) – The simulators for scenario A and B. If None, these must be provided later, using the set_scenario_A and set_scenario_B methods.
• evaluator (fdsim.evaluation.Evaluator, optional, default=None) – The Evaluator object that should be used to analyze the simulation runs. This class specifies which metrics will be compared between the two scenarios.
• **kwargs (key-value pairs) – Any parameters that should be passed to the BaseSimulator (see BaseExperiment) for the available parameters.

analyze(results_A, results_B, to_minutes=True, with_std=True)

Analyze the output of the evaluator on the two simulation logs.

plot_distributions(metric_set_name, return_fig=True)

Plot figures showing the probability of exceeding any given value for a measure.

Parameters:

• metric_set_name (str) – The name of the metric set in the Evaluator object that you want to plot.
• return_fig (bool, optional, default=True) – Whether to return the figure or just print it.

print_results(to_latex=False, plot=True)

Print the results of the experiments to stdout.

Parameters:

• to_latex (bool, optional, default=True) – Whether to print tables as LaTeX Tabular environments.
• plot (bool, optional, default=True) – Whether to print distribution plots as well.

run()

Run the specified experiment.

save_simulation_logs(dirpath='.', prefix='experiment1')

Write simulation logs to disk.

Parameters:

• dirpath (str) – Path to directory to save the result in.
• prefix (str) – String to prepend to ‘_results_A’ and ‘results_B’ for the two logs.

class experiments.BaseExperiment(forced_runs=None, max_runs=1000, effect_size=0.5, alpha=0.05, power=0.8, name=None, description=None, to_minutes=True, with_std=True, pvalue_zero_threshold=1e-08, verbose=True)

Base class for experimentation with fdsim. Not useful to instantiate on its own.

Parameters:

• forced_runs (int or None, optional, default=None) – A fixed number of simulation runs to use for the entire experiment (so this number will be divided over the number of scenarios.
• max_runs (int, optional, default=1000) – The maximum number of runs to use in the experiment.
• effect_size (float, optional, default=0.5) – The effect size, normalized by the standard deviation, that you want to be able to detect in the experiment. The lower the value, the more runs will be used (up to max_runs and provided that forced_runs=None).
• alpha (float, optional, default=0.05) – The allowed probability of falsely discovering a significant effect. One minus the significance level. Using \(\alpha=0.05\) is conventional in statistical experiments.
• power (float, optional, default=0.8) – The statistical power, i.e., the probability of finding a significant effect if it exists. One minus the power is the probability of a Type II error (false negatives).
• name (str, optional, default=None) – The name of the experiment. Will be used when printing results.
• description (str, optional, default=None) – Description of the experiment. Will be used when printing results.
• to_minutes (bool, optional, default=True) – Whether to translate the results to minutes (mm:ss) or to leave it in full seconds.
• with_std (bool, optional, default=True) – Whether to include the standard deviation of a metric among several runs with the same settings in the results tables.
• pvalue_zero_threshold (float, optional, default=1e-8) – From which value to print p-values as zeros in results tables.
• verbose (bool, optional, default=True) – Whether to print progress during experiment runs.

analyze()

Analyze the simulation output by performing statistical tests.

static anova_table(aov)

Add effect sizes $eta^2 and omega^2$$ to an ANOVA table from statsmodels.

print_results()

Print the results of the last experiment.

run(n_runs=None)

Run the specified experiment.

save_simulation_logs(dirpath='.', prefix='experiment1')

Save the raw simulation logs to disk.

class experiments.MultiFactorExperiment(base_simulator, evaluator=None, cache_path='cache', **kwargs)

Experiment to evaluate combinations of values for multiple decision variables / factors.

This has a slightly more complex API due to the fact that different factor levels should be applied to a Simulator in order to define every possible setting. How this works is explained in the add_factor method.

Parameters:

• base_simulator (fdsim.simulation.Simulator) – The base situation to which different factor levels will be applied. Note that for every factor in the experiment, a relevant value should be set in this base situation.
• evaluator (fdsim.evaluation.Evaluator) – The Evaluator object to use to calculate performance measures and metrics.
• cache_path (str) – Directory to store base simulator in, so that it can be reloaded for different runs.
• **kwargs (key-value pairs) – Parameters passed to BaseExperiment.

add_factor(factor_name, factor_type, levels=None)

Add a factor to the experiment. A factor is a variable that can take on at least two levels (categories).

Parameters:

• factor_name (str,) – The name of the factor.
• factor_type (str,) – What type of factor it concerns. One of [“vehicles”, “station_location”, “station_status”, “add_station”, “remove_station”, “crew”].
• levels (dict, optional (default: None),) – A nested dictionary specifying the parameters for the different levels, like: {‘level_name_1’ -> {‘parameter 1’ -> value1, ‘parameter 2’ -> value2, etc.}, ‘level_name_2’ -> {‘parameter 1’ -> value1, ‘parameter 2’ -> value2, etc.}}. If None, then it must be specified later by using self.add_factor_level().

Notes

The base_simulator’s settings are also considered a level. I.e., make sure that the base case includes a relevant setting for each factor that is added.

Examples

>>> # add resources factor
>>> experiment = MultiFactorExperiment(base_simulator)
>>> levels = {"extra-ts": {"station_name": "HENDRIK", "TS": 1, "TS_crew_ft": 1}}
>>> experiment.add_factor("TS-Hendrik", "resources", levels=levels)

Change the location of stations:

>>> # add station_location factor with two alternatives to the base case
>>> levels = {"at N200": {"new_location": "13781452"}}
              "other-loc": {"new_location": "13781363"}}
>>> experiment.add_factor("Osdorp loc", "location", "OSDORP", levels=levels)

Set a status cycle such as closing a station part of the day or having part time crew operate it:

>>> # set a station status cycle
>>> levels = {"Closed at night":
                {"station_name": "VICTOR", "start": 23, "end": 7, "status": "closed"},
              "Office hours":
                {"station_name": "VICTOR", "start": 18, "end": 8, "status": "closed"}
             }
>>> experiment.add_factor("Status Victor", "station_status", levels=levels)

>>> # add a station
>>> levels = {"Bovenkerk":
                {"station_name": "BOVENKERK", location": "13710002", "TS": 1, "TS_ft": 1},
              "Harbor":
                {"station_name": "HARBOR", location": "13710001", "TS": 1, "TS_ft": 1}
             }
>>> experiment.add_factor("New station location", "add_station", levels=levels)

>>> # remove a station
>>> levels = {"Hendrik": {"station_name": "HENDRIK"},
              "Nico": {"station_name": "NICO"}}
>>> experiment.add_factor("Remove station", "remove_station", levels=levels)

add_factor_level(factor_name, level_name, params)

Add a single level to an existing factor.

Parameters:

• factor_name (str,) – The name of the factor to add a level to.
• level_name (str,) – The name of the new level.
• params (dict,) – The parameters to pass to the underlying method of fdsim.simulation.Simulator.

analyze(results_dict, scenario_dict)

Analyze the simulation output by performing statistical tests.

get_significant_results(anova_results)

Find the combinations of independent and dependent variables for which the scenario had a significant impact based on the ANOVA results.

perform_tukey(data_dict, var_dict)

Perform Tukey HSD post-hoc analysis to find the specific pairs of groups that had a significantly different mean.

plot_distributions(metric_set_name, return_fig=True)

Plot figures showing the probability of exceeding any given value for a measure.

Parameters:

• metric_set_name (str) – The name of the metric set in the Evaluator object that you want to plot.
• return_fig (bool, optional, default=True) – Whether to return the figure or just print it.

print_results(to_latex=False, plot=True)

Print the results of the ANOVA and Tukey analysis to give a quick overview of the results.

Parameters:

• to_latex (bool, optional, default=True) – Whether to print tables as LaTeX Tabular environments.
• plot (bool, optional, default=True) – Whether to print distribution plots as well.

run()

Run the experiment.

save_simulation_logs(dirpath='.', prefix='experiment1')

Write simulation logs to disk.

Parameters:

• dirpath (str) – Path to directory to save the result in.
• prefix (str) – String to prepend to ‘_results_A’ and ‘results_B’ for the two logs.

class experiments.MultiScenarioExperiment(evaluator=None, **kwargs)

Class that is used to evaluate more than two alternative scenarios against each other.

Scenarios can be added one by one using the add_scenario method. Since this method takes initialized and configured :code:`fdsim.simulation.Simulator`s as input, no assumptions are made on the setups of the scenarios. Each is treated as an alternative situation that is compared against all other situations/scenarios.

Parameters:

• evaluator (fdsim.evaluation.Evaluator, optional, default=None) – The Evaluator object that should be used to analyze the simulation runs. This class specifies which metrics will be compared between the two scenarios.
• **kwargs (key-value pairs) – Any parameters that should be passed to the BaseSimulator (see BaseExperiment) for the available parameters.

add_scenario(simulator, name, description=None)

Add an alternative scenario to the possible options.

Parameters:

• simulator (fdsim.simulation.Simulator) – The simulator that is configured to simulate the desired scenario.
• name (str) – The name for this scenario. It is used to print and distinguish between scenarios later on and is therefore required.
• description (str, optional, default=None) – An optional description of this scenario that may provide additional context to the user.

analyze(results_dict)

Analyze the results using one-way ANOVA.

Parameters:results_dict (dict) – Dictionary where keys are scenario names and values are outputs of Evaluator.evaluate.

get_significant_results(anova_results)

Find the dependent variables for which the scenario had a significant impact based on the ANOVA results.

perform_tukey(data_dict, target_dict, group_col='scenario')

Perform Tukey HSD post-hoc analysis to find the specific pairs of groups that had a significantly different mean.

plot_distributions(metric_set_name, return_fig=True)

Plot figures showing the probability of exceeding any given value for a measure.

Parameters:

• metric_set_name (str) – The name of the metric set in the Evaluator object that you want to plot.
• return_fig (bool, optional, default=True) – Whether to return the figure or just print it.

print_results(to_latex=False, plot=True)

Print the results of the ANOVA and Tukey analysis to give a quick overview of the results.

Parameters:

• to_latex (bool, optional, default=True) – Whether to print tables as LaTeX Tabular environments.
• plot (bool, optional, default=True) – Whether to print distribution plots as well.

run()

Run the experiment.

save_simulation_logs(dirpath='.', prefix='experiment1')

Write simulation logs to disk.

Parameters:

• dirpath (str) – Path to directory to save the result in.
• prefix (str) – String to prepend to ‘_results_A’ and ‘results_B’ for the two logs.