agrifoodpy.pipeline.pipeline

Pipeline implementation

This class provides methods to build and manage a pipeline for end to end simulations using the agrifoodpy package.

Classes

Pipeline

Class for constructing and running pipelines of functions with

Functions

standalone(input_keys, return_keys)

Decorator to make a pipeline node available as a standalone function

pipeline_node([input_keys])

Decorator to make a function compatible with pipeline execution

Module Contents

class agrifoodpy.pipeline.pipeline.Pipeline(datablock=None)

Class for constructing and running pipelines of functions with individual sets of parameters.

nodes = []
params = []
names = []
classmethod read(filename)

Read a pipeline configuration from a YAML file

Parameters:

filename (str) – The name of the configuration file.

Returns:

pipeline – The pipeline object.

Return type:

Pipeline

datablock_write(path, value)

Writes a single value to the datablock at the specified path.

Parameters:

  • path (list) – The datablock path to the value to be written.

  • value (any) – The value to be written.

add_node(node, params=None, name=None, index=None)

Adds a node to the pipeline, including its function and execution parameters.

Parameters:

  • node (function) – The function to be executed on this node.

  • params (dict, optional) – The parameters to be passed to the node function.

  • name (str, optional) – The name of the node. If not provided, a generic name will be assigned.

  • index (int, optional) – Index of the enw node. If None, the new node is appended at the end of the node list.

remove_node(node)

Remove a node from the pipeline by index or name.

Parameters:

node (int or str) – Index of the node to remove, or its name.

run(from_node=0, to_node=None, skip=None, timing=False)

Runs the pipeline

Parameters:

  • from_node (int, optional) – The index of the first node to be executed. Defaults to 0.

  • to_node (int, optional) – The index of the last node to be executed. If not provided, all nodes will be executed

  • skip (list of int, optional) – List of node indices to skip during execution. Defaults to None.

  • timing (bool, optional) – If True, the execution time of each node will be printed. Defaults to False.

print_nodes(show_params=True)

Prints the list of nodes associated with a Pipeline instance.

Parameters:

show_params (bool, optional) – If True, displays the parameters associated with each node.

agrifoodpy.pipeline.pipeline.standalone(input_keys, return_keys)

Decorator to make a pipeline node available as a standalone function

If datablock is not passed as a kwarg, and datasets are passed directly instead of datablock keys, a temporary datablock is created and the datasets associated with the arguments in input_keys are added to it. The function then returns the specified datasets in return_keys.

Parameters:

  • input_keys (list of strings) – List of dataset keys to be added to the temporary datablock

  • return_keys (list of strings) – List of keys to datablock datasets to be returned by the decorated function.

Returns:

wrapper – The decorated function

Return type:

function

agrifoodpy.pipeline.pipeline.pipeline_node(input_keys=None)

Decorator to make a function compatible with pipeline execution

If a datablock is passed as a kwarg, the function will be executed in pipeline mode, and the values of the parameters named in input_keys will be interpreted as datablock lookup keys. The corresponding objects will be extracted from the datablock and passed to the function. Unregistered keyword arguments will be passed directly to the function. The decorated function takes a “return_key” kwarg to specify the key under which the function output will be stored in the datablock. If not provided, the function name will be used as the return key.

Parameters:

input_keys (string or list of strings, optional) – List of decorated function parameter names whose values will be used as datablock lookup keys in pipeline mode.

Returns:

wrapper – The decorated function

Return type:

function