Graph functions guide

SWMManywhere works by starting with a graph of plausible pipe locations (typically the street network) and iteratively applying functions to transform that network gradually into a UDM. A graph function is actually a class, of type BaseGraphFunction, that can be called with a function that takes a graph (and some arguments) and returns an updated graph.

Using graph functions

Let's look at a graph function that is simply a wrapper for networkx.to_undirected:

Source code in swmmanywhere/graphfcns/network_cleaning_graphfcns.py

@register_graphfcn
class to_undirected(BaseGraphFunction):
    """to_undirected class."""

    def __call__(self, G: nx.Graph, **kwargs) -> nx.Graph:
        """Convert the graph to an undirected graph.

        Args:
            G (nx.Graph): A graph
            **kwargs: Additional keyword arguments are ignored.

        Returns:
            G (nx.Graph): An undirected graph
        """
        # Don't use osmnx.to_undirected! It enables multigraph if the geometries
        # are different, but we have already saved the street cover so don't
        # want this!
        return G.to_undirected()

We can see that this graph function is a class that can be called with a graph and returns a graph. Note that the class has been registered with @register_graphfcn.

Registered graph functions

The GraphFunctionRegistry is a dictionary called graphfcns that contains all registered graph functions to be called from one place.

>>> from swmmanywhere.graph_utilities import graphfcns
>>> print(graphfcns.keys())
dict_keys(['assign_id', 'remove_parallel_edges', 'remove_non_pipe_allowable_links', 
'calculate_streetcover', 'double_directed', 'to_undirected', 'split_long_edges', 
'merge_street_nodes', 'fix_geometries', 'clip_to_catchments', 
'calculate_contributing_area', 'set_elevation', 'set_surface_slope',
'set_chahinian_slope', 'set_chahinian_angle', 'calculate_weights', 'identify_outlets',
'derive_topology', 'pipe_by_pipe'])

We will later demonstrate how to add a new graph function to the registry.

Arguments

In the previous example, we saw that, in addition to a graph, the function takes **kwargs, which are ignored. While this graph function does not require any information that is not contained within the graph, most require parameters or file path information to be completed. A graph function can receive a FilePaths object or any number of parameter categories, which we will briefly explain below. A full explanation of these is outside scope of this guide, but for now you can view the parameters and FilePaths APIs.

We can see an example of using a parameter category with this graph function:

Source code in swmmanywhere/graphfcns/network_cleaning_graphfcns.py

@register_graphfcn
class remove_non_pipe_allowable_links(BaseGraphFunction):
    """remove_non_pipe_allowable_links class."""

    def __call__(
        self, G: nx.Graph, topology_derivation: parameters.TopologyDerivation, **kwargs
    ) -> nx.Graph:
        """Remove non-pipe allowable links.

        This function removes links that are not allowable for pipes. The non-
        allowable links are specified in the `omit_edges` attribute of the
        topology_derivation parameter. There two cases handled:

        1. The `highway` property of the edge. In `osmnx`, `highway` is a category
            that contains the road type, e.g., motorway, trunk, primary. If the
            edge contains a value in the `highway` property that is in `omit_edges`,
            the edge is removed.

        2. Any other properties of the edge that are in `omit_edges`. If the
            property is not null in the edge data, the edge is removed. e.g.,
            if `bridge` is in `omit_edges` and the `bridge` entry of the edge
            is NULL, then the edge is retained, if it is something like 'yes',
            or 'viaduct' then the edge is removed.

        Args:
            G (nx.Graph): A graph
            topology_derivation (parameters.TopologyDerivation): A TopologyDerivation
                parameter object
            **kwargs: Additional keyword arguments are ignored.

        Returns:
            G (nx.Graph): A graph
        """
        edges_to_remove = set()
        for u, v, keys, data in G.edges(data=True, keys=True):
            for omit in topology_derivation.omit_edges:
                if data.get("highway", None) == omit:
                    # Check whether the 'highway' property is 'omit'
                    edges_to_remove.add((u, v, keys))
                elif data.get(omit, None):
                    # Check whether the 'omit' property of edge is not None
                    edges_to_remove.add((u, v, keys))
        for edges in edges_to_remove:
            G.remove_edge(*edges)
        return G

We can see that remove_non_pipe_allowable_links uses the omit_edges parameter, which is contained in the parameters.TopologyDerivation object that the graph function takes as an argument. Although we recommend changing parameter values in SWMManywhere with the configuration file we will give an example below explain how a parameter can be changed 'manually' to better understand what is happening at the graph function level.

>>> from swmmanywhere.examples.data import demo_graph as G
>>> from swmmanywhere.graph_utilities import graphfcns
>>> from swmmanywhere.parameters import TopologyDerivation
>>> G_ = graphfcns.remove_non_pipe_allowable_links(G, TopologyDerivation())
>>> print(f"{len(G.edges) - len(G_.edges)} edges removed")
2 edges removed
>>> G_ = graphfcns.remove_non_pipe_allowable_links(G, 
        TopologyDerivation(omit_edges=["primary", "bridge"])
    )
>>> print(f"{len(G.edges) - len(G_.edges)} edges removed")
16 edges removed

We can see that, by changing the parameter to remove more edge types, the graph function produced a different graph.

Lists of graph functions

Graph functions are intended to be applied in a sequence, gradually transforming the graph. SWMManywhere provides a function to do this called iterate_graphfcns.

For example:

>>> from swmmanywhere.examples.data import demo_graph as G
>>> from swmmanywhere.graph_utilities import iterate_graphfcns
>>> print(len(G.edges))
22
>>> G = iterate_graphfcns(G, ["assign_id", "remove_non_pipe_allowable_links"])
>>> print(len(G.edges))
20

We have applied a list of two graph functions to the graph G, which has made some changes (in this case checking the edge id and removing links as above).

In the configuration file we can specify the list of graph functions to be applied as a graphfcn_list.

In this example we do not provide parameters.TopologyDerivation argument, even though it is needed by remove_non_pipe_allowable_links. If parameters are not provided, iterate_graphfcns uses the default values for all parameters.

Validating graph functions

Furthermore, this graphfcn_list also provides opportunities for validation. For example, see the following graph function:

Source code in swmmanywhere/graphfcns/topology_graphfcns.py

@register_graphfcn
class set_surface_slope(
    BaseGraphFunction,
    required_node_attributes=["surface_elevation"],
    adds_edge_attributes=["surface_slope"],
):
    """set_surface_slope class."""

    def __call__(self, G: nx.Graph, **kwargs) -> nx.Graph:
        """Set the surface slope for each edge.

        This function sets the surface slope for each edge. The surface slope is
        calculated from the elevation data.

        Args:
            G (nx.Graph): A graph
            **kwargs: Additional keyword arguments are ignored.

        Returns:
            G (nx.Graph): A graph
        """
        G = G.copy()
        # Compute the slope for each edge
        slope_dict = {
            (u, v, k): (
                G.nodes[u]["surface_elevation"] - G.nodes[v]["surface_elevation"]
            )
            / d["length"]
            for u, v, k, d in G.edges(data=True, keys=True)
        }

        # Set the 'surface_slope' attribute for all edges
        nx.set_edge_attributes(G, slope_dict, "surface_slope")
        return G

Critically, we can see that the set_surface_slope graph function has a parameter required_node_attributes (not shown above but see also required_edge_attributes), which specify that the node parameters surface_elevation are required to perform the graph function. Although providing this information does not guarantee that the graph function will behave as intended, if it is not provided then the graph function is guaranteed to fail. To check the feasibility of a set of graph functions a-priori, the parameter adds_edge_attributes (not shown above but see also adds_node_attributes), can be used to specify what, if any, parameters are added to the graph by the graph function.

Let us inspect the set_elevation graph function:

Source code in swmmanywhere/graphfcns/topology_graphfcns.py

@register_graphfcn
class set_elevation(
    BaseGraphFunction,
    required_node_attributes=["x", "y"],
    adds_node_attributes=["surface_elevation"],
):
    """set_elevation class."""

    def __call__(self, G: nx.Graph, addresses: FilePaths, **kwargs) -> nx.Graph:
        """Set the elevation for each node.

        This function sets the elevation for each node. The elevation is
        calculated from the elevation data.

        Args:
            G (nx.Graph): A graph
            addresses (FilePaths): An FilePaths parameter object
            **kwargs: Additional keyword arguments are ignored.

        Returns:
            G (nx.Graph): A graph
        """
        G = G.copy()
        x = [d["x"] for x, d in G.nodes(data=True)]
        y = [d["y"] for x, d in G.nodes(data=True)]
        elevations = go.interpolate_points_on_raster(
            x, y, addresses.bbox_paths.elevation
        )
        elevations_dict = {id_: elev for id_, elev in zip(G.nodes, elevations)}
        nx.set_node_attributes(G, elevations_dict, "surface_elevation")
        return G

We can see that set_elevation adds the node attribute surface_elevation, which is required for set_surface_slope. The default order of graphfcn_list has these graph functions in the appropriate order, but we can demonstrate the automatic validation in SWMManywhere by switching their order. We will copy the minimum viable config template and use a short graphfcn_list that places the set_surface_slope graph function before set_elevation.

base_dir: /path/to/base/directory
project: my_first_swmm
bbox: [1.52740,42.50524,1.54273,42.51259]
graphfcn_list:
  - set_surface_slope
  - set_elevation

If we try to run this with:

python -m swmmanywhere --config_path=/path/to/file.yml

Before any graph functions are executed, we will receive the error message:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  ...
ValueError: Graphfcn set_surface_slope requires node attributes
                ['surface_elevation']

Add a new graph function

Adding a custom graph function can be done by creating a graph function in the appropriate style, see below for example to create a new graph function and then specifying to use it with the config file.

Write the graph function

You create a new module that can contain multiple graph functions. See below as a template of that module.

from __future__ import annotations

import networkx as nx

from swmmanywhere.graph_utilities import BaseGraphFunction, register_graphfcn


@register_graphfcn
class new_graphfcn(BaseGraphFunction, adds_edge_attributes=["new_attrib"]):
    """New graphfcn class."""

    def __call__(self, G: nx.graph, **kwargs) -> nx.Graph:
        """Adds new_attrib to the graph."""
        G = G.copy()
        nx.set_edge_attributes(G, "new_value", "new_attrib")
        return G

Adjust config file

We will add the required lines to the minimum viable config template.

base_dir: /path/to/base/directory
project: my_first_swmm
bbox: [1.52740,42.50524,1.54273,42.51259]
custom_graphfcn_modules: 
  - /path/to/custom_graphfcns.py
graphfcn_list: 
  - assign_id
  - fix_geometries
  - remove_non_pipe_allowable_links
  - calculate_streetcover
  - remove_parallel_edges
  - to_undirected
  - split_long_edges
  - merge_street_nodes
  - assign_id
  - clip_to_catchments
  - calculate_contributing_area
  - set_elevation
  - double_directed
  - fix_geometries
  - set_surface_slope
  - set_chahinian_slope
  - set_chahinian_angle
  - calculate_weights
  - identify_outlets
  - derive_topology
  - pipe_by_pipe
  - fix_geometries
  - assign_id
  - new_graphfcn

We can see that we now provide the graphfcn_list with new_graphfcn in the list. This list (except for new_graphfcn) is reproduced from the demo_config.yml. Any number of new graph functions can be inserted at any points in the graphfcn_list. If deviating from the list in demo_config.yml, which provides the default graphfcn_list, then an entire (new) list must be provided.

And we provide the path to the custom_graphfcns.py module that contains our new_graphfcn under the custom_graphfcn_module entry.