Extended Demo¶
Note - this script can also be opened in interactive Python if you wanted to
play around. On the GitHub it is in docs/notebooks
. To run this on your
local machine, you will need to install the optional dependencies for doc
:
pip install swmmanywhere[doc]
Introduction¶
This script demonstrates a simple use case of swmmanywhere
, building on the
quickstart
example, but including plotting and alterations.
Since this is a notebook, we will define config
as a dictionary rather than a yaml
file, but the same principles apply.
Initial run¶
Here we will run the quickstart configuration, keeping everything in a temporary directory.
# Imports
from __future__ import annotations
import tempfile
from pathlib import Path
import folium
import geopandas as gpd
import pandas as pd
from swmmanywhere.logging import set_verbose
from swmmanywhere.swmmanywhere import swmmanywhere
from swmmanywhere.utilities import plot_map
# Create temporary directory
temp_dir = tempfile.TemporaryDirectory()
base_dir = Path(temp_dir.name)
# Define minimum viable config (with shorter duration so better inspect results)
config = {
"base_dir": base_dir,
"project": "my_first_swmm",
"bbox": [1.52740, 42.50524, 1.54273, 42.51259],
"run_settings": {"duration": 3600},
}
# Run SWMManywhere
outputs = swmmanywhere(config)
# Verify the output
model_file = outputs[0]
if not model_file.exists():
raise FileNotFoundError(f"Model file not created: {model_file}")
/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/networkx/readwrite/json_graph/node_link.py:287: FutureWarning: The default value will be changed to `edges="edges" in NetworkX 3.6. To make this warning go away, explicitly set the edges kwarg, e.g.: nx.node_link_graph(data, edges="links") to preserve current behavior, or nx.node_link_graph(data, edges="edges") for forward compatibility. warnings.warn(
Skipping field osmid: unsupported OGR type: 1
Skipping field highway: unsupported OGR type: 5
Skipping field lanes: unsupported OGR type: 5
Skipping field name: unsupported OGR type: 5
Skipping field reversed: unsupported OGR type: 1
Plotting output¶
If you do not have a real UDM, the majority of your interpretation will be
around the synthesised nodes
and edges
. These are
created in the same directory as the model_file
. Let's have a look at them.
Note that the outfall
that each node drains to is specified in the outfall
attribute, we will plot these in red and other nodes in black with the built-
in swmmanywhere.utilities.plot_map
function.
# Create a folium map and add the nodes and edges
plot_map(model_file.parent)
Skipping field osmid: unsupported OGR type: 1
Skipping field highway: unsupported OGR type: 5
Skipping field lanes: unsupported OGR type: 5
Skipping field name: unsupported OGR type: 5
Skipping field reversed: unsupported OGR type: 1
OK, it's done something! Though perhaps we're not super satisfied with the output.
Customising outputs¶
Some things stick out on first glance,
- Probably we do not need pipes in the hills to the South, these seem to be along
pedestrian routes, which can be adjusted with the
allowable_networks
parameter. - We will also remove any types under the
omit_edges
entry, here you can specify to not allow pipes to cross bridges, tunnels, motorways, etc., however, this is such a small area we probably don't want to restrict things so much. - We have far too few outfalls, it seems implausible that so many riverside streets
would not have outfalls. Furthermore, there are points that are quite far from the
river that have been assigned as outfalls. We can reduce the
river_buffer_distance
to make nodes nearer the river more likely to be outfalls, but also reduce theoutfall_length
distance parameter to enableswmmanywhere
to more freely select outfalls that are adjacent to the river.
Let's just demonstrate that using the
parameter_overrides
functionality.
config["parameter_overrides"] = {
"topology_derivation": {
"allowable_networks": ["drive"],
"omit_edges": ["bridge"],
},
"outfall_derivation": {
"outfall_length": 5,
"river_buffer_distance": 30,
},
}
outputs = swmmanywhere(config)
plot_map(outputs[0].parent)
/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/networkx/readwrite/json_graph/node_link.py:287: FutureWarning: The default value will be changed to `edges="edges" in NetworkX 3.6. To make this warning go away, explicitly set the edges kwarg, e.g.: nx.node_link_graph(data, edges="links") to preserve current behavior, or nx.node_link_graph(data, edges="edges") for forward compatibility. warnings.warn(
Skipping field osmid: unsupported OGR type: 1
Skipping field lanes: unsupported OGR type: 5
Skipping field name: unsupported OGR type: 5
Skipping field osmid: unsupported OGR type: 1
Skipping field lanes: unsupported OGR type: 5
Skipping field name: unsupported OGR type: 5
OK that clearly helped, although we have appear to have stranded pipes (e.g., along
Carrer dels Canals in North West), presumably due to some mistake in the
OSM specifying that it
is connected via a pedestrian route. We won't remedy this in the tutorial, but you can
manually provide your
starting_graph
via the configuration file to address such mistakes.
More importantly we can see some distinctive unconnected network in the South West. What is going on there? To explain this we will have to turn on verbosity to print the intermediate files used in model derivation.
To do this with a command line call we simply add the flag --verbose=True
.
Though in code we will have to use set_verbose
from the logging
module.
# Make verbose
set_verbose(True) # Set verbosity
# Run again
outputs = swmmanywhere(config)
model_dir = outputs[0].parent
m = plot_map(model_dir)
2024/11/29 09:54:28 | Creating project structure.
2024/11/29 09:54:28 | Project structure created at /tmp/tmpiww78mna
2024/11/29 09:54:28 | Project name: my_first_swmm
2024/11/29 09:54:28 | Bounding box: [1.5274, 42.50524, 1.54273, 42.51259],
number: 1
2024/11/29 09:54:28 | Model number: 3
2024/11/29 09:54:28 | Loading and setting parameters.
2024/11/29 09:54:28 | Setting topology_derivation allowable_networks to ['drive']
2024/11/29 09:54:28 | Setting topology_derivation omit_edges to ['bridge']
2024/11/29 09:54:28 | Setting outfall_derivation outfall_length to 5
2024/11/29 09:54:28 | Setting outfall_derivation river_buffer_distance to 30
2024/11/29 09:54:28 | Allowable networks have been changed, removing old street graph.
2024/11/29 09:54:28 | Running downloads.
2024/11/29 09:54:28 | downloading network to /tmp/tmpiww78mna/my_first_swmm/bbox_1/download/street.parquet
2024/11/29 09:54:29 | Iterating graphs.
2024/11/29 09:54:29 | Iterating graph functions.
2024/11/29 09:54:29 | graphfcn: assign_id completed.
2024/11/29 09:54:29 | graphfcn: fix_geometries completed.
2024/11/29 09:54:29 | graphfcn: remove_non_pipe_allowable_links completed.
/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/networkx/readwrite/json_graph/node_link.py:287: FutureWarning: The default value will be changed to `edges="edges" in NetworkX 3.6. To make this warning go away, explicitly set the edges kwarg, e.g.: nx.node_link_graph(data, edges="links") to preserve current behavior, or nx.node_link_graph(data, edges="edges") for forward compatibility. warnings.warn(
2024/11/29 09:54:29 | graphfcn: calculate_streetcover completed.
2024/11/29 09:54:29 | graphfcn: remove_parallel_edges completed.
2024/11/29 09:54:29 | graphfcn: to_undirected completed.
2024/11/29 09:54:30 | graphfcn: split_long_edges completed.
2024/11/29 09:54:30 | graphfcn: merge_street_nodes completed.
2024/11/29 09:54:30 | graphfcn: assign_id completed.
Extracted WhiteboxTools to /tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn/WBT
Starting WhiteboxTools session with source directory: /tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn
Running WhiteboxTools version: 2.4.0
Command: /tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn/WBT/whitebox_tools --run=BreachDepressions --wd=/tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn --compress_rasters=false --max_procs=1 -i=dem.tif --fillpits -o=dem_corr.tif
WhiteboxTools output:
******************************** * Welcome to BreachDepressions * * Powered by WhiteboxTools * * www.whiteboxgeo.com * ******************************** Reading data... Progress: 0% Progress: 1% Progress: 2% Progress: 3% Progress: 4% Progress: 5% Progress: 6% Progress: 7% Progress: 8% Progress: 9% Progress: 10% Progress: 11% Progress: 12% Progress: 13% Progress: 14% Progress: 15% Progress: 16% Progress: 17% Progress: 18% Progress: 19% Progress: 20% Progress: 21% Progress: 22% Progress: 23% Progress: 24% Progress: 25% Progress: 26% Progress: 27% Progress: 28% Progress: 29% Progress: 30% Progress: 31% Progress: 32% Progress: 33% Progress: 34% Progress: 35% Progress: 36% Progress: 37% Progress: 38% Progress: 39% Progress: 40% Progress: 41% Progress: 42% Progress: 43% Progress: 44% Progress: 45% Progress: 46% Progress: 47% Progress: 48% Progress: 49% Progress: 50% Progress: 51% Progress: 52% Progress: 53% Progress: 54% Progress: 55% Progress: 56% Progress: 57% Progress: 58% Progress: 59% Progress: 60% Progress: 61% Progress: 62% Progress: 63% Progress: 64% Progress: 65% Progress: 66% Progress: 67% Progress: 68% Progress: 69% Progress: 70% Progress: 71% Progress: 72% Progress: 73% Progress: 74% Progress: 75% Progress: 76% Progress: 77% Progress: 78% Progress: 79% Progress: 80% Progress: 81% Progress: 82% Progress: 83% Progress: 84% Progress: 85% Progress: 86% Progress: 87% Progress: 88% Progress: 89% Progress: 90% Progress: 91% Progress: 92% Progress: 93% Progress: 94% Progress: 95% Progress: 96% Progress: 97% Progress: 98% Progress: 99% Progress: 100% Progress: 101% Progress: 102% Progress: 103% Progress: 104% Progress: 105% Progress: 106% Progress: 107% Progress: 108% Progress: 109% Saving data... Output file written Elapsed Time (excluding I/O): 0.0s
Running WhiteboxTools version: 2.4.0
Command: /tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn/WBT/whitebox_tools --run=D8Pointer --wd=/tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn --compress_rasters=false --max_procs=1 -i=dem_corr.tif -o=fdir.tif
WhiteboxTools output:
**************************** * Welcome to D8Pointer * * Powered by WhiteboxTools * * www.whiteboxgeo.com * **************************** Reading data... Progress: 0% Progress: 2% Progress: 5% Progress: 8% Progress: 11% Progress: 14% Progress: 17% Progress: 20% Progress: 23% Progress: 26% Progress: 29% Progress: 32% Progress: 35% Progress: 38% Progress: 41% Progress: 44% Progress: 47% Progress: 50% Progress: 52% Progress: 55% Progress: 58% Progress: 61% Progress: 64% Progress: 67% Progress: 70% Progress: 73% Progress: 76% Progress: 79% Progress: 82% Progress: 85% Progress: 88% Progress: 91% Progress: 94% Progress: 97% Progress: 100% Saving data... Output file written Elapsed Time (excluding I/O): 0.0s
Completed WhiteboxTools session with source directory: /tmp/tmpiww78mna/my_first_swmm/bbox_1/download/tmp5tsczbhn
2024/11/29 09:54:31 | graphfcn: clip_to_catchments completed.
Downloading WhiteboxTools from https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
Extracted WhiteboxTools to /tmp/tmp2bh0yb6e/tmpap15dy_l/WBT
Starting WhiteboxTools session with source directory: /tmp/tmp2bh0yb6e/tmpap15dy_l
Running WhiteboxTools version: 2.4.0
Command: /tmp/tmp2bh0yb6e/tmpap15dy_l/WBT/whitebox_tools --run=BreachDepressions --wd=/tmp/tmp2bh0yb6e/tmpap15dy_l --compress_rasters=false --max_procs=1 -i=dem.tif --fillpits -o=dem_corr.tif
WhiteboxTools output:
******************************** * Welcome to BreachDepressions * * Powered by WhiteboxTools * * www.whiteboxgeo.com * ******************************** Reading data... Progress: 0% Progress: 1% Progress: 2% Progress: 3% Progress: 4% Progress: 5% Progress: 6% Progress: 7% Progress: 8% Progress: 9% Progress: 10% Progress: 11% Progress: 12% Progress: 13% Progress: 14% Progress: 15% Progress: 16% Progress: 17% Progress: 18% Progress: 19% Progress: 20% Progress: 21% Progress: 22% Progress: 23% Progress: 24% Progress: 25% Progress: 26% Progress: 27% Progress: 28% Progress: 29% Progress: 30% Progress: 31% Progress: 32% Progress: 33% Progress: 34% Progress: 35% Progress: 36% Progress: 37% Progress: 38% Progress: 39% Progress: 40% Progress: 41% Progress: 42% Progress: 43% Progress: 44% Progress: 45% Progress: 46% Progress: 47% Progress: 48% Progress: 49% Progress: 50% Progress: 51% Progress: 52% Progress: 53% Progress: 54% Progress: 55% Progress: 56% Progress: 57% Progress: 58% Progress: 59% Progress: 60% Progress: 61% Progress: 62% Progress: 63% Progress: 64% Progress: 65% Progress: 66% Progress: 67% Progress: 68% Progress: 69% Progress: 70% Progress: 71% Progress: 72% Progress: 73% Progress: 74% Progress: 75% Progress: 76% Progress: 77% Progress: 78% Progress: 79% Progress: 80% Progress: 81% Progress: 82% Progress: 83% Progress: 84% Progress: 85% Progress: 86% Progress: 87% Progress: 88% Progress: 89% Progress: 90% Progress: 91% Progress: 92% Progress: 93% Progress: 94% Progress: 95% Progress: 96% Progress: 97% Progress: 98% Progress: 99% Progress: 100% Progress: 101% Progress: 102% Progress: 103% Progress: 104% Progress: 105% Progress: 106% Progress: 107% Progress: 108% Progress: 109% Saving data... Output file written Elapsed Time (excluding I/O): 0.0s
Running WhiteboxTools version: 2.4.0
Command: /tmp/tmp2bh0yb6e/tmpap15dy_l/WBT/whitebox_tools --run=D8Pointer --wd=/tmp/tmp2bh0yb6e/tmpap15dy_l --compress_rasters=false --max_procs=1 -i=dem_corr.tif -o=fdir.tif
WhiteboxTools output:
**************************** * Welcome to D8Pointer * * Powered by WhiteboxTools * * www.whiteboxgeo.com * **************************** Reading data... Progress: 0% Progress: 2% Progress: 5% Progress: 8% Progress: 11% Progress: 14% Progress: 17% Progress: 20% Progress: 23% Progress: 26% Progress: 29% Progress: 32% Progress: 35% Progress: 38% Progress: 41% Progress: 44% Progress: 47% Progress: 50% Progress: 52% Progress: 55% Progress: 58% Progress: 61% Progress: 64% Progress: 67% Progress: 70% Progress: 73% Progress: 76% Progress: 79% Progress: 82% Progress: 85% Progress: 88% Progress: 91% Progress: 94% Progress: 97% Progress: 100% Saving data... Output file written Elapsed Time (excluding I/O): 0.0s
Completed WhiteboxTools session with source directory: /tmp/tmp2bh0yb6e/tmpap15dy_l
2024/11/29 09:54:33 | graphfcn: calculate_contributing_area completed.
2024/11/29 09:54:33 | graphfcn: set_elevation completed.
2024/11/29 09:54:33 | graphfcn: double_directed completed.
2024/11/29 09:54:33 | graphfcn: fix_geometries completed.
2024/11/29 09:54:33 | graphfcn: set_surface_slope completed.
2024/11/29 09:54:34 | graphfcn: set_chahinian_slope completed.
2024/11/29 09:54:34 | graphfcn: set_chahinian_angle completed.
2024/11/29 09:54:34 | graphfcn: calculate_weights completed.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
893, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
858, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
456, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
703, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
1312, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
779, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
816, using this node as outfall.
2024/11/29 09:54:34 | No outfalls found for subgraph containing
1599, using this node as outfall.
2024/11/29 09:54:34 | graphfcn: identify_outfalls completed.
2024/11/29 09:54:35 | Total graph weight 281.1776365743678.
2024/11/29 09:54:35 | graphfcn: derive_topology completed.
0%| | 0/421 [00:00<?, ?it/s]
11%|█ | 45/421 [00:00<00:00, 432.52it/s]
22%|██▏ | 91/421 [00:00<00:00, 443.90it/s]
32%|███▏ | 136/421 [00:00<00:00, 397.89it/s]
42%|████▏ | 177/421 [00:00<00:00, 341.74it/s]
51%|█████ | 213/421 [00:00<00:00, 311.73it/s]
58%|█████▊ | 246/421 [00:00<00:00, 303.79it/s]
66%|██████▌ | 277/421 [00:00<00:00, 268.27it/s]
72%|███████▏ | 305/421 [00:01<00:00, 253.36it/s]
80%|████████ | 338/421 [00:01<00:00, 272.50it/s]
87%|████████▋ | 367/421 [00:01<00:00, 274.19it/s]
94%|█████████▍| 396/421 [00:01<00:00, 268.39it/s]
100%|██████████| 421/421 [00:01<00:00, 297.58it/s]
2024/11/29 09:54:36 | graphfcn: pipe_by_pipe completed.
2024/11/29 09:54:36 | graphfcn: fix_geometries completed.
2024/11/29 09:54:36 | graphfcn: assign_id completed.
2024/11/29 09:54:36 | Saving final graph and writing inp file.
Skipping field osmid: unsupported OGR type: 1
Skipping field lanes: unsupported OGR type: 5
Skipping field name: unsupported OGR type: 5
2024/11/29 09:54:37 | Running the synthetic model.
2024/11/29 09:54:37 | /tmp/tmpiww78mna/my_first_swmm/bbox_1/model_3/model_3.inp initialised in pyswmm
2024/11/29 09:54:37 | Starting simulation for: /tmp/tmpiww78mna/my_first_swmm/bbox_1/model_3/model_3.inp
0%| | 0/3600 [00:00<?, ?it/s]
13%|█▎ | 483.0/3600 [00:00<00:00, 4755.56it/s]
27%|██▋ | 959.0/3600 [00:00<00:01, 2273.28it/s]
35%|███▍ | 1253.0/3600 [00:00<00:01, 2187.14it/s]
44%|████▍ | 1586.0/3600 [00:00<00:00, 2490.35it/s]
60%|██████ | 2178.0/3600 [00:00<00:00, 3431.70it/s]
83%|████████▎ | 2993.0/3600 [00:00<00:00, 4759.07it/s]
2024/11/29 09:54:38 | Model run complete.
3603.0it [00:00, 3870.74it/s]
2024/11/29 09:54:38 | Writing synthetic results.
2024/11/29 09:54:38 | No real network provided, returning SWMM .inp file.
Skipping field osmid: unsupported OGR type: 1
Skipping field lanes: unsupported OGR type: 5
Skipping field name: unsupported OGR type: 5
That's a lot of information! However, the reason we are currently interested
in this is because the files associated with
each workflow step are saved when verbose=True
.
We will load a file called subbasins
and add it to the map.
subbasins = gpd.read_file(model_dir / "subbasins.geoparquet")
folium.GeoJson(subbasins, fill_opacity=0, color="blue", weight=2).add_to(m)
m