Skip to content

Using Convolutional Neural Networks to extract information from the Cosmic Dawn

Description

This exemplar will explain and demonstrate the steps required to go from image-based data to a finished Convolutional Neural Network (CNN) pipeline, which can be used to extract relevant information from the images. While demonstrating how to solve this machine learning problem, I will also explain how to prototype code in Jupyter notebooks. I will start by explaining how to analyse the statistics of the data to create appropriate training, validation and testing sets; here I will emphasize the importance of uniform parameter spaces. The exemplar will then go through the process of setting up the architecture of the network and how to train it. Once the network is trained I will discuss what possible next steps are, and which is the most appropriate. Finally, I will go through how to convert the code prototyped in Jupyter notebooks into a useable package

Learning Outcomes

  • Use a Jupyter Lab notebook to prototype code
  • Use tensorflow to create a CNN to infer parameters from simulated images
  • Convert that prototyped code into a runable script that can then be scaled up to be run on something like the HPC
Task Time
Introduction to CNNs 1 hours
FirstDawn 3 hours

Requirements

Academic

  • Familiarity with Python 3
  • Have used Jupyter Lab before
  • Very little command line knowledge

System

  • 4GB of disk space for datasets
  • Python 3.11 or newer
  • Access to the HPC (optional)

Getting Started

To get started, first clone this repo, then change directories into it: 

  git clone https://github.com/kimeels/ReCode_FirstDawn.git
  cd ReCode_FirstDawn

If python virtualenv and Jupyter Lab isn't already installed on your system, install it using:

  python3 -m pip install virtualenv
  python3 -m pip install jupyterlab

Then create a virtual environment for this exemplar:

python3 -m venv venv_recode_firstdawn

Source into your new virtual environment using:

  source venv_recode_firstdawn/bin/activate

Run this line to install all the necessary packages:

  pip install -r requirements.txt
Finally, run these two lines to setup your virtual env with jupyter lab. 
  pip install ipykernel
  python -m ipykernel install --user --name=venv_recode_firstdawn

Project Structure

.
├── docs
├── notebooks
    ├── figures
    ├── Converting to a python script.ipynb
    ├── FirstDawn.ipynb
    └── Introduction to CNNs.ipynb
│   
└── requirements.txt

License

This project is licensed under the BSD-3-Clause license