{ "cells": [ { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuromatch/climate-course-content/blob/main/tutorials/W2D4_AIandClimateChange/instructor/W2D4_Tutorial5.ipynb)   \"Open" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Tutorial 5: Testing generalization to new scenarios\n", "\n", "**Week 2, Day 4, AI and Climate Change**\n", "\n", "**Content creators:** Deepak Mewada, Grace Lindsay\n", "\n", "**Content reviewers:** Mujeeb Abdulfatai, Nkongho Ayuketang Arreyndip, Jeffrey N. A. Aryee, Paul Heubel, Jenna Pearson, Abel Shibu\n", "\n", "**Content editors:** Deepak Mewada, Grace Lindsay\n", "\n", "**Production editors:** Paul Heubel, Konstantine Tsafatinos\n", "\n", "**Our 2024 Sponsors:** CMIP, NFDI4Earth" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Tutorial Objectives\n", "\n", "*Estimated timing of tutorial:* 20 minutes\n", "\n", "In this tutorial, you will\n", "* Learn about a different type of out-of-distribution test of our model.\n", "* Evaluate the model's performance.\n" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {}, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# imports\n", "\n", "import matplotlib.pyplot as plt # For plotting graphs\n", "import pandas as pd # For data manipulation\n", "# Import specific machine learning models and tools\n", "from sklearn.model_selection import train_test_split # For splitting dataset into train and test sets\n", "from sklearn.ensemble import RandomForestRegressor # For Random Forest Regression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Install and import feedback gadget\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Install and import feedback gadget\n", "\n", "!pip3 install vibecheck datatops --quiet\n", "\n", "from vibecheck import DatatopsContentReviewContainer\n", "def content_review(notebook_section: str):\n", " return DatatopsContentReviewContainer(\n", " \"\", # No text prompt\n", " notebook_section,\n", " {\n", " \"url\": \"https://pmyvdlilci.execute-api.us-east-1.amazonaws.com/klab\",\n", " \"name\": \"comptools_4clim\",\n", " \"user_key\": \"l5jpxuee\",\n", " },\n", " ).render()\n", "\n", "\n", "feedback_prefix = \"W2D4_T5\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Figure Settings\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Figure Settings\n", "import ipywidgets as widgets # interactive display\n", "\n", "%config InlineBackend.figure_format = 'retina'\n", "plt.style.use(\n", " \"https://raw.githubusercontent.com/neuromatch/climate-course-content/main/cma.mplstyle\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Helper functions\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Helper functions\n", "\n", "# Load and Prepare the Data\n", "url_Climatebench_train_val = \"https://osf.io/y2pq7/download\" # Dataset URL\n", "training_data = pd.read_csv(url_Climatebench_train_val) # load the training data from the provided URL\n", "training_data.pop('scenario') # drop the 'scenario' column as it's just a label and won't be passed into the model\n", "target = training_data.pop('tas_FINAL') # extract the target variable 'tas_FINAL' which we aim to predict\n", "\n", "url_spatial_test_data = \"https://osf.io/7tr49/download\" # test data with different location\n", "spatial_test_data = pd.read_csv(url_spatial_test_data) # load spatial test data from the provided URL\n", "spatial_test_data.pop('scenario') # drop the `scenario` column from the data as it is just a label, but will not be passed into the model.\n", "spatial_test_target = spatial_test_data.pop('tas_FINAL') # extract the target variable 'tas_FINAL'\n", "\n", "# Split the data into training and testing sets: 80%/20%\n", "X_train, X_test, y_train, y_test = train_test_split(training_data, target, test_size=0.2, random_state=1)\n", "\n", "# Training the model on the training data\n", "rf_regressor = RandomForestRegressor(random_state=42, n_estimators=80, max_depth=50)\n", "rf_regressor.fit(X_train, y_train)\n", "\n", "spatial_test_score = rf_regressor.score(spatial_test_data,spatial_test_target)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set random seed\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Executing `set_seed(seed=seed)` you are setting the seed\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Set random seed\n", "\n", "# @markdown Executing `set_seed(seed=seed)` you are setting the seed\n", "\n", "# Call `set_seed` function in the exercises to ensure reproducibility.\n", "import random\n", "import numpy as np\n", "\n", "def set_seed(seed=None):\n", " if seed is None:\n", " seed = np.random.choice(2 ** 32)\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " print(f'Random seed {seed} has been set.')\n", "\n", "# Set a global seed value for reproducibility\n", "random_state = 42 # change 42 with any number you like\n", "\n", "set_seed(seed=random_state)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Video 1: Testing generalization to new scenarios\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "remove-input" ] }, "outputs": [], "source": [ "# @title Video 1: Testing generalization to new scenarios\n", "\n", "from ipywidgets import widgets\n", "from IPython.display import YouTubeVideo\n", "from IPython.display import IFrame\n", "from IPython.display import display\n", "\n", "\n", "class PlayVideo(IFrame):\n", " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", " self.id = id\n", " if source == 'Bilibili':\n", " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", " elif source == 'Osf':\n", " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", "\n", "\n", "def display_videos(video_ids, W=400, H=300, fs=1):\n", " tab_contents = []\n", " for i, video_id in enumerate(video_ids):\n", " out = widgets.Output()\n", " with out:\n", " if video_ids[i][0] == \"Youtube\":\n", " video = YouTubeVideo(\n", " id=video_ids[i][1], width=W, height=H, fs=fs, rel=0\n", " )\n", " print(f\"Video available at https://youtube.com/watch?v={video.id}\")\n", " else:\n", " video = PlayVideo(\n", " id=video_ids[i][1],\n", " source=video_ids[i][0],\n", " width=W,\n", " height=H,\n", " fs=fs,\n", " autoplay=False,\n", " )\n", " if video_ids[i][0] == \"Bilibili\":\n", " print(\n", " f\"Video available at https://www.bilibili.com/video/{video.id}\"\n", " )\n", " elif video_ids[i][0] == \"Osf\":\n", " print(f\"Video available at https://osf.io/{video.id}\")\n", " display(video)\n", " tab_contents.append(out)\n", " return tab_contents\n", "\n", "video_ids = [('Youtube', 'GGy_kFLzBiw'), ('Bilibili', 'BV1UJGDeLETs'), ('Osf', '')]\n", "tab_contents = display_videos(video_ids, W=730, H=410)\n", "tabs = widgets.Tab()\n", "tabs.children = tab_contents\n", "for i in range(len(tab_contents)):\n", " tabs.set_title(i, video_ids[i][0])\n", "display(tabs)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "Video Summary : \n", "* Discussed how we previously tested generalization to an unseen region. \n", "* Stressed that the real utility of these emulators is the ability to run new scenarios. \n", "* Now we will see if the model generalizes to data from a new scenario.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submit your feedback\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Submit your feedback\n", "content_review(f\"{feedback_prefix}_Testing_generalization_to_new_scenarios_Video\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "remove-input" ] }, "outputs": [], "source": [ "# @markdown\n", "from IPython.display import IFrame\n", "from ipywidgets import widgets\n", "out = widgets.Output()\n", "with out:\n", " print(f\"If you want to download the slides: https://osf.io/download/2rq8x/\")\n", " display(IFrame(src=f\"https://mfr.ca-1.osf.io/render?url=https://osf.io/2rq8x/?direct%26mode=render%26action=download%26mode=render\", width=730, height=410))\n", "display(out)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submit your feedback\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Submit your feedback\n", "content_review(f\"{feedback_prefix}_Testing_generalization_to_new_scenarios_Slides\")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Section 1: Test Generalization to Held-out Emissions Scenario" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Section 1.1: Load the New Testing (Scenario) Data\n", "Load the new dataset and print it. As you can see, the scenario for all of these datapoints is ssp245. This scenario was not included in our initial data set. According to the scenario descriptions included in the table in Tutorial 1, ssp245 represent a \"medium forcing future scenario\". The lat/lon locations are the same as the initial dataset (blue box region)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "url_scenario_test_data = \"https://osf.io/pkbwx/download\" # Dataset URL\n", "scenario_test_data = pd.read_csv(url_scenario_test_data) # Load scenario test data from the provided URL\n", "scenario_test_data" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "Now we will prepare the data to be fed into the pre-trained model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "scenario_test_data.pop('scenario') # remove the 'scenario' column from the dataset\n", "scenario_test_target = scenario_test_data.pop('tas_FINAL') # extract the target variable 'tas_FINAL'\n", "scenario_test_data # display the prepared scenario test data" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Section 1.2: Evaluate the Model on this New (Scenario) Data" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "Now let's evaluate our pre-trained model (`rf_regressor`) to see how well it performs on this new emissions scenario. Use what you know to evaluate the performance and make a scatter plot of predicted vs. true temperature values." ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Coding Exercise 1.2:" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "execution": {} }, "source": [ "```python\n", "def evaluate_and_plot_scenario_performance(rf_regressor, scenario_test_data, scenario_test_target):\n", " \"\"\"Evaluate the performance of the pre-trained model on the new emissions scenario\n", " and create a scatter plot of predicted vs. true temperature values.\n", "\n", " Args:\n", " rf_regressor (RandomForestRegressor): Pre-trained Random Forest regressor model.\n", " scenario_test_data (ndarray): Test features for the new emissions scenario.\n", " scenario_test_target (ndarray): True temperature values of the new emissions scenario.\n", "\n", " Returns:\n", " float: Score of the model on the scenario test data.\n", " \"\"\"\n", "\n", " # predict temperature values for the new emissions scenario\n", " scenario_test_predicted = ...\n", "\n", " # evaluate the model on the new emissions scenario\n", " scenario_test_score = ...\n", " print(\"Scenario Test Score:\", scenario_test_score)\n", "\n", " # implement plt.scatter() to compare predicted and true temperature values\n", " plt.figure()\n", " _ = ...\n", " # implement plt.plot() to plot the diagonal line y=x\n", " _ = ...\n", "\n", " # aesthetics\n", " plt.xlabel('Predicted Temperatures (K)')\n", " plt.ylabel('True Temperatures (K)')\n", " plt.title('Annual mean temperature anomaly\\n(New Emissions Scenario)')\n", " plt.grid(True)\n", " plt.show()\n", "\n", " return scenario_test_score\n", "\n", "# test your function\n", "scenario_test_score = evaluate_and_plot_scenario_performance(rf_regressor, scenario_test_data, scenario_test_target)\n", "\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# to_remove solution\n", "\n", "def evaluate_and_plot_scenario_performance(rf_regressor, scenario_test_data, scenario_test_target):\n", " \"\"\"Evaluate the performance of the pre-trained model on the new emissions scenario\n", " and create a scatter plot of predicted vs. true temperature values.\n", "\n", " Args:\n", " rf_regressor (RandomForestRegressor): Pre-trained Random Forest regressor model.\n", " scenario_test_data (ndarray): Test features for the new emissions scenario.\n", " scenario_test_target (ndarray): True temperature values of the new emissions scenario.\n", "\n", " Returns:\n", " float: Score of the model on the new emissions scenario.\n", " \"\"\"\n", "\n", " # predict temperature values for the new emissions scenario\n", " scenario_test_predicted = rf_regressor.predict(scenario_test_data)\n", "\n", " # evaluate the model on the new emissions scenario\n", " scenario_test_score = rf_regressor.score(scenario_test_data, scenario_test_target)\n", " print(\"Scenario Test Score:\", scenario_test_score)\n", "\n", " # implement plt.scatter() to compare predicted and true temperature values\n", " plt.figure()\n", " _ = plt.scatter(scenario_test_predicted, scenario_test_target, color='b', label='Comparison of Predicted and True Temperatures')\n", " # implement plt.plot() to plot the diagonal line y=x\n", " _ = plt.plot([min(scenario_test_predicted), max(scenario_test_predicted)], [min(scenario_test_target), max(scenario_test_target)], color='r',label='Ideal Line')\n", "\n", " # aesthetics\n", " plt.xlabel('Predicted Temperatures (K)')\n", " plt.ylabel('True Temperatures (K)')\n", " plt.title('Annual mean temperature anomaly\\n(New Emissions Scenario)')\n", " plt.grid(True)\n", " plt.show()\n", "\n", " return scenario_test_score\n", "\n", "# test your function\n", "scenario_test_score = evaluate_and_plot_scenario_performance(rf_regressor, scenario_test_data, scenario_test_target)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# to_remove explanation\n", "\n", "''' For TAs:\n", "If the following error arises: \"ValueError: The feature names should match those that were passed during fit. Feature names unseen at fit time:- tas_FINAL\"\n", "\n", "Re-run all cells from the beginning, e.g. by clicking 'Run' -> 'Run All Above Selected Cell' in Jupyter Lab.\n", "'''" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Submit your feedback\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Submit your feedback\n", "content_review(f\"{feedback_prefix}_Coding_Exercise_1_2\")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Question 1.2: Performance of the Model on New Scenario Data\n", "\n", "1. Again, have you observed a decrease in the score? \n", "2. What do you believe could be the cause of this? \n", "3. What kind of new scenarios might the model perform better for?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# to_remove explanation\n", "\n", "\"\"\"\n", "1. Yes, there appears to be a decrease in score when the model is tested on new scenario data, though it is still well-above 0 suggesting the model has learned something about predicting temperature from emissions.\n", "2. The decrease in score could be due to the model's inability to generalize well to new scenarios. It's possible that the model was trained on a specific set of scenarios and may not perform as accurately when presented with new, unseen scenarios. Factors such as differences in data distribution, environmental conditions, or other variables not captured in the training data could contribute to this decrease in performance.\n", "3. The model might perform better for new scenarios that are similar to the ones it was trained on. Additionally, if the new scenarios have data distributions and patterns that are more aligned with the training data, the model could potentially perform better. However, it's important to note that the model's performance on new scenarios will ultimately depend on how well it can adapt and generalize to the differences present in the new data.\n", "\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Submit your feedback\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Submit your feedback\n", "content_review(f\"{feedback_prefix}_Questions_1_2\")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "For the sake of clarity let's summarize all the result." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {} }, "outputs": [], "source": [ "# summarize results\n", "train_score = rf_regressor.score(X_train, y_train)\n", "test_score = rf_regressor.score(X_test, y_test)\n", "#average_score = (train_score + test_score + spatial_test_score + scenario_test_score) / 4\n", "\n", "print(f\"\\tTraining Data Score : {train_score}\")\n", "print(f\"\\tTesting Data Score on same Scenario/Region : {test_score}\")\n", "print(f\"\\tHeld-out Spatial Region Test Score : {spatial_test_score}\")\n", "print(f\"\\tHeld-out Scenario Test Score : {scenario_test_score}\")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "This shows us that the model does generalize somewhat (i.e. the score is well above zero even in the new regions and in the new scenario). However, it does not generalize very well. That is, it does not perform as well on data that differs from the data it was trained on. Ideally, we would be able to build a model that inherently learns the complex relationship between emissions scenarios and future temperature. A model that truly learned this relationship would be able to generalize to new scenarios and regions." ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "\n", "Do you have any ideas of how to build a better machine learning model to emulate climate models? Many scientists are working on this problem!" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Bonus Section 2: Try other Regression Models\n", "\n", "*Only complete this section if you are well ahead of schedule, or have already completed the final tutorial.*\n", "\n", "Random Forest models are not the only regression models that could be applied to this problem. In this code, we will use scikit-learn to train and evaluate various regression models on the Climate Bench dataset. We will load the data, split it, define models, train them with different settings, and evaluate their performance. We will calculate and print average scores for each model configuration and identify the best-performing model.\n", "\n", "For more information about the models used here and various other models, you can refer to [scikit-learn.org/stable/supervised_learning.html#supervised-learning](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning). \n", "*Note: the following cell may take ~2 minutes to run.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# Import necessary libraries\n", "import matplotlib.pyplot as plt\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, BaggingRegressor\n", "from sklearn.svm import SVR\n", "from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet\n", "from sklearn.linear_model import RidgeCV\n", "import pandas as pd\n", "from sklearn.neural_network import MLPRegressor\n", "\n", "# Load datasets\n", "train_val_data = pd.read_csv(\"https://osf.io/y2pq7/download\")\n", "spatial_test_data = pd.read_csv(\"https://osf.io/7tr49/download\")\n", "scenario_test_data = pd.read_csv(\"https://osf.io/pkbwx/download\")\n", "\n", "# Pop the 'scenario' column from all datasets\n", "train_val_data.pop('scenario')\n", "spatial_test_data.pop('scenario')\n", "scenario_test_data.pop('scenario')\n", "\n", "# Split train_val_data into training and testing sets\n", "X_train, X_test, y_train, y_test = train_test_split(train_val_data.drop(columns=[\"tas_FINAL\"]),\n", " train_val_data[\"tas_FINAL\"],\n", " test_size=0.2,\n", " random_state=1)\n", "\n", "# Define models with different configurations\n", "models = {\n", " \"MLP\": [make_pipeline(StandardScaler(), MLPRegressor(hidden_layer_sizes=(50,), max_iter=1000)),\n", " make_pipeline(StandardScaler(), MLPRegressor(hidden_layer_sizes=(500, 500, 500), random_state=1, max_iter=1000))],\n", " \"RandomForest\": [make_pipeline(StandardScaler(), RandomForestRegressor(n_estimators=100, max_depth=None)),\n", " make_pipeline(StandardScaler(), RandomForestRegressor(n_estimators=50, max_depth=10))],\n", " \"GradientBoosting\": [make_pipeline(StandardScaler(), GradientBoostingRegressor(n_estimators=100, max_depth=3)),\n", " make_pipeline(StandardScaler(), GradientBoostingRegressor(n_estimators=50, max_depth=2))],\n", " \"BaggingRegressor\": [make_pipeline(StandardScaler(), BaggingRegressor(n_estimators=100)),\n", " make_pipeline(StandardScaler(), BaggingRegressor(n_estimators=50))],\n", " \"SVR\": [make_pipeline(StandardScaler(), SVR(kernel=\"linear\")),\n", " make_pipeline(StandardScaler(), SVR(kernel=\"rbf\"))],\n", " \"LinearRegression\": [make_pipeline(StandardScaler(), LinearRegression())],\n", " \"Ridge\": [make_pipeline(StandardScaler(), Ridge())],\n", " \"RidgeCV\":[RidgeCV(alphas=[167], cv=5)],\n", " \"Lasso\": [make_pipeline(StandardScaler(), Lasso())],\n", " \"ElasticNet\": [make_pipeline(StandardScaler(), ElasticNet())]\n", "}\n", "\n", "# Train models and calculate score for each configuration\n", "results = {}\n", "for model_name, model_list in models.items():\n", " model_results = []\n", " for config_num, model in enumerate(model_list): # Add enumeration for configuration number\n", " # Train model\n", " model.fit(X_train, y_train)\n", "\n", " # Calculate scores\n", " train_score = model.score(X_train, y_train)\n", " test_score = model.score(X_test, y_test)\n", " spatial_test_score = model.score(spatial_test_data.drop(columns=[\"tas_FINAL\"]), spatial_test_data[\"tas_FINAL\"])\n", " scenario_test_score = model.score(scenario_test_data.drop(columns=[\"tas_FINAL\"]), scenario_test_data[\"tas_FINAL\"])\n", "\n", " # Append results\n", " model_results.append({\n", " \"Configuration\": config_num, # Add configuration number\n", " \"Training Score\": train_score,\n", " \"Testing Score\": test_score,\n", " \"Spatial Test Score\": spatial_test_score,\n", " \"Scenario Test Score\": scenario_test_score\n", " })\n", "\n", " # Calculate average score for the model\n", " average_score = sum(sum(result.values()) for result in model_results) / (len(model_results) * 4)\n", "\n", " # Store results including average score\n", " results[model_name] = {\"Average Score\": average_score, \"Results\": model_results}\n", "\n", "# Print results including average score for each model\n", "for model_name, model_data in results.items():\n", " print(f\"Model:\\t{model_name}\")\n", " print(f\"Average Score:\\t\\t\\t\\t {model_data['Average Score']}\")\n", " print(\"Configuration-wise Average Scores:\")\n", " for result in model_data['Results']:\n", " print(f\"\\nConfiguration {result['Configuration']}: \"\n", " f\"\\nTraining Score: {result['Training Score']}, \"\n", " f\"\\nTesting Score: {result['Testing Score']}, \"\n", " f\"\\nSpatial Test Score: {result['Spatial Test Score']}, \"\n", " f\"\\nScenario Test Score: {result['Scenario Test Score']}\")\n", " print()\n", "\n", "# Find the best model and its average score\n", "best_model = max(results, key=lambda x: results[x][\"Average Score\"])\n", "best_average_score = results[best_model][\"Average Score\"]\n", "\n", "# Print the best model and its average score\n", "print(f\"\\nBest Model: {best_model}, Average Score: {best_average_score}\")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "Let's plot the result. \n", "*Note: This code will plot the actual score for positive average scores and zero for negative average scores.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Run this cell to see the plot of results!\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title\n", "# @markdown Run this cell to see the plot of results!\n", "\n", "import matplotlib.pyplot as plt\n", "\n", "# Extract model names and average scores from results\n", "model_names = list(results.keys())\n", "average_scores = [results[model_name][\"Average Score\"] for model_name in model_names]\n", "\n", "# Adjust scores to plot zero for negative scores\n", "adjusted_scores = [score if score > 0 else 0 for score in average_scores]\n", "\n", "# Plotting\n", "plt.figure()\n", "plt.bar(model_names, adjusted_scores, color=['skyblue' if score > 0 else 'lightgray' for score in average_scores])\n", "plt.xlabel('Model')\n", "plt.ylabel('Average Score')\n", "plt.title('Average Score of Different Regression Models')\n", "plt.xticks(rotation=45, ha='right') # Rotate x-axis labels for better readability\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "This quick sweep of models suggests Random Forest is a good choice, but recall that most of these models have hyperparameters. Varying these hyperparameters may lead to different results!\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Summary\n", "\n", "In this tutorial, we explored how machine learning models adapt to unfamiliar emissions scenarios. Evaluating model performance on datasets representing different emission scenarios provided insights into the models' capabilities in predicting climate variables under diverse environmental conditions.\n" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Resources\n", "\n", "* [ClimateBench v1.0: A Benchmark for Data-Driven Climate Projections](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021MS002954) \n", "* [Scikit-learn.org, Supervised Learning](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "include_colab_link": true, "name": "W2D4_Tutorial5", "provenance": [], "toc_visible": true }, "kernel": { "display_name": "Python 3", "language": "python", "name": "python3" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" } }, "nbformat": 4, "nbformat_minor": 4 }