Baysian_Optimization_in_FastAI.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Baysian Optimization in FastAI.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/muellerzr/BaysianOptimizationFastAI/blob/master/Baysian_Optimization_in_FastAI.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JfT6ajc1288E",
        "colab_type": "text"
      },
      "source": [
        "# Notebook"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cpLkEr6H2-_Z",
        "colab_type": "text"
      },
      "source": [
        "This notebook shows a working implementation of using a BaysianOptimization library within Fast.AI. Essentially we modify fit_with to have whatever hyperparameters we want to use, and their respective ranges. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dlmeOZMF3bBC",
        "colab_type": "code",
        "outputId": "a93cf169-e3e5-4b4b-ae9b-a6d0c7854051",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "!pip install bayesian-optimization"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting bayesian-optimization\n",
            "  Downloading https://files.pythonhosted.org/packages/72/0c/173ac467d0a53e33e41b521e4ceba74a8ac7c7873d7b857a8fbdca88302d/bayesian-optimization-1.0.1.tar.gz\n",
            "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from bayesian-optimization) (1.16.4)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from bayesian-optimization) (1.3.0)\n",
            "Requirement already satisfied: scikit-learn>=0.18.0 in /usr/local/lib/python3.6/dist-packages (from bayesian-optimization) (0.21.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.18.0->bayesian-optimization) (0.13.2)\n",
            "Building wheels for collected packages: bayesian-optimization\n",
            "  Building wheel for bayesian-optimization (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/1d/0d/3b/6b9d4477a34b3905f246ff4e7acf6aafd4cc9b77d473629b77\n",
            "Successfully built bayesian-optimization\n",
            "Installing collected packages: bayesian-optimization\n",
            "Successfully installed bayesian-optimization-1.0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4D4zEkKK25c3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fastai import *\n",
        "from fastai.tabular import *\n",
        "from bayes_opt import BayesianOptimization\n",
        "from fastprogress import *"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CBQOksSd3pmb",
        "colab_type": "text"
      },
      "source": [
        "Here the example will be the Adults dataset, where we will adjust the weight decay, learning rate, and dropout"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4ygX7iWV6VV3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "path = untar_data(URLs.ADULT_SAMPLE)\n",
        "df = pd.read_csv(path/'adult.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_o2Mw8pg6WsT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dep_var = 'salary'\n",
        "cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race']\n",
        "cont_names = ['age', 'fnlwgt', 'education-num']\n",
        "procs = [FillMissing, Categorify, Normalize]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8xTFXIU26X-6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = (TabularList.from_df(df, path=path, cat_names=cat_names, cont_names=cont_names, procs=procs)\n",
        "                           .split_by_idx(list(range(800,1000)))\n",
        "                           .label_from_df(cols=dep_var)\n",
        "                           .add_test(test)\n",
        "                           .databunch())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XPi65G3Q6bM1",
        "colab_type": "text"
      },
      "source": [
        "Anything you want adjusted goes into `fit_with`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iNAWeoqz3oAj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fit_with(lr, wd, dp):\n",
        "  \n",
        "  # Create our learner with the parameters\n",
        "  learn = tabular_learner(data, layers=[200,100], metrics=accuracy, emb_drop=dp, wd=wd)\n",
        "  \n",
        "  # train the model at the specified learning rate\n",
        "  with progress_disabled_ctx(learn) as learn:\n",
        "    learn.fit_one_cycle(3, max_lr=lr)\n",
        "    \n",
        "  # save, print, and return the model's overall accuracy\n",
        "  acc = float(learn.validate(learn.data.valid_dl)[1])\n",
        "  \n",
        "  # Small change to the standard, we are only returning accuracy\n",
        "  \n",
        "  return acc"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JWhCOMdx4cAd",
        "colab_type": "text"
      },
      "source": [
        "Lastly we need to dictate the upper and lower bounds we want to examine"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UicFp63y4boy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pbounds = {'lr': (1e-5, 1e-2), 'wd': (4e-4, 0.4), 'dp': (0.01, 0.5)}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vhxOsLz4r3K",
        "colab_type": "text"
      },
      "source": [
        "Now we make the optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tAnPHSDM4p09",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = BayesianOptimization( \n",
        "    f = fit_with, # use our custom fit function    \n",
        "    pbounds=pbounds, # use our limits\n",
        "    verbose = 2, # 1 prints a maximum only when it is observed, 0 is completely silent\n",
        "    random_state=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GLKrybGI5Ivn",
        "colab_type": "text"
      },
      "source": [
        "Now we run it! It can take a while depending. Then we can print the best one!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VghGmuyi4-ja",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer.maximize()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CQbtxpiI5OG_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(optimizer.max)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}