{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 6: Population Level Modeling (with PopNet)\n",
    "\n",
    "In this tutorial we will focus on modeling of populations and population firing rates. This is done with the PopNet simulator application of bmtk which uses [DiPDE](https://github.com/AllenInstitute/dipde) engine as a backend. We will first build our networks using the bmtk NetworkBuilder and save them into the SONATA data format. Then we will show how to simulate the firing rates over a given time-source.\n",
    "\n",
    "Requirements:\n",
    "* BMTK\n",
    "* DiPDE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Building the network\n",
    "\n",
    "\n",
    "#### Converting existing networks\n",
    "Like BioNet for biophysically detailed modeling, and PointNet with point-based networks, PopNet stores networks in the SONATA data format. PopNet supports simulating networks of individual cells at the population level. First thing you have to do is modify the node-types and edge-types of an existing network to use Population level models (rather than models of individual cells. \n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "**WARNING** - Converting a network of individual nodes into population of nodes is good for a quick and naive simulation, but for faster and more reliable results it's best to build a network from scratch (next section).\n",
    "</div>\n",
    "\n",
    "Here is the node-types csv file of a network set to work with BioNet "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>node_type_id</th>\n",
       "      <th>ei</th>\n",
       "      <th>morphology_file</th>\n",
       "      <th>model_processing</th>\n",
       "      <th>pop_name</th>\n",
       "      <th>location</th>\n",
       "      <th>model_template</th>\n",
       "      <th>model_type</th>\n",
       "      <th>dynamics_params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>e</td>\n",
       "      <td>Scnn1a.swc</td>\n",
       "      <td>aibs_perisomatic</td>\n",
       "      <td>Scnn1a</td>\n",
       "      <td>L4</td>\n",
       "      <td>ctdb:Biophys1.hoc</td>\n",
       "      <td>biophysical</td>\n",
       "      <td>472363762_fit.json</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>101</td>\n",
       "      <td>i</td>\n",
       "      <td>Pvalb.swc</td>\n",
       "      <td>aibs_perisomatic</td>\n",
       "      <td>PV</td>\n",
       "      <td>L4</td>\n",
       "      <td>ctdb:Biophys1.hoc</td>\n",
       "      <td>biophysical</td>\n",
       "      <td>472912177_fit.json</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>102</td>\n",
       "      <td>e</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LIF_exc</td>\n",
       "      <td>VisL4</td>\n",
       "      <td>nrn:IntFire1</td>\n",
       "      <td>point_process</td>\n",
       "      <td>IntFire1_exc_1.json</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>103</td>\n",
       "      <td>i</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LIF_inh</td>\n",
       "      <td>VisL4</td>\n",
       "      <td>nrn:IntFire1</td>\n",
       "      <td>point_process</td>\n",
       "      <td>IntFire1_inh_1.json</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   node_type_id ei morphology_file  model_processing pop_name location  \\\n",
       "0           100  e      Scnn1a.swc  aibs_perisomatic   Scnn1a       L4   \n",
       "1           101  i       Pvalb.swc  aibs_perisomatic       PV       L4   \n",
       "2           102  e             NaN               NaN  LIF_exc    VisL4   \n",
       "3           103  i             NaN               NaN  LIF_inh    VisL4   \n",
       "\n",
       "      model_template     model_type      dynamics_params  \n",
       "0  ctdb:Biophys1.hoc    biophysical   472363762_fit.json  \n",
       "1  ctdb:Biophys1.hoc    biophysical   472912177_fit.json  \n",
       "2       nrn:IntFire1  point_process  IntFire1_exc_1.json  \n",
       "3       nrn:IntFire1  point_process  IntFire1_inh_1.json  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.read_csv('sources/chapter06/converted_network/V1_node_types_bionet.csv', sep=' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "vs the equivelent form for PopNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>node_type_id</th>\n",
       "      <th>ei</th>\n",
       "      <th>morphology_file</th>\n",
       "      <th>model_processing</th>\n",
       "      <th>pop_name</th>\n",
       "      <th>location</th>\n",
       "      <th>model_template</th>\n",
       "      <th>model_type</th>\n",
       "      <th>dynamics_params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>e</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Scnn1a</td>\n",
       "      <td>L4</td>\n",
       "      <td>dipde:Internal</td>\n",
       "      <td>population</td>\n",
       "      <td>472363762_pop.json</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>101</td>\n",
       "      <td>i</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PV</td>\n",
       "      <td>L4</td>\n",
       "      <td>dipde:Internal</td>\n",
       "      <td>population</td>\n",
       "      <td>472912177_pop.json</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>102</td>\n",
       "      <td>e</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LIF_exc</td>\n",
       "      <td>VisL4</td>\n",
       "      <td>dipde:Internal</td>\n",
       "      <td>population</td>\n",
       "      <td>IntFire1_exc_pop.json</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>103</td>\n",
       "      <td>i</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LIF_inh</td>\n",
       "      <td>VisL4</td>\n",
       "      <td>dipde:Internal</td>\n",
       "      <td>population</td>\n",
       "      <td>IntFire1_inh_pop.json</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   node_type_id ei  morphology_file  model_processing pop_name location  \\\n",
       "0           100  e              NaN               NaN   Scnn1a       L4   \n",
       "1           101  i              NaN               NaN       PV       L4   \n",
       "2           102  e              NaN               NaN  LIF_exc    VisL4   \n",
       "3           103  i              NaN               NaN  LIF_inh    VisL4   \n",
       "\n",
       "   model_template  model_type        dynamics_params  \n",
       "0  dipde:Internal  population     472363762_pop.json  \n",
       "1  dipde:Internal  population     472912177_pop.json  \n",
       "2  dipde:Internal  population  IntFire1_exc_pop.json  \n",
       "3  dipde:Internal  population  IntFire1_inh_pop.json  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('sources/chapter06/converted_network/V1_node_types_popnet.csv', sep=' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some things to note:\n",
    "* **model_type** is now a population for all nodes, rather than individual biophysical/point types\n",
    "* We have set **model_template** to dipde:Internal which will tell the simulator to use special DiPDE model types\n",
    "* We are using new **dynamic_params** files with parameters that have been adjusted to appropiate range for DiPDE models.\n",
    "* **morophology_file** and **model_processing**, which were used to set and processes individual cell morphologies, is no longer applicable.\n",
    "\n",
    "We must make similar adjustments to our edge_types.csv files. And finally when we run the simulation we must tell PopNet to cluster nodes together using the **group_by** property\n",
    "\n",
    "```python\n",
    "network = popnet.PopNetwork.from_config(configure, group_by='node_type_id')\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building a network\n",
    "\n",
    "We will create a network of two populations, one population of excitatory cells and another of inhibitory cells. Then we will save the network into SONATA formated data files.\n",
    "\n",
    "The first step is to use the NetworkBuilder to instantiate a new network with two populations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bmtk.builder import NetworkBuilder\n",
    "\n",
    "\n",
    "net = NetworkBuilder('V1')\n",
    "net.add_nodes(\n",
    "    pop_name='excitatory',  # name of specific population optional\n",
    "    ei='e',  # Optional\n",
    "    location='VisL4',  # Optional\n",
    "    model_type='population',  # Required, indicates what types of cells are being model\n",
    "    model_template='dipde:Internal',  # Required, instructs what DiPDE objects will be created\n",
    "    dynamics_params='exc_model.json'  # Required, contains parameters used by DiPDE during initialization of object\n",
    ")\n",
    "\n",
    "net.add_nodes(\n",
    "    pop_name='inhibitory',\n",
    "    ei='i',\n",
    "    model_type='population',\n",
    "    model_template='dipde:Internal',\n",
    "    dynamics_params='inh_model.json'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will create connections between the two populations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bmtk.builder.connection_map.ConnectionMap at 0x7f41abb8fb38>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.add_edges(\n",
    "    source={'ei': 'e'}, target={'ei': 'i'},\n",
    "    syn_weight=0.005,\n",
    "    nsyns=20,\n",
    "    delay=0.002,\n",
    "    dynamics_params='ExcToInh.json'\n",
    ")\n",
    "\n",
    "net.add_edges(\n",
    "    source={'ei': 'i'}, target={'ei': 'e'},\n",
    "    syn_weight=-0.002,\n",
    "    nsyns=10,\n",
    "    delay=0.002,\n",
    "    dynamics_params='InhToExc.json'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and finally we must build and save the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.build()\n",
    "net.save(output_dir='sim_ch06/network')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### External Nodes\n",
    "\n",
    "The *dipde:Internal* nodes we created don't carry intrinsic firing rates, and instead we will use External Populations to drive the network activity. To do this we will create a separate network of 'virtual' populations, or alternativly use model_type=dipde:External, that connect to our excitatory population.  \n",
    "\n",
    "Note: we could add 'virtual' populations directly to our V1 network. However creating them as a separate network provides a great advantage if/when we want to replace our external connections with a different model (Or if we want to remove the reccurrent connections and simulation with only feed-foward activity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_net = NetworkBuilder('LGN')\n",
    "input_net.add_nodes(\n",
    "    pop_name='tON',\n",
    "    ei='e',\n",
    "    model_type='virtual'\n",
    ")\n",
    "\n",
    "input_net.add_edges(\n",
    "    target=net.nodes(ei='e'),\n",
    "    syn_weight=0.0025,\n",
    "    nsyns=10,\n",
    "    delay=0.002,\n",
    "    dynamics_params='input_ExcToExc.json'\n",
    ")\n",
    "\n",
    "input_net.build()\n",
    "input_net.save(output_dir='sim_ch06/network')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Setting up the PopNet environment\n",
    "\n",
    "Before running the simulation we need to set up our simulation environment, inlcuding setting up run-scripts, configuration parameters, and placing our parameter files in their appropiate location. The easiest way to do this is through the command-line:\n",
    "\n",
    "```bash\n",
    "$ python -m bmtk.utils.sim_setup -n network --run-time 1500.0 popnet\n",
    "```\n",
    "\n",
    "Which creates initial files to run a 1500 ms simulation using the network files found in our ./network directory.\n",
    "\n",
    "#### Inputs\n",
    "\n",
    "We next need to set the firing rates of the External Population. There are multiple ways to set this value which will be discussed later. The best way is to set the firing rates using a input-rates file for each External Population, we can fetch an existing one using the command:\n",
    "\n",
    "```bash\n",
    " $ cd sim_ch06\n",
    " $ wget https://github.com/AllenInstitute/bmtk/raw/develop/examples/pop_2pops/external_rates.csv -O lgn_rates.csv\n",
    "\n",
    "```\n",
    "\n",
    "Then we must open the simulation_config.json file with a text editor and add the lgn_rates.csv file as a part of our inputs:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"inputs\": {\n",
    "    \"LGN_pop_rates\": {\n",
    "      \"input_type\": \"csv\",\n",
    "      \"module\": \"pop_rates\",\n",
    "      \"rates\": \"${BASE_DIR}/lgn_rates.csv\",\n",
    "      \"node_set\": \"LGN\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Running the simulation\n",
    "\n",
    "The call to sim_setup created a file run_pointnet.py which we can run directly in a command line:\n",
    "```bash\n",
    "$ python run_popnet.py config.json\n",
    "```\n",
    "\n",
    "Or we can run it directly using the following python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-02-18 09:13:19,953 [WARNING] Directory /home/kael/Workspace/bmtk/docs/tutorial/sim_ch06/output already exists.\n",
      "2022-02-18 09:13:19,954 [INFO] Created log file\n",
      "2022-02-18 09:13:20,000 [INFO] Building cells.\n",
      "2022-02-18 09:13:20,001 [INFO] Building recurrent connections\n",
      "2022-02-18 09:13:20,007 [INFO] Build virtual cell stimulations for LGN_pop_rates\n",
      "2022-02-18 09:13:20,017 [INFO] Network created.\n",
      "2022-02-18 09:13:20,018 [INFO] Running simulation.\n",
      "2022-02-18 09:13:23,629 [INFO] Finished simulation.\n"
     ]
    }
   ],
   "source": [
    "from bmtk.simulator import popnet\n",
    "\n",
    "configure = popnet.config.from_json('sim_ch06/config.json')\n",
    "configure.build_env()\n",
    "network = popnet.PopNetwork.from_config(configure)\n",
    "sim = popnet.PopSimulator.from_config(configure, network)\n",
    "sim.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Analyzing results\n",
    "\n",
    "As specified in the \"output\" section of simulation_config.json, the results will be written to ouput/spike_rates.csv. The BMTK analyzer includes code for ploting and analyzing the firing rates of our network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 12056 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from bmtk.analyzer.firing_rates import plot_rates_popnet\n",
    "\n",
    "plot_rates_popnet('sim_ch06/network/V1_node_types.csv', 'sim_ch06/output/spike_rates.csv', model_keys='pop_name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
