{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# DAG Creation and Submission"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Launch this tutorial in a Jupyter Notebook on Binder: \n",
        "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/htcondor/htcondor-python-bindings-tutorials/master?urlpath=lab/tree/DAG-Creation-And-Submission.ipynb)\n",
        "\n",
        "In this tutorial, we will learn how to use `htcondor.dags` to create and submit an HTCondor DAGMan workflow.\n",
        "Our goal will be to create an image of the Mandelbrot set.\n",
        "This is a perfect problem for high-throughput computing because each point in the image can be calculated completely independently of any other point, so we are free to divide the image creation up into patches, each created by a single HTCondor job.\n",
        "DAGMan will enter the picture to coordinate stitching the image patches we create back into a single image."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Making a Mandelbrot set image locally\n",
        "\n",
        "We'll use `goatbrot` (https://github.com/beejjorgensen/goatbrot) to make the image.\n",
        "`goatbrot` can be run from the command line, and takes a series of options to specify which part of the Mandelbrot set to draw, as well as the properties of the image itself.\n",
        "\n",
        "`goatbrot` options:\n",
        "- `-i 1000` The number of iterations.\n",
        "- `-c 0,0` The center point of the image region.\n",
        "- `-w 3` The width of the image region.\n",
        "- `-s 1000,1000` The pixel dimensions of the image.\n",
        "- `-o test.ppm` The name of the output file to generate.\n",
        "\n",
        "We can run a shell command from Jupyter by prefixing it with a `!`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Complex image:\r\n",
            "            Center: 0 + 0i\r\n",
            "             Width: 3\r\n",
            "            Height: 3\r\n",
            "        Upper Left: -1.5 + 1.5i\r\n",
            "       Lower Right: 1.5 + -1.5i\r\n",
            "\r\n",
            "Output image:\r\n",
            "          Filename: test.ppm\r\n",
            "     Width, Height: 500, 500\r\n",
            "             Theme: beej\r\n",
            "       Antialiased: no\r\n",
            "\r\n",
            "Mandelbrot:\r\n",
            "    Max Iterations: 10\r\n",
            "        Continuous: no\r\n",
            "\r\n",
            "Goatbrot:\r\n",
            "    Multithreading: not supported in this build\r\n",
            "\r\n",
            "Completed: 100.0%\r\n"
          ]
        }
      ],
      "source": [
        "! ./goatbrot -i 10 -c 0,0 -w 3 -s 500,500 -o test.ppm\n",
        "! convert test.ppm test.png"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's take a look at the test image. It won't be very good, because we didn't run for very many iterations.\n",
        "We'll use HTCondor to produce a better image!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 9160 characters]\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from IPython.display import Image\n",
        "\n",
        "Image('test.png')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## What is the workflow?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can parallelize this calculation by drawing rectangular sub-regions of the full region (\"tiles\") we want and stitching them together into a single image using `montage`.\n",
        "Let's draw this out as a graph, showing how data (image patches) will flow through the system.\n",
        "(Don't worry about this code, unless you want to know how to make dot diagrams in Python!)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.43.0 (0)\n",
              " -->\n",
              "<!-- Title: %3 Pages: 1 -->\n",
              "<svg width=\"370pt\" height=\"116pt\"\n",
              " viewBox=\"0.00 0.00 369.69 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\n",
              "<title>%3</title>\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-112 365.69,-112 365.69,4 -4,4\"/>\n",
              "<!-- montage -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>montage</title>\n",
              "<ellipse fill=\"none\" stroke=\"black\" cx=\"180.35\" cy=\"-18\" rx=\"41.69\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"180.35\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">montage</text>\n",
              "</g>\n",
              "<!-- tile_0&#45;0 -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>tile_0&#45;0</title>\n",
              "<ellipse fill=\"none\" stroke=\"black\" cx=\"38.35\" cy=\"-90\" rx=\"38.19\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"38.35\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">tile_0&#45;0</text>\n",
              "</g>\n",
              "<!-- tile_0&#45;0&#45;&gt;montage -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>tile_0&#45;0&#45;&gt;montage</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M63.73,-76.49C86.23,-65.4 119.32,-49.09 144.5,-36.67\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"146.31,-39.68 153.73,-32.12 143.22,-33.4 146.31,-39.68\"/>\n",
              "</g>\n",
              "<!-- tile_0&#45;1 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>tile_0&#45;1</title>\n",
              "<ellipse fill=\"none\" stroke=\"black\" cx=\"133.35\" cy=\"-90\" rx=\"38.19\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"133.35\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">tile_0&#45;1</text>\n",
              "</g>\n",
              "<!-- tile_0&#45;1&#45;&gt;montage -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>tile_0&#45;1&#45;&gt;montage</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M144.48,-72.41C150.22,-63.87 157.33,-53.28 163.7,-43.79\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"166.62,-45.72 169.29,-35.47 160.81,-41.82 166.62,-45.72\"/>\n",
              "</g>\n",
              "<!-- tile_1&#45;0 -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>tile_1&#45;0</title>\n",
              "<ellipse fill=\"none\" stroke=\"black\" cx=\"228.35\" cy=\"-90\" rx=\"38.19\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"228.35\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">tile_1&#45;0</text>\n",
              "</g>\n",
              "<!-- tile_1&#45;0&#45;&gt;montage -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>tile_1&#45;0&#45;&gt;montage</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M217.21,-72.76C211.36,-64.23 204.06,-53.58 197.5,-44.02\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"200.29,-41.9 191.75,-35.63 194.52,-45.86 200.29,-41.9\"/>\n",
              "</g>\n",
              "<!-- tile_1&#45;1 -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>tile_1&#45;1</title>\n",
              "<ellipse fill=\"none\" stroke=\"black\" cx=\"323.35\" cy=\"-90\" rx=\"38.19\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"323.35\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">tile_1&#45;1</text>\n",
              "</g>\n",
              "<!-- tile_1&#45;1&#45;&gt;montage -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>tile_1&#45;1&#45;&gt;montage</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M297.78,-76.49C275.13,-65.4 241.81,-49.09 216.44,-36.67\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"217.67,-33.37 207.15,-32.12 214.59,-39.66 217.67,-33.37\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.graphs.Digraph at 0x7fceed231850>"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from graphviz import Digraph\n",
        "import itertools\n",
        "\n",
        "num_tiles_per_side = 2\n",
        "\n",
        "dot = Digraph()\n",
        "\n",
        "dot.node('montage')\n",
        "for x, y in itertools.product(range(num_tiles_per_side), repeat = 2):\n",
        "    n = f'tile_{x}-{y}'\n",
        "    dot.node(n)\n",
        "    dot.edge(n, 'montage')\n",
        "\n",
        "dot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Since we can chop the image up however we'd like, we have as many tiles per side as we'd like (try changing `num_tiles_per_side` above).\n",
        "The \"shape\" of the DAG is the same: there is a \"layer\" of `goatbrot` jobs that calculate tiles, which all feed into `montage`.\n",
        "Now that we know the structure of the problem, we can start describing it to HTCondor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Describing `goatbrot` as an HTCondor job\n",
        "\n",
        "We describe a job using a `Submit` object. \n",
        "It corresponds to the submit *file* used by the command line tools.\n",
        "It mostly behaves like a standard Python dictionary, where the keys and values correspond to submit descriptors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "executable = goatbrot\n",
            "arguments = -i 10000 -c $(x),$(y) -w $(w) -s 500,500 -o tile_$(tile_x)-$(tile_y).ppm\n",
            "log = mandelbrot.log\n",
            "output = goatbrot.out.$(tile_x)_$(tile_y)\n",
            "error = goatbrot.err.$(tile_x)_$(tile_y)\n",
            "request_cpus = 1\n",
            "request_memory = 128MB\n",
            "request_disk = 1GB\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import htcondor\n",
        "\n",
        "tile_description = htcondor.Submit(\n",
        "    executable = 'goatbrot',  # the program we want to run\n",
        "    arguments = '-i 10000 -c $(x),$(y) -w $(w) -s 500,500 -o tile_$(tile_x)-$(tile_y).ppm',  # the arguments to pass to the executable\n",
        "    log = 'mandelbrot.log',  # the HTCondor job event log\n",
        "    output = 'goatbrot.out.$(tile_x)_$(tile_y)',  # stdout from the job goes here\n",
        "    error = 'goatbrot.err.$(tile_x)_$(tile_y)',   # stderr from the job goes here\n",
        "    request_cpus = '1',        # resource requests; we don't need much per job for this problem\n",
        "    request_memory = '128MB',\n",
        "    request_disk = '1GB',\n",
        ")\n",
        "\n",
        "print(tile_description)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Notice the heavy use of macros like `$(x)` to specify the tile.\n",
        "Those aren't built-in submit macros; instead, we will plan on passing their values in through **vars**.\n",
        "Vars will let us customize each individual job in the tile layer by filling in those macros individually.\n",
        "Each job will recieve a dictionary of macro values; our next goal is to make a list of those dictionaries.\n",
        "\n",
        "We will do this using a function that takes the number of tiles per side as an argument.\n",
        "As mentioned above, the **structure** of the DAG is the same no matter how \"wide\" the tile layer is.\n",
        "This is why we define a function to produce the tile vars instead of just calculating them once: we can vary the width of the DAG by passing different arguments to `make_tile_vars`.\n",
        "More customizations could be applied to make different images (for example, you could make it possible to set the center point of the image)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "def make_tile_vars(num_tiles_per_side, width = 3):\n",
        "    width_per_tile = width / num_tiles_per_side\n",
        "    \n",
        "    centers = [\n",
        "        width_per_tile * (n + 0.5 - (num_tiles_per_side / 2)) \n",
        "        for n in range(num_tiles_per_side)\n",
        "    ]\n",
        "    \n",
        "    vars = []\n",
        "    for (tile_y, y), (tile_x, x) in itertools.product(enumerate(centers), repeat = 2):\n",
        "        var = dict(\n",
        "            w = width_per_tile,\n",
        "            x = x,\n",
        "            y = -y,  # image coordinates vs. Cartesian coordinates\n",
        "            tile_x = str(tile_x).rjust(5, '0'),\n",
        "            tile_y = str(tile_y).rjust(5, '0'),\n",
        "        )\n",
        "        \n",
        "        vars.append(var)\n",
        "        \n",
        "    return vars"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'w': 1.5, 'x': -0.75, 'y': 0.75, 'tile_x': '00000', 'tile_y': '00000'}\n",
            "{'w': 1.5, 'x': 0.75, 'y': 0.75, 'tile_x': '00001', 'tile_y': '00000'}\n",
            "{'w': 1.5, 'x': -0.75, 'y': -0.75, 'tile_x': '00000', 'tile_y': '00001'}\n",
            "{'w': 1.5, 'x': 0.75, 'y': -0.75, 'tile_x': '00001', 'tile_y': '00001'}\n"
          ]
        }
      ],
      "source": [
        "tile_vars = make_tile_vars(2)\n",
        "for var in tile_vars:\n",
        "    print(var)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we want to increase the number of tiles per side, we just pass in a larger number.\n",
        "Because the `tile_description` is **parameterized** in terms of these variables, it will work the same way no matter what we pass in as `vars`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'w': 0.75, 'x': -1.125, 'y': 1.125, 'tile_x': '00000', 'tile_y': '00000'}\n",
            "{'w': 0.75, 'x': -0.375, 'y': 1.125, 'tile_x': '00001', 'tile_y': '00000'}\n",
            "{'w': 0.75, 'x': 0.375, 'y': 1.125, 'tile_x': '00002', 'tile_y': '00000'}\n",
            "{'w': 0.75, 'x': 1.125, 'y': 1.125, 'tile_x': '00003', 'tile_y': '00000'}\n",
            "{'w': 0.75, 'x': -1.125, 'y': 0.375, 'tile_x': '00000', 'tile_y': '00001'}\n",
            "{'w': 0.75, 'x': -0.375, 'y': 0.375, 'tile_x': '00001', 'tile_y': '00001'}\n",
            "{'w': 0.75, 'x': 0.375, 'y': 0.375, 'tile_x': '00002', 'tile_y': '00001'}\n",
            "{'w': 0.75, 'x': 1.125, 'y': 0.375, 'tile_x': '00003', 'tile_y': '00001'}\n",
            "{'w': 0.75, 'x': -1.125, 'y': -0.375, 'tile_x': '00000', 'tile_y': '00002'}\n",
            "{'w': 0.75, 'x': -0.375, 'y': -0.375, 'tile_x': '00001', 'tile_y': '00002'}\n",
            "{'w': 0.75, 'x': 0.375, 'y': -0.375, 'tile_x': '00002', 'tile_y': '00002'}\n",
            "{'w': 0.75, 'x': 1.125, 'y': -0.375, 'tile_x': '00003', 'tile_y': '00002'}\n",
            "{'w': 0.75, 'x': -1.125, 'y': -1.125, 'tile_x': '00000', 'tile_y': '00003'}\n",
            "{'w': 0.75, 'x': -0.375, 'y': -1.125, 'tile_x': '00001', 'tile_y': '00003'}\n",
            "{'w': 0.75, 'x': 0.375, 'y': -1.125, 'tile_x': '00002', 'tile_y': '00003'}\n",
            "{'w': 0.75, 'x': 1.125, 'y': -1.125, 'tile_x': '00003', 'tile_y': '00003'}\n"
          ]
        }
      ],
      "source": [
        "tile_vars = make_tile_vars(4)\n",
        "for var in tile_vars:\n",
        "    print(var)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Describing montage as an HTCondor job\n",
        "\n",
        "Now we can write the `montage` job description. \n",
        "The problem is that the arguments and input files depend on how many tiles we have, which we don't know ahead-of-time.\n",
        "We'll take the brute-force approach of just writing a function that takes the tile `vars` we made in the previous section and using them to build the `montage` job description.\n",
        "\n",
        "Not that some of the work of building up the submit description is done in Python.\n",
        "This is a major advantage of communicating with HTCondor via Python: you can do the hard work in Python instead of in submit language!\n",
        "\n",
        "One area for possible improvement here is to remove the duplication of the format of the input file names, which is repeated here from when it was first used in the `goatbrot` submit object. When building a larger, more complicated workflow, it is important to reduce duplication of information to make it easier to modify the workflow in the future."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "def make_montage_description(tile_vars):\n",
        "    num_tiles_per_side = int(len(tile_vars) ** .5)\n",
        "    \n",
        "    input_files = [f'tile_{d[\"tile_x\"]}-{d[\"tile_y\"]}.ppm' for d in tile_vars]\n",
        "    \n",
        "    return htcondor.Submit(\n",
        "        executable = '/usr/bin/montage',\n",
        "        arguments = f'{\" \".join(input_files)} -mode Concatenate -tile {num_tiles_per_side}x{num_tiles_per_side} mandelbrot.png',\n",
        "        transfer_input_files = ', '.join(input_files),\n",
        "        log = 'mandelbrot.log',\n",
        "        output = 'montage.out',\n",
        "        error = 'montage.err',\n",
        "        request_cpus = '1',\n",
        "        request_memory = '128MB',\n",
        "        request_disk = '1GB',\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "executable = /usr/bin/montage\n",
            "arguments = tile_00000-00000.ppm tile_00001-00000.ppm tile_00000-00001.ppm tile_00001-00001.ppm -mode Concatenate -tile 2x2 mandelbrot.png\n",
            "transfer_input_files = tile_00000-00000.ppm, tile_00001-00000.ppm, tile_00000-00001.ppm, tile_00001-00001.ppm\n",
            "log = mandelbrot.log\n",
            "output = montage.out\n",
            "error = montage.err\n",
            "request_cpus = 1\n",
            "request_memory = 128MB\n",
            "request_disk = 1GB\n",
            "\n"
          ]
        }
      ],
      "source": [
        "montage_description = make_montage_description(make_tile_vars(2))\n",
        "\n",
        "print(montage_description)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Describing the DAG using `htcondor.dags`\n",
        "\n",
        "Now that we have the job descriptions, all we have to do is use `htcondor.dags` to tell DAGMan about the dependencies between them.\n",
        "`htcondor.dags` is a subpackage of the HTCondor Python bindings that lets you write DAG descriptions using a higher-level language than raw DAG description file syntax.\n",
        "Incidentally, it also lets you use Python to drive the creation process, increasing your flexibility.\n",
        "\n",
        "**Important Concept:** the code from `dag = dags.DAG()` onwards only defines the **topology** (or **structure**) of the DAG. \n",
        "The `tile` layer can be flexibly grown or shrunk by adjusting the `tile_vars` without changing the topology, and this can be clearly expressed in the code.\n",
        "The `tile_vars` are driving the creation of the DAG. Try changing `num_tiles_per_side` to some other value!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [],
      "source": [
        "from htcondor import dags\n",
        "\n",
        "num_tiles_per_side = 2\n",
        "\n",
        "# create the tile vars early, since we need to pass them to multiple places later\n",
        "tile_vars = make_tile_vars(num_tiles_per_side)\n",
        "\n",
        "dag = dags.DAG()\n",
        "\n",
        "# create the tile layer, passing in the submit description for a tile job and the tile vars\n",
        "tile_layer = dag.layer(\n",
        "    name = 'tile',\n",
        "    submit_description = tile_description,\n",
        "    vars = tile_vars,\n",
        ")\n",
        "\n",
        "# create the montage \"layer\" (it only has one job in it, so no need for vars)\n",
        "# note that the submit description is created \"on the fly\"!\n",
        "montage_layer = tile_layer.child_layer(\n",
        "    name = 'montage',\n",
        "    submit_description = make_montage_description(tile_vars),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can get a textual description of the DAG structure by calling the `describe` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Type     Name   # Nodes  # Children  Parents\n",
            "Layer    tile      4         1                       \n",
            "Layer  montage     1         0       tile[ManyToMany]\n"
          ]
        }
      ],
      "source": [
        "print(dag.describe())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Write the DAG to disk\n",
        "\n",
        "We still need to write the DAG to disk to get DAGMan to work with it.\n",
        "We also need to move some files around so that the jobs know where to find them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "DAG directory: /home/jovyan/tutorials/mandelbrot-dag\n",
            "DAG description file: /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag\n"
          ]
        }
      ],
      "source": [
        "from pathlib import Path\n",
        "import shutil\n",
        "\n",
        "dag_dir = (Path.cwd() / 'mandelbrot-dag').absolute()\n",
        "\n",
        "# blow away any old files\n",
        "shutil.rmtree(dag_dir, ignore_errors = True)\n",
        "\n",
        "# make the magic happen!\n",
        "dag_file = dags.write_dag(dag, dag_dir)\n",
        "\n",
        "# the submit files are expecting goatbrot to be next to them, so copy it into the dag directory\n",
        "shutil.copy2('goatbrot', dag_dir)\n",
        "\n",
        "print(f'DAG directory: {dag_dir}')\n",
        "print(f'DAG description file: {dag_file}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Submit the DAG via the Python bindings\n",
        "\n",
        "Now that we have written out the DAG description file, we can submit it for execution using the standard Python bindings submit mechanism.\n",
        "The `Submit` class has a static method which can read a DAG description and generate a corresponding `Submit` object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "universe = scheduler\n",
            "executable = /usr/bin/condor_dagman\n",
            "getenv = CONDOR_CONFIG,_CONDOR_*,PATH,PYTHONPATH,PERL*,PEGASUS_*,TZ,HOME,USER,LANG,LC_ALL\n",
            "output = /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag.lib.out\n",
            "error = /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag.lib.err\n",
            "log = /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag.dagman.log\n",
            "remove_kill_sig = SIGUSR1\n",
            "MY.OtherJobRemoveRequirements = \"DAGManJobId =?= $(cluster)\"\n",
            "on_exit_remove = (ExitSignal =?= 11 || (ExitCode =!= UNDEFINED && ExitCode >=0 && ExitCode <= 2))\n",
            "arguments = \"-p 0 -f -l . -Lockfile /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag.lock -AutoRescue 1 -DoRescueFrom 0 -Dag /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag -Suppress_notification -CsdVersion $CondorVersion:' '10.7.0' '2023-07-31' 'BuildID:' 'UW_Python_Wheel_Build' '$ -Dagman /usr/bin/condor_dagman\"\n",
            "environment = \"_CONDOR_DAGMAN_LOG=/home/jovyan/tutorials/mandelbrot-dag/dagfile.dag.dagman.out _CONDOR_MAX_DAGMAN_LOG=0\"\n",
            "\n"
          ]
        }
      ],
      "source": [
        "dag_submit = htcondor.Submit.from_dag(str(dag_file), {'force': 1})\n",
        "\n",
        "print(dag_submit)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can enter the DAG directory and submit the DAGMan job, which will execute the graph:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "DAGMan job cluster is 1\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "os.chdir(dag_dir)\n",
        "\n",
        "schedd = htcondor.Schedd()\n",
        "cluster_id = schedd.submit(dag_submit).cluster()\n",
        "    \n",
        "print(f\"DAGMan job cluster is {cluster_id}\")\n",
        "\n",
        "os.chdir('..')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's wait for the DAGMan job to complete by reading it's event log:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "DAG job log file is /home/jovyan/tutorials/mandelbrot-dag/dagfile.dag.dagman.log\n"
          ]
        }
      ],
      "source": [
        "dag_job_log = f\"{dag_file}.dagman.log\"\n",
        "print(f\"DAG job log file is {dag_job_log}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "000 (001.000.000) 2023-09-19 21:41:54 Job submitted from host: <172.17.0.2:9618?addrs=172.17.0.2-9618&alias=fa6c829ace67&noUDP&sock=schedd_82_2342>\n",
            "\n",
            "001 (001.000.000) 2023-09-19 21:41:57 Job executing on host: <172.17.0.2:9618?addrs=172.17.0.2-9618&alias=fa6c829ace67&noUDP&sock=schedd_82_2342>\n",
            "\n",
            "005 (001.000.000) 2023-09-19 21:42:23 Job terminated.\n",
            "\t(1) Normal termination (return value 0)\n",
            "\t\tUsr 0 00:00:00, Sys 0 00:00:00  -  Run Remote Usage\n",
            "\t\tUsr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage\n",
            "\t\tUsr 0 00:00:00, Sys 0 00:00:00  -  Total Remote Usage\n",
            "\t\tUsr 0 00:00:00, Sys 0 00:00:00  -  Total Local Usage\n",
            "\t0  -  Run Bytes Sent By Job\n",
            "\t0  -  Run Bytes Received By Job\n",
            "\t0  -  Total Bytes Sent By Job\n",
            "\t0  -  Total Bytes Received By Job\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# read events from the log, waiting forever for the next event\n",
        "dagman_job_events = htcondor.JobEventLog(str(dag_job_log)).events(None)\n",
        "\n",
        "# this event stream only contains the events for the DAGMan job itself, not the jobs it submits\n",
        "for event in dagman_job_events:\n",
        "    print(event)\n",
        "    \n",
        "    # stop waiting when we see the terminate event\n",
        "    if event.type is htcondor.JobEventType.JOB_TERMINATED and event.cluster == cluster_id:\n",
        "        break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's look at the final image!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 221124 characters]\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "Image(dag_dir / \"mandelbrot.png\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
