{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "05473d59-5b2e-4511-a2ce-428930f2af24",
   "metadata": {},
   "source": [
    "# Designing a Jupyter Extensions with IPyDrawio"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a849e0-cdfe-4c8f-8417-d232999cc1b4",
   "metadata": {},
   "source": [
    "Diagrams can effectively support the many phases of the software development cycle. What IPyDrawio, and the underlying general-purpose drawio UI, lack in features versus more specialized tools, when used in the right context to help a software team communicate between itself, and sometimes more importantly, its users."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6517292-f934-4365-a6e3-3c223f14bca9",
   "metadata": {},
   "source": [
    "```{note}\n",
    "This tutorial looks at how different kinds of diagram artifacts can be created that support the process of building _Jupyter Extensions_, using IPyDrawio itself as a subject. Each of the images below rendered directly in the browser for this documentation, as well as editable within IPyDrawio.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a5ef077-3d16-4e61-ad47-39b03eb60612",
   "metadata": {},
   "source": [
    "Extensions for [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html) and [Jupyter Server](https://jupyter-server.readthedocs.io/en/latest/developers/extensions.html?highlight=distributing#distributing-a-server-extension) often developed, and can be distributed, together as a single Python package. While a single author can build, test, document and ship new features _without_ any kinds of pictures, individuals and especially team- and community-driven processes can be enhanced by different visual artifacts that _live next to the code_."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "150c79c8-4349-4fde-bc3c-d7dfa1c74d1a",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "Using an editable, but widely-viewable format like `.dio.png` or `.dio.svg` means these assets can be used \"at-rest,\" though without advanced features like multiple pages and interactivity, and works well with the visual diff tools on e.g. GitHub.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c9e0cd8-e623-4c15-9ca9-43f2f35bca8b",
   "metadata": {},
   "source": [
    "## Gathering Ideas\n",
    "\n",
    "As you start thinking about your extension, who will use it, and how you might build it, IPyDrawio diagrams can capture structured problem-solving approaches effective at capturing, and then leveraging, visual concepts to capture goals and trade-offs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1e0fe74-ddc5-4892-bf73-827bba9eeb94",
   "metadata": {},
   "source": [
    "### Mind Maps\n",
    "[Mind Maps](https://en.wikipedia.org/wiki/Mind_map) are an effective way to capture \"stream of consciousness\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "605ed2a3-1a8b-416f-82b1-d1493c986cfe",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "Search for `mind` in the _Custom Diagram..._ panel. This task also lends itself to the `sketch` theme, which trades UI simplicity for hiding some of the more advanced features.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93667624-a6e6-4e58-bb97-3d76c312f353",
   "metadata": {},
   "source": [
    "[![a mind map of ipydrawio][mindmap]][mindmap]\n",
    "\n",
    "[mindmap]: ../../_static/tutorials/designing-jupyter-extensions/mindmap.dio.svg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec77aa2-80cf-4702-b222-5c40e021b4a1",
   "metadata": {},
   "source": [
    "### Seven Management and Planning Tools\n",
    "The [Seven Management and Planning Tools](https://en.wikipedia.org/wiki/Seven_management_and_planning_tools) are suitable for large and small projects.\n",
    "\n",
    "```{note}\n",
    "While there are no \"prebuilt\" templates for these diagrams, many of the \"basic\" and and \"tree\" shape palettes have ready-made shapes with pre-configured connectors that preserve intent, and work well with the auto-layout tools.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de02a784-aa8c-49a5-a1c2-e321c42912bd",
   "metadata": {},
   "source": [
    "[![an affinity diagram of ipydrawio][affinity]][affinity]\n",
    "\n",
    "[affinity]: ../../_static/tutorials/designing-jupyter-extensions/affinity.dio.svg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d8c47d4-84dd-4907-95ca-5d8c898c4c6b",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "Use the <kbd>Ctrl</kbd> (or <kbd>Cmd</kbd> on MacOS) with existing arrows to quickly make copies.  \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14fe2c2b-de91-4aee-92d1-179d9754468b",
   "metadata": {},
   "source": [
    "### Mermaid Diagrams\n",
    "\n",
    "The [mermaid.js](https://mermaid-js.github.io/mermaid) drawio plugin (enabled by default) provides a way to write pithy text descriptions, including:\n",
    "\n",
    "> _Flowchart, Sequence diagram, Class Diagram, State Diagram, Entity Relationship Diagram, User Journey, Gantt, Pie Chart, Requirement Diagram_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7db97c9-fe37-49f2-b08a-c24b9bdff6b7",
   "metadata": {},
   "source": [
    "```{note}\n",
    "From the drawio menu, select _Insert_ ▸ _Advanced_ ▸ _Mermaid_. Paste some mermaid there. The resulting object will _not_ be editable with shape-editing tools, but you _will_ be able to adjust the mermaid syntax later by clicking on it, which can be worth _much_ more over time. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eab9baf5-f7c5-4ae3-992c-b682943f9fbf",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "Here are some more interactive editing experience than the `textarea` provided by drawio.\n",
    "- [jupyterlab-markup](https://github.com/agoose77/jupyterlab-markup)\n",
    "- [mermaid live editor](https://mermaid-js.github.io/mermaid-live-editor)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e4be3cd-a8d6-41a7-913a-645ef4487a5f",
   "metadata": {},
   "source": [
    "[![mermaid diagrams][mermaid]][mermaid]\n",
    "\n",
    "[mermaid]: ../../_static/tutorials/designing-jupyter-extensions/mermaid.dio.svg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57bd8c7-6576-4938-a102-1eeaf33b2626",
   "metadata": {},
   "source": [
    "If these kinds of diagrams are helpful for your extension, consider more advanced tools like [Papyrus](https://www.eclipse.org/papyrus) that support more formal models and diagram standards."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f30c2f-f937-4c67-98a1-1cc4670f2709",
   "metadata": {},
   "source": [
    "## Visual Design\n",
    "\n",
    "While IPyDrawio isn't as full-featured as properietary tools, or even open source tools such as the excellent [Inkscape](https://inkscape.org), it can be effective, and in some ways, more approachable, than full design suites."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d22eebd-b5b7-4a8f-89a5-977b61852562",
   "metadata": {},
   "source": [
    "### Wireframing\n",
    "\n",
    "Low-fidelity wireframes, which often include filler text like _lorem ipsum_, placeholder images, etc. provide a quick way to capture the essential views of a novel UI without getting a team hung up on details."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69f3416e-884b-4292-84cf-59e54e40a592",
   "metadata": {},
   "source": [
    "[![a wireframe of the ipydrawio custom diagram panel][wireframe]][wireframe]\n",
    "\n",
    "\n",
    "[wireframe]: ../../_static/tutorials/designing-jupyter-extensions/wireframe.dio.svg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "202e9e97-1a90-4c9f-829f-c2fd55c96f13",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "Use primarily _basic shapes_: a box with a word in it says \"button\" just as well as something with gradients and carefully-kerned fonts.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da515eab-71a5-488f-b40a-c361ea26ae71",
   "metadata": {},
   "source": [
    "```{note}\n",
    "For single-page/stage views, the `sketch` theme can work well, while `min` provides a nice balance between unobtrusiveness and power.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9cede37-b50d-4c49-805f-e904db806715",
   "metadata": {},
   "source": [
    "### Comps\n",
    "\n",
    "High-fidelity mockups, or \"comps,\" eventually provide the language used between the designer role and the developer role... even if these roles are filled by the same person."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ab27c36-9864-4348-b649-d43970a553d5",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "Search for `jupyter` in the _Custom Diagram..._ panel. The _JupyterLab Mockups_ contains recreations of a number of common views, such as the _Launcher_ and _Notebook_, which can be recombined, remixed, or expanded to suggest new features. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ab07900-ff38-4502-839f-d2e31efc6129",
   "metadata": {},
   "source": [
    "```{note}\n",
    "One of the \"full\" themes (e.g. `kennedy`, `dark` or `atlas`) is recommended, as using multiple pages, cross-page links, and auto-layouts are more readily discoverable than with `min` or `sketch`.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed75f4f-d1ae-4d30-a082-1f7f5c3ce326",
   "metadata": {},
   "source": [
    "[![a comp of a future feature for IPyDrawio][comp]][comp]\n",
    "\n",
    "[comp]: ../../_static/tutorials/designing-jupyter-extensions/comp.dio.svg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98e88667-f9ef-4ae3-8746-348a20b86598",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "While not a _particularly_ accessible semantic model, native drawio documents and SVG _are_ well-behaved XML, and far less opaque than \"professional\" UML/SysML tools. Especially when using some of the richer shapes, such as those in _Trees_, some data will be preserved, and can be used, for example, to validate repository integrity.\n",
    "\n",
    "\n",
    "```{note}\n",
    "For example, if you are building an extension composed of many classes, you can use either simple string comparison, regular expressions, or XPath queries (provided by both [browser JS](https://developer.mozilla.org/en-US/docs/Web/XPath) and [python](https://docs.python.org/3/library/xml.etree.elementtree.html#elementtree-xpath)) to verify all classes are included in a key diagram.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1ef92c-69cd-4197-ba34-a25dd428f160",
   "metadata": {},
   "source": [
    "[![a deployment diagram of ipydrawio packages][deployment]][deployment]\n",
    "\n",
    "[deployment]: ../../_static/tutorials/designing-jupyter-extensions/deployment.dio.svg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9976fbb6-e3fb-43b4-a30d-a0ef1d182bbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lxml.etree as ET\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97be3409-eb2b-4766-a873-e956a8e139b3",
   "metadata": {},
   "source": [
    "Because the sources and targets of the dotted edges are not preserved on the `path` SVG elements, we use the fill colors to fill in the missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "168ffd51-051f-40b2-83f5-c1d3c08a3a8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "packages = {\n",
    "    \"#dae8fc\": \"ipydrawio\",\n",
    "    \"#f8cecc\": \"ipydrawio_export\",\n",
    "    \"#d5e8d4\": \"ipydrawio_mathjax\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45a41a7b-3f73-4e74-8c85-9cf2d7d8b012",
   "metadata": {},
   "outputs": [],
   "source": [
    "ships = {\n",
    "    py_pkg: {\n",
    "        js_pkg[\"dest\"]\n",
    "        for js_pkg\n",
    "        in __import__(py_pkg)._jupyter_labextension_paths()\n",
    "    }\n",
    "    for fill, py_pkg in packages.items()\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebbd6b25-bcf2-4047-8c72-adde387ec0ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_deployment(\n",
    "    ns=\"@deathbeds/\",\n",
    "    diagram=\"../../_static/tutorials/designing-jupyter-extensions/deployment.dio.svg\"\n",
    "):\n",
    "    deployment = Path(diagram)\n",
    "    et = ET.fromstring(deployment.read_text(encoding=\"utf-8\"))\n",
    "    # get everything with a data-label, which gets preserved\n",
    "    should_ship_in = {\n",
    "        f\"\"\"{ns}{e.attrib[\"data-label\"]}\"\"\":\n",
    "            packages.get(e.xpath(\"./*[1]/@fill\")[0])\n",
    "        for e in et.xpath(\"//*[@data-label]\")\n",
    "    }\n",
    "    for py_package, ships_exts in ships.items():\n",
    "        print(py_package)\n",
    "        print(\"... observed\", sorted(ships_exts))\n",
    "        should_ship_exts = {\n",
    "            js_name for js_name, py_name in should_ship_in.items()\n",
    "            if py_name == py_package\n",
    "        }\n",
    "        print(\"... expected\", sorted(should_ship_exts))\n",
    "        mismatches = ships_exts ^ should_ship_exts\n",
    "        assert not mismatches, f\"mismatches: {mismatches}\"\n",
    "        print(f\"... {py_package} is 🚢-shape!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "458ea4b9-1365-48d1-9bf9-7d7aa8d04645",
   "metadata": {},
   "source": [
    "```{note}\n",
    "For documentation purposes, we just run this directly here, but this could be easily run as part of a regular unit test suite, and an \"offending\" diagram included in a rich test output such as [pytest-html](https://github.com/pytest-dev/pytest-html).\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42de7083-e97a-4b52-a647-9dd718d20b08",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_deployment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd02cf8-959a-4698-9749-34e3422ef9fc",
   "metadata": {},
   "source": [
    "Now, if a new package is added to either the code, or the diagram, and the other side is _not_ updated, this test will fail, helping to keep the documentation up-to-date."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0178416c-7f47-4ae2-81bf-850e14c8d22a",
   "metadata": {},
   "source": [
    "## Documentation\n",
    "\n",
    "Once these diagrams exist, they can be used directly within documentation sites. While alternatives exist for working directly with `.drawio` files, such as [sphinxcontrib-drawio](https://pypi.org/project/sphinxcontrib-drawio), again keeping the files in standards-compliant, but editable, formats such as `.dio.png` and `.dio.svg` are more likely to stay editable and viewable over time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0165bbc2-700b-47c8-b0b6-fbc9789f2cb5",
   "metadata": {},
   "source": [
    "```{hint}\n",
    "[myst-nb](https://github.com/executablebooks/MyST-NB) is highly recommended (and used on this documentation site) for embedding various kinds of images without a lot of the hassle of some of the finer points of `.rst` editing.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63e1049d-cfa4-49d2-9dea-db554172899a",
   "metadata": {},
   "source": [
    "Another approach is generating diagrams, with tools such as:\n",
    "\n",
    "- [graphviz2drawio](https://github.com/hbmartin/graphviz2drawio) \n",
    "- [n2g](https://github.com/dmulyalin/N2G)\n",
    "- [drawio-network-plot](https://github.com/amroashram/drawio_network_plot)\n",
    "- [python-drawio](https://github.com/chmduquesne/python-drawio)\n",
    "\n",
    "These offer an intermediate representation, as code or some other grammar, which can be more effective at keeping documentation up-to-date than hand editing diagrams."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}