{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Github Copilot Usage\n", "\n", "### Introduction\n", "\n", "This notebook will go over usage of github Copilot for Open Source projects which\n", "are part of a eco-system of other projects within a standards body. \n", "\n", "An initial aim is to find out it's usefulness for MaterialX \n", "code development. The hope is that since it is using [OpenAI Codex](https://openai.com/blog/openai-codex) \n", "that it can leverage existing open source github repositories for MaterialX as well as\n", "related ASWF projects.\n", "\n", "We concentrate on source code since that appears to be the main focus of Copilot .\n", "When asked non programming questions Copilot will reply: `Sorry, but I can only assist with programming related questions.` Strangely the prompt `What is the latest version of MaterialX`` returns this answer?\n", "\n", "As a complimentary tool, the public beta of Copilot Chat was also installed.\n", "\n", "> See [Sectup section](#setup) at the end test setup for Visual Studio / Visual Studio Code.\n", "\n", "### Gating Factors\n", "\n", "We will look at the following factors\n", "\n", "- **Validity** and Correctness: Can code and documentation be generated that is correct and from a validated source?\n", "- **Accountability**: If code is generated can it be traced back to the original source? \n", "- **Efficiency**: Can it save time and effort in development?\n", "- **Standards**: Does it follow the standards of the project, standard body?\n", "- **Security**: Does it introduce security risks?\n", "- **Relevance**: Does it provide suggestions that are up-to-date and relevant to the project?\n", "- **Repeatability**: Can it be used to repeat the same task over and over again with consistent results?\n", "\n", "Note that this notebook was written with Copilot enabled to test it's usefulness in assisting with writing documentation.\n", "\n", "The general suggestion is to selectively enable Copilot as it can branch off and suggest text and code which can have no relevance to the task / topic at hand. This is due to lack of context provided in the instruction for Copilot. This includes\n", "currently logged issues where only the visible editor text is scanned for context.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Correctness and Repeatability\n", "\n", "In it's current state this is one of it's greatest failings. For the most part without providing\n", "any custom training data Copilot will:\n", "- Generate code or documentation\n", "- Complete code or documentation for APIs which do exist or uses code from an unrelated API.\n", "\n", "This is not just returning the incorrect version of a call but instead creating non-existent code calls.\n", "\n", "This not only extends to MaterialX specific code, but also to code from other projects and even\n", "established APIs such as Python and Javascript. It sometimes will generate calls for one API\n", "based on existing calls for another API which greatly lowers the trust factor.\n", "\n", "Below is a trivial example using comments as hints for code generation.\n", "\n", "`getDefinitions()` does not exist in the API but is suggested by Copilot. The same is true when asking to create\n", "a definition." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "ename": "AttributeError", "evalue": "'MaterialX.PyMaterialXCore.Document' object has no attribute 'getDefinitions'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn[5], line 7\u001b[0m\n\u001b[0;32m 4\u001b[0m doc \u001b[39m=\u001b[39m mx\u001b[39m.\u001b[39mcreateDocument()\n\u001b[0;32m 6\u001b[0m \u001b[39m# Find all definitions in the document.\u001b[39;00m\n\u001b[1;32m----> 7\u001b[0m def_list \u001b[39m=\u001b[39m doc\u001b[39m.\u001b[39;49mgetDefinitions()\n\u001b[0;32m 9\u001b[0m \u001b[39m# Create a new definition.\u001b[39;00m\n\u001b[0;32m 10\u001b[0m def_ \u001b[39m=\u001b[39m doc\u001b[39m.\u001b[39maddDefinition(\u001b[39m\"\u001b[39m\u001b[39mmy_def\u001b[39m\u001b[39m\"\u001b[39m)\n", "\u001b[1;31mAttributeError\u001b[0m: 'MaterialX.PyMaterialXCore.Document' object has no attribute 'getDefinitions'" ] } ], "source": [ "import MaterialX as mx\n", "\n", "# Create a document.\n", "doc = mx.createDocument()\n", "\n", "# Find all definitions in the document.\n", "def_list = doc.getDefinitions()\n", "\n", "# Create a new definition.\n", "def_ = doc.addDefinition(\"my_def\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One useful task would be for generation of test code for existing APIs. But this cannot even be considered\n", "since the code generated will mostly likely be invalid." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In terms of code completion, it fairs a bit better. At this time of writing the MaterialX Python\n", "libraries are not documented. After using Co-Pilot for a while it was able to suggest API calls and\n", "argument completions. An example is shown below of additional suggestions added for `Document` calls.`\n", "\n", "\n", "\n", "For the example below, it was able to suggest the correct call node graph creation, but the complete\n", "code including the import and doc creation must be visible on screen and within the same\n", "notebook cell for this to work. Sometimes it will just not come up with anything so there is\n", "a repeatability issue.\n", "\n", "When correct it can save quite a bit of time, but it can waste just as much time by inserting calls or arguments which do not exist in the API. The hope is that correctness and repeatability will improve over time." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "my_graph\n" ] } ], "source": [ "import MaterialX as mx\n", "doc = mx.createDocument()\n", "\n", "# Create a new node graph.\n", "graph = doc.addNodeGraph(\"my_graph\")\n", "# Print graph name\n", "print(graph.getName())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use the Co-Pilot action to \"fix\" or explain code is mostly of no use as it will return generic information about language or environment syntax and not anything specific to the code.\n", "\n", "For example querying to \"explain\" the document `doc.` returns generic information about the need to call a member function, and the \"fix\" action actually fairs no better, reusing the same non-existent API calls.\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using Chat is not recommended. It is unknown how the response is generated but it is unusable. As an example the prompt was on how to create a node definition. The example XML returned was:\n", "```xml\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "```\n", "For example the embedding of shader code inside is purely fictitious as is it's explanation:\n", "Using Chat is not recommended. It is unknown how the response is generated but it is pretty well always | unusable. As an example the prompt was on how to create a node definition. The example XML returned was:\n", "\n", "> The `ShaderStage` element specifies the shader stage where the node will be used, and the `ShaderCode` element contains the shader code that implements the node's functionality. In this case, the node simply adds its two input values together and outputs the result. Note that this is just a simple example, and MaterialX supports much more complex node definitions with multiple shader stages, conditional branching, and other advanced features.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accountability\n", "\n", "It is important to be able to provide accreditation for any code that is used. This is not possible with Co-Pilot. It is unknown how the code is generated and it is not possible to trace the code back to the original source. \n", "\n", "### Efficiency\n", "\n", "For efficiency and accuracy, it is much faster to build symbols for a project and use the existing non-Co-Pilot code completion. There are no assumptions as to what code API and version of code to use as this is part of the project itself.\n", "\n", "### Standards\n", "\n", "There does not seem to be any way to provide formatting rules for Co-Pilot (e.g. such as a Clang format file). When asked to format code it uses some unknown formatting convention. For this aspect it is not worth using Co-Pilot for such as task. Naming convention also seems not possible to perform. Other small consistency items such as adding in a default copyright appears to also not be possible.\n", "\n", "There is no way to add in requirements for version of compiler, platform, and other build dependencies including what test harness is being used. As such it is possible to have code which is compliant being suggested which will not compile or run. \n", "\n", "For examine move semantics are suggested for performance on a given function, but this may not be available.\n", "(For this example it could not parse the function signature correctly and suggested acting on\n", "non-existent arguments.)\n", "\n", "\n", "\n", "### Relevance\n", "\n", "Using fixed data is one of the major drawbacks. It appears to be possible to add in additional data but\n", "this has not been attempted for Copilot or any other LLM.\n", "\n", "### Repeatability\n", "\n", "This is a general issue that tuning can help to alleviate but just from Visual Studio (Code)\n", "integration it is not possible to tune these parameters.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Appendix: Setup\n", "\n", "As a setup step, the Copilot and Chat plugins need to be installed.\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "The chat interface for Copilot is available by signing up for the beta program at this time of writing. The chat feature can be used in a similar way to ChatGPT but within the context of a working environment. This has the added advantage of avoiding some context switching. (There are ChatGPT and other unofficial add-ons which are available\n", "but not considered here.) \n", "\n", "The following is a snapshot with and chat add-ons installed:\n", "\n", "\n", "
\n", "
\n", "
\n", "
\n", " \"Image\n", "
\n", "
\n", "
\n", "
\n", "\n", "If chat is setup, for the Visual Studio Code integration a new tab on the left is available where prompts may be entered.\n", "\n", "Unlike code completion, the responses need to be migrated to their intended location. Options (highlighted on the left) include:\n", "\n", "* Inserting into the editor at a location\n", "* Export to file\n", "* Running from terminal.\n", "\n", "The last option (Running from terminal) only makes sense for code which can run as a command. At the current time the response is not checked to see if it's runnable.\n", "\n", "Copilot can be turned on or off via the icon highlighted on the bottom right." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }