Add files using upload-large-folder tool

stanza/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,29 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Environment (please complete the following information):**
+ - OS: [e.g. Windows, Ubuntu, CentOS, MacOS]
+ - Python version: [e.g. Python 3.6.8 from Anaconda]
+ - Stanza version: [e.g., 1.0.0]
+
+**Additional context**
+Add any other context about the problem here.

stanza/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

stanza/.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,18 @@
+---
+name: Question
+about: 'Question about general usage. '
+title: "[QUESTION]"
+labels: question
+assignees: ''
+
+---
+
+Before you start, make sure to check out:
+* Our documentation: https://stanfordnlp.github.io/stanza/
+* Our FAQ: https://stanfordnlp.github.io/stanza/faq.html
+* Github issues (especially closed ones)
+Your question might have an answer in these places!
+
+If you still couldn't find the answer to your question, feel free to delete this text and write down your question. The more information you provide with your question, the faster we will be able to help you!
+
+If you have a question about an issue you're facing when using Stanza, please try to provide a detailed step-by-step guide to reproduce the issue you're facing. Try to at least provide a minimal code sample to reproduce the problem you are facing, instead of just describing it. That would greatly help us in locating the issue faster and help you resolve it!

stanza/.github/pull_request_template.md

@@ -0,0 +1,16 @@
+**BEFORE YOU START**: please make sure your pull request is against the `dev` branch. 
+We cannot accept pull requests against the `main` branch. 
+See our [contributing guide](https://github.com/stanfordnlp/stanza/blob/main/CONTRIBUTING.md) for details.
+
+## Description
+A brief and concise description of what your pull request is trying to accomplish.
+
+## Fixes Issues
+A list of issues/bugs with # references. (e.g., #123)
+
+## Unit test coverage
+Are there unit tests in place to make sure your code is functioning correctly?
+(see [here](https://github.com/stanfordnlp/stanza/blob/master/tests/test_tagger.py) for a simple example)
+
+## Known breaking changes/behaviors
+Does this break anything in Stanza's existing user interface? If so, what is it and how is it addressed?

stanza/.github/stale.yml

@@ -0,0 +1,21 @@
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 60
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 7
+# Issues with these labels will never be considered stale
+exemptLabels:
+  - pinned
+  - security
+  - fixed on dev
+  - bug
+  - enhancement
+# Label to use when marking an issue as stale
+staleLabel: stale
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you
+  for your contributions.
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: >
+  This issue has been automatically closed due to inactivity.

stanza/.github/workflows/stanza-tests.yaml

@@ -0,0 +1,43 @@
+name: Run Stanza Tests
+on: [push]
+jobs:
+  Run-Stanza-Tests:
+    runs-on: self-hosted
+    steps:
+      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
+      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
+      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
+      - name: Check out repository code
+        uses: actions/checkout@v2
+      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
+      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
+      - name: Run Stanza Tests
+        run: |
+          # set up environment
+          echo "Setting up environment..."
+          bash
+          #. $CONDA_PREFIX/etc/profile.d/conda.sh
+          . /home/stanzabuild/miniconda3/etc/profile.d/conda.sh
+          conda activate stanza
+          export STANZA_TEST_HOME=/scr/stanza_test
+          export CORENLP_HOME=$STANZA_TEST_HOME/corenlp_dir
+          export CLASSPATH=$CORENLP_HOME/*:
+          echo CORENLP_HOME=$CORENLP_HOME
+          echo CLASSPATH=$CLASSPATH
+          # install from stanza repo being evaluated
+          echo PWD: $pwd
+          echo PATH: $PATH
+          pip3 install -e .
+          pip3 install -e .[test]
+          pip3 install -e .[transformers]
+          pip3 install -e .[tokenizers]
+          # set up for tests
+          echo "Running stanza test set up..."
+          rm -rf $STANZA_TEST_HOME
+          python3 stanza/tests/setup.py
+          # run tests
+          echo "Running tests..."
+          export CUDA_VISIBLE_DEVICES=2
+          pytest stanza/tests
+          
+      - run: echo "🍏 This job's status is ${{ job.status }}."

stanza/demo/CONLL_Dependency_Visualizer_Example.ipynb

@@ -0,0 +1,70 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c0fd86c8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.conll_deprel_visualization import conll_to_visual\n",
+    "\n",
+    "# load necessary conllu files - expected to be in the demo directory along with the notebook\n",
+    "en_file = \"en_test.conllu.txt\"\n",
+    "\n",
+    "# testing left to right languages\n",
+    "conll_to_visual(en_file, \"en\", sent_count=2)\n",
+    "conll_to_visual(en_file, \"en\", sent_count=10)\n",
+    "#conll_to_visual(en_file, \"en\", display_all=True)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fc4b3f9b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.conll_deprel_visualization import conll_to_visual\n",
+    "\n",
+    "jp_file = \"japanese_test.conllu.txt\"\n",
+    "conll_to_visual(jp_file, \"ja\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6852b8e8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.conll_deprel_visualization import conll_to_visual\n",
+    "\n",
+    "# testing right to left languages\n",
+    "ar_file = \"arabic_test.conllu.txt\"\n",
+    "conll_to_visual(ar_file, \"ar\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

stanza/demo/Dependency_Visualization_Testing.ipynb

@@ -0,0 +1,78 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "64b2a9e0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.dependency_visualization import visualize_strings\n",
+    "\n",
+    "ar_strings = ['برلين ترفض حصول شركة اميركية على رخصة تصنيع دبابة \"ليوبارد\" الالمانية', \"هل بإمكاني مساعدتك؟\", \n",
+    "              \"أراك في مابعد\", \"لحظة من فضلك\"]\n",
+    "# Testing with right to left language\n",
+    "visualize_strings(ar_strings, \"ar\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "35ef521b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.dependency_visualization import visualize_strings\n",
+    "\n",
+    "en_strings = [\"This is a sentence.\", \n",
+    "              \"He is wearing a red shirt\",\n",
+    "              \"Barack Obama was born in Hawaii. He was elected President of the United States in 2008.\"]\n",
+    "# Testing with left to right languages\n",
+    "visualize_strings(en_strings, \"en\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f3cf10ba",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.dependency_visualization import visualize_strings\n",
+    "\n",
+    "zh_strings = [\"中国是一个很有意思的国家。\"]\n",
+    "# Testing with right to left language\n",
+    "visualize_strings(zh_strings, \"zh\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d2b9b574",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

stanza/demo/NER_Visualization.ipynb

@@ -0,0 +1,88 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "abf300bb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.ner_visualization import visualize_strings\n",
+    "\n",
+    "en_strings = ['''Samuel Jackson, a Christian man from Utah, went to the JFK Airport for a flight to New York.\n",
+    "                 He was thinking of attending the US Open, his favorite tennis tournament besides Wimbledon.\n",
+    "                 That would be a dream trip, certainly not possible since it is $5000 attendance and 5000 miles away.\n",
+    "                 On the way there, he watched the Super Bowl for 2 hours and read War and Piece by Tolstoy for 1 hour.\n",
+    "                 In New York, he crossed the Brooklyn Bridge and listened to the 5th symphony of Beethoven as well as\n",
+    "                 \"All I want for Christmas is You\" by Mariah Carey.''', \n",
+    "              \"Barack Obama was born in Hawaii. He was elected President of the United States in 2008\"]\n",
+    "    \n",
+    "visualize_strings(en_strings, \"en\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5670921a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.ner_visualization import visualize_strings\n",
+    "\n",
+    "zh_strings = ['''来自犹他州的基督徒塞缪尔杰克逊前往肯尼迪机场搭乘航班飞往纽约。\n",
+    "                 他正在考虑参加美国公开赛，这是除了温布尔登之外他最喜欢的网球赛事。\n",
+    "                 那将是一次梦想之旅，当然不可能，因为它的出勤费为 5000 美元，距离 5000 英里。\n",
+    "                 在去的路上，他看了 2 个小时的超级碗比赛，看了 1 个小时的托尔斯泰的《战争与碎片》。\n",
+    "                 在纽约，他穿过布鲁克林大桥，聆听了贝多芬的第五交响曲以及 玛丽亚凯莉的“圣诞节我想要的就是你”。''',\n",
+    "              \"我觉得罗家费德勒住在加州, 在美国里面。\"]\n",
+    "visualize_strings(zh_strings, \"zh\", colors={\"PERSON\": \"yellow\", \"DATE\": \"red\", \"GPE\": \"blue\"})\n",
+    "visualize_strings(zh_strings, \"zh\", select=['PERSON', 'DATE'])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b8d96072",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stanza.utils.visualization.ner_visualization import visualize_strings\n",
+    "\n",
+    "ar_strings = [\".أعيش في سان فرانسيسكو ، كاليفورنيا. اسمي أليكس وأنا ألتحق بجامعة ستانفورد. أنا أدرس علوم الكمبيوتر وأستاذي هو كريس مانينغ\"\n",
+    "             , \"اسمي أليكس ، أنا من الولايات المتحدة.\",  \n",
+    "               '''صامويل جاكسون ، رجل مسيحي من ولاية يوتا ، ذهب إلى مطار جون كنيدي في رحلة إلى نيويورك. كان يفكر في حضور بطولة الولايات المتحدة المفتوحة للتنس ، بطولة التنس المفضلة لديه إلى جانب بطولة ويمبلدون. ستكون هذه رحلة الأحلام ، وبالتأكيد ليست ممكنة لأنها تبلغ 5000 دولار للحضور و 5000 ميل. في الطريق إلى هناك ، شاهد Super Bowl لمدة ساعتين وقرأ War and Piece by Tolstoy لمدة ساعة واحدة. في نيويورك ، عبر جسر بروكلين واستمع إلى السيمفونية الخامسة لبيتهوفن وكذلك \"كل ما أريده في عيد الميلاد هو أنت\" لماريا كاري.''']\n",
+    "\n",
+    "visualize_strings(ar_strings, \"ar\", colors={\"PER\": \"pink\", \"LOC\": \"linear-gradient(90deg, #aa9cfc, #fc9ce7)\", \"ORG\": \"yellow\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "22489b27",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

stanza/demo/Stanza_Beginners_Guide.ipynb

@@ -0,0 +1,358 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "name": "Stanza-Beginners-Guide.ipynb",
+      "provenance": [],
+      "collapsed_sections": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "56LiYCkPM7V_",
+        "colab_type": "text"
+      },
+      "source": [
+        "# Welcome to Stanza!\n",
+        "\n",
+        "![Latest Version](https://img.shields.io/pypi/v/stanza.svg?colorB=bc4545)\n",
+        "![Python Versions](https://img.shields.io/pypi/pyversions/stanza.svg?colorB=bc4545)\n",
+        "\n",
+        "Stanza is a Python NLP toolkit that supports 60+ human languages. It is built with highly accurate neural network components that enable efficient training and evaluation with your own annotated data, and offers pretrained models on 100 treebanks. Additionally, Stanza provides a stable, officially maintained Python interface to Java Stanford CoreNLP Toolkit.\n",
+        "\n",
+        "In this tutorial, we will demonstrate how to set up Stanza and annotate text with its native neural network NLP models. For the use of the Python CoreNLP interface, please see other tutorials."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yQff4Di5Nnq0",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 1. Installing Stanza\n",
+        "\n",
+        "Note that Stanza only supports Python 3.6 and above. Installing and importing Stanza are as simple as running the following commands:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "owSj1UtdEvSU",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Install; note that the prefix \"!\" is not needed if you are running in a terminal\n",
+        "!pip install stanza\n",
+        "\n",
+        "# Import the package\n",
+        "import stanza"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4ixllwEKeCJg",
+        "colab_type": "text"
+      },
+      "source": [
+        "### More Information\n",
+        "\n",
+        "For common troubleshooting, please visit our [troubleshooting page](https://stanfordnlp.github.io/stanfordnlp/installation_usage.html#troubleshooting)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aeyPs5ARO79d",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 2. Downloading Models\n",
+        "\n",
+        "You can download models with the `stanza.download` command. The language can be specified with either a full language name (e.g., \"english\"), or a short code (e.g., \"en\"). \n",
+        "\n",
+        "By default, models will be saved to your `~/stanza_resources` directory. If you want to specify your own path to save the model files, you can pass a `dir=your_path` argument.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "HDwRm-KXGcYo",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Download an English model into the default directory\n",
+        "print(\"Downloading English model...\")\n",
+        "stanza.download('en')\n",
+        "\n",
+        "# Similarly, download a (simplified) Chinese model\n",
+        "# Note that you can use verbose=False to turn off all printed messages\n",
+        "print(\"Downloading Chinese model...\")\n",
+        "stanza.download('zh', verbose=False)"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7HCfQ0SfdmsU",
+        "colab_type": "text"
+      },
+      "source": [
+        "### More Information\n",
+        "\n",
+        "Pretrained models are provided for 60+ different languages. For all languages, available models and the corresponding short language codes, please check out the [models page](https://stanfordnlp.github.io/stanza/models.html).\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b3-WZJrzWD2o",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 3. Processing Text\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XrnKl2m3fq2f",
+        "colab_type": "text"
+      },
+      "source": [
+        "### Constructing Pipeline\n",
+        "\n",
+        "To process a piece of text, you'll need to first construct a `Pipeline` with different `Processor` units. The pipeline is language-specific, so again you'll need to first specify the language (see examples).\n",
+        "\n",
+        "- By default, the pipeline will include all processors, including tokenization, multi-word token expansion, part-of-speech tagging, lemmatization, dependency parsing and named entity recognition (for supported languages). However, you can always specify what processors you want to include with the `processors` argument.\n",
+        "\n",
+        "- Stanza's pipeline is CUDA-aware, meaning that a CUDA-device will be used whenever it is available, otherwise CPUs will be used when a GPU is not found. You can force the pipeline to use CPU regardless by setting `use_gpu=False`.\n",
+        "\n",
+        "- Again, you can suppress all printed messages by setting `verbose=False`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "HbiTSBDPG53o",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Build an English pipeline, with all processors by default\n",
+        "print(\"Building an English pipeline...\")\n",
+        "en_nlp = stanza.Pipeline('en')\n",
+        "\n",
+        "# Build a Chinese pipeline, with customized processor list and no logging, and force it to use CPU\n",
+        "print(\"Building a Chinese pipeline...\")\n",
+        "zh_nlp = stanza.Pipeline('zh', processors='tokenize,lemma,pos,depparse', verbose=False, use_gpu=False)"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Go123Bx8e1wt",
+        "colab_type": "text"
+      },
+      "source": [
+        "### Annotating Text\n",
+        "\n",
+        "After a pipeline is successfully constructed, you can get annotations of a piece of text simply by passing the string into the pipeline object. The pipeline will return a `Document` object, which can be used to access detailed annotations from. For example:\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "k_p0h1UTHDMm",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Processing English text\n",
+        "en_doc = en_nlp(\"Barack Obama was born in Hawaii.  He was elected president in 2008.\")\n",
+        "print(type(en_doc))\n",
+        "\n",
+        "# Processing Chinese text\n",
+        "zh_doc = zh_nlp(\"达沃斯世界经济论坛是每年全球政商界领袖聚在一起的年度盛事。\")\n",
+        "print(type(zh_doc))"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "DavwCP9egzNZ",
+        "colab_type": "text"
+      },
+      "source": [
+        "### More Information\n",
+        "\n",
+        "For more information on how to construct a pipeline and information on different processors, please visit our [pipeline page](https://stanfordnlp.github.io/stanfordnlp/pipeline.html)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "O_PYLEGziQWR",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 4. Accessing Annotations\n",
+        "\n",
+        "Annotations can be accessed from the returned `Document` object. \n",
+        "\n",
+        "A `Document` contains a list of `Sentence`s, and a `Sentence` contains a list of `Token`s and `Word`s. For the most part `Token`s and `Word`s overlap, but some tokens can be divided into mutiple words, for instance the French token `aux` is divided into the words `à` and `les`, while in English a word and a token are equivalent. Note that dependency parses are derived over `Word`s.\n",
+        "\n",
+        "Additionally, a `Span` object is used to represent annotations that are part of a document, such as named entity mentions.\n",
+        "\n",
+        "\n",
+        "The following example iterate over all English sentences and words, and print the word information one by one:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "B5691SpFHFZ6",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "for i, sent in enumerate(en_doc.sentences):\n",
+        "    print(\"[Sentence {}]\".format(i+1))\n",
+        "    for word in sent.words:\n",
+        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{:d}\\t{:12s}\".format(\\\n",
+        "              word.text, word.lemma, word.pos, word.head, word.deprel))\n",
+        "    print(\"\")"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-AUkCkNIrusq",
+        "colab_type": "text"
+      },
+      "source": [
+        "The following example iterate over all extracted named entity mentions and print out their character spans and types."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "5Uu0-WmvsnlK",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "print(\"Mention text\\tType\\tStart-End\")\n",
+        "for ent in en_doc.ents:\n",
+        "    print(\"{}\\t{}\\t{}-{}\".format(ent.text, ent.type, ent.start_char, ent.end_char))"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Ql1SZlZOnMLo",
+        "colab_type": "text"
+      },
+      "source": [
+        "And similarly for the Chinese text:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "XsVcEO9tHKPG",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "for i, sent in enumerate(zh_doc.sentences):\n",
+        "    print(\"[Sentence {}]\".format(i+1))\n",
+        "    for word in sent.words:\n",
+        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{:d}\\t{:12s}\".format(\\\n",
+        "              word.text, word.lemma, word.pos, word.head, word.deprel))\n",
+        "    print(\"\")"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dUhWAs8pnnHT",
+        "colab_type": "text"
+      },
+      "source": [
+        "Alternatively, you can directly print a `Word` object to view all its annotations as a Python dict:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "6_UafNb7HHIg",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "word = en_doc.sentences[0].words[0]\n",
+        "print(word)"
+      ],
+      "execution_count": 0,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TAQlOsuRoq2V",
+        "colab_type": "text"
+      },
+      "source": [
+        "### More Information\n",
+        "\n",
+        "For all information on different data objects, please visit our [data objects page](https://stanfordnlp.github.io/stanza/data_objects.html)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hiiWHxYPpmhd",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 5. Resources\n",
+        "\n",
+        "Apart from this interactive tutorial, we also provide tutorials on our website that cover a variety of use cases such as how to use different model \"packages\" for a language, how to use spaCy as a tokenizer, how to process pretokenized text without running the tokenizer, etc. For these tutorials please visit [our Tutorials page](https://stanfordnlp.github.io/stanza/tutorials.html).\n",
+        "\n",
+        "Other resources that you may find helpful include:\n",
+        "\n",
+        "- [Stanza Homepage](https://stanfordnlp.github.io/stanza/index.html)\n",
+        "- [FAQs](https://stanfordnlp.github.io/stanza/faq.html)\n",
+        "- [GitHub Repo](https://github.com/stanfordnlp/stanza)\n",
+        "- [Reporting Issues](https://github.com/stanfordnlp/stanza/issues)\n",
+        "- [Stanza System Description Paper](http://arxiv.org/abs/2003.07082)\n"
+      ]
+    }
+  ]
+}

stanza/demo/arabic_test.conllu.txt

@@ -0,0 +1,127 @@
+# newdoc id = assabah.20041005.0017
+# newpar id = assabah.20041005.0017:p1
+# sent_id = assabah.20041005.0017:p1u1
+# text = سوريا: تعديل وزاري واسع يشمل 8 حقائب
+# orig_file_sentence ASB_ARB_20041005.0017#1
+1	سوريا	سُورِيَا	X	X---------	Foreign=Yes	0	root	0:root	SpaceAfter=No|Vform=سُورِيَا|Gloss=Syria|Root=sUr|Translit=sūriyā|LTranslit=sūriyā
+2	:	:	PUNCT	G---------	_	1	punct	1:punct	Vform=:|Translit=:
+3	تعديل	تَعدِيل	NOUN	N------S1I	Case=Nom|Definite=Ind|Number=Sing	6	nsubj	6:nsubj	Vform=تَعدِيلٌ|Gloss=adjustment,change,modification,amendment|Root=`_d_l|Translit=taʿdīlun|LTranslit=taʿdīl
+4	وزاري	وِزَارِيّ	ADJ	A-----MS1I	Case=Nom|Definite=Ind|Gender=Masc|Number=Sing	3	amod	3:amod	Vform=وِزَارِيٌّ|Gloss=ministry,ministerial|Root=w_z_r|Translit=wizārīyun|LTranslit=wizārīy
+5	واسع	وَاسِع	ADJ	A-----MS1I	Case=Nom|Definite=Ind|Gender=Masc|Number=Sing	3	amod	3:amod	Vform=وَاسِعٌ|Gloss=wide,extensive,broad|Root=w_s_`|Translit=wāsiʿun|LTranslit=wāsiʿ
+6	يشمل	شَمِل	VERB	VIIA-3MS--	Aspect=Imp|Gender=Masc|Mood=Ind|Number=Sing|Person=3|VerbForm=Fin|Voice=Act	1	parataxis	1:parataxis	Vform=يَشمَلُ|Gloss=comprise,include,contain|Root=^s_m_l|Translit=yašmalu|LTranslit=šamil
+7	8	8	NUM	Q---------	NumForm=Digit	6	obj	6:obj	Vform=٨|Translit=8
+8	حقائب	حَقِيبَة	NOUN	N------P2I	Case=Gen|Definite=Ind|Number=Plur	7	nmod	7:nmod:gen	Vform=حَقَائِبَ|Gloss=briefcase,suitcase,portfolio,luggage|Root=.h_q_b|Translit=ḥaqāʾiba|LTranslit=ḥaqībat
+
+# newpar id = assabah.20041005.0017:p2
+# sent_id = assabah.20041005.0017:p2u1
+# text = دمشق (وكالات الانباء) - اجرى الرئيس السوري بشار الاسد تعديلا حكومياً واسعا تم بموجبه إقالة وزيري الداخلية والاعلام عن منصبيها في حين ظل محمد ناجي العطري رئيساً للحكومة.
+# orig_file_sentence ASB_ARB_20041005.0017#2
+1	دمشق	دمشق	X	U---------	_	0	root	0:root	Vform=دمشق|Root=OOV|Translit=dmšq
+2	(	(	PUNCT	G---------	_	3	punct	3:punct	SpaceAfter=No|Vform=(|Translit=(
+3	وكالات	وِكَالَة	NOUN	N------P1R	Case=Nom|Definite=Cons|Number=Plur	1	dep	1:dep	Vform=وِكَالَاتُ|Gloss=agency|Root=w_k_l|Translit=wikālātu|LTranslit=wikālat
+4	الانباء	نَبَأ	NOUN	N------P2D	Case=Gen|Definite=Def|Number=Plur	3	nmod	3:nmod:gen	SpaceAfter=No|Vform=اَلأَنبَاءِ|Gloss=news_item,report|Root=n_b_'|Translit=al-ʾanbāʾi|LTranslit=nabaʾ
+5	)	)	PUNCT	G---------	_	3	punct	3:punct	Vform=)|Translit=)
+6	-	-	PUNCT	G---------	_	1	punct	1:punct	Vform=-|Translit=-
+7	اجرى	أَجرَى	VERB	VP-A-3MS--	Aspect=Perf|Gender=Masc|Number=Sing|Person=3|Voice=Act	1	advcl	1:advcl:فِي_حِينَ	Vform=أَجرَى|Gloss=conduct,carry_out,perform|Root=^g_r_y|Translit=ʾaǧrā|LTranslit=ʾaǧrā
+8	الرئيس	رَئِيس	NOUN	N------S1D	Case=Nom|Definite=Def|Number=Sing	7	nsubj	7:nsubj	Vform=اَلرَّئِيسُ|Gloss=president,head,chairman|Root=r_'_s|Translit=ar-raʾīsu|LTranslit=raʾīs
+9	السوري	سُورِيّ	ADJ	A-----MS1D	Case=Nom|Definite=Def|Gender=Masc|Number=Sing	8	amod	8:amod	Vform=اَلسُّورِيُّ|Gloss=Syrian|Root=sUr|Translit=as-sūrīyu|LTranslit=sūrīy
+10	بشار	بشار	X	U---------	_	11	nmod	11:nmod	Vform=بشار|Root=OOV|Translit=bšār
+11	الاسد	الاسد	X	U---------	_	8	nmod	8:nmod	Vform=الاسد|Root=OOV|Translit=ālāsd
+12	تعديلا	تَعدِيل	NOUN	N------S4I	Case=Acc|Definite=Ind|Number=Sing	7	obj	7:obj	Vform=تَعدِيلًا|Gloss=adjustment,change,modification,amendment|Root=`_d_l|Translit=taʿdīlan|LTranslit=taʿdīl
+13	حكومياً	حُكُومِيّ	ADJ	A-----MS4I	Case=Acc|Definite=Ind|Gender=Masc|Number=Sing	12	amod	12:amod	Vform=حُكُومِيًّا|Gloss=governmental,state,official|Root=.h_k_m|Translit=ḥukūmīyan|LTranslit=ḥukūmīy
+14	واسعا	وَاسِع	ADJ	A-----MS4I	Case=Acc|Definite=Ind|Gender=Masc|Number=Sing	12	amod	12:amod	Vform=وَاسِعًا|Gloss=wide,extensive,broad|Root=w_s_`|Translit=wāsiʿan|LTranslit=wāsiʿ
+15	تم	تَمّ	VERB	VP-A-3MS--	Aspect=Perf|Gender=Masc|Number=Sing|Person=3|Voice=Act	12	acl	12:acl	Vform=تَمَّ|Gloss=conclude,take_place|Root=t_m_m|Translit=tamma|LTranslit=tamm
+16-18	بموجبه	_	_	_	_	_	_	_	_
+16	ب	بِ	ADP	P---------	AdpType=Prep	18	case	18:case	Vform=بِ|Gloss=by,with|Root=bi|Translit=bi|LTranslit=bi
+17	موجب	مُوجِب	NOUN	N------S2R	Case=Gen|Definite=Cons|Number=Sing	16	fixed	16:fixed	Vform=مُوجِبِ|Gloss=reason,motive|Root=w_^g_b|Translit=mūǧibi|LTranslit=mūǧib
+18	ه	هُوَ	PRON	SP---3MS2-	Case=Gen|Gender=Masc|Number=Sing|Person=3|PronType=Prs	15	nmod	15:nmod:بِ_مُوجِب:gen	Vform=هِ|Gloss=he,she,it|Translit=hi|LTranslit=huwa
+19	إقالة	إِقَالَة	NOUN	N------S1R	Case=Nom|Definite=Cons|Number=Sing	15	nsubj	15:nsubj	Vform=إِقَالَةُ|Gloss=dismissal,discharge|Root=q_y_l|Translit=ʾiqālatu|LTranslit=ʾiqālat
+20	وزيري	وَزِير	NOUN	N------D2R	Case=Gen|Definite=Cons|Number=Dual	19	nmod	19:nmod:gen	Vform=وَزِيرَي|Gloss=minister|Root=w_z_r|Translit=wazīray|LTranslit=wazīr
+21	الداخلية	دَاخِلِيّ	ADJ	A-----FS2D	Case=Gen|Definite=Def|Gender=Fem|Number=Sing	20	amod	20:amod	Vform=اَلدَّاخِلِيَّةِ|Gloss=internal,domestic,interior,of_state|Root=d__h_l|Translit=ad-dāḫilīyati|LTranslit=dāḫilīy
+22-23	والاعلام	_	_	_	_	_	_	_	_
+22	و	وَ	CCONJ	C---------	_	23	cc	23:cc	Vform=وَ|Gloss=and|Root=wa|Translit=wa|LTranslit=wa
+23	الإعلام	إِعلَام	NOUN	N------S2D	Case=Gen|Definite=Def|Number=Sing	21	conj	20:amod|21:conj	Vform=اَلإِعلَامِ|Gloss=information,media|Root=`_l_m|Translit=al-ʾiʿlāmi|LTranslit=ʾiʿlām
+24	عن	عَن	ADP	P---------	AdpType=Prep	25	case	25:case	Vform=عَن|Gloss=about,from|Root=`an|Translit=ʿan|LTranslit=ʿan
+25-26	منصبيها	_	_	_	_	_	_	_	_
+25	منصبي	مَنصِب	NOUN	N------D2R	Case=Gen|Definite=Cons|Number=Dual	19	nmod	19:nmod:عَن:gen	Vform=مَنصِبَي|Gloss=post,position,office|Root=n_.s_b|Translit=manṣibay|LTranslit=manṣib
+26	ها	هُوَ	PRON	SP---3FS2-	Case=Gen|Gender=Fem|Number=Sing|Person=3|PronType=Prs	25	nmod	25:nmod:gen	Vform=هَا|Gloss=he,she,it|Translit=hā|LTranslit=huwa
+27	في	فِي	ADP	P---------	AdpType=Prep	7	mark	7:mark	Vform=فِي|Gloss=in|Root=fI|Translit=fī|LTranslit=fī
+28	حين	حِينَ	ADP	PI------2-	AdpType=Prep|Case=Gen	7	mark	7:mark	Vform=حِينِ|Gloss=when|Root=.h_y_n|Translit=ḥīni|LTranslit=ḥīna
+29	ظل	ظَلّ	VERB	VP-A-3MS--	Aspect=Perf|Gender=Masc|Number=Sing|Person=3|Voice=Act	7	parataxis	7:parataxis	Vform=ظَلَّ|Gloss=remain,continue|Root=.z_l_l|Translit=ẓalla|LTranslit=ẓall
+30	محمد	محمد	X	U---------	_	32	nmod	32:nmod	Vform=محمد|Root=OOV|Translit=mḥmd
+31	ناجي	ناجي	X	U---------	_	32	nmod	32:nmod	Vform=ناجي|Root=OOV|Translit=nāǧy
+32	العطري	العطري	X	U---------	_	29	nsubj	29:nsubj	Vform=العطري|Root=OOV|Translit=ālʿṭry
+33	رئيساً	رَئِيس	NOUN	N------S4I	Case=Acc|Definite=Ind|Number=Sing	29	xcomp	29:xcomp	Vform=رَئِيسًا|Gloss=president,head,chairman|Root=r_'_s|Translit=raʾīsan|LTranslit=raʾīs
+34-35	للحكومة	_	_	_	_	_	_	_	SpaceAfter=No
+34	ل	لِ	ADP	P---------	AdpType=Prep	35	case	35:case	Vform=لِ|Gloss=for,to|Root=l|Translit=li|LTranslit=li
+35	الحكومة	حُكُومَة	NOUN	N------S2D	Case=Gen|Definite=Def|Number=Sing	33	nmod	33:nmod:لِ:gen	Vform=اَلحُكُومَةِ|Gloss=government,administration|Root=.h_k_m|Translit=al-ḥukūmati|LTranslit=ḥukūmat
+36	.	.	PUNCT	G---------	_	1	punct	1:punct	Vform=.|Translit=.
+
+# newpar id = assabah.20041005.0017:p3
+# sent_id = assabah.20041005.0017:p3u1
+# text = واضافت المصادر ان مهدي دخل الله رئيس تحرير صحيفة الحزب الحاكم والليبرالي التوجهات تسلم منصب وزير الاعلام خلفا لاحمد الحسن فيما تسلم اللواء غازي كنعان رئيس شعبة الامن السياسي منصب وزير الداخلية.
+# orig_file_sentence ASB_ARB_20041005.0017#3
+1-2	واضافت	_	_	_	_	_	_	_	_
+1	و	وَ	CCONJ	C---------	_	0	root	0:root	Vform=وَ|Gloss=and|Root=wa|Translit=wa|LTranslit=wa
+2	أضافت	أَضَاف	VERB	VP-A-3FS--	Aspect=Perf|Gender=Fem|Number=Sing|Person=3|Voice=Act	1	parataxis	1:parataxis	Vform=أَضَافَت|Gloss=add,attach,receive_as_guest|Root=.d_y_f|Translit=ʾaḍāfat|LTranslit=ʾaḍāf
+3	المصادر	مَصدَر	NOUN	N------P1D	Case=Nom|Definite=Def|Number=Plur	2	nsubj	2:nsubj	Vform=اَلمَصَادِرُ|Gloss=source|Root=.s_d_r|Translit=al-maṣādiru|LTranslit=maṣdar
+4	ان	أَنَّ	SCONJ	C---------	_	16	mark	16:mark	Vform=أَنَّ|Gloss=that|Root='_n|Translit=ʾanna|LTranslit=ʾanna
+5	مهدي	مهدي	X	U---------	_	6	nmod	6:nmod	Vform=مهدي|Root=OOV|Translit=mhdy
+6	دخل	دخل	X	U---------	_	16	nsubj	16:nsubj	Vform=دخل|Root=OOV|Translit=dḫl
+7	الله	الله	X	U---------	_	6	nmod	6:nmod	Vform=الله|Root=OOV|Translit=āllh
+8	رئيس	رَئِيس	NOUN	N------S4R	Case=Acc|Definite=Cons|Number=Sing	6	nmod	6:nmod:acc	Vform=رَئِيسَ|Gloss=president,head,chairman|Root=r_'_s|Translit=raʾīsa|LTranslit=raʾīs
+9	تحرير	تَحرِير	NOUN	N------S2R	Case=Gen|Definite=Cons|Number=Sing	8	nmod	8:nmod:gen	Vform=تَحرِيرِ|Gloss=liberation,liberating,editorship,editing|Root=.h_r_r|Translit=taḥrīri|LTranslit=taḥrīr
+10	صحيفة	صَحِيفَة	NOUN	N------S2R	Case=Gen|Definite=Cons|Number=Sing	9	nmod	9:nmod:gen	Vform=صَحِيفَةِ|Gloss=newspaper,sheet,leaf|Root=.s_.h_f|Translit=ṣaḥīfati|LTranslit=ṣaḥīfat
+11	الحزب	حِزب	NOUN	N------S2D	Case=Gen|Definite=Def|Number=Sing	10	nmod	10:nmod:gen	Vform=اَلحِزبِ|Gloss=party,band|Root=.h_z_b|Translit=al-ḥizbi|LTranslit=ḥizb
+12	الحاكم	حَاكِم	NOUN	N------S2D	Case=Gen|Definite=Def|Number=Sing	11	nmod	11:nmod:gen	Vform=اَلحَاكِمِ|Gloss=ruler,governor|Root=.h_k_m|Translit=al-ḥākimi|LTranslit=ḥākim
+13-14	والليبرالي	_	_	_	_	_	_	_	_
+13	و	وَ	CCONJ	C---------	_	6	cc	6:cc	Vform=وَ|Gloss=and|Root=wa|Translit=wa|LTranslit=wa
+14	الليبرالي	لِيبِرَالِيّ	ADJ	A-----MS4D	Case=Acc|Definite=Def|Gender=Masc|Number=Sing	6	amod	6:amod	Vform=اَللِّيبِرَالِيَّ|Gloss=liberal|Root=lIbirAl|Translit=al-lībirālīya|LTranslit=lībirālīy
+15	التوجهات	تَوَجُّه	NOUN	N------P2D	Case=Gen|Definite=Def|Number=Plur	14	nmod	14:nmod:gen	Vform=اَلتَّوَجُّهَاتِ|Gloss=attitude,approach|Root=w_^g_h|Translit=at-tawaǧǧuhāti|LTranslit=tawaǧǧuh
+16	تسلم	تَسَلَّم	VERB	VP-A-3MS--	Aspect=Perf|Gender=Masc|Number=Sing|Person=3|Voice=Act	2	ccomp	2:ccomp	Vform=تَسَلَّمَ|Gloss=receive,assume|Root=s_l_m|Translit=tasallama|LTranslit=tasallam
+17	منصب	مَنصِب	NOUN	N------S4R	Case=Acc|Definite=Cons|Number=Sing	16	obj	16:obj	Vform=مَنصِبَ|Gloss=post,position,office|Root=n_.s_b|Translit=manṣiba|LTranslit=manṣib
+18	وزير	وَزِير	NOUN	N------S2R	Case=Gen|Definite=Cons|Number=Sing	17	nmod	17:nmod:gen	Vform=وَزِيرِ|Gloss=minister|Root=w_z_r|Translit=wazīri|LTranslit=wazīr
+19	الاعلام	عَلَم	NOUN	N------P2D	Case=Gen|Definite=Def|Number=Plur	18	nmod	18:nmod:gen	Vform=اَلأَعلَامِ|Gloss=flag,banner,badge|Root=`_l_m|Translit=al-ʾaʿlāmi|LTranslit=ʿalam
+20	خلفا	خَلَف	NOUN	N------S4I	Case=Acc|Definite=Ind|Number=Sing	16	obl	16:obl:acc	Vform=خَلَفًا|Gloss=substitute,scion|Root=_h_l_f|Translit=ḫalafan|LTranslit=ḫalaf
+21-22	لاحمد	_	_	_	_	_	_	_	_
+21	ل	لِ	ADP	P---------	AdpType=Prep	23	case	23:case	Vform=لِ|Gloss=for,to|Root=l|Translit=li|LTranslit=li
+22	أحمد	أَحمَد	NOUN	N------S2I	Case=Gen|Definite=Ind|Number=Sing	23	nmod	23:nmod:gen	Vform=أَحمَدَ|Gloss=Ahmad|Root=.h_m_d|Translit=ʾaḥmada|LTranslit=ʾaḥmad
+23	الحسن	الحسن	X	U---------	_	20	nmod	20:nmod:لِ	Vform=الحسن|Root=OOV|Translit=ālḥsn
+24	فيما	فِيمَا	CCONJ	C---------	_	25	cc	25:cc	Vform=فِيمَا|Gloss=while,during_which|Root=fI|Translit=fīmā|LTranslit=fīmā
+25	تسلم	تَسَلَّم	VERB	VP-A-3MS--	Aspect=Perf|Gender=Masc|Number=Sing|Person=3|Voice=Act	16	conj	2:ccomp|16:conj	Vform=تَسَلَّمَ|Gloss=receive,assume|Root=s_l_m|Translit=tasallama|LTranslit=tasallam
+26	اللواء	لِوَاء	NOUN	N------S1D	Case=Nom|Definite=Def|Number=Sing	25	nsubj	25:nsubj	Vform=اَللِّوَاءُ|Gloss=banner,flag|Root=l_w_y|Translit=al-liwāʾu|LTranslit=liwāʾ
+27	غازي	غازي	X	U---------	_	28	nmod	28:nmod	Vform=غازي|Root=OOV|Translit=ġāzy
+28	كنعان	كنعان	X	U---------	_	26	nmod	26:nmod	Vform=كنعان|Root=OOV|Translit=knʿān
+29	رئيس	رَئِيس	NOUN	N------S1R	Case=Nom|Definite=Cons|Number=Sing	26	nmod	26:nmod:nom	Vform=رَئِيسُ|Gloss=president,head,chairman|Root=r_'_s|Translit=raʾīsu|LTranslit=raʾīs
+30	شعبة	شُعبَة	NOUN	N------S2R	Case=Gen|Definite=Cons|Number=Sing	29	nmod	29:nmod:gen	Vform=شُعبَةِ|Gloss=branch,subdivision|Root=^s_`_b|Translit=šuʿbati|LTranslit=šuʿbat
+31	الامن	أَمن	NOUN	N------S2D	Case=Gen|Definite=Def|Number=Sing	30	nmod	30:nmod:gen	Vform=اَلأَمنِ|Gloss=security,safety|Root='_m_n|Translit=al-ʾamni|LTranslit=ʾamn
+32	السياسي	سِيَاسِيّ	ADJ	A-----MS2D	Case=Gen|Definite=Def|Gender=Masc|Number=Sing	31	amod	31:amod	Vform=اَلسِّيَاسِيِّ|Gloss=political|Root=s_w_s|Translit=as-siyāsīyi|LTranslit=siyāsīy
+33	منصب	مَنصِب	NOUN	N------S4R	Case=Acc|Definite=Cons|Number=Sing	25	obj	25:obj	Vform=مَنصِبَ|Gloss=post,position,office|Root=n_.s_b|Translit=manṣiba|LTranslit=manṣib
+34	وزير	وَزِير	NOUN	N------S2R	Case=Gen|Definite=Cons|Number=Sing	33	nmod	33:nmod:gen	Vform=وَزِيرِ|Gloss=minister|Root=w_z_r|Translit=wazīri|LTranslit=wazīr
+35	الداخلية	دَاخِلِيّ	ADJ	A-----FS2D	Case=Gen|Definite=Def|Gender=Fem|Number=Sing	34	amod	34:amod	SpaceAfter=No|Vform=اَلدَّاخِلِيَّةِ|Gloss=internal,domestic,interior,of_state|Root=d__h_l|Translit=ad-dāḫilīyati|LTranslit=dāḫilīy
+36	.	.	PUNCT	G---------	_	1	punct	1:punct	Vform=.|Translit=.
+
+# newpar id = assabah.20041005.0017:p4
+# sent_id = assabah.20041005.0017:p4u1
+# text = وذكرت وكالة الانباء السورية ان التعديل شمل ثماني حقائب بينها وزارتا الداخلية والاقتصاد.
+# orig_file_sentence ASB_ARB_20041005.0017#4
+1-2	وذكرت	_	_	_	_	_	_	_	_
+1	و	وَ	CCONJ	C---------	_	0	root	0:root	Vform=وَ|Gloss=and|Root=wa|Translit=wa|LTranslit=wa
+2	ذكرت	ذَكَر	VERB	VP-A-3FS--	Aspect=Perf|Gender=Fem|Number=Sing|Person=3|Voice=Act	1	parataxis	1:parataxis	Vform=ذَكَرَت|Gloss=mention,cite,remember|Root=_d_k_r|Translit=ḏakarat|LTranslit=ḏakar
+3	وكالة	وِكَالَة	NOUN	N------S1R	Case=Nom|Definite=Cons|Number=Sing	2	nsubj	2:nsubj	Vform=وِكَالَةُ|Gloss=agency|Root=w_k_l|Translit=wikālatu|LTranslit=wikālat
+4	الانباء	نَبَأ	NOUN	N------P2D	Case=Gen|Definite=Def|Number=Plur	3	nmod	3:nmod:gen	Vform=اَلأَنبَاءِ|Gloss=news_item,report|Root=n_b_'|Translit=al-ʾanbāʾi|LTranslit=nabaʾ
+5	السورية	سُورِيّ	ADJ	A-----FS1D	Case=Nom|Definite=Def|Gender=Fem|Number=Sing	3	amod	3:amod	Vform=اَلسُّورِيَّةُ|Gloss=Syrian|Root=sUr|Translit=as-sūrīyatu|LTranslit=sūrīy
+6	ان	أَنَّ	SCONJ	C---------	_	8	mark	8:mark	Vform=أَنَّ|Gloss=that|Root='_n|Translit=ʾanna|LTranslit=ʾanna
+7	التعديل	تَعدِيل	NOUN	N------S4D	Case=Acc|Definite=Def|Number=Sing	8	obl	8:obl:acc	Vform=اَلتَّعدِيلَ|Gloss=adjustment,change,modification,amendment|Root=`_d_l|Translit=at-taʿdīla|LTranslit=taʿdīl
+8	شمل	شَمِل	VERB	VP-A-3MS--	Aspect=Perf|Gender=Masc|Number=Sing|Person=3|Voice=Act	2	ccomp	2:ccomp	Vform=شَمِلَ|Gloss=comprise,include,contain|Root=^s_m_l|Translit=šamila|LTranslit=šamil
+9	ثماني	ثَمَانُون	NUM	QL------4R	Case=Acc|Definite=Cons|NumForm=Word	8	obj	8:obj	Vform=ثَمَانِي|Gloss=eighty|Root=_t_m_n|Translit=ṯamānī|LTranslit=ṯamānūn
+10	حقائب	حَقِيبَة	NOUN	N------P2I	Case=Gen|Definite=Ind|Number=Plur	9	nmod	9:nmod:gen	Vform=حَقَائِبَ|Gloss=briefcase,suitcase,portfolio,luggage|Root=.h_q_b|Translit=ḥaqāʾiba|LTranslit=ḥaqībat
+11-12	بينها	_	_	_	_	_	_	_	_
+11	بين	بَينَ	ADP	PI------4-	AdpType=Prep|Case=Acc	12	case	12:case	Vform=بَينَ|Gloss=between,among|Root=b_y_n|Translit=bayna|LTranslit=bayna
+12	ها	هُوَ	PRON	SP---3FS2-	Case=Gen|Gender=Fem|Number=Sing|Person=3|PronType=Prs	10	obl	10:obl:بَينَ:gen	Vform=هَا|Gloss=he,she,it|Translit=hā|LTranslit=huwa
+13	وزارتا	وِزَارَة	NOUN	N------D1R	Case=Nom|Definite=Cons|Number=Dual	12	nsubj	12:nsubj	Vform=وِزَارَتَا|Gloss=ministry|Root=w_z_r|Translit=wizāratā|LTranslit=wizārat
+14	الداخلية	دَاخِلِيّ	ADJ	A-----FS2D	Case=Gen|Definite=Def|Gender=Fem|Number=Sing	13	amod	13:amod	Vform=اَلدَّاخِلِيَّةِ|Gloss=internal,domestic,interior,of_state|Root=d__h_l|Translit=ad-dāḫilīyati|LTranslit=dāḫilīy
+15-16	والاقتصاد	_	_	_	_	_	_	_	SpaceAfter=No
+15	و	وَ	CCONJ	C---------	_	16	cc	16:cc	Vform=وَ|Gloss=and|Root=wa|Translit=wa|LTranslit=wa
+16	الاقتصاد	اِقتِصَاد	NOUN	N------S2D	Case=Gen|Definite=Def|Number=Sing	14	conj	13:amod|14:conj	Vform=اَلِاقتِصَادِ|Gloss=economy,saving|Root=q_.s_d|Translit=al-i-ʼqtiṣādi|LTranslit=iqtiṣād
+17	.	.	PUNCT	G---------	_	1	punct	1:punct	Vform=.|Translit=.

stanza/demo/corenlp.py

@@ -0,0 +1,95 @@
+from stanza.server import CoreNLPClient
+
+# example text
+print('---')
+print('input text')
+print('')
+
+text = "Chris Manning is a nice person. Chris wrote a simple sentence. He also gives oranges to people."
+
+print(text)
+
+# set up the client
+print('---')
+print('starting up Java Stanford CoreNLP Server...')
+
+# set up the client
+with CoreNLPClient(annotators=['tokenize','ssplit','pos','lemma','ner','parse','depparse','coref'], timeout=60000, memory='16G') as client:
+    # submit the request to the server
+    ann = client.annotate(text)
+
+    # get the first sentence
+    sentence = ann.sentence[0]
+
+    # get the dependency parse of the first sentence
+    print('---')
+    print('dependency parse of first sentence')
+    dependency_parse = sentence.basicDependencies
+    print(dependency_parse)
+ 
+    # get the constituency parse of the first sentence
+    print('---')
+    print('constituency parse of first sentence')
+    constituency_parse = sentence.parseTree
+    print(constituency_parse)
+
+    # get the first subtree of the constituency parse
+    print('---')
+    print('first subtree of constituency parse')
+    print(constituency_parse.child[0])
+
+    # get the value of the first subtree
+    print('---')
+    print('value of first subtree of constituency parse')
+    print(constituency_parse.child[0].value)
+
+    # get the first token of the first sentence
+    print('---')
+    print('first token of first sentence')
+    token = sentence.token[0]
+    print(token)
+
+    # get the part-of-speech tag
+    print('---')
+    print('part of speech tag of token')
+    token.pos
+    print(token.pos)
+
+    # get the named entity tag
+    print('---')
+    print('named entity tag of token')
+    print(token.ner)
+
+    # get an entity mention from the first sentence
+    print('---')
+    print('first entity mention in sentence')
+    print(sentence.mentions[0])
+
+    # access the coref chain
+    print('---')
+    print('coref chains for the example')
+    print(ann.corefChain)
+
+    # Use tokensregex patterns to find who wrote a sentence.
+    pattern = '([ner: PERSON]+) /wrote/ /an?/ []{0,3} /sentence|article/'
+    matches = client.tokensregex(text, pattern)
+    # sentences contains a list with matches for each sentence.
+    assert len(matches["sentences"]) == 3
+    # length tells you whether or not there are any matches in this
+    assert matches["sentences"][1]["length"] == 1
+    # You can access matches like most regex groups.
+    matches["sentences"][1]["0"]["text"] == "Chris wrote a simple sentence"
+    matches["sentences"][1]["0"]["1"]["text"] == "Chris"
+
+    # Use semgrex patterns to directly find who wrote what.
+    pattern = '{word:wrote} >nsubj {}=subject >obj {}=object'
+    matches = client.semgrex(text, pattern)
+    # sentences contains a list with matches for each sentence.
+    assert len(matches["sentences"]) == 3
+    # length tells you whether or not there are any matches in this
+    assert matches["sentences"][1]["length"] == 1
+    # You can access matches like most regex groups.
+    matches["sentences"][1]["0"]["text"] == "wrote"
+    matches["sentences"][1]["0"]["$subject"]["text"] == "Chris"
+    matches["sentences"][1]["0"]["$object"]["text"] == "sentence"
+

stanza/demo/japanese_test.conllu.txt

@@ -0,0 +1,82 @@
+# newdoc id = test-s1
+# sent_id = test-s1
+# text = これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。
+1	これ	此れ	PRON	代名詞	_	6	obl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=代名詞|SpaceAfter=No|UnidicInfo=,此れ,これ,これ,コレ,,,コレ,コレ,此れ
+2	に	に	ADP	助詞-格助詞	_	1	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,に,に,に,ニ,,,ニ,ニ,に
+3	不快	不快	NOUN	名詞-普通名詞-形状詞可能	_	4	compound	_	BunsetuBILabel=B|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,不快,不快,不快,フカイ,,,フカイ,フカイカン,不快感
+4	感	感	NOUN	名詞-普通名詞-一般	_	6	obj	_	BunsetuBILabel=I|BunsetuPositionType=SEM_HEAD|LUWBILabel=I|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,感,感,感,カン,,,カン,フカイカン,不快感
+5	を	を	ADP	助詞-格助詞	_	4	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,を,を,を,オ,,,ヲ,ヲ,を
+6	示す	示す	VERB	動詞-一般-五段-サ行	_	7	acl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=動詞-一般-五段-サ行|SpaceAfter=No|UnidicInfo=,示す,示す,示す,シメス,,,シメス,シメス,示す
+7	住民	住民	NOUN	名詞-普通名詞-一般	_	9	nsubj	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,住民,住民,住民,ジューミン,,,ジュウミン,ジュウミン,住民
+8	は	は	ADP	助詞-係助詞	_	7	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-係助詞|SpaceAfter=No|UnidicInfo=,は,は,は,ワ,,,ハ,ハ,は
+9	い	居る	VERB	動詞-非自立可能-上一段-ア行	_	29	advcl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=動詞-一般-上一段-ア行|PrevUDLemma=いる|SpaceAfter=No|UnidicInfo=,居る,い,いる,イ,,,イル,イル,居る
+10	まし	ます	AUX	助動詞-助動詞-マス	_	9	aux	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助動詞-助動詞-マス|SpaceAfter=No|UnidicInfo=,ます,まし,ます,マシ,,,マス,マス,ます
+11	た	た	AUX	助動詞-助動詞-タ	_	9	aux	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助動詞-助動詞-タ|SpaceAfter=No|UnidicInfo=,た,た,た,タ,,,タ,タ,た
+12	が	が	SCONJ	助詞-接続助詞	_	9	mark	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助詞-接続助詞|SpaceAfter=No|UnidicInfo=,が,が,が,ガ,,,ガ,ガ,が
+13	,	,	PUNCT	補助記号-読点	_	9	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-読点|SpaceAfter=No|UnidicInfo=,，,,,,,,,,，
+14	現在	現在	ADV	名詞-普通名詞-副詞可能	_	16	advmod	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=副詞|SpaceAfter=No|UnidicInfo=,現在,現在,現在,ゲンザイ,,,ゲンザイ,ゲンザイ,現在
+15	,	,	PUNCT	補助記号-読点	_	14	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-読点|SpaceAfter=No|UnidicInfo=,，,,,,,,,,，
+16	表立っ	表立つ	VERB	動詞-一般-五段-タ行	_	24	advcl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=動詞-一般-五段-タ行|SpaceAfter=No|UnidicInfo=,表立つ,表立っ,表立つ,オモテダッ,,,オモテダツ,オモテダツ,表立つ
+17	て	て	SCONJ	助詞-接続助詞	_	16	mark	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-接続助詞|SpaceAfter=No|UnidicInfo=,て,て,て,テ,,,テ,テ,て
+18	反対	反対	NOUN	名詞-普通名詞-サ変形状詞可能	_	20	nmod	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,反対,反対,反対,ハンタイ,,,ハンタイ,ハンタイ,反対
+19	や	や	ADP	助詞-副助詞	_	18	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-副助詞|SpaceAfter=No|UnidicInfo=,や,や,や,ヤ,,,ヤ,ヤ,や
+20	抗議	抗議	NOUN	名詞-普通名詞-サ変可能	_	22	nmod	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,抗議,抗議,抗議,コーギ,,,コウギ,コウギ,抗議
+21	の	の	ADP	助詞-格助詞	_	20	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,の,の,の,ノ,,,ノ,ノ,の
+22	声	声	NOUN	名詞-普通名詞-一般	_	24	obj	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,声,声,声,コエ,,,コエ,コエ,声
+23	を	を	ADP	助詞-格助詞	_	22	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,を,を,を,オ,,,ヲ,ヲ,を
+24	挙げ	上げる	VERB	動詞-非自立可能-下一段-ガ行	_	27	acl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=動詞-一般-下一段-ガ行|SpaceAfter=No|UnidicInfo=,上げる,挙げ,挙げる,アゲ,,,アゲル,アゲル,上げる
+25	て	て	SCONJ	助詞-接続助詞	_	24	mark	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助動詞-上一段-ア行|SpaceAfter=No|UnidicInfo=,て,て,て,テ,,,テ,テイル,ている
+26	いる	居る	VERB	動詞-非自立可能-上一段-ア行	_	25	fixed	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=I|LUWPOS=助動詞-上一段-ア行|PrevUDLemma=いる|SpaceAfter=No|UnidicInfo=,居る,いる,いる,イル,,,イル,テイル,ている
+27	住民	住民	NOUN	名詞-普通名詞-一般	_	29	nsubj	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,住民,住民,住民,ジューミン,,,ジュウミン,ジュウミン,住民
+28	は	は	ADP	助詞-係助詞	_	27	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-係助詞|SpaceAfter=No|UnidicInfo=,は,は,は,ワ,,,ハ,ハ,は
+29	い	居る	VERB	動詞-非自立可能-上一段-ア行	_	0	root	_	BunsetuBILabel=B|BunsetuPositionType=ROOT|LUWBILabel=B|LUWPOS=動詞-一般-上一段-ア行|PrevUDLemma=いる|SpaceAfter=No|UnidicInfo=,居る,い,いる,イ,,,イル,イル,居る
+30	ない	ない	AUX	助動詞-助動詞-ナイ	Polarity=Neg	29	aux	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助動詞-助動詞-ナイ|SpaceAfter=No|UnidicInfo=,ない,ない,ない,ナイ,,,ナイ,ナイ,ない
+31	よう	様	AUX	形状詞-助動詞語幹	_	29	aux	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=形状詞-助動詞語幹|PrevUDLemma=よう|SpaceAfter=No|UnidicInfo=,様,よう,よう,ヨー,,,ヨウ,ヨウ,様
+32	です	です	AUX	助動詞-助動詞-デス	_	29	aux	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助動詞-助動詞-デス|PrevUDLemma=だ|SpaceAfter=No|UnidicInfo=,です,です,です,デス,,,デス,デス,です
+33	。	。	PUNCT	補助記号-句点	_	29	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-句点|SpaceAfter=Yes|UnidicInfo=,。,。,。,,,,,,。
+
+# newdoc id = test-s2
+# sent_id = test-s2
+# text = 幸福の科学側からは,特にどうしてほしいという要望はいただいていません。
+1	幸福	幸福	NOUN	名詞-普通名詞-形状詞可能	_	4	nmod	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,幸福,幸福,幸福,コーフク,,,コウフク,コウフクノカガクガワ,幸福の科学側
+2	の	の	ADP	助詞-格助詞	_	1	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=I|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,の,の,の,ノ,,,ノ,コウフクノカガクガワ,幸福の科学側
+3	科学	科学	NOUN	名詞-普通名詞-サ変可能	_	4	compound	_	BunsetuBILabel=B|BunsetuPositionType=CONT|LUWBILabel=I|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,科学,科学,科学,カガク,,,カガク,コウフクノカガクガワ,幸福の科学側
+4	側	側	NOUN	名詞-普通名詞-一般	_	17	obl	_	BunsetuBILabel=I|BunsetuPositionType=SEM_HEAD|LUWBILabel=I|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,側,側,側,ガワ,,,ガワ,コウフクノカガクガワ,幸福の科学側
+5	から	から	ADP	助詞-格助詞	_	4	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,から,から,から,カラ,,,カラ,カラ,から
+6	は	は	ADP	助詞-係助詞	_	4	case	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助詞-係助詞|SpaceAfter=No|UnidicInfo=,は,は,は,ワ,,,ハ,ハ,は
+7	,	,	PUNCT	補助記号-読点	_	4	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-読点|SpaceAfter=No|UnidicInfo=,，,,,,,,,,，
+8	特に	特に	ADV	副詞	_	17	advmod	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=副詞|SpaceAfter=No|UnidicInfo=,特に,特に,特に,トクニ,,,トクニ,トクニ,特に
+9	どう	どう	ADV	副詞	_	15	advcl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=動詞-一般-サ行変格|SpaceAfter=No|UnidicInfo=,どう,どう,どう,ドー,,,ドウ,ドウスル,どうする
+10	し	為る	AUX	動詞-非自立可能-サ行変格	_	9	aux	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=I|LUWPOS=動詞-一般-サ行変格|PrevUDLemma=する|SpaceAfter=No|UnidicInfo=,為る,し,する,シ,,,スル,ドウスル,どうする
+11	て	て	SCONJ	助詞-接続助詞	_	9	mark	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助動詞-形容詞|SpaceAfter=No|UnidicInfo=,て,て,て,テ,,,テ,テホシイ,てほしい
+12	ほしい	欲しい	AUX	形容詞-非自立可能-形容詞	_	11	fixed	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=I|LUWPOS=助動詞-形容詞|PrevUDLemma=ほしい|SpaceAfter=No|UnidicInfo=,欲しい,ほしい,ほしい,ホシー,,,ホシイ,テホシイ,てほしい
+13	と	と	ADP	助詞-格助詞	_	9	case	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,と,と,と,ト,,,ト,トイウ,という
+14	いう	言う	VERB	動詞-一般-五段-ワア行	_	13	fixed	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=I|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,言う,いう,いう,イウ,,,イウ,トイウ,という
+15	要望	要望	NOUN	名詞-普通名詞-サ変可能	_	17	nsubj	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,要望,要望,要望,ヨーボー,,,ヨウボウ,ヨウボウ,要望
+16	は	は	ADP	助詞-係助詞	_	15	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-係助詞|SpaceAfter=No|UnidicInfo=,は,は,は,ワ,,,ハ,ハ,は
+17	いただい	頂く	VERB	動詞-非自立可能-五段-カ行	_	0	root	_	BunsetuBILabel=B|BunsetuPositionType=ROOT|LUWBILabel=B|LUWPOS=動詞-一般-五段-カ行|PrevUDLemma=いただく|SpaceAfter=No|UnidicInfo=,頂く,いただい,いただく,イタダイ,,,イタダク,イタダク,頂く
+18	て	て	SCONJ	助詞-接続助詞	_	17	mark	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助動詞-上一段-ア行|SpaceAfter=No|UnidicInfo=,て,て,て,テ,,,テ,テイル,ている
+19	い	居る	VERB	動詞-非自立可能-上一段-ア行	_	18	fixed	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=I|LUWPOS=助動詞-上一段-ア行|PrevUDLemma=いる|SpaceAfter=No|UnidicInfo=,居る,い,いる,イ,,,イル,テイル,ている
+20	ませ	ます	AUX	助動詞-助動詞-マス	_	17	aux	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=B|LUWPOS=助動詞-助動詞-マス|SpaceAfter=No|UnidicInfo=,ます,ませ,ます,マセ,,,マス,マス,ます
+21	ん	ず	AUX	助動詞-助動詞-ヌ	Polarity=Neg	17	aux	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助動詞-助動詞-ヌ|PrevUDLemma=ぬ|SpaceAfter=No|UnidicInfo=,ず,ん,ぬ,ン,,,ヌ,ズ,ず
+22	。	。	PUNCT	補助記号-句点	_	17	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-句点|SpaceAfter=Yes|UnidicInfo=,。,。,。,,,,,,。
+
+# newdoc id = test-s3
+# sent_id = test-s3
+# text = 星取り参加は当然とされ,不参加は白眼視される。
+1	星取り	星取り	NOUN	名詞-普通名詞-一般	_	2	compound	_	BunsetuBILabel=B|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,星取り,星取り,星取り,ホシトリ,,,ホシトリ,ホシトリサンカ,星取り参加
+2	参加	参加	NOUN	名詞-普通名詞-サ変可能	_	4	nsubj	_	BunsetuBILabel=I|BunsetuPositionType=SEM_HEAD|LUWBILabel=I|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,参加,参加,参加,サンカ,,,サンカ,ホシトリサンカ,星取り参加
+3	は	は	ADP	助詞-係助詞	_	2	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-係助詞|SpaceAfter=No|UnidicInfo=,は,は,は,ワ,,,ハ,ハ,は
+4	当然	当然	ADJ	形状詞-一般	_	6	advcl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=形状詞-一般|SpaceAfter=No|UnidicInfo=,当然,当然,当然,トーゼン,,,トウゼン,トウゼン,当然
+5	と	と	ADP	助詞-格助詞	_	4	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-格助詞|SpaceAfter=No|UnidicInfo=,と,と,と,ト,,,ト,ト,と
+6	さ	為る	VERB	動詞-非自立可能-サ行変格	_	13	acl	_	BunsetuBILabel=B|BunsetuPositionType=SEM_HEAD|LUWBILabel=B|LUWPOS=動詞-一般-サ行変格|PrevUDLemma=する|SpaceAfter=No|UnidicInfo=,為る,さ,する,サ,,,スル,スル,する
+7	れ	れる	AUX	助動詞-助動詞-レル	_	6	aux	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助動詞-助動詞-レル|SpaceAfter=No|UnidicInfo=,れる,れ,れる,レ,,,レル,レル,れる
+8	,	,	PUNCT	補助記号-読点	_	6	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-読点|SpaceAfter=No|UnidicInfo=,，,,,,,,,,，
+9	不	不	NOUN	接頭辞	Polarity=Neg	10	compound	_	BunsetuBILabel=B|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,不,不,不,フ,,,フ,フサンカ,不参加
+10	参加	参加	NOUN	名詞-普通名詞-サ変可能	_	13	nsubj	_	BunsetuBILabel=I|BunsetuPositionType=SEM_HEAD|LUWBILabel=I|LUWPOS=名詞-普通名詞-一般|SpaceAfter=No|UnidicInfo=,参加,参加,参加,サンカ,,,サンカ,フサンカ,不参加
+11	は	は	ADP	助詞-係助詞	_	10	case	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助詞-係助詞|SpaceAfter=No|UnidicInfo=,は,は,は,ワ,,,ハ,ハ,は
+12	白眼	白眼	NOUN	名詞-普通名詞-一般	_	13	compound	_	BunsetuBILabel=B|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=動詞-一般-サ行変格|SpaceAfter=No|UnidicInfo=,白眼,白眼,白眼,ハクガン,,,ハクガン,ハクガンシスル,白眼視する
+13	視	視	NOUN	接尾辞-名詞的-サ変可能	_	0	root	_	BunsetuBILabel=I|BunsetuPositionType=ROOT|LUWBILabel=I|LUWPOS=動詞-一般-サ行変格|SpaceAfter=No|UnidicInfo=,視,視,視,シ,,,シ,ハクガンシスル,白眼視する
+14	さ	為る	AUX	動詞-非自立可能-サ行変格	_	13	aux	_	BunsetuBILabel=I|BunsetuPositionType=FUNC|LUWBILabel=I|LUWPOS=動詞-一般-サ行変格|PrevUDLemma=する|SpaceAfter=No|UnidicInfo=,為る,さ,する,サ,,,スル,ハクガンシスル,白眼視する
+15	れる	れる	AUX	助動詞-助動詞-レル	_	13	aux	_	BunsetuBILabel=I|BunsetuPositionType=SYN_HEAD|LUWBILabel=B|LUWPOS=助動詞-助動詞-レル|SpaceAfter=No|UnidicInfo=,れる,れる,れる,レル,,,レル,レル,れる
+16	。	。	PUNCT	補助記号-句点	_	13	punct	_	BunsetuBILabel=I|BunsetuPositionType=CONT|LUWBILabel=B|LUWPOS=補助記号-句点|SpaceAfter=Yes|UnidicInfo=,。,。,。,,,,,,。

stanza/demo/pipeline_demo.py

@@ -0,0 +1,52 @@
+"""
+A basic demo of the Stanza neural pipeline.
+"""
+
+import sys
+import argparse
+import os
+
+import stanza
+from stanza.resources.common import DEFAULT_MODEL_DIR
+
+
+if __name__ == '__main__':
+    # get arguments
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-d', '--models_dir', help='location of models files | default: ~/stanza_resources',
+                        default=DEFAULT_MODEL_DIR)
+    parser.add_argument('-l', '--lang', help='Demo language',
+                        default="en")
+    parser.add_argument('-c', '--cpu', action='store_true', help='Use cpu as the device.')
+    args = parser.parse_args()
+
+    example_sentences = {"en": "Barack Obama was born in Hawaii.  He was elected president in 2008.",
+            "zh": "中国文化经历上千年的历史演变，是各区域、各民族古代文化长期相互交流、借鉴、融合的结果。",
+            "fr": "Van Gogh grandit au sein d'une famille de l'ancienne bourgeoisie. Il tente d'abord de faire carrière comme marchand d'art chez Goupil & C.",
+            "vi": "Trận Trân Châu Cảng (hay Chiến dịch Hawaii theo cách gọi của Bộ Tổng tư lệnh Đế quốc Nhật Bản) là một đòn tấn công quân sự bất ngờ được Hải quân Nhật Bản thực hiện nhằm vào căn cứ hải quân của Hoa Kỳ tại Trân Châu Cảng thuộc tiểu bang Hawaii vào sáng Chủ Nhật, ngày 7 tháng 12 năm 1941, dẫn đến việc Hoa Kỳ sau đó quyết định tham gia vào hoạt động quân sự trong Chiến tranh thế giới thứ hai."}
+
+    if args.lang not in example_sentences:
+        print(f'Sorry, but we don\'t have a demo sentence for "{args.lang}" for the moment. Try one of these languages: {list(example_sentences.keys())}')
+        sys.exit(1)
+
+    # download the models
+    stanza.download(args.lang, dir=args.models_dir)
+    # set up a pipeline
+    print('---')
+    print('Building pipeline...')
+    pipeline = stanza.Pipeline(lang=args.lang, dir=args.models_dir, use_gpu=(not args.cpu))
+    # process the document
+    doc = pipeline(example_sentences[args.lang])
+    # access nlp annotations
+    print('')
+    print('Input: {}'.format(example_sentences[args.lang]))
+    print("The tokenizer split the input into {} sentences.".format(len(doc.sentences)))
+    print('---')
+    print('tokens of first sentence: ')
+    doc.sentences[0].print_tokens()
+    print('')
+    print('---')
+    print('dependency parse of first sentence: ')
+    doc.sentences[0].print_dependencies()
+    print('')
+

stanza/demo/scenegraph.py

@@ -0,0 +1,20 @@
+"""
+Very short demo for the SceneGraph interface in the CoreNLP server
+
+Requires CoreNLP >= 4.5.5, Stanza >= 1.5.1
+"""
+
+import json
+
+from stanza.server import CoreNLPClient
+
+# start_server=None if you have the server running in another process on the same host
+# you can start it with whatever normal options CoreNLPClient has
+#
+# preload=False avoids having the server unnecessarily load annotators
+# if you don't plan on using them
+with CoreNLPClient(preload=False) as client:
+    result = client.scenegraph("Jennifer's antennae are on her head.")
+    print(json.dumps(result, indent=2))
+
+

stanza/demo/semgrex.py

@@ -0,0 +1,18 @@
+import stanza
+from stanza.server.semgrex import Semgrex
+
+nlp = stanza.Pipeline("en", processors="tokenize,pos,lemma,depparse")
+
+doc = nlp("Banning opal removed all artifact decks from the meta.  I miss playing lantern.")
+with Semgrex(classpath="$CLASSPATH") as sem:
+    semgrex_results = sem.process(doc,
+                                  "{pos:NN}=object <obl {}=action",
+                                  "{cpos:NOUN}=thing <obj {cpos:VERB}=action")
+    print("COMPLETE RESULTS")
+    print(semgrex_results)
+
+    print("Number of matches in graph 0 ('Banning opal...') for semgrex query 1 (thing <obj action): %d" % len(semgrex_results.result[0].result[1].match))
+    for match_idx, match in enumerate(semgrex_results.result[0].result[1].match):
+        print("Match {}:\n-----------\n{}".format(match_idx, match))
+
+    print("graph 1 for semgrex query 0 is an empty match: len %d" % len(semgrex_results.result[1].result[0].match))

stanza/demo/semgrex_sample.conllu

@@ -0,0 +1,24 @@
+
+# sent_id = reviews-181748-0003
+# text = My experience was awful though.
+1	My	my	PRON	PRP$	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	2	nmod:poss	2:nmod:poss	_
+2	experience	experience	NOUN	NN	Number=Sing	4	nsubj	4:nsubj	_
+3	was	be	AUX	VBD	Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin	4	cop	4:cop	_
+4	awful	awful	ADJ	JJ	Degree=Pos	0	root	0:root	_
+5	though	though	ADV	RB	_	4	advmod	4:advmod	SpaceAfter=No
+6	.	.	PUNCT	.	_	4	punct	4:punct	_
+
+
+
+# sent_id = reviews-117115-0005
+# text = The intruders slit the screen of the window.
+1	The	the	DET	DT	Definite=Def|PronType=Art	2	det	2:det	_
+2	intruders	intruder	NOUN	NNS	Number=Plur	3	nsubj	3:nsubj	_
+3	slit	slit	VERB	VBD	Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin	0	root	0:root	_
+4	the	the	DET	DT	Definite=Def|PronType=Art	5	det	5:det	_
+5	screen	screen	NOUN	NN	Number=Sing	3	obj	3:obj	_
+6	of	of	ADP	IN	_	8	case	8:case	_
+7	the	the	DET	DT	Definite=Def|PronType=Art	8	det	8:det	_
+8	window	window	NOUN	NN	Number=Sing	5	nmod	5:nmod:of	SpaceAfter=No
+9	.	.	PUNCT	.	_	3	punct	3:punct	_
+

stanza/demo/ssurgeon_script.txt

@@ -0,0 +1,18 @@
+# To run this, use the stanza/server/ssurgeon.py main file.
+# For example:
+# python3 stanza/server/ssurgeon.py  --edit_file demo/ssurgeon_script.txt --no_print_input --input_file ../data/ud2_11/UD_English-Pronouns/en_pronouns-ud-test.conllu > en_pronouns.updated.conllu
+# This script updates the UD 2.11 version of UD_English-Pronouns to
+# better match punctuation attachments, MWT, and no double subjects.
+
+# This turns unwanted csubj into advcl
+{}=source >nsubj {} >csubj=bad {}
+relabelNamedEdge -edge bad -reln advcl
+
+# This detects punctuations which are not attached to the root and reattaches them
+{word:/[.]/}=punct <punct=bad {}=parent << {$}=root : {}=parent << {}=root
+removeNamedEdge -edge bad
+addEdge -gov root -dep punct -reln punct
+
+# This detects the specific MWT found in the 2.11 dataset
+{}=first . {word:/'s|n't|'ll/}=second
+combineMWT -node first -node second

stanza/doc/CoreNLP.proto

@@ -0,0 +1,873 @@
+syntax = "proto2";
+
+package edu.stanford.nlp.pipeline;
+
+option java_package = "edu.stanford.nlp.pipeline";
+option java_outer_classname = "CoreNLPProtos";
+
+//
+// From JAVANLP_HOME, you can build me with the command:
+//
+//  protoc -I=src/edu/stanford/nlp/pipeline/ --java_out=src src/edu/stanford/nlp/pipeline/CoreNLP.proto
+//
+
+//
+// To do the python version:
+//
+//  protoc -I=./doc --python_out=./stanza/protobuf ./doc/CoreNLP.proto
+//
+
+//
+// An enumeration for the valid languages allowed in CoreNLP
+//
+enum Language {
+  Unknown  = 0;
+  Any      = 1;
+  Arabic   = 2;
+  Chinese  = 3;
+  English  = 4;
+  German   = 5;
+  French   = 6;
+  Hebrew   = 7;
+  Spanish  = 8;
+  UniversalEnglish = 9;
+  UniversalChinese = 10;
+}
+
+//
+// A document; that is, the equivalent of an Annotation.
+//
+message Document {
+  required string     text        = 1;
+  repeated Sentence   sentence    = 2;
+  repeated CorefChain corefChain  = 3;
+  optional string     docID       = 4;
+  optional string     docDate     = 7;
+  optional uint64     calendar    = 8;
+
+  /**
+   * A peculiar field, for the corner case when a Document is
+   * serialized without any sentences. Otherwise
+   */
+  repeated Token      sentencelessToken = 5;
+  repeated Token      character = 10;
+
+  repeated Quote      quote = 6;
+
+  /**
+   * This field is for entity mentions across the document.
+   */
+  repeated NERMention mentions = 9;
+  optional bool hasEntityMentionsAnnotation = 13; // used to differentiate between null and empty list
+
+  /**
+   * xml information
+   */
+  optional bool    xmlDoc = 11;
+  repeated Section sections = 12;
+
+  /** coref mentions for entire document **/
+  repeated Mention         mentionsForCoref                    = 14;
+  optional bool hasCorefMentionAnnotation = 15;
+  optional bool hasCorefAnnotation = 16;
+  repeated int32 corefMentionToEntityMentionMappings = 17;
+  repeated int32 entityMentionToCorefMentionMappings = 18;
+
+  extensions 100 to 255;
+}
+
+//
+// The serialized version of a CoreMap representing a sentence.
+//
+message Sentence {
+  repeated Token            token                               = 1;
+  required uint32           tokenOffsetBegin                    = 2;
+  required uint32           tokenOffsetEnd                      = 3;
+  optional uint32           sentenceIndex                       = 4;
+  optional uint32           characterOffsetBegin                = 5;
+  optional uint32           characterOffsetEnd                  = 6;
+  optional ParseTree        parseTree                           = 7;
+  optional ParseTree        binarizedParseTree                  = 31;
+  optional ParseTree        annotatedParseTree                  = 32;
+  optional string           sentiment                           = 33;
+  repeated ParseTree        kBestParseTrees                     = 34;
+  optional DependencyGraph  basicDependencies                   = 8;
+  optional DependencyGraph  collapsedDependencies               = 9;
+  optional DependencyGraph  collapsedCCProcessedDependencies    = 10;
+  optional DependencyGraph  alternativeDependencies             = 13;
+  repeated RelationTriple   openieTriple                        = 14;   // The OpenIE triples in the sentence
+  repeated RelationTriple   kbpTriple                           = 16;   // The KBP triples in this sentence
+  repeated SentenceFragment entailedSentence                    = 15;   // The entailed sentences, by natural logic
+  repeated SentenceFragment entailedClause                      = 35;   // The entailed clauses, by natural logic
+  optional DependencyGraph  enhancedDependencies                = 17;
+  optional DependencyGraph  enhancedPlusPlusDependencies        = 18;
+  repeated Token            character                           = 19;
+
+  optional uint32           paragraph                           = 11;
+
+  optional string           text                                = 12;   // Only needed if we're only saving the sentence.
+
+  optional uint32           lineNumber                          = 20;
+
+  // Fields set by other annotators in CoreNLP
+  optional bool            hasRelationAnnotations              = 51;
+  repeated Entity          entity                              = 52;
+  repeated Relation        relation                            = 53;
+  optional bool            hasNumerizedTokensAnnotation        = 54;
+  repeated NERMention      mentions                            = 55;
+  repeated Mention         mentionsForCoref                    = 56;
+  optional bool            hasCorefMentionsAnnotation          = 57;
+
+  optional string          sentenceID                          = 58;  // Useful when storing sentences (e.g. ForEach)
+  optional string          sectionDate                         = 59;  // date of section
+  optional uint32          sectionIndex                        = 60;  // section index for this sentence's section
+  optional string          sectionName                         = 61;  // name of section
+  optional string          sectionAuthor                       = 62;  // author of section
+  optional string          docID                               = 63;  // doc id
+  optional bool            sectionQuoted                       = 64;  // is this sentence in an xml quote in a post
+
+  optional bool            hasEntityMentionsAnnotation         = 65;  // check if there are entity mentions
+  optional bool            hasKBPTriplesAnnotation             = 68;  // check if there are KBP triples
+  optional bool            hasOpenieTriplesAnnotation          = 69;  // check if there are OpenIE triples
+
+  // quote stuff
+  optional uint32             chapterIndex                     = 66;
+  optional uint32             paragraphIndex                   = 67;
+  // the quote annotator can soometimes add merged sentences
+  optional Sentence           enhancedSentence                 = 70;
+
+  // speaker stuff
+  optional string          speaker                             = 71;  // The speaker speaking this sentence
+  optional string          speakerType                         = 72;  // The type of speaker speaking this sentence
+
+  extensions 100 to 255;
+}
+
+//
+// The serialized version of a Token (a CoreLabel).
+//
+message Token {
+  // Fields set by the default annotators [new CoreNLP(new Properties())]
+  optional string word              = 1;    // the word's gloss (post-tokenization)
+  optional string pos               = 2;    // The word's part of speech tag
+  optional string value             = 3;    // The word's 'value', (e.g., parse tree node)
+  optional string category          = 4;    // The word's 'category' (e.g., parse tree node)
+  optional string before            = 5;    // The whitespace/xml before the token
+  optional string after             = 6;    // The whitespace/xml after the token
+  optional string originalText      = 7;    // The original text for this token
+  optional string ner               = 8;    // The word's NER tag
+  optional string coarseNER         = 62;   // The word's coarse NER tag
+  optional string fineGrainedNER    = 63;   // The word's fine-grained NER tag
+  repeated string nerLabelProbs     = 66;   // listing of probs
+  optional string normalizedNER     = 9;    // The word's normalized NER tag
+  optional string lemma             = 10;   // The word's lemma
+  optional uint32 beginChar         = 11;   // The character offset begin, in the document
+  optional uint32 endChar           = 12;   // The character offset end, in the document
+  optional uint32 utterance         = 13;   // The utterance tag used in dcoref
+  optional string speaker           = 14;   // The speaker speaking this word
+  optional string speakerType       = 77;   // The type of speaker speaking this word
+  optional uint32 beginIndex        = 15;   // The begin index of, e.g., a span
+  optional uint32 endIndex          = 16;   // The begin index of, e.g., a span
+  optional uint32 tokenBeginIndex   = 17;   // The begin index of the token
+  optional uint32 tokenEndIndex     = 18;   // The end index of the token
+  optional Timex  timexValue        = 19;   // The time this word refers to
+  optional bool   hasXmlContext     = 21;   // Used by clean xml annotator
+  repeated string xmlContext        = 22;   // Used by clean xml annotator
+  optional uint32 corefClusterID    = 23;   // The [primary] cluster id for this token
+  optional string answer            = 24;   // A temporary annotation which is occasionally left in
+  //  optional string projectedCategory = 25;   // The syntactic category of the maximal constituent headed by the word. Not used anywhere, so deleted.
+  optional uint32    headWordIndex  = 26;   // The index of the head word of this word.
+  optional Operator  operator       = 27;   // If this is an operator, which one is it and what is its scope (as per Natural Logic)?
+  optional Polarity  polarity       = 28;   // The polarity of this word, according to Natural Logic
+  optional string    polarity_dir   = 39;   // The polarity of this word, either "up", "down", or "flat"
+  optional Span      span           = 29;   // The span of a leaf node of a tree
+  optional string    sentiment      = 30;   // The final sentiment of the sentence
+  optional int32     quotationIndex = 31;   // The index of the quotation this token refers to
+  optional MapStringString conllUFeatures = 32;
+  optional string coarseTag         = 33; //  The coarse POS tag (used to store the UPOS tag)
+  optional Span conllUTokenSpan     = 34;
+  optional string conllUMisc        = 35;
+  optional MapStringString conllUSecondaryDeps = 36;
+  optional string   wikipediaEntity = 37;
+  optional bool     isNewline = 38;
+
+
+  // Fields set by other annotators in CoreNLP
+  optional string gender          = 51;  // gender annotation (machine reading)
+  optional string trueCase        = 52;  // true case type of token
+  optional string trueCaseText    = 53;  // true case gloss of token
+
+  //  Chinese character info
+  optional string chineseChar     = 54;
+  optional string chineseSeg      = 55;
+  optional string chineseXMLChar  = 60;
+
+  //  Arabic character info
+  optional string arabicSeg       = 76;
+
+  // Section info
+  optional string sectionName     = 56;
+  optional string sectionAuthor   = 57;
+  optional string sectionDate     = 58;
+  optional string sectionEndLabel = 59;
+
+  // French tokens have parents
+  optional string parent          = 61;
+
+  // mention index info
+  repeated uint32 corefMentionIndex = 64;
+  optional uint32 entityMentionIndex = 65;
+
+  // mwt stuff
+  optional bool isMWT = 67;
+  optional bool isFirstMWT = 68;
+  optional string mwtText = 69;
+  // setting this to a map might be nice, but there are a couple issues
+  // for one, there can be values with no key
+  // for another, it's a pain to correctly parse, since different treebanks
+  // can have different standards for how to write out the misc field
+  optional string mwtMisc = 78;
+
+  // number info
+  optional uint64 numericValue = 70;
+  optional string numericType = 71;
+  optional uint64 numericCompositeValue = 72;
+  optional string numericCompositeType = 73;
+
+  optional uint32 codepointOffsetBegin   = 74;
+  optional uint32 codepointOffsetEnd     = 75;
+
+  // Fields in the CoreLabel java class that are moved elsewhere
+  //       string text           @see Document#text + character offsets
+  //       uint32 sentenceIndex  @see Sentence#sentenceIndex
+  //       string docID          @see Document#docID
+  //       uint32 paragraph      @see Sentence#paragraph
+
+  // Most serialized annotations will not have this
+  // Some code paths may not correctly process this if serialized,
+  // since many places will read the index off the position in a sentence
+  // In particular, deserializing a Document using ProtobufAnnotationSerializer
+  // will clobber any index value
+  // But Semgrex and Ssurgeon in particular need a way
+  // to pass around nodes where the node's index is not strictly 1, 2, 3, ...
+  // thanks to the empty nodes in UD treebanks such as
+  // English EWT or Estonian EWT (not related to each other)
+  optional uint32 index          = 79;
+  optional uint32 emptyIndex     = 80;
+
+  extensions 100 to 255;
+}
+
+//
+// An enumeration of valid sentiment values for the sentiment classifier.
+//
+enum Sentiment {
+  STRONG_NEGATIVE   = 0;
+  WEAK_NEGATIVE     = 1;
+  NEUTRAL           = 2;
+  WEAK_POSITIVE     = 3;
+  STRONG_POSITIVE   = 4;
+}
+
+//
+// A quotation marker in text
+//
+message Quote {
+  optional string text           = 1;
+  optional uint32 begin          = 2;
+  optional uint32 end            = 3;
+  optional uint32 sentenceBegin  = 5;
+  optional uint32 sentenceEnd    = 6;
+  optional uint32 tokenBegin     = 7;
+  optional uint32 tokenEnd       = 8;
+  optional string docid          = 9;
+  optional uint32 index          = 10;
+  optional string author         = 11;
+  optional string mention        = 12;
+  optional uint32 mentionBegin   = 13;
+  optional uint32 mentionEnd     = 14;
+  optional string mentionType    = 15;
+  optional string mentionSieve   = 16;
+  optional string speaker        = 17;
+  optional string speakerSieve   = 18;
+  optional string canonicalMention = 19;
+  optional uint32 canonicalMentionBegin = 20;
+  optional uint32 canonicalMentionEnd = 21;
+  optional DependencyGraph attributionDependencyGraph = 22;
+}
+
+//
+// A syntactic parse tree, with scores.
+//
+message ParseTree {
+  repeated ParseTree child           = 1;
+  optional string    value           = 2;
+  optional uint32    yieldBeginIndex = 3;
+  optional uint32    yieldEndIndex   = 4;
+  optional double    score           = 5;
+  optional Sentiment sentiment       = 6;
+}
+
+//
+// A dependency graph representation.
+//
+message DependencyGraph {
+  message Node {
+    required uint32 sentenceIndex  = 1;
+    required uint32 index          = 2;
+    optional uint32 copyAnnotation = 3;
+    optional uint32 emptyIndex     = 4;
+  }
+
+  message Edge {
+    required uint32 source      = 1;
+    required uint32 target      = 2;
+    optional string dep         = 3;
+    optional bool   isExtra     = 4;
+    optional uint32 sourceCopy  = 5;
+    optional uint32 targetCopy  = 6;
+    optional uint32 sourceEmpty = 8;
+    optional uint32 targetEmpty = 9;
+    optional Language language  = 7 [default=Unknown];
+  }
+
+  repeated Node     node     = 1;
+  repeated Edge     edge     = 2;
+  repeated uint32   root     = 3 [packed=true];
+  // optional: if this graph message is not part of a larger context,
+  // the tokens will help reconstruct the actual sentence
+  repeated Token    token    = 4;
+  // The values in this field will index directly into the node list
+  // This is useful so that additional information such as emptyIndex
+  // can be considered without having to pass it around a second time
+  repeated uint32   rootNode = 5 [packed=true];
+}
+
+//
+// A coreference chain.
+// These fields are not *really* optional. CoreNLP will crash without them.
+//
+message CorefChain {
+  message CorefMention {
+    optional int32  mentionID          = 1;
+    optional string mentionType        = 2;
+    optional string number             = 3;
+    optional string gender             = 4;
+    optional string animacy            = 5;
+    optional uint32 beginIndex         = 6;
+    optional uint32 endIndex           = 7;
+    optional uint32 headIndex          = 9;
+    optional uint32 sentenceIndex      = 10;
+    optional uint32 position           = 11;  // the second element of position
+  }
+
+  required int32        chainID        = 1;
+  repeated CorefMention mention        = 2;
+  required uint32       representative = 3;
+}
+
+//
+// a mention
+//
+
+message Mention {
+  optional int32 mentionID             = 1;
+  optional string mentionType          = 2;
+  optional string number               = 3;
+  optional string gender               = 4;
+  optional string animacy              = 5;
+  optional string person               = 6;
+  optional uint32 startIndex           = 7;
+  optional uint32 endIndex             = 9;
+  optional int32 headIndex             = 10;
+  optional string headString           = 11;
+  optional string nerString            = 12;
+  optional int32 originalRef           = 13;
+  optional int32 goldCorefClusterID    = 14;
+  optional int32 corefClusterID        = 15;
+  optional int32 mentionNum            = 16;
+  optional int32 sentNum               = 17;
+  optional int32 utter                 = 18;
+  optional int32 paragraph             = 19;
+  optional bool isSubject              = 20;
+  optional bool isDirectObject         = 21;
+  optional bool isIndirectObject       = 22;
+  optional bool isPrepositionObject    = 23;
+  optional bool hasTwin                = 24;
+  optional bool generic                = 25;
+  optional bool isSingleton            = 26;
+  optional bool hasBasicDependency     = 27;
+  optional bool hasEnhancedDependency  = 28;
+  optional bool hasContextParseTree    = 29;
+  optional IndexedWord headIndexedWord = 30;
+  optional IndexedWord   dependingVerb = 31;
+  optional IndexedWord       headWord  = 32;
+  optional SpeakerInfo    speakerInfo  = 33;
+
+  repeated IndexedWord sentenceWords   = 50;
+  repeated IndexedWord originalSpan    = 51;
+  repeated string dependents           = 52;
+  repeated string preprocessedTerms    = 53;
+  repeated int32 appositions           = 54;
+  repeated int32 predicateNominatives  = 55;
+  repeated int32 relativePronouns      = 56;
+  repeated int32 listMembers           = 57;
+  repeated int32 belongToLists         = 58;
+
+}
+
+//
+// store the position (sentence, token index) of a CoreLabel
+//
+
+message IndexedWord {
+  optional  int32 sentenceNum          = 1;
+  optional  int32 tokenIndex           = 2;
+  optional  int32 docID                = 3;
+  optional uint32 copyCount            = 4;
+}
+
+//
+// speaker info, this is used for Mentions
+//
+
+message SpeakerInfo {
+  optional string speakerName          = 1;
+  repeated int32 mentions              = 2;
+}
+
+//
+// A Span of text
+//
+message Span {
+  required uint32 begin      = 1;
+  required uint32 end        = 2;
+}
+
+//
+// A Timex object, representing a temporal expression (TIMe EXpression)
+// These fields are not *really* optional. CoreNLP will crash without them.
+//
+message Timex {
+  optional string value      = 1;
+  optional string altValue   = 2;
+  optional string text       = 3;
+  optional string type       = 4;
+  optional string tid        = 5;
+  optional uint32 beginPoint = 6;
+  optional uint32 endPoint   = 7;
+}
+
+//
+// A representation of an entity in a relation.
+// This corresponds to the EntityMention, and more broadly the
+// ExtractionObject classes.
+//
+message Entity {
+  optional uint32 headStart      = 6;
+  optional uint32 headEnd        = 7;
+  optional string mentionType    = 8;
+  optional string normalizedName = 9;
+  optional uint32 headTokenIndex = 10;
+  optional string corefID        = 11;
+  // inherited from ExtractionObject
+  optional string objectID       = 1;
+  optional uint32 extentStart    = 2;
+  optional uint32 extentEnd      = 3;
+  optional string type           = 4;
+  optional string subtype        = 5;
+  // Implicit
+  //       uint32 sentence       @see implicit in sentence
+}
+
+//
+// A representation of a relation, mirroring RelationMention
+//
+message Relation {
+  repeated string argName   = 6;
+  repeated Entity arg       = 7;
+  optional string signature = 8;
+  // inherited from ExtractionObject
+  optional string objectID = 1;
+  optional uint32 extentStart    = 2;
+  optional uint32 extentEnd      = 3;
+  optional string type           = 4;
+  optional string subtype        = 5;
+  // Implicit
+  //       uint32 sentence       @see implicit in sentence
+}
+
+//
+// A Natural Logic operator
+//
+message Operator {
+  required string name                = 1;
+  required int32  quantifierSpanBegin = 2;
+  required int32  quantifierSpanEnd   = 3;
+  required int32  subjectSpanBegin    = 4;
+  required int32  subjectSpanEnd      = 5;
+  required int32  objectSpanBegin     = 6;
+  required int32  objectSpanEnd       = 7;
+}
+
+//
+// The seven informative Natural Logic relations
+//
+enum NaturalLogicRelation {
+  EQUIVALENCE        = 0;
+  FORWARD_ENTAILMENT = 1;
+  REVERSE_ENTAILMENT = 2;
+  NEGATION           = 3;
+  ALTERNATION        = 4;
+  COVER              = 5;
+  INDEPENDENCE       = 6;
+}
+
+//
+// The polarity of a word, according to Natural Logic
+//
+message Polarity {
+  required NaturalLogicRelation projectEquivalence       = 1;
+  required NaturalLogicRelation projectForwardEntailment = 2;
+  required NaturalLogicRelation projectReverseEntailment = 3;
+  required NaturalLogicRelation projectNegation          = 4;
+  required NaturalLogicRelation projectAlternation       = 5;
+  required NaturalLogicRelation projectCover             = 6;
+  required NaturalLogicRelation projectIndependence      = 7;
+}
+
+//
+// An NER mention in the text
+//
+message NERMention {
+  optional uint32 sentenceIndex                 = 1;
+  required uint32 tokenStartInSentenceInclusive = 2;
+  required uint32 tokenEndInSentenceExclusive   = 3;
+  required string ner                           = 4;
+  optional string normalizedNER                 = 5;
+  optional string entityType                    = 6;
+  optional Timex  timex                         = 7;
+  optional string wikipediaEntity               = 8;
+  optional string gender                        = 9;
+  optional uint32 entityMentionIndex            = 10;
+  optional uint32 canonicalEntityMentionIndex   = 11;
+  optional string entityMentionText             = 12;
+}
+
+//
+// An entailed sentence fragment.
+// Created by the openie annotator.
+//
+message SentenceFragment {
+  repeated uint32 tokenIndex     = 1;
+  optional uint32 root           = 2;
+  optional bool   assumedTruth   = 3;
+  optional double score          = 4;
+}
+
+
+//
+// The index of a token in a document, including the sentence
+// index and the offset.
+//
+message TokenLocation {
+ optional uint32 sentenceIndex = 1;
+ optional uint32 tokenIndex    = 2;
+
+}
+
+
+//
+// An OpenIE relation triple.
+// Created by the openie annotator.
+//
+message RelationTriple {
+  optional string          subject        = 1;   // The surface form of the subject
+  optional string          relation       = 2;   // The surface form of the relation (required)
+  optional string          object         = 3;   // The surface form of the object
+  optional double          confidence     = 4;   // The [optional] confidence of the extraction
+  repeated TokenLocation   subjectTokens  = 13; // The tokens comprising the subject of the triple
+  repeated TokenLocation   relationTokens = 14; // The tokens comprising the relation of the triple
+  repeated TokenLocation   objectTokens   = 15; // The tokens comprising the object of the triple
+  optional DependencyGraph tree           = 8;   // The dependency graph fragment for this triple
+  optional bool            istmod         = 9;   // If true, this expresses an implicit tmod relation
+  optional bool            prefixBe       = 10;  // If true, this relation string is missing a 'be' prefix
+  optional bool            suffixBe       = 11;  // If true, this relation string is missing a 'be' suffix
+  optional bool            suffixOf       = 12;  // If true, this relation string is missing a 'of' prefix
+}
+
+
+//
+// A map from strings to strings.
+// Used, minimally, in the CoNLLU featurizer
+//
+message MapStringString {
+  repeated string key   = 1;
+  repeated string value = 2;
+}
+
+//
+// A map from integers to strings.
+// Used, minimally, in the CoNLLU featurizer
+//
+message MapIntString {
+  repeated uint32 key   = 1;
+  repeated string value = 2;
+}
+
+//
+// Store section info
+//
+
+message Section {
+  required uint32 charBegin         = 1;
+  required uint32 charEnd           = 2;
+  optional string author            = 3;
+  repeated uint32 sentenceIndexes   = 4;
+  optional string datetime          = 5;
+  repeated Quote quotes             = 6;
+  optional uint32 authorCharBegin   = 7;
+  optional uint32 authorCharEnd     = 8;
+  required Token xmlTag             = 9;
+}
+
+
+
+// A message for requesting a semgrex
+// Each sentence stores information about the tokens making up the
+// corresponding graph
+// An alternative would have been to use the existing Document or
+// Sentence classes, but the problem with that is it would be
+// ambiguous which dependency object to use.
+message SemgrexRequest {
+  message Dependencies {
+    repeated Token           token       = 1;
+    required DependencyGraph graph       = 2;
+  }
+
+  repeated string            semgrex     = 1;
+  repeated Dependencies      query       = 2;
+}
+
+// The response from running a semgrex
+// If you pass in M semgrex expressions and N dependency graphs,
+// this returns MxN nested results.  Each SemgrexResult can match
+// multiple times in one graph
+//
+// You may want to send multiple semgrexes per query because
+// translating large numbers of dependency graphs to protobufs
+// will be expensive, so doing several queries at once will save time
+message SemgrexResponse {
+  message NamedNode {
+    required string          name        = 1;
+    required int32           matchIndex  = 2;
+  }
+
+  message NamedRelation {
+    required string          name        = 1;
+    required string          reln        = 2;
+  }
+
+  message NamedEdge {
+    required string          name        = 1;
+    required int32           source      = 2;
+    required int32           target      = 3;
+    optional string          reln        = 4;
+    optional bool            isExtra     = 5;
+    optional uint32          sourceCopy  = 6;
+    optional uint32          targetCopy  = 7;
+  }
+
+  message Match {
+    required int32           matchIndex   = 1;
+    repeated NamedNode       node         = 2;
+    repeated NamedRelation   reln         = 3;
+    repeated NamedEdge       edge         = 6;
+
+    // when processing multiple dependency graphs at once,
+    // which dependency graph this applies to
+    // indexed from 0
+    optional int32           graphIndex   = 4;
+    // index of the semgrex expression this match applies to
+    // indexed from 0
+    optional int32           semgrexIndex = 5;
+  }
+
+  message SemgrexResult {
+    repeated Match           match       = 1;
+  }
+
+  message GraphResult {
+    repeated SemgrexResult   result      = 1;
+  }
+
+  repeated GraphResult       result      = 1;
+}
+
+
+// A message for processing an Ssurgeon
+// Each sentence stores information about the tokens making up the
+// corresponding graph
+// An alternative would have been to use the existing Document or
+// Sentence classes, but the problem with that is it would be
+// ambiguous which dependency object to use.  Another problem
+// is that if the intent is to use multiple graphs from a
+// Sentence, then edits to the nodes of one graph would show up
+// in the nodes of the other graph (same backing CoreLabels)
+// and the operations themselves may not have the intended effect.
+// The Ssurgeon is composed of two pieces, the semgrex and the
+// ssurgeon operations, along with some optional documentation.
+message SsurgeonRequest {
+  message Ssurgeon {
+    optional string          semgrex     = 1;
+    repeated string          operation   = 2;
+    optional string          id          = 3;
+    optional string          notes       = 4;
+    optional string          language    = 5;
+  }
+
+  repeated Ssurgeon          ssurgeon    = 1;
+  repeated DependencyGraph   graph       = 2;
+}
+
+message SsurgeonResponse {
+  message SsurgeonResult {
+    optional DependencyGraph graph      = 1;
+    optional bool            changed    = 2;
+  }
+
+  repeated SsurgeonResult    result      = 1;
+}
+
+// It's possible to send in a whole document, but we
+// only care about the Sentences and Tokens
+message TokensRegexRequest {
+  required Document          doc         = 1;
+  repeated string            pattern     = 2;
+}
+
+// The result will be a nested structure:
+// repeated PatternMatch, one for each pattern
+// each PatternMatch has a repeated Match,
+// which tells you which sentence matched and where
+message TokensRegexResponse {
+  message MatchLocation {
+    optional string          text        = 1;
+    optional int32           begin       = 2;
+    optional int32           end         = 3;
+  }
+
+  message Match {
+    required int32           sentence    = 1;
+    required MatchLocation   match       = 2;
+    repeated MatchLocation   group       = 3;
+  }
+
+  message PatternMatch {
+    repeated Match           match       = 1;
+  }
+
+  repeated PatternMatch      match       = 1;
+}
+
+// A protobuf which allows to pass in a document with basic
+// dependencies to be converted to enhanced
+message DependencyEnhancerRequest {
+  required Document          document           = 1;
+
+  oneof ref {
+    Language          language           = 2;
+    // The expected value of this is a regex which matches relative pronouns
+    string            relativePronouns   = 3;
+  }
+}
+
+// A version of ParseTree with a flattened structure so that deep trees
+// don't exceed the protobuf stack depth
+message FlattenedParseTree {
+  message Node {
+    oneof contents {
+      bool              openNode           = 1;
+      bool              closeNode          = 2;
+      string            value              = 3;
+    }
+
+    optional double     score              = 4;
+  }
+
+  repeated Node         nodes              = 1;
+}
+
+// A protobuf for calling the java constituency parser evaluator from elsewhere
+message EvaluateParserRequest {
+  message ParseResult {
+    required FlattenedParseTree         gold           = 1;
+    // repeated so you can send in kbest parses, if your parser handles that
+    // note that this already includes a score field
+    repeated FlattenedParseTree         predicted      = 2;
+  }
+
+  repeated ParseResult         treebank       = 1;
+}
+
+message EvaluateParserResponse {
+  required double              f1             = 1;
+  optional double              kbestF1        = 2;
+  // keep track of the individual tree F1 scores
+  repeated double              treeF1         = 3;
+}
+
+
+// A protobuf for running Tsurgeon operations on constituency trees
+message TsurgeonRequest {
+  message Operation {
+    required string                tregex         = 1;
+    repeated string                tsurgeon       = 2;
+  }
+  repeated Operation               operations     = 1;
+  repeated FlattenedParseTree      trees          = 2;
+}
+
+// The results of the Tsurgeon operation
+message TsurgeonResponse {
+  repeated FlattenedParseTree      trees          = 1;
+}
+
+// Sent in Morphology requests - a stream of sentences with tagged words
+message MorphologyRequest {
+  message TaggedWord {
+    required string                word           = 1;
+    optional string                xpos           = 2;
+  }
+
+  repeated TaggedWord              words          = 1;
+}
+
+// Sent back from the Morphology request - the words and their tags
+message MorphologyResponse {
+  message WordTagLemma {
+    required string                word           = 1;
+    optional string                xpos           = 2;
+    required string                lemma          = 3;
+  }
+
+  repeated WordTagLemma            words          = 1;
+}
+
+
+// A request for converting constituency trees to dependency graphs
+message DependencyConverterRequest {
+  repeated FlattenedParseTree      trees          = 1;
+}
+
+// The result of using the CoreNLP dependency converter.
+// One graph per tree
+message DependencyConverterResponse {
+  message DependencyConversion {
+    required DependencyGraph       graph          = 1;
+    optional FlattenedParseTree    tree           = 2;
+  }
+
+  repeated DependencyConversion    conversions         = 1;
+}
+

stanza/scripts/config.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Set environment variables for the training and testing of stanza modules.
+
+# Set UDBASE to the location of UD data folder
+# The data should be CoNLL-U format
+# For details, see
+#   http://universaldependencies.org/conll18/data.html (CoNLL-18 UD data)
+#   https://universaldependencies.org/
+# When rebuilding models based on Universal Dependencies, download the
+#   UD data to some directory, set UDBASE to that directory, and
+#   uncomment this line.  Alternatively, put UDBASE in your shell
+#   config, Windows env variables, etc as relevant.
+# export UDBASE=/path/to/UD
+
+# Set NERBASE to the location of NER data folder
+# The data should be BIO format or convertable to that format
+# For details, see https://www.aclweb.org/anthology/W03-0419.pdf (CoNLL-03 NER paper)
+# There are other NER datasets, supported in
+#   stanza/utils/datasets/ner/prepare_ner_dataset.py
+# If rebuilding NER data, choose a location for the NER directory
+#   and set NERBASE to that variable.
+# export NERBASE=/path/to/NER
+
+# Set CONSTITUENCY_BASE to the location of NER data folder
+# The data will be in some dataset-specific format
+# There is a conversion script which will turn this
+#   into a PTB style format
+#   stanza/utils/datasets/constituency/prepare_con_dataset.py
+# If processing constituency data, choose a location for the CON data
+#   and set CONSTITUENCY_BASE to that variable.
+# export CONSTITUENCY_BASE=/path/to/CON
+
+# Set directories to store processed training/evaluation files
+# $DATA_ROOT is a default home for where all the outputs from the
+#   preparation scripts will go.  The training scripts will then look
+#   for the stanza formatted data in that directory.
+export DATA_ROOT=./data
+export TOKENIZE_DATA_DIR=$DATA_ROOT/tokenize
+export MWT_DATA_DIR=$DATA_ROOT/mwt
+export LEMMA_DATA_DIR=$DATA_ROOT/lemma
+export POS_DATA_DIR=$DATA_ROOT/pos
+export DEPPARSE_DATA_DIR=$DATA_ROOT/depparse
+export ETE_DATA_DIR=$DATA_ROOT/ete
+export NER_DATA_DIR=$DATA_ROOT/ner
+export CHARLM_DATA_DIR=$DATA_ROOT/charlm
+export CONSTITUENCY_DATA_DIR=$DATA_ROOT/constituency
+export SENTIMENT_DATA_DIR=$DATA_ROOT/sentiment
+
+# Set directories to store external word vector data
+export WORDVEC_DIR=./extern_data/wordvec

stanza/scripts/download_vectors.sh

@@ -0,0 +1,93 @@
+#!/bin/bash
+#
+# Download word vector files for all supported languages. Run as:
+#   ./download_vectors.sh WORDVEC_DIR
+# where WORDVEC_DIR is the target directory to store the word vector data.
+
+# check arguments
+: ${1?"Usage: $0 WORDVEC_DIR"}
+WORDVEC_DIR=$1
+
+# constants and functions
+CONLL17_URL="https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-1989/word-embeddings-conll17.tar"
+CONLL17_TAR="word-embeddings-conll17.tar"
+
+FASTTEXT_BASE_URL="https://dl.fbaipublicfiles.com/fasttext/vectors-wiki"
+
+# TODO: some fasttext vectors are now at
+# https://fasttext.cc/docs/en/pretrained-vectors.html
+# there are also vectors for
+# Welsh, Icelandic, Thai, Sanskrit
+# https://fasttext.cc/docs/en/crawl-vectors.html
+
+# We get the Armenian word vectors from here:
+# https://github.com/ispras-texterra/word-embeddings-eval-hy
+# https://arxiv.org/ftp/arxiv/papers/1906/1906.03134.pdf
+# In particular, the glove model (dogfooding):
+# https://at.ispras.ru/owncloud/index.php/s/pUUiS1l1jGKNax3/download
+# These vectors improved F1 by about 1 on various tasks for Armenian
+# and had much better coverage of Western Armenian
+
+# For Eryza, we use word vectors available here:
+# https://github.com/mokha/semantics
+# @incollection{Alnajjar_2021,
+#   doi = {10.31885/9789515150257.24},
+#   url = {https://doi.org/10.31885%2F9789515150257.24},
+#   year = 2021,
+#   month = {mar},
+#   publisher = {University of Helsinki},
+#   pages = {275--288},
+#   author = {Khalid Alnajjar},
+#   title = {When Word Embeddings Become Endangered},
+#   booktitle = {Multilingual Facilitation}
+# }
+
+declare -a FASTTEXT_LANG=("Afrikaans" "Breton" "Buryat" "Chinese" "Faroese" "Gothic" "Kurmanji" "North_Sami" "Serbian" "Upper_Sorbian")
+declare -a FASTTEXT_CODE=("af" "br" "bxr" "zh" "fo" "got" "ku" "se" "sr" "hsb")
+declare -a LOCAL_CODE=("af" "br" "bxr" "zh" "fo" "got" "kmr" "sme" "sr" "hsb")
+
+color_green='\033[32;1m'
+color_clear='\033[0m' # No Color
+function msg() {
+    echo -e "${color_green}$@${color_clear}"
+}
+
+function prepare_fasttext_vec() {
+    lang=$1
+    ftcode=$2
+    code=$3
+
+    cwd=$(pwd)
+    mkdir -p $lang
+    cd $lang
+    msg "=== Downloading fasttext vector file for ${lang}..."
+    url="${FASTTEXT_BASE_URL}/wiki.${ftcode}.vec"
+    fname="${code}.vectors"
+    wget $url -O $fname
+
+    msg "=== Compressing file ${fname}..."
+    xz $fname
+    cd $cwd
+}
+
+# do the actual work
+mkdir -p $WORDVEC_DIR
+cd $WORDVEC_DIR
+
+msg "Downloading CONLL17 word vectors. This may take a while..."
+wget $CONLL17_URL -O $CONLL17_TAR
+
+msg "Extracting CONLL17 word vector files..."
+tar -xvf $CONLL17_TAR
+rm $CONLL17_TAR
+
+msg "Preparing fasttext vectors for the rest of the languages."
+for (( i=0; i<${#FASTTEXT_LANG[*]}; ++i)); do
+    prepare_fasttext_vec ${FASTTEXT_LANG[$i]} ${FASTTEXT_CODE[$i]} ${LOCAL_CODE[$i]}
+done
+
+# handle old french
+mkdir Old_French
+ln -s French/fr.vectors.xz Old_French/fro.vectors.xz
+
+msg "All done."

stanza/stanza/_version.py

@@ -0,0 +1,4 @@
+""" Single source of truth for version number """
+
+__version__ = "1.10.1"
+__resources_version__ = '1.10.0'

stanza/stanza/models/_training_logging.py

@@ -0,0 +1,4 @@
+import logging
+
+logger = logging.getLogger('stanza')
+logger.setLevel(logging.DEBUG)

stanza/stanza/models/classifier.py

@@ -0,0 +1,660 @@
+import argparse
+import ast
+import logging
+import os
+import random
+import re
+from enum import Enum
+
+import torch
+import torch.nn as nn
+
+from stanza.models.common import loss
+from stanza.models.common import utils
+from stanza.models.pos.vocab import CharVocab
+
+import stanza.models.classifiers.data as data
+from stanza.models.classifiers.trainer import Trainer
+from stanza.models.classifiers.utils import WVType, ExtraVectors, ModelType
+from stanza.models.common.peft_config import add_peft_args, resolve_peft_args
+
+from stanza.utils.confusion import format_confusion, confusion_to_accuracy, confusion_to_macro_f1
+
+
+class Loss(Enum):
+    CROSS = 1
+    WEIGHTED_CROSS = 2
+    LOG_CROSS = 3
+    FOCAL = 4
+
+class DevScoring(Enum):
+    ACCURACY = 'ACC'
+    WEIGHTED_F1 = 'WF'
+
+logger = logging.getLogger('stanza')
+tlogger = logging.getLogger('stanza.classifiers.trainer')
+
+logging.getLogger('elmoformanylangs').setLevel(logging.WARNING)
+
+DEFAULT_TRAIN='data/sentiment/en_sstplus.train.txt'
+DEFAULT_DEV='data/sentiment/en_sst3roots.dev.txt'
+DEFAULT_TEST='data/sentiment/en_sst3roots.test.txt'
+
+"""A script for training and testing classifier models, especially on the SST.
+
+If you run the script with no arguments, it will start trying to train
+a sentiment model.
+
+python3 -m stanza.models.classifier
+
+This requires the sentiment dataset to be in an `extern_data`
+directory, such as by symlinking it from somewhere else.
+
+The default model is a CNN where the word vectors are first mapped to
+channels with filters of a few different widths, those channels are
+maxpooled over the entire sentence, and then the resulting pools have
+fully connected layers until they reach the number of classes in the
+training data.  You can see the defaults in the options below.
+
+https://arxiv.org/abs/1408.5882
+
+(Currently the CNN is the only sentence classifier implemented.)
+
+To train with a more complicated CNN arch:
+
+nohup python3 -u -m stanza.models.classifier --max_epochs 400 --filter_channels 1000 --fc_shapes 400,100 > FC41.out 2>&1 &
+
+You can train models with word vectors other than the default word2vec.  For example:
+
+ nohup python3 -u -m stanza.models.classifier  --wordvec_type google --wordvec_dir extern_data/google --max_epochs 200 --filter_channels 1000 --fc_shapes 200,100 --base_name FC21_google > FC21_google.out 2>&1 &
+
+A model trained on the 5 class dataset can be tested on the 2 class dataset with a command line like this:
+
+python3 -u -m stanza.models.classifier  --no_train --load_name saved_models/classifier/sst_en_ewt_FS_3_4_5_C_1000_FC_400_100_classifier.E0165-ACC41.87.pt --test_file data/sentiment/en_sst2roots.test.txt --test_remap_labels "{0:0, 1:0, 3:1, 4:1}"
+
+python3 -u -m stanza.models.classifier  --wordvec_type google --wordvec_dir extern_data/google --no_train --load_name saved_models/classifier/FC21_google_en_ewt_FS_3_4_5_C_1000_FC_200_100_classifier.E0189-ACC45.87.pt --test_file data/sentiment/en_sst2roots.test.txt --test_remap_labels "{0:0, 1:0, 3:1, 4:1}"
+
+A model trained on the 3 class dataset can be tested on the 2 class dataset with a command line like this:
+
+python3 -u -m stanza.models.classifier  --wordvec_type google --wordvec_dir extern_data/google --no_train --load_name saved_models/classifier/FC21_3C_google_en_ewt_FS_3_4_5_C_1000_FC_200_100_classifier.E0101-ACC68.94.pt --test_file data/sentiment/en_sst2roots.test.txt --test_remap_labels "{0:0, 2:1}"
+
+To train models on combined 3 class datasets:
+
+nohup python3 -u -m stanza.models.classifier --max_epochs 400 --filter_channels 1000 --fc_shapes 400,100 --base_name FC41_3class  --extra_wordvec_method CONCAT --extra_wordvec_dim 200  --train_file data/sentiment/en_sstplus.train.txt --dev_file data/sentiment/en_sst3roots.dev.txt --test_file data/sentiment/en_sst3roots.test.txt > FC41_3class.out 2>&1 &
+
+This tests that model:
+
+python3 -u -m stanza.models.classifier --no_train --load_name en_sstplus.pt --test_file data/sentiment/en_sst3roots.test.txt
+
+Here is an example for training a model in a different language:
+
+nohup python3 -u -m stanza.models.classifier --max_epochs 400 --filter_channels 1000 --fc_shapes 400,100 --base_name FC41_german  --train_file data/sentiment/de_sb10k.train.txt --dev_file data/sentiment/de_sb10k.dev.txt --test_file data/sentiment/de_sb10k.test.txt --shorthand de_sb10k --min_train_len 3 --extra_wordvec_method CONCAT --extra_wordvec_dim 100 > de_sb10k.out 2>&1 &
+
+This uses more data, although that wound up being worse for the German model:
+
+nohup python3 -u -m stanza.models.classifier --max_epochs 400 --filter_channels 1000 --fc_shapes 400,100 --base_name FC41_german  --train_file data/sentiment/de_sb10k.train.txt,data/sentiment/de_scare.train.txt,data/sentiment/de_usage.train.txt --dev_file data/sentiment/de_sb10k.dev.txt --test_file data/sentiment/de_sb10k.test.txt --shorthand de_sb10k --min_train_len 3 --extra_wordvec_method CONCAT --extra_wordvec_dim 100 > de_sb10k.out 2>&1 &
+
+nohup python3 -u -m stanza.models.classifier --max_epochs 400 --filter_channels 1000 --fc_shapes 400,100 --base_name FC41_chinese --train_file data/sentiment/zh_ren.train.txt --dev_file data/sentiment/zh_ren.dev.txt --test_file data/sentiment/zh_ren.test.txt --shorthand zh_ren --wordvec_type fasttext --extra_wordvec_method SUM --wordvec_pretrain_file ../stanza_resources/zh-hans/pretrain/gsdsimp.pt > zh_ren.out 2>&1 &
+
+nohup python3 -u -m stanza.models.classifier --max_epochs 400 --filter_channels 1000 --fc_shapes 400,100 --save_name vi_vsfc.pt  --train_file data/sentiment/vi_vsfc.train.json --dev_file data/sentiment/vi_vsfc.dev.json --test_file data/sentiment/vi_vsfc.test.json --shorthand vi_vsfc --wordvec_pretrain_file ../stanza_resources/vi/pretrain/vtb.pt --wordvec_type word2vec --extra_wordvec_method SUM --dev_eval_scoring WEIGHTED_F1 > vi_vsfc.out 2>&1 &
+
+python3 -u -m stanza.models.classifier --no_train --test_file extern_data/sentiment/vietnamese/_UIT-VSFC/test.txt --shorthand vi_vsfc --wordvec_pretrain_file ../stanza_resources/vi/pretrain/vtb.pt --wordvec_type word2vec --load_name vi_vsfc.pt
+"""
+
+def convert_fc_shapes(arg):
+    """
+    Returns a tuple of sizes to use in FC layers.
+
+    For examples, converts "100" -> (100,)
+    "100,200" -> (100,200)
+    """
+    arg = arg.strip()
+    if not arg:
+        return ()
+    arg = ast.literal_eval(arg)
+    if isinstance(arg, int):
+        return (arg,)
+    if isinstance(arg, tuple):
+        return arg
+    return tuple(arg)
+
+# For the most part, these values are for the constituency parser.
+# Only the WD for adadelta is originally for sentiment
+# Also LR for adadelta and madgrad
+
+# madgrad learning rate experiment on sstplus
+# note that the hyperparameters are not cross-validated in tandem, so
+# later changes may make some earlier experiments slightly out of date
+# LR
+#   0.01         failed to converge
+#   0.004        failed to converge
+#   0.003        0.5572
+#   0.002        failed to converge
+#   0.001        0.6857
+#   0.0008       0.6799
+#   0.0005       0.6849
+#   0.00025      0.6749
+#   0.0001       0.6746
+#   0.00001      0.6536
+#   0.000001     0.6267
+# LR 0.001 produced the best model, but it does occasionally fail to
+# converge to a working model, so we set the default to 0.0005 instead
+DEFAULT_LEARNING_RATES = { "adamw": 0.0002, "adadelta": 1.0, "sgd": 0.001, "adabelief": 0.00005, "madgrad": 0.0005, "sgd": 0.001 }
+DEFAULT_LEARNING_EPS = { "adabelief": 1e-12, "adadelta": 1e-6, "adamw": 1e-8 }
+DEFAULT_LEARNING_RHO = 0.9
+DEFAULT_MOMENTUM = { "madgrad": 0.9, "sgd": 0.9 }
+DEFAULT_WEIGHT_DECAY = { "adamw": 0.05, "adadelta": 0.0001, "sgd": 0.01, "adabelief": 1.2e-6, "madgrad": 2e-6 }
+
+def build_argparse():
+    """
+    Build the argparse for the classifier.
+
+    Refactored so that other utility scripts can use the same parser if needed.
+    """
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument('--train', dest='train', default=True, action='store_true', help='Train the model (default)')
+    parser.add_argument('--no_train', dest='train', action='store_false', help="Don't train the model")
+
+    parser.add_argument('--shorthand', type=str, default='en_ewt', help="Treebank shorthand, eg 'en' for English")
+
+    parser.add_argument('--load_name', type=str, default=None, help='Name for loading an existing model')
+    parser.add_argument('--save_dir', type=str, default='saved_models/classifier', help='Root dir for saving models.')
+    parser.add_argument('--save_name', type=str, default="{shorthand}_{embedding}_{bert_finetuning}_{classifier_type}_classifier.pt", help='Name for saving the model')
+
+    parser.add_argument('--checkpoint_save_name', type=str, default=None, help="File name to save the most recent checkpoint")
+    parser.add_argument('--no_checkpoint', dest='checkpoint', action='store_false', help="Don't save checkpoints")
+
+    parser.add_argument('--save_intermediate_models', default=False, action='store_true',
+                        help='Save all intermediate models - this can be a lot!')
+
+    parser.add_argument('--train_file', type=str, default=DEFAULT_TRAIN, help='Input file(s) to train a model from.  Each line is an example.  Should go <label> <tokenized sentence>.  Comma separated list.')
+    parser.add_argument('--dev_file', type=str, default=DEFAULT_DEV, help='Input file(s) to use as the dev set.')
+    parser.add_argument('--test_file', type=str, default=DEFAULT_TEST, help='Input file(s) to use as the test set.')
+    parser.add_argument('--output_predictions', default=False, action='store_true', help='Output predictions when running the test set')
+    parser.add_argument('--max_epochs', type=int, default=100)
+    parser.add_argument('--tick', type=int, default=50)
+
+    parser.add_argument('--model_type', type=lambda x: ModelType[x.upper()], default=ModelType.CNN,
+                        help='Model type to use.  Options: %s' % " ".join(x.name for x in ModelType))
+
+    parser.add_argument('--filter_sizes', default=(3,4,5), type=ast.literal_eval, help='Filter sizes for the layer after the word vectors')
+    parser.add_argument('--filter_channels', default=1000, type=ast.literal_eval, help='Number of channels for layers after the word vectors.  Int for same number of channels (scaled by width) for each filter, or tuple/list for exact lengths for each filter')
+    parser.add_argument('--fc_shapes', default="400,100", type=convert_fc_shapes, help='Extra fully connected layers to put after the initial filters.  If set to blank, will FC directly from the max pooling to the output layer.')
+    parser.add_argument('--dropout', default=0.5, type=float, help='Dropout value to use')
+
+    parser.add_argument('--batch_size', default=50, type=int, help='Batch size when training')
+    parser.add_argument('--batch_single_item', default=200, type=int, help='Items of this size go in their own batch')
+    parser.add_argument('--dev_eval_batches', default=2000, type=int, help='Run the dev set after this many train batches.  Set to 0 to only do it once per epoch')
+    parser.add_argument('--dev_eval_scoring', type=lambda x: DevScoring[x.upper()], default=DevScoring.WEIGHTED_F1,
+                        help=('Scoring method to use for choosing the best model.  Options: %s' %
+                              " ".join(x.name for x in DevScoring)))
+
+    parser.add_argument('--weight_decay', default=None, type=float, help='Weight decay (eg, l2 reg) to use in the optimizer')
+    parser.add_argument('--learning_rate', default=None, type=float, help='Learning rate to use in the optimizer')
+    parser.add_argument('--momentum', default=None, type=float, help='Momentum to use in the optimizer')
+
+    parser.add_argument('--optim', default='adadelta', choices=['adadelta', 'madgrad', 'sgd'], help='Optimizer type: SGD, Adadelta, or madgrad.  Highly recommend to install madgrad and use that')
+
+    parser.add_argument('--test_remap_labels', default=None, type=ast.literal_eval,
+                        help='Map of which label each classifier label should map to.  For example, "{0:0, 1:0, 3:1, 4:1}" to map a 5 class sentiment test to a 2 class.  Any labels not mapped will be considered wrong')
+    parser.add_argument('--forgive_unmapped_labels', dest='forgive_unmapped_labels', default=True, action='store_true',
+                        help='When remapping labels, such as from 5 class to 2 class, pick a different label if the first guess is not remapped.')
+    parser.add_argument('--no_forgive_unmapped_labels', dest='forgive_unmapped_labels', action='store_false',
+                        help="When remapping labels, such as from 5 class to 2 class, DON'T pick a different label if the first guess is not remapped.")
+
+    parser.add_argument('--loss', type=lambda x: Loss[x.upper()], default=Loss.CROSS,
+                        help="Whether to use regular cross entropy or scale it by 1/log(quantity)")
+    parser.add_argument('--loss_focal_gamma', default=2, type=float, help='gamma value for a focal loss')
+    parser.add_argument('--min_train_len', type=int, default=0,
+                        help="Filter sentences less than this length")
+
+    parser.add_argument('--pretrain_max_vocab', type=int, default=-1)
+    parser.add_argument('--wordvec_pretrain_file', type=str, default=None, help='Exact name of the pretrain file to read')
+    parser.add_argument('--wordvec_raw_file', type=str, default=None, help='Exact name of the raw wordvec file to read')
+    parser.add_argument('--wordvec_dir', type=str, default='extern_data/wordvec', help='Directory of word vectors')
+    parser.add_argument('--wordvec_type', type=lambda x: WVType[x.upper()], default='word2vec', help='Different vector types have different options, such as google 300d replacing numbers with #')
+    parser.add_argument('--extra_wordvec_dim', type=int, default=0, help="Extra dim of word vectors - will be trained")
+    parser.add_argument('--extra_wordvec_method', type=lambda x: ExtraVectors[x.upper()], default='sum', help='How to train extra dimensions of word vectors, if at all')
+    parser.add_argument('--extra_wordvec_max_norm', type=float, default=None, help="Max norm for initializing the extra vectors")
+
+    parser.add_argument('--charlm_forward_file', type=str, default=None, help="Exact path to use for forward charlm")
+    parser.add_argument('--charlm_backward_file', type=str, default=None, help="Exact path to use for backward charlm")
+    parser.add_argument('--charlm_projection', type=int, default=None, help="Project the charlm values to this dimension")
+    parser.add_argument('--char_lowercase', dest='char_lowercase', action='store_true', help="Use lowercased characters in character model.")
+
+    parser.add_argument('--elmo_model', default='extern_data/manyelmo/english', help='Directory with elmo model')
+    parser.add_argument('--use_elmo', dest='use_elmo', default=False, action='store_true', help='Use an elmo model as a source of parameters')
+    parser.add_argument('--elmo_projection', type=int, default=None, help='Project elmo to this many dimensions')
+
+    parser.add_argument('--bert_model', type=str, default=None, help="Use an external bert model (requires the transformers package)")
+    parser.add_argument('--no_bert_model', dest='bert_model', action="store_const", const=None, help="Don't use bert")
+    parser.add_argument('--bert_finetune', default=False, action='store_true', help="Finetune the Bert model")
+    parser.add_argument('--bert_learning_rate', default=0.01, type=float, help='Scale the learning rate for transformer finetuning by this much')
+    parser.add_argument('--bert_weight_decay', default=0.0001, type=float, help='Scale the weight decay for transformer finetuning by this much')
+    parser.add_argument('--bert_hidden_layers', type=int, default=4, help="How many layers of hidden state to use from the transformer")
+    parser.add_argument('--bert_hidden_layers_original', action='store_const', const=None, dest='bert_hidden_layers', help='Use layers 2,3,4 of the Bert embedding')
+
+    parser.add_argument('--bilstm', dest='bilstm', action='store_true', default=True, help="Use a bilstm after the inputs, before the convs.  Using bilstm is about as accurate and significantly faster (because of dim reduction) than going straight to the filters")
+    parser.add_argument('--no_bilstm', dest='bilstm', action='store_false', help="Don't use a bilstm after the inputs, before the convs.")
+    # somewhere between 200-300 seems to be the sweet spot for a couple datasets:
+    # dev set macro f1 scores on 3 class problems
+    # note that these were only run once each
+    # more trials might narrow down which ones works best
+    # es_tass2020:
+    #   150        0.5580
+    #   200        0.5629
+    #   250        0.5586
+    #   300        0.5642    <---
+    #   400        0.5525
+    #   500        0.5579
+    #   750        0.5585
+    # en_sstplus:
+    #   150        0.6816
+    #   200        0.6721
+    #   250        0.6915    <---
+    #   300        0.6824
+    #   400        0.6757
+    #   500        0.6770
+    #   750        0.6781
+    # de_sb10k
+    #   150        0.6745
+    #   200        0.6798    <---
+    #   250        0.6459
+    #   300        0.6665
+    #   400        0.6521
+    #   500        0.6584
+    #   750        0.6447
+    parser.add_argument('--bilstm_hidden_dim', type=int, default=300, help="Dimension of the bilstm to use")
+
+    parser.add_argument('--maxpool_width', type=int, default=1, help="Width of the maxpool kernel to use")
+
+    parser.add_argument('--no_constituency_backprop', dest='constituency_backprop', default=True, action='store_false', help="When using a constituency parser, backprop into the parser's weights if True")
+    parser.add_argument('--constituency_model', type=str, default="/home/john/stanza_resources/it/constituency/vit_bert.pt", help="Which constituency model to use.  TODO: make this more user friendly")
+    parser.add_argument('--constituency_batch_norm', default=False, action='store_true', help='Add a LayerNorm between the output of the parser and the classifier layers')
+    parser.add_argument('--constituency_node_attn', default=False, action='store_true', help='True means to make an attn layer out of the tree, with the words as key and nodes as query')
+    parser.add_argument('--no_constituency_node_attn', dest='constituency_node_attn', action='store_false', help='True means to make an attn layer out of the tree, with the words as key and nodes as query')
+    parser.add_argument('--constituency_top_layer', dest='constituency_top_layer', default=False, action='store_true', help='True means use the top (ROOT) layer of the constituents.  Otherwise, the next layer down (S, usually) will be used')
+    parser.add_argument('--no_constituency_top_layer', dest='constituency_top_layer', action='store_false', help='True means use the top (ROOT) layer of the constituents.  Otherwise, the next layer down (S, usually) will be used')
+    parser.add_argument('--constituency_all_words', default=False, action='store_true', help='Use all word positions in the constituency classifier')
+    parser.add_argument('--no_constituency_all_words', dest='constituency_all_words', default=False, action='store_false', help='Use the start and end word embeddings as inputs to the constituency classifier')
+
+    parser.add_argument('--log_norms', default=False, action='store_true', help='Log the parameters norms while training.  A very noisy option')
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+
+    parser.add_argument('--seed', default=None, type=int, help='Random seed for model')
+
+    add_peft_args(parser)
+    utils.add_device_args(parser)
+
+    return parser
+
+def build_model_filename(args):
+    shape = "FS_%s" % "_".join([str(x) for x in args.filter_sizes])
+    shape = shape + "_C_%d_" % args.filter_channels
+    if args.fc_shapes:
+        shape = shape + "_FC_%s_" % "_".join([str(x) for x in args.fc_shapes])
+
+    model_save_file = utils.standard_model_file_name(vars(args), "classifier", shape=shape, classifier_type=args.model_type.name)
+    logger.info("Expanded save_name: %s", model_save_file)
+    return model_save_file
+
+def parse_args(args=None):
+    """
+    Add arguments for building the classifier.
+    Parses command line args and returns the result.
+    """
+    parser = build_argparse()
+    args = parser.parse_args(args)
+    resolve_peft_args(args, tlogger)
+
+    if args.wandb_name:
+        args.wandb = True
+
+    args.optim = args.optim.lower()
+    if args.weight_decay is None:
+        args.weight_decay = DEFAULT_WEIGHT_DECAY.get(args.optim, None)
+    if args.momentum is None:
+        args.momentum = DEFAULT_MOMENTUM.get(args.optim, None)
+    if args.learning_rate is None:
+        args.learning_rate = DEFAULT_LEARNING_RATES.get(args.optim, None)
+
+    return args
+
+
+def dataset_predictions(model, dataset):
+    model.eval()
+    index_label_map = {x: y for (x, y) in enumerate(model.labels)}
+
+    dataset_lengths = data.sort_dataset_by_len(dataset, keep_index=True)
+
+    predictions = []
+    o_idx = []
+    for length in dataset_lengths.keys():
+        batch = dataset_lengths[length]
+        output = model([x[0] for x in batch])
+        for i in range(len(batch)):
+            predicted = torch.argmax(output[i])
+            predicted_label = index_label_map[predicted.item()]
+            predictions.append(predicted_label)
+            o_idx.append(batch[i][1])
+
+    predictions = utils.unsort(predictions, o_idx)
+    return predictions
+
+def confusion_dataset(predictions, dataset, labels):
+    """
+    Returns a confusion matrix
+
+    First key: gold
+    Second key: predicted
+    so: confusion_matrix[gold][predicted]
+    """
+    confusion_matrix = {}
+    for label in labels:
+        confusion_matrix[label] = {}
+
+    for predicted_label, datum in zip(predictions, dataset):
+        expected_label = datum.sentiment
+        confusion_matrix[expected_label][predicted_label] = confusion_matrix[expected_label].get(predicted_label, 0) + 1
+
+    return confusion_matrix
+
+
+def score_dataset(model, dataset, label_map=None,
+                  remap_labels=None, forgive_unmapped_labels=False):
+    """
+    remap_labels: a dict from old label to new label to use when
+    testing a classifier on a dataset with a simpler label set.
+    For example, a model trained on 5 class sentiment can be tested
+    on a binary distribution with {"0": "0", "1": "0", "3": "1", "4": "1"}
+
+    forgive_unmapped_labels says the following: in the case that the
+    model predicts "2" in the above example for remap_labels, instead
+    treat the model's prediction as whichever label it gave the
+    highest score
+    """
+    model.eval()
+    if label_map is None:
+        label_map = {x: y for (y, x) in enumerate(model.labels)}
+    correct = 0
+    dataset_lengths = data.sort_dataset_by_len(dataset)
+
+    for length in dataset_lengths.keys():
+        # TODO: possibly break this up into smaller batches
+        batch = dataset_lengths[length]
+        expected_labels = [label_map[x.sentiment] for x in batch]
+
+        output = model(batch)
+
+        for i in range(len(expected_labels)):
+            predicted = torch.argmax(output[i])
+            predicted_label = predicted.item()
+            if remap_labels:
+                if predicted_label in remap_labels:
+                    predicted_label = remap_labels[predicted_label]
+                else:
+                    found = False
+                    if forgive_unmapped_labels:
+                        items = []
+                        for j in range(len(output[i])):
+                            items.append((output[i][j].item(), j))
+                        items.sort(key=lambda x: -x[0])
+                        for _, item in items:
+                            if item in remap_labels:
+                                predicted_label = remap_labels[item]
+                                found = True
+                                break
+                    # if slack guesses allowed, none of the existing
+                    # labels matched, so we count it wrong.  if slack
+                    # guesses not allowed, just count it wrong
+                    if not found:
+                        continue
+
+            if predicted_label == expected_labels[i]:
+                correct = correct + 1
+    return correct
+
+def score_dev_set(model, dev_set, dev_eval_scoring):
+    predictions = dataset_predictions(model, dev_set)
+    confusion_matrix = confusion_dataset(predictions, dev_set, model.labels)
+    logger.info("Dev set confusion matrix:\n{}".format(format_confusion(confusion_matrix, model.labels)))
+    correct, total = confusion_to_accuracy(confusion_matrix)
+    macro_f1 = confusion_to_macro_f1(confusion_matrix)
+    logger.info("Dev set: %d correct of %d examples.  Accuracy: %f" %
+                (correct, len(dev_set), correct / len(dev_set)))
+    logger.info("Macro f1: {}".format(macro_f1))
+
+    accuracy = correct / total
+    if dev_eval_scoring is DevScoring.ACCURACY:
+        return accuracy, accuracy, macro_f1
+    elif dev_eval_scoring is DevScoring.WEIGHTED_F1:
+        return macro_f1, accuracy, macro_f1
+    else:
+        raise ValueError("Unknown scoring method {}".format(dev_eval_scoring))
+
+def intermediate_name(filename, epoch, dev_scoring, score):
+    """
+    Build an informative intermediate checkpoint name from a base name, epoch #, and accuracy
+    """
+    root, ext = os.path.splitext(filename)
+    return root + ".E{epoch:04d}-{score_type}{acc:05.2f}".format(**{"epoch": epoch, "score_type": dev_scoring.value, "acc": score * 100}) + ext
+
+def log_param_sizes(model):
+    logger.debug("--- Model parameter sizes ---")
+    total_size = 0
+    for name, param in model.named_parameters():
+        param_size = param.element_size() * param.nelement()
+        total_size += param_size
+        logger.debug("  %s %d %d %d", name, param.element_size(), param.nelement(), param_size)
+    logger.debug("  Total size: %d", total_size)
+
+def train_model(trainer, model_file, checkpoint_file, args, train_set, dev_set, labels):
+    tlogger.setLevel(logging.DEBUG)
+
+    # TODO: use a (torch) dataloader to possibly speed up the GPU usage
+    model = trainer.model
+    optimizer = trainer.optimizer
+
+    device = next(model.parameters()).device
+    logger.info("Current device: %s" % device)
+
+    label_map = {x: y for (y, x) in enumerate(labels)}
+    label_tensors = {x: torch.tensor(y, requires_grad=False, device=device)
+                     for (y, x) in enumerate(labels)}
+
+    process_outputs = lambda x: x
+    if args.loss == Loss.CROSS:
+        logger.info("Creating CrossEntropyLoss")
+        loss_function = nn.CrossEntropyLoss()
+    elif args.loss == Loss.WEIGHTED_CROSS:
+        logger.info("Creating weighted cross entropy loss w/o log")
+        loss_function = loss.weighted_cross_entropy_loss([label_map[x[0]] for x in train_set], log_dampened=False)
+    elif args.loss == Loss.LOG_CROSS:
+        logger.info("Creating weighted cross entropy loss w/ log")
+        loss_function = loss.weighted_cross_entropy_loss([label_map[x[0]] for x in train_set], log_dampened=True)
+    elif args.loss == Loss.FOCAL:
+        try:
+            from focal_loss.focal_loss import FocalLoss
+        except ImportError:
+            raise ImportError("focal_loss not installed.  Must `pip install focal_loss_torch` to use the --loss=focal feature")
+        logger.info("Creating FocalLoss with loss %f", args.loss_focal_gamma)
+        process_outputs = lambda x: torch.softmax(x, dim=1)
+        loss_function = FocalLoss(gamma=args.loss_focal_gamma)
+    else:
+        raise ValueError("Unknown loss function {}".format(args.loss))
+    loss_function.to(device)
+
+    train_set_by_len = data.sort_dataset_by_len(train_set)
+
+    if trainer.global_step > 0:
+        # We reloaded the model, so let's report its current dev set score
+        _ = score_dev_set(model, dev_set, args.dev_eval_scoring)
+        logger.info("Reloaded model for continued training.")
+        if trainer.best_score is not None:
+            logger.info("Previous best score: %.5f", trainer.best_score)
+
+    log_param_sizes(model)
+
+    # https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html
+    if args.wandb:
+        import wandb
+        wandb_name = args.wandb_name if args.wandb_name else "%s_classifier" % args.shorthand
+        wandb.init(name=wandb_name, config=args)
+        wandb.run.define_metric('accuracy', summary='max')
+        wandb.run.define_metric('macro_f1', summary='max')
+        wandb.run.define_metric('epoch_loss', summary='min')
+
+    for opt_name, opt in optimizer.items():
+        current_lr = opt.param_groups[0]['lr']
+        logger.info("optimizer %s learning rate: %s", opt_name, current_lr)
+
+    # if this is a brand new training run, and we're saving all intermediate models, save the start model as well
+    if args.save_intermediate_models and trainer.epochs_trained == 0:
+        intermediate_file = intermediate_name(model_file, trainer.epochs_trained, args.dev_eval_scoring, 0.0)
+        trainer.save(intermediate_file, save_optimizer=False)
+    for trainer.epochs_trained in range(trainer.epochs_trained, args.max_epochs):
+        running_loss = 0.0
+        epoch_loss = 0.0
+        shuffled_batches = data.shuffle_dataset(train_set_by_len, args.batch_size, args.batch_single_item)
+
+        model.train()
+        logger.info("Starting epoch %d", trainer.epochs_trained)
+        if args.log_norms:
+            model.log_norms()
+
+        for batch_num, batch in enumerate(shuffled_batches):
+            # logger.debug("Batch size %d max len %d" % (len(batch), max(len(x.text) for x in batch)))
+            trainer.global_step += 1
+            logger.debug("Starting batch: %d step %d", batch_num, trainer.global_step)
+
+            batch_labels = torch.stack([label_tensors[x.sentiment] for x in batch])
+
+            # zero the parameter gradients
+            for opt in optimizer.values():
+                opt.zero_grad()
+
+            outputs = model(batch)
+            outputs = process_outputs(outputs)
+            batch_loss = loss_function(outputs, batch_labels)
+            batch_loss.backward()
+            for opt in optimizer.values():
+                opt.step()
+
+            # print statistics
+            running_loss += batch_loss.item()
+            if (batch_num + 1) % args.tick == 0: # print every so many batches
+                train_loss = running_loss / args.tick
+                logger.info('[%d, %5d] Average loss: %.3f', trainer.epochs_trained + 1, batch_num + 1, train_loss)
+                if args.wandb:
+                    wandb.log({'train_loss': train_loss}, step=trainer.global_step)
+                if args.dev_eval_batches > 0 and (batch_num + 1) % args.dev_eval_batches == 0:
+                    logger.info('---- Interim analysis ----')
+                    dev_score, accuracy, macro_f1 = score_dev_set(model, dev_set, args.dev_eval_scoring)
+                    if args.wandb:
+                        wandb.log({'accuracy': accuracy, 'macro_f1': macro_f1}, step=trainer.global_step)
+                    if trainer.best_score is None or dev_score > trainer.best_score:
+                        trainer.best_score = dev_score
+                        trainer.save(model_file, save_optimizer=False)
+                        logger.info("Saved new best score model!  Accuracy %.5f   Macro F1 %.5f   Epoch %5d   Batch %d" % (accuracy, macro_f1, trainer.epochs_trained+1, batch_num+1))
+                    model.train()
+                    if args.log_norms:
+                        trainer.model.log_norms()
+                epoch_loss += running_loss
+                running_loss = 0.0
+        # Add any leftover loss to the epoch_loss
+        epoch_loss += running_loss
+
+        logger.info("Finished epoch %d  Total loss %.3f" % (trainer.epochs_trained + 1, epoch_loss))
+        dev_score, accuracy, macro_f1 = score_dev_set(model, dev_set, args.dev_eval_scoring)
+        if args.wandb:
+            wandb.log({'accuracy': accuracy, 'macro_f1': macro_f1, 'epoch_loss': epoch_loss}, step=trainer.global_step)
+        if checkpoint_file:
+            trainer.save(checkpoint_file, epochs_trained = trainer.epochs_trained + 1)
+        if args.save_intermediate_models:
+            intermediate_file = intermediate_name(model_file, trainer.epochs_trained + 1, args.dev_eval_scoring, dev_score)
+            trainer.save(intermediate_file, save_optimizer=False)
+        if trainer.best_score is None or dev_score > trainer.best_score:
+            trainer.best_score = dev_score
+            trainer.save(model_file, save_optimizer=False)
+            logger.info("Saved new best score model!  Accuracy %.5f   Macro F1 %.5f   Epoch %5d" % (accuracy, macro_f1, trainer.epochs_trained+1))
+
+    if args.wandb:
+        wandb.finish()
+
+def main(args=None):
+    args = parse_args(args)
+    seed = utils.set_random_seed(args.seed)
+    logger.info("Using random seed: %d" % seed)
+
+    utils.ensure_dir(args.save_dir)
+
+    save_name = build_model_filename(args)
+
+    # TODO: maybe the dataset needs to be in a torch data loader in order to
+    # make cuda operations faster
+    checkpoint_file = None
+    if args.train:
+        train_set = data.read_dataset(args.train_file, args.wordvec_type, args.min_train_len)
+        logger.info("Using training set: %s" % args.train_file)
+        logger.info("Training set has %d labels" % len(data.dataset_labels(train_set)))
+        tlogger.setLevel(logging.DEBUG)
+
+        tlogger.info("Saving checkpoints: %s", args.checkpoint)
+        if args.checkpoint:
+            checkpoint_file = utils.checkpoint_name(args.save_dir, save_name, args.checkpoint_save_name)
+            tlogger.info("Checkpoint filename: %s", checkpoint_file)
+    elif not args.load_name:
+        if save_name:
+            args.load_name = save_name
+        else:
+            raise ValueError("No model provided and not asked to train a model.  This makes no sense")
+    else:
+        train_set = None
+
+    if args.train and checkpoint_file is not None and os.path.exists(checkpoint_file):
+        trainer = Trainer.load(checkpoint_file, args, load_optimizer=args.train)
+    elif args.load_name:
+        trainer = Trainer.load(args.load_name, args, load_optimizer=args.train)
+    else:
+        trainer = Trainer.build_new_model(args, train_set)
+
+    trainer.model.log_configuration()
+
+    if args.train:
+        utils.log_training_args(args, logger)
+
+        dev_set = data.read_dataset(args.dev_file, args.wordvec_type, min_len=None)
+        logger.info("Using dev set: %s", args.dev_file)
+        logger.info("Training set has %d items", len(train_set))
+        logger.info("Dev set has %d items", len(dev_set))
+        data.check_labels(trainer.model.labels, dev_set)
+
+        train_model(trainer, save_name, checkpoint_file, args, train_set, dev_set, trainer.model.labels)
+
+    if args.log_norms:
+        trainer.model.log_norms()
+    test_set = data.read_dataset(args.test_file, args.wordvec_type, min_len=None)
+    logger.info("Using test set: %s" % args.test_file)
+    data.check_labels(trainer.model.labels, test_set)
+
+    if args.test_remap_labels is None:
+        predictions = dataset_predictions(trainer.model, test_set)
+        confusion_matrix = confusion_dataset(predictions, test_set, trainer.model.labels)
+        if args.output_predictions:
+            logger.info("List of predictions: %s", predictions)
+        logger.info("Confusion matrix:\n{}".format(format_confusion(confusion_matrix, trainer.model.labels)))
+        correct, total = confusion_to_accuracy(confusion_matrix)
+        logger.info("Macro f1: {}".format(confusion_to_macro_f1(confusion_matrix)))
+    else:
+        correct = score_dataset(trainer.model, test_set,
+                                remap_labels=args.test_remap_labels,
+                                forgive_unmapped_labels=args.forgive_unmapped_labels)
+        total = len(test_set)
+    logger.info("Test set: %d correct of %d examples.  Accuracy: %f" %
+                (correct, total, correct / total))
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/parser.py

@@ -0,0 +1,406 @@
+"""
+Entry point for training and evaluating a dependency parser.
+
+This implementation combines a deep biaffine graph-based parser with linearization and distance features.
+For details please refer to paper: https://nlp.stanford.edu/pubs/qi2018universal.pdf.
+"""
+
+"""
+Training and evaluation for the parser.
+"""
+
+import sys
+import os
+import copy
+import shutil
+import time
+import argparse
+import logging
+import numpy as np
+import random
+import torch
+from torch import nn, optim
+
+import stanza.models.depparse.data as data
+from stanza.models.depparse.data import DataLoader
+from stanza.models.depparse.trainer import Trainer
+from stanza.models.depparse import scorer
+from stanza.models.common import utils
+from stanza.models.common import pretrain
+from stanza.models.common.data import augment_punct
+from stanza.models.common.doc import *
+from stanza.models.common.peft_config import add_peft_args, resolve_peft_args
+from stanza.utils.conll import CoNLL
+from stanza.models import _training_logging
+
+logger = logging.getLogger('stanza')
+
+def build_argparse():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default='data/depparse', help='Root dir for saving models.')
+    parser.add_argument('--wordvec_dir', type=str, default='extern_data/word2vec', help='Directory of word vectors.')
+    parser.add_argument('--wordvec_file', type=str, default=None, help='Word vectors filename.')
+    parser.add_argument('--wordvec_pretrain_file', type=str, default=None, help='Exact name of the pretrain file to read')
+    parser.add_argument('--train_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--eval_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--output_file', type=str, default=None, help='Output CoNLL-U file.')
+    parser.add_argument('--no_gold_labels', dest='gold_labels', action='store_false', help="Don't score the eval file - perhaps it has no gold labels, for example.  Cannot be used at training time")
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--lang', type=str, help='Language')
+    parser.add_argument('--shorthand', type=str, help="Treebank shorthand")
+
+    parser.add_argument('--hidden_dim', type=int, default=400)
+    parser.add_argument('--char_hidden_dim', type=int, default=400)
+    parser.add_argument('--deep_biaff_hidden_dim', type=int, default=400)
+    parser.add_argument('--composite_deep_biaff_hidden_dim', type=int, default=100)
+    parser.add_argument('--word_emb_dim', type=int, default=75)
+    parser.add_argument('--char_emb_dim', type=int, default=100)
+    parser.add_argument('--tag_emb_dim', type=int, default=50)
+    parser.add_argument('--no_upos', dest='use_upos', action='store_false', default=True, help="Don't use upos tags as part of the tag embedding")
+    parser.add_argument('--no_xpos', dest='use_xpos', action='store_false', default=True, help="Don't use xpos tags as part of the tag embedding")
+    parser.add_argument('--no_ufeats', dest='use_ufeats', action='store_false', default=True, help="Don't use ufeats as part of the tag embedding")
+    parser.add_argument('--transformed_dim', type=int, default=125)
+    parser.add_argument('--num_layers', type=int, default=3)
+    parser.add_argument('--char_num_layers', type=int, default=1)
+    parser.add_argument('--checkpoint_save_name', type=str, default=None, help="File name to save the most recent checkpoint")
+    parser.add_argument('--no_checkpoint', dest='checkpoint', action='store_false', help="Don't save checkpoints")
+    parser.add_argument('--pretrain_max_vocab', type=int, default=250000)
+    parser.add_argument('--word_dropout', type=float, default=0.33)
+    parser.add_argument('--dropout', type=float, default=0.5)
+    parser.add_argument('--rec_dropout', type=float, default=0, help="Recurrent dropout")
+    parser.add_argument('--char_rec_dropout', type=float, default=0, help="Recurrent dropout")
+
+    parser.add_argument('--no_char', dest='char', action='store_false', help="Turn off character model.")
+    parser.add_argument('--charlm', action='store_true', help="Turn on contextualized char embedding using pretrained character-level language model.")
+    parser.add_argument('--charlm_save_dir', type=str, default='saved_models/charlm', help="Root dir for pretrained character-level language model.")
+    parser.add_argument('--charlm_shorthand', type=str, default=None, help="Shorthand for character-level language model training corpus.")
+    parser.add_argument('--charlm_forward_file', type=str, default=None, help="Exact path to use for forward charlm")
+    parser.add_argument('--charlm_backward_file', type=str, default=None, help="Exact path to use for backward charlm")
+
+    parser.add_argument('--bert_model', type=str, default=None, help="Use an external bert model (requires the transformers package)")
+    parser.add_argument('--no_bert_model', dest='bert_model', action="store_const", const=None, help="Don't use bert")
+    parser.add_argument('--bert_hidden_layers', type=int, default=4, help="How many layers of hidden state to use from the transformer")
+    parser.add_argument('--bert_hidden_layers_original', action='store_const', const=None, dest='bert_hidden_layers', help='Use layers 2,3,4 of the Bert embedding')
+    parser.add_argument('--bert_finetune', default=False, action='store_true', help='Finetune the bert (or other transformer)')
+    parser.add_argument('--no_bert_finetune', dest='bert_finetune', action='store_false', help="Don't finetune the bert (or other transformer)")
+    parser.add_argument('--bert_finetune_layers', default=None, type=int, help='Only finetune this many layers from the transformer')
+    parser.add_argument('--bert_learning_rate', default=1.0, type=float, help='Scale the learning rate for transformer finetuning by this much')
+    parser.add_argument('--second_bert_learning_rate', default=1e-3, type=float, help='Secondary stage transformer finetuning learning rate scale')
+    parser.add_argument('--bert_start_finetuning', default=200, type=int, help='When to start finetuning the transformer')
+    parser.add_argument('--bert_warmup_steps', default=200, type=int, help='How many steps for a linear warmup when finetuning the transformer')
+    parser.add_argument('--bert_weight_decay', default=0.0, type=float, help='Weight decay bert parameters by this much')
+
+    parser.add_argument('--no_pretrain', dest='pretrain', action='store_false', help="Turn off pretrained embeddings.")
+    parser.add_argument('--no_linearization', dest='linearization', action='store_false', help="Turn off linearization term.")
+    parser.add_argument('--no_distance', dest='distance', action='store_false', help="Turn off distance term.")
+
+    parser.add_argument('--sample_train', type=float, default=1.0, help='Subsample training data.')
+    parser.add_argument('--optim', type=str, default='adam', help='sgd, adagrad, adam or adamax.')
+    parser.add_argument('--second_optim', type=str, default=None, help='sgd, adagrad, adam or adamax.')
+    parser.add_argument('--lr', type=float, default=3e-3, help='Learning rate')
+    parser.add_argument('--second_lr', type=float, default=3e-4, help='Secondary stage learning rate')
+    parser.add_argument('--weight_decay', type=float, default=None, help='Weight decay for the first optimizer')
+    parser.add_argument('--beta2', type=float, default=0.95)
+    parser.add_argument('--second_optim_start_step', type=int, default=None, help='If set, switch to the second optimizer when stalled or at this step regardless of performance.  Normally, the optimizer only switches when the dev scores have stalled for --max_steps_before_stop steps')
+    parser.add_argument('--second_warmup_steps', type=int, default=200, help="If set, give the 2nd optimizer a linear warmup.  Idea being that the optimizer won't have a good grasp on the initial gradients and square gradients when it first starts")
+
+    parser.add_argument('--max_steps', type=int, default=50000)
+    parser.add_argument('--eval_interval', type=int, default=100)
+    parser.add_argument('--checkpoint_interval', type=int, default=500)
+    parser.add_argument('--max_steps_before_stop', type=int, default=1000)
+    parser.add_argument('--batch_size', type=int, default=5000)
+    parser.add_argument('--second_batch_size', type=int, default=None, help='Use a different batch size for the second optimizer.  Can be relevant for models with different transformer finetuning settings between optimizers, for example, where the larger batch size is impossible for FT the transformer"')
+    parser.add_argument('--max_grad_norm', type=float, default=1.0, help='Gradient clipping.')
+    parser.add_argument('--log_step', type=int, default=20, help='Print log every k steps.')
+    parser.add_argument('--log_norms', action='store_true', default=False, help='Log the norms of all the parameters (noisy!)')
+    parser.add_argument('--save_dir', type=str, default='saved_models/depparse', help='Root dir for saving models.')
+    parser.add_argument('--save_name', type=str, default="{shorthand}_{embedding}_parser.pt", help="File name to save the model")
+    parser.add_argument('--continue_from', type=str, default=None, help="File name to preload the model to continue training from")
+
+    parser.add_argument('--seed', type=int, default=1234)
+    add_peft_args(parser)
+    utils.add_device_args(parser)
+
+    parser.add_argument('--augment_nopunct', type=float, default=None, help='Augment the training data by copying this fraction of punct-ending sentences as non-punct.  Default of None will aim for roughly 10%%')
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+    return parser
+
+def parse_args(args=None):
+    parser = build_argparse()
+    args = parser.parse_args(args=args)
+    resolve_peft_args(args, logger)
+
+    if args.wandb_name:
+        args.wandb = True
+
+    args = vars(args)
+    return args
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    utils.set_random_seed(args['seed'])
+
+    logger.info("Running parser in {} mode".format(args['mode']))
+
+    if args['mode'] == 'train':
+        return train(args)
+    else:
+        evaluate(args)
+
+def model_file_name(args):
+    return utils.standard_model_file_name(args, "parser")
+
+# TODO: refactor with everywhere
+def load_pretrain(args):
+    pt = None
+    if args['pretrain']:
+        pretrain_file = pretrain.find_pretrain_file(args['wordvec_pretrain_file'], args['save_dir'], args['shorthand'], args['lang'])
+        if os.path.exists(pretrain_file):
+            vec_file = None
+        else:
+            vec_file = args['wordvec_file'] if args['wordvec_file'] else utils.get_wordvec_file(args['wordvec_dir'], args['shorthand'])
+        pt = pretrain.Pretrain(pretrain_file, vec_file, args['pretrain_max_vocab'])
+    return pt
+
+def predict_dataset(trainer, dev_batch):
+    dev_preds = []
+    if len(dev_batch) > 0:
+        for batch in dev_batch:
+            preds = trainer.predict(batch)
+            dev_preds += preds
+        dev_preds = utils.unsort(dev_preds, dev_batch.data_orig_idx)
+    return dev_preds
+
+def train(args):
+    model_file = model_file_name(args)
+    utils.ensure_dir(os.path.split(model_file)[0])
+
+    # load pretrained vectors if needed
+    pretrain = load_pretrain(args)
+
+    # TODO: refactor.  the exact same thing is done in the tagger
+    if args['charlm']:
+        if args['charlm_shorthand'] is None:
+            raise ValueError("CharLM Shorthand is required for loading pretrained CharLM model...")
+        logger.info('Using pretrained contextualized char embedding')
+        if not args['charlm_forward_file']:
+            args['charlm_forward_file'] = '{}/{}_forward_charlm.pt'.format(args['charlm_save_dir'], args['charlm_shorthand'])
+        if not args['charlm_backward_file']:
+            args['charlm_backward_file'] = '{}/{}_backward_charlm.pt'.format(args['charlm_save_dir'], args['charlm_shorthand'])
+
+    # load data
+    logger.info("Loading data with batch size {}...".format(args['batch_size']))
+    train_data, _, _ = CoNLL.conll2dict(input_file=args['train_file'])
+    # possibly augment the training data with some amount of fake data
+    # based on the options chosen
+    logger.info("Original data size: {}".format(len(train_data)))
+    train_data.extend(augment_punct(train_data, args['augment_nopunct'],
+                                    keep_original_sentences=False))
+    logger.info("Augmented data size: {}".format(len(train_data)))
+    train_doc = Document(train_data)
+    train_batch = DataLoader(train_doc, args['batch_size'], args, pretrain, evaluation=False)
+    vocab = train_batch.vocab
+    dev_doc = CoNLL.conll2doc(input_file=args['eval_file'])
+    dev_batch = DataLoader(dev_doc, args['batch_size'], args, pretrain, vocab=vocab, evaluation=True, sort_during_eval=True)
+
+    # pred path
+    system_pred_file = args['output_file']
+
+    # skip training if the language does not have training or dev data
+    if len(train_batch) == 0 or len(dev_batch) == 0:
+        logger.info("Skip training because no data available...")
+        sys.exit(0)
+
+    if args['wandb']:
+        import wandb
+        wandb_name = args['wandb_name'] if args['wandb_name'] else "%s_depparse" % args['shorthand']
+        wandb.init(name=wandb_name, config=args)
+        wandb.run.define_metric('train_loss', summary='min')
+        wandb.run.define_metric('dev_score', summary='max')
+
+    logger.info("Training parser...")
+    checkpoint_file = None
+    if args.get("checkpoint"):
+        # calculate checkpoint file name from the save filename
+        checkpoint_file = utils.checkpoint_name(args.get("save_dir"), model_file, args.get("checkpoint_save_name"))
+        args["checkpoint_save_name"] = checkpoint_file
+
+    if args.get("checkpoint") and os.path.exists(args["checkpoint_save_name"]):
+        trainer = Trainer(args=args, pretrain=pretrain, vocab=vocab, model_file=args["checkpoint_save_name"], device=args['device'], ignore_model_config=True)
+        if len(trainer.dev_score_history) > 0:
+            logger.info("Continuing from checkpoint %s  Model was previously trained for %d steps, with a best dev score of %.4f", args["checkpoint_save_name"], trainer.global_step, max(trainer.dev_score_history))
+    elif args["continue_from"]:
+        if not os.path.exists(args["continue_from"]):
+            raise FileNotFoundError("--continue_from specified, but the file %s does not exist" % args["continue_from"])
+        trainer = Trainer(args=args, pretrain=pretrain, vocab=vocab, model_file=args["continue_from"], device=args['device'], ignore_model_config=True, reset_history=True)
+    else:
+        trainer = Trainer(args=args, vocab=vocab, pretrain=pretrain, device=args['device'])
+
+    max_steps = args['max_steps']
+    current_lr = args['lr']
+    global_start_time = time.time()
+    format_str = 'Finished STEP {}/{}, loss = {:.6f} ({:.3f} sec/batch), lr: {:.6f}'
+
+    is_second_stage = False
+    # start training
+    train_loss = 0
+    if args['log_norms']:
+        trainer.model.log_norms()
+    while True:
+        do_break = False
+        for i, batch in enumerate(train_batch):
+            start_time = time.time()
+            trainer.global_step += 1
+            loss = trainer.update(batch, eval=False) # update step
+            train_loss += loss
+
+            # will checkpoint if we switch optimizers or score a new best score
+            force_checkpoint = False
+            if trainer.global_step % args['log_step'] == 0:
+                duration = time.time() - start_time
+                logger.info(format_str.format(trainer.global_step, max_steps, loss, duration, current_lr))
+
+            if trainer.global_step % args['eval_interval'] == 0:
+                # eval on dev
+                logger.info("Evaluating on dev set...")
+                dev_preds = predict_dataset(trainer, dev_batch)
+
+                dev_batch.doc.set([HEAD, DEPREL], [y for x in dev_preds for y in x])
+                CoNLL.write_doc2conll(dev_batch.doc, system_pred_file)
+                _, _, dev_score = scorer.score(system_pred_file, args['eval_file'])
+
+                train_loss = train_loss / args['eval_interval'] # avg loss per batch
+                logger.info("step {}: train_loss = {:.6f}, dev_score = {:.4f}".format(trainer.global_step, train_loss, dev_score))
+
+                if args['wandb']:
+                    wandb.log({'train_loss': train_loss, 'dev_score': dev_score})
+
+                train_loss = 0
+
+                # save best model
+                trainer.dev_score_history += [dev_score]
+                if dev_score >= max(trainer.dev_score_history):
+                    trainer.last_best_step = trainer.global_step
+                    trainer.save(model_file)
+                    logger.info("new best model saved.")
+                    force_checkpoint = True
+
+                for scheduler_name, scheduler in trainer.scheduler.items():
+                    logger.info('scheduler %s learning rate: %s', scheduler_name, scheduler.get_last_lr())
+                if args['log_norms']:
+                    trainer.model.log_norms()
+
+            if not is_second_stage and args.get('second_optim', None) is not None:
+                if trainer.global_step - trainer.last_best_step >= args['max_steps_before_stop'] or (args['second_optim_start_step'] is not None and trainer.global_step >= args['second_optim_start_step']):
+                    logger.info("Switching to second optimizer: {}".format(args.get('second_optim', None)))
+                    global_step = trainer.global_step
+                    args["second_stage"] = True
+                    # if the loader gets a model file, it uses secondary optimizer
+                    # (because of the second_stage = True argument)
+                    trainer = Trainer(args=args, vocab=trainer.vocab, pretrain=pretrain,
+                                      model_file=model_file, device=args['device'])
+                    logger.info('Reloading best model to continue from current local optimum')
+
+                    dev_preds = predict_dataset(trainer, dev_batch)
+                    dev_batch.doc.set([HEAD, DEPREL], [y for x in dev_preds for y in x])
+                    CoNLL.write_doc2conll(dev_batch.doc, system_pred_file)
+                    _, _, dev_score = scorer.score(system_pred_file, args['eval_file'])
+                    logger.info("Reloaded model with dev score %.4f", dev_score)
+
+                    is_second_stage = True
+                    trainer.global_step = global_step
+                    trainer.last_best_step = global_step
+                    if args['second_batch_size'] is not None:
+                        train_batch.set_batch_size(args['second_batch_size'])
+                    force_checkpoint = True
+            else:
+                if trainer.global_step - trainer.last_best_step >= args['max_steps_before_stop']:
+                    do_break = True
+                    break
+
+            if trainer.global_step % args['eval_interval'] == 0 or force_checkpoint:
+                # if we need to save checkpoint, do so
+                # (save after switching the optimizer, if applicable, so that
+                # the new optimizer is the optimizer used if a restart happens)
+                if checkpoint_file is not None:
+                    trainer.save(checkpoint_file, save_optimizer=True)
+                    logger.info("new model checkpoint saved.")
+
+            if trainer.global_step >= args['max_steps']:
+                do_break = True
+                break
+
+        if do_break: break
+
+        train_batch.reshuffle()
+
+    logger.info("Training ended with {} steps.".format(trainer.global_step))
+
+    if args['wandb']:
+        wandb.finish()
+
+    if len(trainer.dev_score_history) > 0:
+        # TODO: technically the iteration position will be wrong if
+        # the eval_interval changed when running from a checkpoint
+        # could fix this by saving step & score instead of just score
+        best_f, best_eval = max(trainer.dev_score_history)*100, np.argmax(trainer.dev_score_history)+1
+        logger.info("Best dev F1 = {:.2f}, at iteration = {}".format(best_f, best_eval * args['eval_interval']))
+    else:
+        logger.info("Dev set never evaluated.  Saving final model.")
+        trainer.save(model_file)
+
+    return trainer
+
+def evaluate(args):
+    model_file = model_file_name(args)
+    # load pretrained vectors if needed
+    pretrain = load_pretrain(args)
+
+    load_args = {'charlm_forward_file': args.get('charlm_forward_file', None),
+                 'charlm_backward_file': args.get('charlm_backward_file', None)}
+
+    # load model
+    logger.info("Loading model from: {}".format(model_file))
+    trainer = Trainer(pretrain=pretrain, model_file=model_file, device=args['device'], args=load_args)
+    return evaluate_trainer(args, trainer, pretrain)
+
+def evaluate_trainer(args, trainer, pretrain):
+    system_pred_file = args['output_file']
+    loaded_args, vocab = trainer.args, trainer.vocab
+
+    # load config
+    for k in args:
+        if k.endswith('_dir') or k.endswith('_file') or k in ['shorthand'] or k == 'mode':
+            loaded_args[k] = args[k]
+
+    # load data
+    logger.info("Loading data with batch size {}...".format(args['batch_size']))
+    doc = CoNLL.conll2doc(input_file=args['eval_file'])
+    batch = DataLoader(doc, args['batch_size'], loaded_args, pretrain, vocab=vocab, evaluation=True, sort_during_eval=True)
+
+    preds = predict_dataset(trainer, batch)
+
+    # write to file and score
+    batch.doc.set([HEAD, DEPREL], [y for x in preds for y in x])
+    CoNLL.write_doc2conll(batch.doc, system_pred_file)
+
+    if args['gold_labels']:
+        gold_doc = CoNLL.conll2doc(input_file=args['eval_file'])
+
+        # Check for None ... otherwise an inscrutable error occurs later in the scorer
+        for sent_idx, sentence in enumerate(gold_doc.sentences):
+            for word_idx, word in enumerate(sentence.words):
+                if word.deprel is None:
+                    raise ValueError("Gold document {} has a None at sentence {} word {}\n{:C}".format(args['eval_file'], sent_idx, word_idx, sentence))
+
+        scorer.score_named_dependencies(batch.doc, gold_doc)
+        _, _, score = scorer.score(system_pred_file, args['eval_file'])
+
+        logger.info("Parser score:")
+        logger.info("{} {:.2f}".format(args['shorthand'], score*100))
+
+if __name__ == '__main__':
+    main()

stanza/stanza/server/java_protobuf_requests.py

@@ -0,0 +1,357 @@
+from collections import deque
+import subprocess
+
+from stanza.models.common.utils import misc_to_space_after
+from stanza.models.constituency.parse_tree import Tree
+from stanza.protobuf import DependencyGraph, FlattenedParseTree
+from stanza.server.client import resolve_classpath
+
+def send_request(request, response_type, java_main, classpath=None):
+    """
+    Use subprocess to run a Java protobuf processor on the given request
+
+    Returns the protobuf response
+    """
+    classpath = resolve_classpath(classpath)
+    if classpath is None:
+        raise ValueError("Classpath is None,  Perhaps you need to set the $CLASSPATH or $CORENLP_HOME environment variable to point to a CoreNLP install.")
+    pipe = subprocess.run(["java", "-cp", classpath, java_main],
+                          input=request.SerializeToString(),
+                          stdout=subprocess.PIPE,
+                          check=True)
+    response = response_type()
+    response.ParseFromString(pipe.stdout)
+    return response
+
+def add_tree_nodes(proto_tree, tree, score):
+    # add an open node
+    node = proto_tree.nodes.add()
+    node.openNode = True
+    if score is not None:
+        node.score = score
+
+    # add the content of this node
+    node = proto_tree.nodes.add()
+    node.value = tree.label
+
+    # add all children...
+    # leaves get just one node
+    # branches are called recursively
+    for child in tree.children:
+        if child.is_leaf():
+            node = proto_tree.nodes.add()
+            node.value = child.label
+        else:
+            add_tree_nodes(proto_tree, child, None)
+
+    node = proto_tree.nodes.add()
+    node.closeNode = True
+
+def build_tree(tree, score):
+    """
+    Builds a FlattenedParseTree from CoreNLP.proto
+
+    Populates the value field from tree.label and iterates through the
+    children via tree.children.  Should work on any tree structure
+    which follows that layout
+
+    The score will be added to the top node (if it is not None)
+
+    Operates by recursively calling add_tree_nodes
+    """
+    proto_tree = FlattenedParseTree()
+    add_tree_nodes(proto_tree, tree, score)
+    return proto_tree
+
+def from_tree(proto_tree):
+    """
+    Convert a FlattenedParseTree back into a Tree
+
+    returns Tree, score
+      (score might be None if it is missing)
+    """
+    score = None
+    stack = deque()
+    for node in proto_tree.nodes:
+        if node.HasField("score") and score is None:
+            score = node.score
+
+        if node.openNode:
+            if len(stack) > 0 and isinstance(stack[-1], FlattenedParseTree.Node) and stack[-1].openNode:
+                raise ValueError("Got a proto with no label on a node: {}".format(proto_tree))
+            stack.append(node)
+            continue
+        if not node.closeNode:
+            child = Tree(label=node.value)
+            # TODO: do something with the score
+            stack.append(child)
+            continue
+
+        # must be a close operation...
+        if len(stack) <= 1:
+            raise ValueError("Got a proto with too many close operations: {}".format(proto_tree))
+        # on a close operation, pop until we hit the open
+        # then turn everything in that span into a new node
+        children = []
+        nextNode = stack.pop()
+        while not isinstance(nextNode, FlattenedParseTree.Node):
+            children.append(nextNode)
+            nextNode = stack.pop()
+        if len(children) == 0:
+            raise ValueError("Got a proto with an open immediately followed by a close: {}".format(proto_tree))
+        children.reverse()
+        label = children[0]
+        children = children[1:]
+        subtree = Tree(label=label.label, children=children)
+        stack.append(subtree)
+
+    if len(stack) > 1:
+        raise ValueError("Got a proto which does not close all of the nodes: {}".format(proto_tree))
+    tree = stack.pop()
+    if not isinstance(tree, Tree):
+        raise ValueError("Got a proto which was just one Open operation: {}".format(proto_tree))
+    return tree, score
+
+def add_token(token_list, word, token):
+    """
+    Add a token to a proto request.
+
+    CoreNLP tokens have components of both word and token from stanza.
+
+    We pass along "after" but not "before"
+    """
+    if token is None and isinstance(word.id, int):
+        raise AssertionError("Only expected word w/o token for 'extra' words")
+
+    query_token = token_list.add()
+    query_token.word = word.text
+    query_token.value = word.text
+    if word.lemma is not None:
+        query_token.lemma = word.lemma
+    if word.xpos is not None:
+        query_token.pos = word.xpos
+    if word.upos is not None:
+        query_token.coarseTag = word.upos
+    if word.feats and word.feats != "_":
+        for feature in word.feats.split("|"):
+            key, value = feature.split("=", maxsplit=1)
+            query_token.conllUFeatures.key.append(key)
+            query_token.conllUFeatures.value.append(value)
+    if token is not None:
+        if token.ner is not None:
+            query_token.ner = token.ner
+        if token is not None and len(token.id) > 1:
+            query_token.mwtText = token.text
+            query_token.isMWT = True
+            query_token.isFirstMWT = token.id[0] == word.id
+        if token.id[-1] != word.id:
+            # if we are not the last word of an MWT token
+            # we are absolutely not followed by space
+            pass
+        else:
+            query_token.after = token.spaces_after
+
+        query_token.index = word.id
+    else:
+        # presumably empty words won't really be written this way,
+        # but we can still keep track of it
+        query_token.after = misc_to_space_after(word.misc)
+
+        query_token.index = word.id[0]
+        query_token.emptyIndex = word.id[1]
+
+    if word.misc and word.misc != "_":
+        query_token.conllUMisc = word.misc
+    if token is not None and token.misc and token.misc != "_":
+        query_token.mwtMisc = token.misc
+
+def add_sentence(request_sentences, sentence, num_tokens):
+    """
+    Add the tokens for this stanza sentence to a list of protobuf sentences
+    """
+    request_sentence = request_sentences.add()
+    request_sentence.tokenOffsetBegin = num_tokens
+    request_sentence.tokenOffsetEnd = num_tokens + sum(len(token.words) for token in sentence.tokens)
+    for token in sentence.tokens:
+        for word in token.words:
+            add_token(request_sentence.token, word, token)
+    return request_sentence
+
+def add_word_to_graph(graph, word, sent_idx, word_idx):
+    """
+    Add a node and possibly an edge for a word in a basic dependency graph.
+    """
+    node = graph.node.add()
+    node.sentenceIndex = sent_idx+1
+    node.index = word_idx+1
+
+    if word.head != 0 and word.head is not None:
+        edge = graph.edge.add()
+        edge.source = word.head
+        edge.target = word_idx+1
+        if word.deprel is not None:
+            edge.dep = word.deprel
+        else:
+            # the receiving side doesn't like null as a dependency
+            edge.dep = "_"
+
+def convert_networkx_graph(graph_proto, sentence, sent_idx):
+    """
+    Turns a networkx graph into a DependencyGraph from the proto file
+    """
+    for token in sentence.tokens:
+        for word in token.words:
+            add_token(graph_proto.token, word, token)
+    for word in sentence.empty_words:
+        add_token(graph_proto.token, word, None)
+
+    dependencies = sentence._enhanced_dependencies
+    for target in dependencies:
+        if target == 0:
+            # don't need to send the explicit root
+            continue
+        for source in dependencies.predecessors(target):
+            if source == 0:
+                # unlike with basic, we need to send over the roots,
+                # as the enhanced can have loops
+                graph_proto.rootNode.append(len(graph_proto.node))
+                continue
+            for deprel in dependencies.get_edge_data(source, target):
+                edge = graph_proto.edge.add()
+                if isinstance(source, int):
+                    edge.source = source
+                else:
+                    edge.source = source[0]
+                    if source[1] != 0:
+                        edge.sourceEmpty = source[1]
+                if isinstance(target, int):
+                    edge.target = target
+                else:
+                    edge.target = target[0]
+                    if target[1] != 0:
+                        edge.targetEmpty = target[1]
+                edge.dep = deprel
+        node = graph_proto.node.add()
+        node.sentenceIndex = sent_idx + 1
+        # the nodes in the networkx graph are indexed from 1, not counting the root
+        if isinstance(target, int):
+            node.index = target
+        else:
+            node.index = target[0]
+            if target[1] != 0:
+                node.emptyIndex = target[1]
+    return graph_proto
+
+def features_to_string(features):
+    if not features:
+        return None
+    if len(features.key) == 0:
+        return None
+    return "|".join("%s=%s" % (key, value) for key, value in zip(features.key, features.value))
+
+def misc_space_pieces(misc):
+    """
+    Return only the space-related misc pieces
+    """
+    if misc is None or misc == "" or misc == "_":
+        return misc
+    pieces = misc.split("|")
+    pieces = [x for x in pieces if x.split("=", maxsplit=1)[0] in ("SpaceAfter", "SpacesAfter", "SpacesBefore")]
+    if len(pieces) > 0:
+        return "|".join(pieces)
+    return None
+
+def remove_space_misc(misc):
+    """
+    Remove any pieces from misc which are space-related
+    """
+    if misc is None or misc == "" or misc == "_":
+        return misc
+    pieces = misc.split("|")
+    pieces = [x for x in pieces if x.split("=", maxsplit=1)[0] not in ("SpaceAfter", "SpacesAfter", "SpacesBefore")]
+    if len(pieces) > 0:
+        return "|".join(pieces)
+    return None
+
+def substitute_space_misc(misc, space_misc):
+    space_misc_pieces = space_misc.split("|") if space_misc else []
+    space_misc_after = None
+    space_misc_before = None
+    for piece in space_misc_pieces:
+        if piece.startswith("SpaceBefore"):
+            space_misc_before = piece
+        elif piece.startswith("SpaceAfter") or piece.startswith("SpacesAfter"):
+            space_misc_after = piece
+        else:
+            raise AssertionError("An unknown piece wound up in the misc space fields: %s" % piece)
+
+    pieces = misc.split("|")
+    new_pieces = []
+    for piece in pieces:
+        if piece.startswith("SpaceBefore"):
+            if space_misc_before:
+                new_pieces.append(space_misc_before)
+                space_misc_before = None
+        elif piece.startswith("SpaceAfter") or piece.startswith("SpacesAfter"):
+            if space_misc_after:
+                new_pieces.append(space_misc_after)
+                space_misc_after = None
+        else:
+            new_pieces.append(piece)
+    if space_misc_after:
+        new_pieces.append(space_misc_after)
+    if space_misc_before:
+        new_pieces.append(space_misc_before)
+    if len(new_pieces) == 0:
+        return None
+    return "|".join(new_pieces)
+
+class JavaProtobufContext(object):
+    """
+    A generic context for sending requests to a java program using protobufs in a subprocess
+    """
+    def __init__(self, classpath, build_response, java_main, extra_args=None):
+        self.classpath = resolve_classpath(classpath)
+        self.build_response = build_response
+        self.java_main = java_main
+
+        if extra_args is None:
+            extra_args = []
+        self.extra_args = extra_args
+        self.pipe = None
+
+    def open_pipe(self):
+        self.pipe = subprocess.Popen(["java", "-cp", self.classpath, self.java_main, "-multiple"] + self.extra_args,
+                                     stdin=subprocess.PIPE,
+                                     stdout=subprocess.PIPE)
+
+    def close_pipe(self):
+        if self.pipe.poll() is None:
+            self.pipe.stdin.write((0).to_bytes(4, 'big'))
+            self.pipe.stdin.flush()
+            self.pipe = None
+
+    def __enter__(self):
+        self.open_pipe()
+        return self
+
+    def __exit__(self, type, value, traceback):
+        self.close_pipe()
+
+    def process_request(self, request):
+        if self.pipe is None:
+            raise RuntimeError("Pipe to java process is not open or was closed")
+
+        text = request.SerializeToString()
+        self.pipe.stdin.write(len(text).to_bytes(4, 'big'))
+        self.pipe.stdin.write(text)
+        self.pipe.stdin.flush()
+        response_length = self.pipe.stdout.read(4)
+        if len(response_length) < 4:
+            raise BrokenPipeError("Could not communicate with java process!")
+        response_length = int.from_bytes(response_length, "big")
+        response_text = self.pipe.stdout.read(response_length)
+        response = self.build_response()
+        response.ParseFromString(response_text)
+        return response
+

stanza/stanza/server/main.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Simple shell program to pipe in 
+"""
+
+import corenlp
+
+import json
+import re
+import csv
+import sys
+from collections import namedtuple, OrderedDict
+
+FLOAT_RE = re.compile(r"\d*\.\d+")
+INT_RE = re.compile(r"\d+")
+
+def dictstr(arg):
+    """
+    Parse a key=value string as a tuple (key, value) that can be provided as an argument to dict()
+    """
+    key, value = arg.split("=")
+
+    if value.lower() == "true" or value.lower() == "false":
+        value = bool(value)
+    elif INT_RE.match(value):
+        value = int(value)
+    elif FLOAT_RE.match(value):
+        value = float(value)
+    return (key, value)
+
+
+def do_annotate(args):
+    args.props = dict(args.props) if args.props else {}
+    if args.sentence_mode:
+        args.props["ssplit.isOneSentence"] = True
+
+    with corenlp.CoreNLPClient(annotators=args.annotators, properties=args.props, be_quiet=not args.verbose_server) as client:
+        for line in args.input:
+            if line.startswith("#"): continue
+
+            ann = client.annotate(line.strip(), output_format=args.format)
+
+            if args.format == "json":
+                if args.sentence_mode:
+                    ann = ann["sentences"][0]
+
+                args.output.write(json.dumps(ann))
+                args.output.write("\n")
+
+def main():
+    import argparse
+    parser = argparse.ArgumentParser(description='Annotate data')
+    parser.add_argument('-i', '--input', type=argparse.FileType('r'), default=sys.stdin, help="Input file to process; each line contains one document (default: stdin)")
+    parser.add_argument('-o', '--output', type=argparse.FileType('w'), default=sys.stdout, help="File to write annotations to (default: stdout)")
+    parser.add_argument('-f', '--format', choices=["json",], default="json", help="Output format")
+    parser.add_argument('-a', '--annotators', nargs="+", type=str, default=["tokenize ssplit lemma pos"], help="A list of annotators")
+    parser.add_argument('-s', '--sentence-mode', action="store_true",help="Assume each line of input is a sentence.")
+    parser.add_argument('-v', '--verbose-server', action="store_true",help="Server is made verbose")
+    parser.add_argument('-m', '--memory', type=str, default="4G", help="Memory to use for the server")
+    parser.add_argument('-p', '--props', nargs="+", type=dictstr, help="Properties as a list of key=value pairs")
+    parser.set_defaults(func=do_annotate)
+
+    ARGS = parser.parse_args()
+    if ARGS.func is None:
+        parser.print_help()
+        sys.exit(1)
+    else:
+        ARGS.func(ARGS)
+
+if __name__ == "__main__":
+    main()

stanza/stanza/server/morphology.py

@@ -0,0 +1,81 @@
+"""
+Direct pipe connection to the Java CoreNLP Morphology class
+
+Only effective for English.  Must be supplied with PTB scheme xpos, not upos
+"""
+
+
+from stanza.protobuf import MorphologyRequest, MorphologyResponse
+from stanza.server.java_protobuf_requests import send_request, JavaProtobufContext
+
+
+MORPHOLOGY_JAVA = "edu.stanford.nlp.process.ProcessMorphologyRequest"
+
+def send_morphology_request(request):
+    return send_request(request, MorphologyResponse, MORPHOLOGY_JAVA)
+
+def build_request(words, xpos_tags):
+    """
+    Turn a list of words and a list of tags into a request
+
+    tags must be xpos, not upos
+    """
+    request = MorphologyRequest()
+    for word, tag in zip(words, xpos_tags):
+        tagged_word = request.words.add()
+        tagged_word.word = word
+        tagged_word.xpos = tag
+    return request
+
+
+def process_text(words, xpos_tags):
+    """
+    Get the lemmata for each word/tag pair
+
+    Currently the return is a MorphologyResponse from CoreNLP.proto
+
+    tags must be xpos, not upos
+    """
+    request = build_request(words, xpos_tags)
+
+    return send_morphology_request(request)
+
+
+
+class Morphology(JavaProtobufContext):
+    """
+    Morphology context window
+
+    This is a context window which keeps a process open.  Should allow
+    for multiple requests without launching new java processes each time.
+
+    (much faster than calling process_text over and over)
+    """
+    def __init__(self, classpath=None):
+        super(Morphology, self).__init__(classpath, MorphologyResponse, MORPHOLOGY_JAVA)
+
+    def process(self, words, xpos_tags):
+        """
+        Get the lemmata for each word/tag pair
+        """
+        request = build_request(words, xpos_tags)
+        return self.process_request(request)
+
+
+def main():
+    # TODO: turn this into a unit test, once a new CoreNLP is released
+    words    = ["Jennifer", "has",  "the", "prettiest", "antennae"]
+    tags     = ["NNP",      "VBZ",  "DT",  "JJS",       "NNS"]
+    expected = ["Jennifer", "have", "the", "pretty",    "antenna"]
+    result = process_text(words, tags)
+    lemma = [x.lemma for x in result.words]
+    print(lemma)
+    assert lemma == expected
+
+    with Morphology() as morph:
+        result = morph.process(words, tags)
+        lemma = [x.lemma for x in result.words]
+        assert lemma == expected
+
+if __name__ == '__main__':
+    main()

stanza/stanza/server/parser_eval.py

@@ -0,0 +1,89 @@
+"""
+This class runs a Java process to evaluate a treebank prediction using CoreNLP
+"""
+
+from collections import namedtuple
+import sys
+
+import stanza
+from stanza.protobuf import EvaluateParserRequest, EvaluateParserResponse
+from stanza.server.java_protobuf_requests import send_request, build_tree, JavaProtobufContext
+from stanza.models.constituency.tree_reader import read_treebank
+
+EVALUATE_JAVA = "edu.stanford.nlp.parser.metrics.EvaluateExternalParser"
+
+ParseResult = namedtuple("ParseResult", ['gold', 'predictions', 'state', 'constituents'])
+ScoredTree = namedtuple("ScoredTree", ['tree', 'score'])
+
+def build_request(treebank):
+    """
+    treebank should be a list of pairs:  [gold, predictions]
+      each predictions is a list of tuples (prediction, score, state)
+      state is ignored and can be None
+    Note that for now, only one tree is measured, but this may be extensible in the future
+    Trees should be in the form of a Tree from parse_tree.py
+    """
+    request = EvaluateParserRequest()
+    for raw_result in treebank:
+        gold = raw_result.gold
+        predictions = raw_result.predictions
+        parse_result = request.treebank.add()
+        parse_result.gold.CopyFrom(build_tree(gold, None))
+        for pred in predictions:
+            if isinstance(pred, tuple):
+                prediction, score = pred
+            else:
+                prediction = pred
+                score = None
+            try:
+                parse_result.predicted.append(build_tree(prediction, score))
+            except Exception as e:
+                raise RuntimeError("Unable to build parser request from tree {}".format(pred)) from e
+
+    return request
+
+def collate(gold_treebank, predictions_treebank):
+    """
+    Turns a list of gold and prediction into a evaluation object
+    """
+    treebank = []
+    for gold, prediction in zip(gold_treebank, predictions_treebank):
+        result = ParseResult(gold, [prediction], None, None)
+        treebank.append(result)
+    return treebank
+
+
+class EvaluateParser(JavaProtobufContext):
+    """
+    Parser evaluation context window
+
+    This is a context window which keeps a process open.  Should allow
+    for multiple requests without launching new java processes each time.
+    """
+    def __init__(self, classpath=None, kbest=None, silent=False):
+        if kbest is not None:
+            extra_args = ["-evalPCFGkBest", "{}".format(kbest), "-evals", "pcfgTopK"]
+        else:
+            extra_args = []
+
+        if silent:
+            extra_args.extend(["-evals", "summary=False"])
+
+        super(EvaluateParser, self).__init__(classpath, EvaluateParserResponse, EVALUATE_JAVA, extra_args=extra_args)
+
+    def process(self, treebank):
+        request = build_request(treebank)
+        return self.process_request(request)
+
+
+def main():
+    gold = read_treebank(sys.argv[1])
+    predictions = read_treebank(sys.argv[2])
+    treebank = collate(gold, predictions)
+
+    with EvaluateParser() as ep:
+        ep.process(treebank)
+
+
+if __name__ == '__main__':
+    main()

stanza/stanza/server/tokensregex.py

@@ -0,0 +1,44 @@
+"""Invokes the Java tokensregex on a document
+
+This operates tokensregex on docs processed with stanza models.
+
+https://nlp.stanford.edu/software/tokensregex.html
+
+A minimal example is the main method of this module.
+"""
+
+import stanza
+
+from stanza.protobuf import TokensRegexRequest, TokensRegexResponse
+from stanza.server.java_protobuf_requests import send_request, add_sentence
+
+def send_tokensregex_request(request):
+    return send_request(request, TokensRegexResponse,
+                        "edu.stanford.nlp.ling.tokensregex.ProcessTokensRegexRequest")
+
+def process_doc(doc, *patterns):
+    request = TokensRegexRequest()
+    for pattern in patterns:
+        request.pattern.append(pattern)
+
+    request_doc = request.doc
+    request_doc.text = doc.text
+    num_tokens = 0
+    for sentence in doc.sentences:
+        add_sentence(request_doc.sentence, sentence, num_tokens)
+        num_tokens = num_tokens + sum(len(token.words) for token in sentence.tokens)
+
+    return send_tokensregex_request(request)
+
+def main():
+    #nlp = stanza.Pipeline('en',
+    #                      processors='tokenize,pos,lemma,ner')
+    nlp = stanza.Pipeline('en',
+                          processors='tokenize')
+
+    doc = nlp('Uro ruined modern.  Fortunately, Wotc banned him')
+    print(process_doc(doc, "him", "ruined"))
+
+
+if __name__ == '__main__':
+    main()

stanza/stanza/server/tsurgeon.py

@@ -0,0 +1,95 @@
+"""Invokes the Java tsurgeon on a list of trees
+
+Included with CoreNLP is a mechanism for modifying trees based on
+existing patterns within a tree.  The patterns are found using tregex:
+
+https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/trees/tregex/TregexPattern.html
+
+The modifications are then performed using tsurgeon:
+
+https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/trees/tregex/tsurgeon/Tsurgeon.html
+
+This module accepts Tree objects as produced by the conparser and
+returns the modified trees that result from one or more tsurgeon
+operations.
+"""
+
+from stanza.models.constituency import tree_reader
+from stanza.models.constituency.parse_tree import Tree
+from stanza.protobuf import TsurgeonRequest, TsurgeonResponse
+from stanza.server.java_protobuf_requests import send_request, build_tree, from_tree, JavaProtobufContext
+
+TSURGEON_JAVA = "edu.stanford.nlp.trees.tregex.tsurgeon.ProcessTsurgeonRequest"
+
+def send_tsurgeon_request(request):
+    return send_request(request, TsurgeonResponse, TSURGEON_JAVA)
+
+
+def build_request(trees, operations):
+    """
+    Build the TsurgeonRequest object
+
+    trees: a list of trees
+    operations: a list of (tregex, tsurgeon, tsurgeon, ...)
+    """
+    if isinstance(trees, Tree):
+        trees = (trees,)
+
+    request = TsurgeonRequest()
+    for tree in trees:
+        request.trees.append(build_tree(tree, 0.0))
+    if all(isinstance(x, str) for x in operations):
+        operations = (operations,)
+    for operation in operations:
+        if len(operation) == 1:
+            raise ValueError("Expected [tregex, tsurgeon, ...] but just got a tregex")
+        operation_request = request.operations.add()
+        operation_request.tregex = operation[0]
+        for tsurgeon in operation[1:]:
+            operation_request.tsurgeon.append(tsurgeon)
+    return request
+
+
+def process_trees(trees, *operations):
+    """
+    Returns the result of processing the given tsurgeon operations on the given trees
+
+    Returns a list of modified trees, eg, the result is already processed
+    """
+    request = build_request(trees, operations)
+    result = send_tsurgeon_request(request)
+
+    return [from_tree(t)[0] for t in result.trees]
+
+
+class Tsurgeon(JavaProtobufContext):
+    """
+    Tsurgeon context window
+
+    This is a context window which keeps a process open.  Should allow
+    for multiple requests without launching new java processes each time.
+    """
+    def __init__(self, classpath=None):
+        super(Tsurgeon, self).__init__(classpath, TsurgeonResponse, TSURGEON_JAVA)
+
+    def process(self, trees, *operations):
+        request = build_request(trees, operations)
+        result = self.process_request(request)
+        return [from_tree(t)[0] for t in result.trees]
+
+
+def main():
+    """
+    A small demonstration of a tsurgeon operation
+    """
+    text="( (SBARQ (WHNP (WP Who)) (SQ (VP (VBZ sits) (PP (IN in) (NP (DT this) (NN seat))))) (. ?)))"
+    trees = tree_reader.read_trees(text)
+
+    tregex = "WP=wp"
+    tsurgeon = "relabel wp WWWPPP"
+
+    result = process_trees(trees, (tregex, tsurgeon))
+    print(result)
+
+if __name__ == '__main__':
+    main()

stanza/stanza/server/ud_enhancer.py

@@ -0,0 +1,81 @@
+
+
+import stanza
+from stanza.protobuf import DependencyEnhancerRequest, Document, Language
+from stanza.server.java_protobuf_requests import send_request, add_sentence, JavaProtobufContext
+
+ENHANCER_JAVA = "edu.stanford.nlp.trees.ud.ProcessUniversalEnhancerRequest"
+
+def build_enhancer_request(doc, language, pronouns_pattern):
+    if bool(language) == bool(pronouns_pattern):
+        raise ValueError("Should set exactly one of language and pronouns_pattern")
+
+    request = DependencyEnhancerRequest()
+    if pronouns_pattern:
+        request.setRelativePronouns(pronouns_pattern)
+    elif language.lower() in ("en", "english"):
+        request.language = Language.UniversalEnglish
+    elif language.lower() in ("zh", "zh-hans", "chinese"):
+        request.language = Language.UniversalChinese
+    else:
+        raise ValueError("Sorry, but language " + language + " is not supported yet.  Either set a pronouns pattern or file an issue at https://stanfordnlp.github.io/stanza suggesting a mechanism for converting this language")
+
+    request_doc = request.document
+    request_doc.text = doc.text
+    num_tokens = 0
+    for sent_idx, sentence in enumerate(doc.sentences):
+        request_sentence = add_sentence(request_doc.sentence, sentence, num_tokens)
+        num_tokens = num_tokens + sum(len(token.words) for token in sentence.tokens)
+
+        graph = request_sentence.basicDependencies
+        nodes = []
+        word_index = 0
+        for token in sentence.tokens:
+            for word in token.words:
+                # TODO: refactor with the bit in java_protobuf_requests
+                word_index = word_index + 1
+                node = graph.node.add()
+                node.sentenceIndex = sent_idx
+                node.index = word_index
+
+                if word.head != 0:
+                    edge = graph.edge.add()
+                    edge.source = word.head
+                    edge.target = word_index
+                    edge.dep = word.deprel
+
+    return request
+
+def process_doc(doc, language=None, pronouns_pattern=None):
+    request = build_enhancer_request(doc, language, pronouns_pattern)
+    return send_request(request, Document, ENHANCER_JAVA)
+
+class UniversalEnhancer(JavaProtobufContext):
+    """
+    UniversalEnhancer context window
+
+    This is a context window which keeps a process open.  Should allow
+    for multiple requests without launching new java processes each time.
+    """
+    def __init__(self, language=None, pronouns_pattern=None, classpath=None):
+        super(UniversalEnhancer, self).__init__(classpath, Document, ENHANCER_JAVA)
+        if bool(language) == bool(pronouns_pattern):
+            raise ValueError("Should set exactly one of language and pronouns_pattern")
+        self.language = language
+        self.pronouns_pattern = pronouns_pattern
+
+    def process(self, doc):
+        request = build_enhancer_request(doc, self.language, self.pronouns_pattern)
+        return self.process_request(request)
+
+def main():
+    nlp = stanza.Pipeline('en',
+                          processors='tokenize,pos,lemma,depparse')
+
+    with UniversalEnhancer(language="en") as enhancer:
+        doc = nlp("This is the car that I bought")
+        result = enhancer.process(doc)
+        print(result.sentence[0].enhancedDependencies)
+
+if __name__ == '__main__':
+    main()

stanza/stanza/tests/pytest.ini

@@ -0,0 +1,5 @@
+[pytest]
+markers =
+    travis: all tests that will be run in travis CI
+    client: all tests that are related to the CoreNLP client interface
+    pipeline: all tests that are related to the Stanza neural pipeline

stanza/stanza/tests/setup.py

@@ -0,0 +1,58 @@
+import glob
+import logging
+import os
+import shutil
+import stanza
+from stanza.resources import installation
+from stanza.tests import TEST_HOME_VAR, TEST_DIR_BASE_NAME
+
+logger = logging.getLogger('stanza')
+
+test_dir = os.getenv(TEST_HOME_VAR, None)
+if not test_dir:
+    test_dir = os.path.join(os.getcwd(), TEST_DIR_BASE_NAME)
+    logger.info("STANZA_TEST_HOME not set.  Will assume $PWD/stanza_test = %s", test_dir)
+    logger.info("To use a different directory, export or set STANZA_TEST_HOME=...")
+
+in_dir = os.path.join(test_dir, "in")
+out_dir = os.path.join(test_dir, "out")
+scripts_dir = os.path.join(test_dir, "scripts")
+models_dir=os.path.join(test_dir, "models")
+corenlp_dir=os.path.join(test_dir, "corenlp_dir")
+
+os.makedirs(test_dir, exist_ok=True)
+os.makedirs(in_dir, exist_ok=True)
+os.makedirs(out_dir, exist_ok=True)
+os.makedirs(scripts_dir, exist_ok=True)
+os.makedirs(models_dir, exist_ok=True)
+os.makedirs(corenlp_dir, exist_ok=True)
+
+logger.info("COPYING FILES")
+
+shutil.copy("stanza/tests/data/external_server.properties", scripts_dir)
+shutil.copy("stanza/tests/data/example_french.json", out_dir)
+shutil.copy("stanza/tests/data/aws_annotations.zip", in_dir)
+for emb_file in glob.glob("stanza/tests/data/tiny_emb.*"):
+    shutil.copy(emb_file, in_dir)
+
+logger.info("DOWNLOADING MODELS")
+
+stanza.download(lang='en', model_dir=models_dir, logging_level='info')
+stanza.download(lang="en", model_dir=models_dir, package=None, processors={"ner":"ncbi_disease"})
+stanza.download(lang='fr', model_dir=models_dir, logging_level='info')
+# Latin ITTB has no case information for the lemmatizer
+stanza.download(lang='la', model_dir=models_dir, package='ittb', logging_level='info')
+stanza.download(lang='zh', model_dir=models_dir, logging_level='info')
+# useful not just for verifying RtL, but because the default Arabic has a unique style of xpos tags
+stanza.download(lang='ar', model_dir=models_dir, logging_level='info')
+stanza.download(lang='multilingual', model_dir=models_dir, logging_level='info')
+
+logger.info("DOWNLOADING CORENLP")
+
+installation.install_corenlp(dir=corenlp_dir)
+installation.download_corenlp_models(model="french", version="main", dir=corenlp_dir)
+installation.download_corenlp_models(model="german", version="main", dir=corenlp_dir)
+installation.download_corenlp_models(model="italian", version="main", dir=corenlp_dir)
+installation.download_corenlp_models(model="spanish", version="main", dir=corenlp_dir)
+
+logger.info("Test setup completed.")

stanza/stanza/utils/confusion.py

@@ -0,0 +1,216 @@
+
+from collections import defaultdict, namedtuple
+
+F1Result = namedtuple("F1Result", ['precision', 'recall', 'f1'])
+
+def condense_ner_labels(confusion, gold_labels, pred_labels):
+    new_confusion = defaultdict(lambda: defaultdict(int))
+    new_gold_labels = []
+    new_pred_labels = []
+    for l1 in gold_labels:
+        if l1.find("-") >= 0:
+            new_l1 = l1.split("-", 1)[1]
+        else:
+            new_l1 = l1
+        if new_l1 not in new_gold_labels:
+            new_gold_labels.append(new_l1)
+        for l2 in pred_labels:
+            if l2.find("-") >= 0:
+                new_l2 = l2.split("-", 1)[1]
+            else:
+                new_l2 = l2
+            if new_l2 not in new_pred_labels:
+                new_pred_labels.append(new_l2)
+
+            old_value = confusion.get(l1, {}).get(l2, 0)
+            new_confusion[new_l1][new_l2] = new_confusion[new_l1][new_l2] + old_value
+    return new_confusion, new_gold_labels, new_pred_labels
+
+
+def format_confusion(confusion, labels=None, hide_zeroes=False, hide_blank=False, transpose=False):
+    """
+    pretty print for confusion matrixes
+    adapted from https://gist.github.com/zachguo/10296432
+
+    The matrix should look like this:
+      confusion[gold][pred]
+    """
+    def sort_labels(labels):
+        """
+        Sorts the labels in the list, respecting BIES if all labels are BIES, putting O at the front
+        """
+        labels = set(labels)
+        if 'O' in labels:
+            had_O = True
+            labels.remove('O')
+        else:
+            had_O = False
+
+        if not all(isinstance(x, str) and len(x) > 2 and x[0] in ('B', 'I', 'E', 'S') and x[1] in ('-', '_') for x in labels):
+            labels = sorted(labels)
+        else:
+            # sort first by the body of the lable, then by BEIS
+            labels = sorted(labels, key=lambda x: (x[2:], x[0]))
+        if had_O:
+            labels = ['O'] + labels
+        return labels
+
+    if transpose:
+        new_confusion = defaultdict(lambda: defaultdict(int))
+        for label1 in confusion.keys():
+            for label2 in confusion[label1].keys():
+                new_confusion[label2][label1] = confusion[label1][label2]
+        confusion = new_confusion
+
+    if labels is None:
+        gold_labels = set(confusion.keys())
+        if hide_blank:
+            gold_labels = set(x for x in gold_labels if any(confusion[x][key] != 0 for key in confusion[x].keys()))
+
+        pred_labels = set()
+        for key in confusion.keys():
+            if hide_blank:
+                new_pred_labels = set(x for x in confusion[key].keys() if confusion[key][x] != 0)
+            else:
+                new_pred_labels = confusion[key].keys()
+            pred_labels = pred_labels.union(new_pred_labels)
+
+        if not hide_blank:
+            gold_labels = gold_labels.union(pred_labels)
+            pred_labels = gold_labels
+
+        gold_labels = sort_labels(gold_labels)
+        pred_labels = sort_labels(pred_labels)
+    else:
+        gold_labels = labels
+        pred_labels = labels
+
+    columnwidth = max([len(str(x)) for x in pred_labels] + [5])  # 5 is value length
+    empty_cell = " " * columnwidth
+
+    # If the numbers are all ints, no need to include the .0 at the end of each entry
+    all_ints = True
+    for i, label1 in enumerate(gold_labels):
+        for j, label2 in enumerate(pred_labels):
+            if not isinstance(confusion.get(label1, {}).get(label2, 0), int):
+                all_ints = False
+                break
+        if not all_ints:
+            break
+
+    if all_ints:
+        format_cell = lambda confusion_cell: "%{0}d".format(columnwidth) % confusion_cell
+    else:
+        format_cell = lambda confusion_cell: "%{0}.1f".format(columnwidth) % confusion_cell
+
+    # make sure the columnwidth can handle long numbers
+    for i, label1 in enumerate(gold_labels):
+        for j, label2 in enumerate(pred_labels):
+            cell = confusion.get(label1, {}).get(label2, 0)
+            columnwidth = max(columnwidth, len(format_cell(cell)))
+
+    # if this is an NER confusion matrix (well, if it has - in the labels)
+    # try to drop a bunch of labels to make the matrix easier to display
+    if columnwidth * len(pred_labels) > 150:
+        confusion, gold_labels, pred_labels = condense_ner_labels(confusion, gold_labels, pred_labels)
+
+    # Print header
+    if transpose:
+        corner_label = "p\\t"
+    else:
+        corner_label = "t\\p"
+    fst_empty_cell = (columnwidth-3)//2 * " " + corner_label + (columnwidth-3)//2 * " "
+    if len(fst_empty_cell) < len(empty_cell):
+        fst_empty_cell = " " * (len(empty_cell) - len(fst_empty_cell)) + fst_empty_cell
+    header = "    " + fst_empty_cell + " "
+    for label in pred_labels:
+        header = header + "%{0}s ".format(columnwidth) % str(label)
+    text = [header.rstrip()]
+
+    # Print rows
+    for i, label1 in enumerate(gold_labels):
+        row = "    %{0}s ".format(columnwidth) % str(label1)
+        for j, label2 in enumerate(pred_labels):
+            confusion_cell = confusion.get(label1, {}).get(label2, 0)
+            cell = format_cell(confusion_cell)
+            if hide_zeroes:
+                cell = cell if confusion_cell else empty_cell
+            row = row + cell + " "
+        text.append(row.rstrip())
+    return "\n".join(text)
+
+
+def confusion_to_accuracy(confusion_matrix):
+    """
+    Given a confusion dictionary, return correct, total
+    """
+    correct = 0
+    total = 0
+    for l1 in confusion_matrix.keys():
+        for l2 in confusion_matrix[l1].keys():
+            if l1 == l2:
+                correct = correct + confusion_matrix[l1][l2]
+            else:
+                total = total + confusion_matrix[l1][l2]
+    return correct, (correct + total)
+
+def confusion_to_f1(confusion_matrix):
+    results = {}
+
+    keys = set()
+    for k in confusion_matrix.keys():
+        keys.add(k)
+        for k2 in confusion_matrix.get(k).keys():
+            keys.add(k2)
+
+    sum_f1 = 0
+    for k in keys:
+        tp = 0
+        fn = 0
+        fp = 0
+        for k2 in keys:
+            if k == k2:
+                tp = confusion_matrix.get(k, {}).get(k, 0)
+            else:
+                fn = fn + confusion_matrix.get(k, {}).get(k2, 0)
+                fp = fp + confusion_matrix.get(k2, {}).get(k, 0)
+        if tp + fp == 0:
+            precision = 0.0
+        else:
+            precision = tp / (tp + fp)
+        if tp + fn == 0:
+            recall = 0.0
+        else:
+            recall = tp / (tp + fn)
+        if precision + recall == 0.0:
+            f1 = 0.0
+        else:
+            f1 = 2 * (precision * recall) / (precision + recall)
+
+        results[k] = F1Result(precision, recall, f1)
+
+    return results
+
+def confusion_to_macro_f1(confusion_matrix):
+    """
+    Return the macro f1 for a confusion matrix.
+    """
+    sum_f1 = 0.0
+    results = confusion_to_f1(confusion_matrix)
+    for k in results.keys():
+        sum_f1 = sum_f1 + results[k].f1
+
+    return sum_f1 / len(results)
+
+def confusion_to_weighted_f1(confusion_matrix, exclude=None):
+    results = confusion_to_f1(confusion_matrix)
+
+    sum_f1 = 0.0
+    total_items = 0
+    for k in results.keys():
+        if exclude is not None and k in exclude:
+            continue
+        k_items = sum(confusion_matrix.get(k, {}).values())
+        total_items += k_items
+        sum_f1 += results[k].f1 * k_items
+    return sum_f1 / total_items

stanza/stanza/utils/conll.py

@@ -0,0 +1,205 @@
+"""
+Utility functions for the loading and conversion of CoNLL-format files.
+"""
+import os
+import io
+from zipfile import ZipFile
+
+from stanza.models.common.doc import Document
+from stanza.models.common.doc import ID, TEXT, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC, NER, START_CHAR, END_CHAR
+from stanza.models.common.doc import FIELD_TO_IDX, FIELD_NUM
+
+class CoNLLError(ValueError):
+    pass
+
+class CoNLL:
+
+    @staticmethod
+    def load_conll(f, ignore_gapping=True):
+        """ Load the file or string into the CoNLL-U format data.
+        Input: file or string reader, where the data is in CoNLL-U format.
+        Output: a tuple whose first element is a list of list of list for each token in each sentence in the data,
+        where the innermost list represents all fields of a token; and whose second element is a list of lists for each
+        comment in each sentence in the data.
+        """
+        # f is open() or io.StringIO()
+        doc, sent = [], []
+        doc_comments, sent_comments = [], []
+        for line_idx, line in enumerate(f):
+            # leave whitespace such as NBSP, in case it is meaningful in the conll-u doc
+            line = line.lstrip().rstrip(' \n\r\t')
+            if len(line) == 0:
+                if len(sent) > 0:
+                    doc.append(sent)
+                    sent = []
+                    doc_comments.append(sent_comments)
+                    sent_comments = []
+            else:
+                if line.startswith('#'): # read comment line
+                    sent_comments.append(line)
+                    continue
+                array = line.split('\t')
+                if ignore_gapping and '.' in array[0]:
+                    continue
+                if len(array) != FIELD_NUM:
+                    raise CoNLLError(f"Cannot parse CoNLL line {line_idx+1}: expecting {FIELD_NUM} fields, {len(array)} found at line {line_idx}\n  {array}")
+                sent += [array]
+        if len(sent) > 0:
+            doc.append(sent)
+            doc_comments.append(sent_comments)
+        return doc, doc_comments
+
+    @staticmethod
+    def convert_conll(doc_conll):
+        """ Convert the CoNLL-U format input data to a dictionary format output data.
+        Input: list of token fields loaded from the CoNLL-U format data, where the outmost list represents a list of sentences, and the inside list represents all fields of a token.
+        Output: a list of list of dictionaries for each token in each sentence in the document.
+        """
+        doc_dict = []
+        doc_empty = []
+        for sent_idx, sent_conll in enumerate(doc_conll):
+            sent_dict = []
+            sent_empty = []
+            for token_idx, token_conll in enumerate(sent_conll):
+                try:
+                    token_dict = CoNLL.convert_conll_token(token_conll)
+                except ValueError as e:
+                    raise CoNLLError("Could not process sentence %d token %d: %s" % (sent_idx, token_idx, str(e))) from e
+                if '.' in token_dict[ID]:
+                    token_dict[ID] = tuple(int(x) for x in token_dict[ID].split(".", maxsplit=1))
+                    sent_empty.append(token_dict)
+                else:
+                    token_dict[ID] = tuple(int(x) for x in token_dict[ID].split("-", maxsplit=1))
+                    sent_dict.append(token_dict)
+            doc_dict.append(sent_dict)
+            doc_empty.append(sent_empty)
+        return doc_dict, doc_empty
+
+    @staticmethod
+    def convert_dict(doc_dict):
+        """ Convert the dictionary format input data to the CoNLL-U format output data.
+
+        This is the reverse function of `convert_conll`, but does not include sentence level annotations or comments.
+
+        Can call this on a Document using `CoNLL.convert_dict(doc.to_dict())`
+
+        Input: dictionary format data, which is a list of list of dictionaries for each token in each sentence in the data.
+        Output: CoNLL-U format data as a list of list of list for each token in each sentence in the data.
+        """
+        doc = Document(doc_dict)
+        text = "{:c}".format(doc)
+        sentences = text.split("\n\n")
+        doc_conll = [[x.split("\t") for x in sentence.split("\n")] for sentence in sentences]
+        return doc_conll
+
+    @staticmethod
+    def convert_conll_token(token_conll):
+        """ Convert the CoNLL-U format input token to the dictionary format output token.
+        Input: a list of all CoNLL-U fields for the token.
+        Output: a dictionary that maps from field name to value.
+        """
+        token_dict = {}
+        for field in FIELD_TO_IDX:
+            value = token_conll[FIELD_TO_IDX[field]]
+            if value != '_':
+                if field == HEAD:
+                    token_dict[field] = int(value)
+                else:
+                    token_dict[field] = value
+            # special case if text is '_'
+            if token_conll[FIELD_TO_IDX[TEXT]] == '_':
+                token_dict[TEXT] = token_conll[FIELD_TO_IDX[TEXT]]
+                token_dict[LEMMA] = token_conll[FIELD_TO_IDX[LEMMA]]
+        return token_dict
+
+    @staticmethod
+    def conll2dict(input_file=None, input_str=None, ignore_gapping=True, zip_file=None):
+        """ Load the CoNLL-U format data from file or string into lists of dictionaries.
+        """
+        assert any([input_file, input_str]) and not all([input_file, input_str]), 'either use input file or input string'
+        if zip_file: assert input_file, 'must provide input_file if zip_file is set'
+
+        if input_str:
+            infile = io.StringIO(input_str)
+            doc_conll, doc_comments = CoNLL.load_conll(infile, ignore_gapping)
+        elif zip_file:
+            with ZipFile(zip_file) as zin:
+                with zin.open(input_file) as fin:
+                    doc_conll, doc_comments = CoNLL.load_conll(io.TextIOWrapper(fin, encoding="utf-8"), ignore_gapping)
+        else:
+            with open(input_file, encoding='utf-8') as fin:
+                doc_conll, doc_comments = CoNLL.load_conll(fin, ignore_gapping)
+
+        doc_dict, doc_empty = CoNLL.convert_conll(doc_conll)
+        return doc_dict, doc_comments, doc_empty
+
+    @staticmethod
+    def conll2doc(input_file=None, input_str=None, ignore_gapping=True, zip_file=None):
+        doc_dict, doc_comments, doc_empty = CoNLL.conll2dict(input_file, input_str, ignore_gapping, zip_file=zip_file)
+        return Document(doc_dict, text=None, comments=doc_comments, empty_sentences=doc_empty)
+
+    @staticmethod
+    def conll2multi_docs(input_file=None, input_str=None, ignore_gapping=True, zip_file=None):
+        doc_dict, doc_comments, doc_empty = CoNLL.conll2dict(input_file, input_str, ignore_gapping, zip_file=zip_file)
+
+        docs = []
+        current_doc = []
+        current_comments = []
+        current_empty = []
+        current_doc_id = None
+        for doc, comments, empty in zip(doc_dict, doc_comments, doc_empty):
+            for comment in comments:
+                if comment.startswith("# doc_id =") or comment.startswith("# newdoc id ="):
+                    doc_id = comment.split("=", maxsplit=1)[1]
+                    if len(current_doc) == 0:
+                        current_doc_id = doc_id
+                    elif doc_id != current_doc_id:
+                        new_doc = Document(current_doc, text=None, comments=current_comments, empty_sentences=current_empty)
+                        if current_doc_id != None:
+                            for i in new_doc.sentences:
+                                i.doc_id = current_doc_id.strip()
+                        docs.append(new_doc)
+                        current_doc_id = doc_id
+                    else:
+                        continue
+                    current_doc = [doc]
+                    current_comments = [comments]
+                    current_empty = [empty]
+                    break
+            else: # no comments defined a new doc_id, so just add it to the current document
+                current_doc.append(doc)
+                current_comments.append(comments)
+                current_empty.append(empty)
+        if len(current_doc) > 0:
+            new_doc = Document(current_doc, text=None, comments=current_comments, empty_sentences=current_empty)
+            if current_doc_id != None:
+                for i in new_doc.sentences:
+                    i.doc_id = current_doc_id.strip()
+            docs.append(new_doc)
+            current_doc_id = doc_id
+
+        return docs
+
+    @staticmethod
+    def dict2conll(doc_dict, filename):
+        """
+        Convert the dictionary format input data to the CoNLL-U format output data and write to a file.
+        """
+        doc = Document(doc_dict)
+        CoNLL.write_doc2conll(doc, filename)
+
+
+    @staticmethod
+    def write_doc2conll(doc, filename, mode='w', encoding='utf-8'):
+        """
+        Writes the doc as a conll file to the given file.
+
+        If passed a string, that filename will be opened.  Otherwise, filename.write() will be called.
+
+        Note that the output needs an extra \n\n at the end to be a legal output file
+        """
+        if hasattr(filename, "write"):
+            filename.write("{:C}\n\n".format(doc))
+        else:
+            with open(filename, mode, encoding=encoding) as outfile:
+                outfile.write("{:C}\n\n".format(doc))

stanza/stanza/utils/conll18_ud_eval.py

@@ -0,0 +1,832 @@
+#!/usr/bin/env python3
+
+# Code from CoNLL 2018 UD shared task updated for evaluation of enhanced
+# dependencies in IWPT 2020 shared task.
+# -- read DEPS, split on '|', compute overlap
+# New metrics ELAS and EULAS.
+# Gosse Bouma
+# New option --enhancements can switch off evaluation of certain types of
+# enhancements: default --enhancements 0 ... evaluate all enhancement types
+# 1 ... no gapping; 2 ... no coord shared parents; 3 ... no coord shared dependents
+# 4 ... no xsubj (control verbs); 5 ... no relative clauses; 6 ... no case info in deprels;
+# combinations: 12 ... both 1 and 2 apply
+
+# Compatible with Python 2.7 and 3.2+, can be used either as a module
+# or a standalone executable.
+#
+# Copyright 2017, 2018 Institute of Formal and Applied Linguistics (UFAL),
+# Faculty of Mathematics and Physics, Charles University, Czech Republic.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+#
+# Authors: Milan Straka, Martin Popel <surname@ufal.mff.cuni.cz>
+#
+# Changelog:
+# - [12 Apr 2018] Version 0.9: Initial release.
+# - [19 Apr 2018] Version 1.0: Fix bug in MLAS (duplicate entries in functional_children).
+#                              Add --counts option.
+# - [02 May 2018] Version 1.1: When removing spaces to match gold and system characters,
+#                              consider all Unicode characters of category Zs instead of
+#                              just ASCII space.
+# - [25 Jun 2018] Version 1.2: Use python3 in the she-bang (instead of python).
+#                              In Python2, make the whole computation use `unicode` strings.
+
+# Command line usage
+# ------------------
+# eval.py [-v] [-c] gold_conllu_file system_conllu_file
+#
+# - if no -v is given, only the official IWPT 2020 Shared Task evaluation metrics
+#   are printed
+# - if -v is given, more metrics are printed (as precision, recall, F1 score,
+#   and in case the metric is computed on aligned words also accuracy on these):
+#   - Tokens: how well do the gold tokens match system tokens
+#   - Sentences: how well do the gold sentences match system sentences
+#   - Words: how well can the gold words be aligned to system words
+#   - UPOS: using aligned words, how well does UPOS match
+#   - XPOS: using aligned words, how well does XPOS match
+#   - UFeats: using aligned words, how well does universal FEATS match
+#   - AllTags: using aligned words, how well does UPOS+XPOS+FEATS match
+#   - Lemmas: using aligned words, how well does LEMMA match
+#   - UAS: using aligned words, how well does HEAD match
+#   - LAS: using aligned words, how well does HEAD+DEPREL(ignoring subtypes) match
+#   - CLAS: using aligned words with content DEPREL, how well does
+#       HEAD+DEPREL(ignoring subtypes) match
+#   - MLAS: using aligned words with content DEPREL, how well does
+#       HEAD+DEPREL(ignoring subtypes)+UPOS+UFEATS+FunctionalChildren(DEPREL+UPOS+UFEATS) match
+#   - BLEX: using aligned words with content DEPREL, how well does
+#       HEAD+DEPREL(ignoring subtypes)+LEMMAS match
+# - if -c is given, raw counts of correct/gold_total/system_total/aligned words are printed
+#   instead of precision/recall/F1/AlignedAccuracy for all metrics.
+
+# API usage
+# ---------
+# - load_conllu(file)
+#   - loads CoNLL-U file from given file object to an internal representation
+#   - the file object should return str in both Python 2 and Python 3
+#   - raises UDError exception if the given file cannot be loaded
+# - evaluate(gold_ud, system_ud)
+#   - evaluate the given gold and system CoNLL-U files (loaded with load_conllu)
+#   - raises UDError if the concatenated tokens of gold and system file do not match
+#   - returns a dictionary with the metrics described above, each metric having
+#     three fields: precision, recall and f1
+
+# Description of token matching
+# -----------------------------
+# In order to match tokens of gold file and system file, we consider the text
+# resulting from concatenation of gold tokens and text resulting from
+# concatenation of system tokens. These texts should match -- if they do not,
+# the evaluation fails.
+#
+# If the texts do match, every token is represented as a range in this original
+# text, and tokens are equal only if their range is the same.
+
+# Description of word matching
+# ----------------------------
+# When matching words of gold file and system file, we first match the tokens.
+# The words which are also tokens are matched as tokens, but words in multi-word
+# tokens have to be handled differently.
+#
+# To handle multi-word tokens, we start by finding "multi-word spans".
+# Multi-word span is a span in the original text such that
+# - it contains at least one multi-word token
+# - all multi-word tokens in the span (considering both gold and system ones)
+#   are completely inside the span (i.e., they do not "stick out")
+# - the multi-word span is as small as possible
+#
+# For every multi-word span, we align the gold and system words completely
+# inside this span using LCS on their FORMs. The words not intersecting
+# (even partially) any multi-word span are then aligned as tokens.
+
+
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import io
+import sys
+import unicodedata
+import unittest
+
+# CoNLL-U column names
+ID, FORM, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC = range(10)
+
+# Content and functional relations
+CONTENT_DEPRELS = {
+    "nsubj", "obj", "iobj", "csubj", "ccomp", "xcomp", "obl", "vocative",
+    "expl", "dislocated", "advcl", "advmod", "discourse", "nmod", "appos",
+    "nummod", "acl", "amod", "conj", "fixed", "flat", "compound", "list",
+    "parataxis", "orphan", "goeswith", "reparandum", "root", "dep"
+}
+
+FUNCTIONAL_DEPRELS = {
+    "aux", "cop", "mark", "det", "clf", "case", "cc"
+}
+
+UNIVERSAL_FEATURES = {
+    "PronType", "NumType", "Poss", "Reflex", "Foreign", "Abbr", "Gender",
+    "Animacy", "Number", "Case", "Definite", "Degree", "VerbForm", "Mood",
+    "Tense", "Aspect", "Voice", "Evident", "Polarity", "Person", "Polite"
+}
+
+# UD Error is used when raising exceptions in this module
+class UDError(Exception):
+    pass
+
+# Conversion methods handling `str` <-> `unicode` conversions in Python2
+def _decode(text):
+    return text if sys.version_info[0] >= 3 or not isinstance(text, str) else text.decode("utf-8")
+
+def _encode(text):
+    return text if sys.version_info[0] >= 3 or not isinstance(text, unicode) else text.encode("utf-8")
+
+CASE_DEPRELS = {'obl','nmod','conj','advcl'}
+UNIVERSAL_DEPREL_EXTENSIONS = {'pass','relcl','xsubj'}
+
+# Modify the set of deps produced by system to be in accordance with gold treebank type.
+# Return a (filtered) list of (hd, dependency_path) tuples.
+def process_enhanced_deps(deps) :
+    edeps = []
+    if deps != '' and deps != '_':
+        for edep in deps.split('|') :
+            (hd, path) = edep.split(':', 1)
+            steps = path.split('>') # collapsing empty nodes gives rise to paths like this : 3:conj:en>obl:voor
+            edeps.append((hd,steps))   # (3,['conj:en','obj:voor'])
+    return edeps
+
+# Load given CoNLL-U file into internal representation.
+# The file parameter is the open file object.
+# The path parameter is needed only for diagnostic messages.
+def load_conllu(file, path, treebank_type):
+    # Internal representation classes
+    class UDRepresentation:
+        def __init__(self):
+            # Characters of all the tokens in the whole file.
+            # Whitespace between tokens is not included.
+            self.characters = []
+            # List of UDSpan instances with start&end indices into `characters`.
+            self.tokens = []
+            # List of UDWord instances.
+            self.words = []
+            # List of UDSpan instances with start&end indices into `characters`.
+            self.sentences = []
+            # File path may be needed in error messages.
+            self.path = ''
+    class UDSpan:
+        def __init__(self, start, end, line):
+            self.start = start
+            # Note that self.end marks the first position **after the end** of span,
+            # so we can use characters[start:end] or range(start, end).
+            self.end = end
+            # Line number (1-based) will be useful if we need to report an error later.
+            self.line = line
+    class UDWord:
+        def __init__(self, span, columns, is_multiword):
+            # Span of this word (or MWT, see below) within ud_representation.characters.
+            self.span = span
+            # 10 columns of the CoNLL-U file: ID, FORM, LEMMA,...
+            self.columns = columns
+            # is_multiword==True means that this word is part of a multi-word token.
+            # In that case, self.span marks the span of the whole multi-word token.
+            self.is_multiword = is_multiword
+            # Reference to the UDWord instance representing the HEAD (or None if root).
+            self.parent = None
+            # List of references to UDWord instances representing functional-deprel children.
+            self.functional_children = []
+            # Only consider universal FEATS.
+            self.columns[FEATS] = "|".join(sorted(feat for feat in columns[FEATS].split("|")
+                                                  if feat.split("=", 1)[0] in UNIVERSAL_FEATURES))
+            # Let's ignore language-specific deprel subtypes.
+            self.columns[DEPREL] = columns[DEPREL].split(":")[0]
+            # Precompute which deprels are CONTENT_DEPRELS and which FUNCTIONAL_DEPRELS
+            self.is_content_deprel = self.columns[DEPREL] in CONTENT_DEPRELS
+            self.is_functional_deprel = self.columns[DEPREL] in FUNCTIONAL_DEPRELS
+            # store enhanced deps --GB
+            # split string positions and enhanced labels as well?
+            self.columns[DEPS] = process_enhanced_deps(columns[DEPS])
+
+    ud = UDRepresentation()
+
+    # Load the CoNLL-U file
+    ud.path = path
+    index, sentence_start = 0, None
+    line_idx = 0
+    while True:
+        line = file.readline()
+        line_idx += 1 # errors will be displayed indexed from 1
+        if not line:
+            break
+        line = _decode(line.rstrip("\r\n"))
+
+        # Handle sentence start boundaries
+        if sentence_start is None:
+            # Skip comments
+            if line.startswith("#"):
+                continue
+            # Start a new sentence
+            ud.sentences.append(UDSpan(index, 0, line_idx))
+            sentence_start = len(ud.words)
+        if not line:
+            # Add parent and children UDWord links and check there are no cycles
+            def process_word(word):
+                if word.parent == "remapping":
+                    raise UDError("There is a cycle in the sentence that ends at line %d" % line_idx)
+                if word.parent is None:
+                    head = int(word.columns[HEAD])
+                    if head < 0 or head > len(ud.words) - sentence_start:
+                        raise UDError("HEAD '{}' points outside of the sentence that ends at line {}".format(_encode(word.columns[HEAD]), line_idx))
+                    if head:
+                        parent = ud.words[sentence_start + head - 1]
+                        word.parent = "remapping"
+                        process_word(parent)
+                        word.parent = parent
+
+            position = sentence_start # need to incrementally keep track of current position for loop detection in relcl
+            for word in ud.words[sentence_start:]:
+                process_word(word)
+                enhanced_deps = word.columns[DEPS]
+                # replace head positions of enhanced dependencies with parent word object -- GB
+                processed_deps = []
+                for (head,steps) in word.columns[DEPS] :       # (3,['conj:en','obj:voor'])
+                    # Empty nodes should have been collapsed during preprocessing.
+                    # If not, we cannot evaluate gapping correctly. However, people
+                    # may care just about basic trees and may not want to bother
+                    # with preprocessing.
+                    if '.' in head:
+                        if treebank_type.get('no_empty_nodes', False):
+                            raise UDError("The collapsed CoNLL-U file still contains references to empty nodes at line {}: {}".format(line_idx, _encode(line)))
+                        else:
+                            continue
+                    hd = int(head)
+                    parent = ud.words[sentence_start + hd -1] if hd else hd  # just assign '0' to parent for root cases
+                    processed_deps.append((parent,steps))
+                enhanced_deps = processed_deps
+
+                # ignore rel>rel dependencies, and instead append the original hd/rel edge
+                # note that this also ignores other extensions (like adding lemma's)
+                # note that this sometimes introduces duplicates (if orig hd/rel was already included in DEPS)
+                if treebank_type.get('no_gapping', False) : # enhancement 1
+                    processed_deps = []
+                    for (parent,steps) in enhanced_deps :
+                        if len(steps) > 1 :
+                            processed_deps.append((word.parent,[word.columns[DEPREL]]))
+                        else :
+                            if (parent,steps) in processed_deps :
+                                True
+                            else :
+                                processed_deps.append((parent,steps))
+                    enhanced_deps = processed_deps
+
+                # for a given conj node, any rel other than conj in DEPS can be ignored
+                if treebank_type.get('no_shared_parents_in_coordination', False) :   # enhancement  2
+                    for (hd,steps) in enhanced_deps :
+                        if len(steps) == 1 and steps[0].startswith('conj') :
+                            enhanced_deps = [(hd,steps)]
+
+                # deprels not matching ud_hd/ud_dep are spurious.
+                #  czech/pud estonian/ewt syntagrus finnish/pud
+                # TO DO: treebanks that do not mark xcomp and relcl subjects
+                if treebank_type.get('no_shared_dependents_in_coordination', False) : # enhancement  3
+                    processed_deps = []
+                    for (hd,steps) in enhanced_deps :
+                        duplicate = 0
+                        for (hd2,steps2) in enhanced_deps :
+                            if steps == steps2 and hd2 == word.columns[HEAD]  and hd != hd2  : # checking only for ud_hd here, check for ud_dep as well?
+                                duplicate = 1
+                        if not(duplicate) :
+                            processed_deps.append((hd,steps))
+                    enhanced_deps = processed_deps
+
+                # if treebank does not have control relations: subjects of xcomp parents in system are to be skipped
+                # note that rel is actually a path sometimes rel1>rel2 in theory rel2 could be subj?
+                # from lassy-small: 7:conj:en>nsubj:pass|7:conj:en>nsubj:xsubj    (7,['conj:en','nsubj:xsubj'])
+                if treebank_type.get('no_control', False) : # enhancement 4
+                    processed_deps = []
+                    for (parent,steps) in enhanced_deps :
+                        include = 1
+                        if ( parent and parent.columns[DEPREL] == 'xcomp') :
+                            for rel in steps:
+                                if rel.startswith('nsubj') :
+                                    include = 0
+                        if include :
+                            processed_deps.append((parent,steps))
+                    enhanced_deps = processed_deps
+
+                if treebank_type.get('no_external_arguments_of_relative_clauses', False) : # enhancement 5
+                    processed_deps = []
+                    for (parent,steps) in enhanced_deps :
+                        if (steps[0] == 'ref') :
+                            processed_deps.append((word.parent,[word.columns[DEPREL]]))  # append the original relation
+                        # ignore external argument link
+                        # external args are deps of an acl:relcl where that acl also is a dependent of external arg (i.e. ext arg introduces a cycle)
+                        elif ( parent and parent.columns[DEPREL].startswith('acl')  and int(parent.columns[HEAD]) == position - sentence_start ) :
+                            #print('removed external argument')
+                            True
+                        else :
+                            processed_deps.append((parent,steps))
+                    enhanced_deps = processed_deps
+
+                # treebanks where no lemma info has been added
+                if treebank_type.get('no_case_info', False) :  # enhancement number 6
+                    processed_deps = []
+                    for (hd,steps) in enhanced_deps :
+                        processed_steps = []
+                        for dep in steps :
+                            depparts = dep.split(':')
+                            if depparts[0] in  CASE_DEPRELS :
+                                if (len(depparts) == 2 and not(depparts[1] in UNIVERSAL_DEPREL_EXTENSIONS )) :
+                                    dep = depparts[0]
+                            processed_steps.append(dep)
+                        processed_deps.append((hd,processed_steps))
+                    enhanced_deps = processed_deps
+
+                position += 1
+                word.columns[DEPS] = enhanced_deps
+
+            # func_children cannot be assigned within process_word
+            # because it is called recursively and may result in adding one child twice.
+            for word in ud.words[sentence_start:]:
+                if word.parent and word.is_functional_deprel:
+                    word.parent.functional_children.append(word)
+
+            if len(ud.words) == sentence_start :
+                raise UDError("There is a sentence with 0 tokens (possibly a double blank line) at line %d" % line_idx)
+
+            # Check there is a single root node
+            if len([word for word in ud.words[sentence_start:] if word.parent is None]) == 0:
+                raise UDError("There are no roots in the sentence that ends at %d" % line_idx)
+            if not treebank_type.get('multiple_roots_okay', False):
+                if len([word for word in ud.words[sentence_start:] if word.parent is None]) > 1:
+                    raise UDError("There are multiple roots in the sentence that ends at %d" % line_idx)
+
+            # End the sentence
+            ud.sentences[-1].end = index
+            sentence_start = None
+            continue
+
+        # Read next token/word
+        columns = line.split("\t")
+        if len(columns) != 10:
+            raise UDError("The CoNLL-U line does not contain 10 tab-separated columns at line {}: '{}'".format(line_idx, _encode(line)))
+
+        # Skip empty nodes
+        # If we are evaluating enhanced graphs, empty nodes should have been collapsed
+        # during preprocessing and should not occur here. However, we cannot raise
+        # an exception if they do because the user may be interested just in the
+        # basic tree and may not want to bother with preprocessing.
+        if "." in columns[ID]:
+            # When launching this script, we can specify that empty nodes should be considered errors.
+            if treebank_type.get('no_empty_nodes', False):
+                raise UDError("The collapsed CoNLL-U line still contains empty nodes at line {}: {}".format(line_idx, _encode(line)))
+            else:
+                continue
+
+        # Delete spaces from FORM, so gold.characters == system.characters
+        # even if one of them tokenizes the space. Use any Unicode character
+        # with category Zs.
+        columns[FORM] = "".join(filter(lambda c: unicodedata.category(c) != "Zs", columns[FORM]))
+        if not columns[FORM]:
+            raise UDError("There is an empty FORM in the CoNLL-U file at line %d" % line_idx)
+
+        # Save token
+        ud.characters.extend(columns[FORM])
+        ud.tokens.append(UDSpan(index, index + len(columns[FORM]), line_idx))
+        index += len(columns[FORM])
+
+        # Handle multi-word tokens to save word(s)
+        if "-" in columns[ID]:
+            try:
+                start, end = map(int, columns[ID].split("-"))
+            except:
+                raise UDError("Cannot parse multi-word token ID '{}' at line {}".format(_encode(columns[ID]), line_idx))
+
+            words_expected = end - start + 1
+            words_found = 0
+            while words_found < words_expected:
+                word_line = _decode(file.readline().rstrip("\r\n"))
+                line_idx += 1
+                word_columns = word_line.split("\t")
+                if len(word_columns) != 10:
+                    raise UDError("The CoNLL-U line does not contain 10 tab-separated columns at line {}: '{}'".format(line_idx, _encode(word_line)))
+                if "." in word_columns[ID]:
+                    if treebank_type.get('no_empty_nodes', False):
+                        raise UDError("The collapsed CoNLL-U line still contains empty nodes at line {}: {}".format(line_idx, _encode(line)))
+                    else:
+                        continue
+                ud.words.append(UDWord(ud.tokens[-1], word_columns, is_multiword=True))
+                words_found += 1
+
+        # Basic tokens/words
+        else:
+            try:
+                word_id = int(columns[ID])
+            except:
+                raise UDError("Cannot parse word ID '{}' at line {}".format(_encode(columns[ID]), line_idx))
+            if word_id != len(ud.words) - sentence_start + 1:
+                raise UDError("Incorrect word ID '{}' for word '{}', expected '{}' at line {}".format(
+                    _encode(columns[ID]), _encode(columns[FORM]), len(ud.words) - sentence_start + 1, line_idx))
+
+            try:
+                head_id = int(columns[HEAD])
+            except ValueError as e:
+                raise UDError("Cannot parse HEAD '{}' at line {}".format(_encode(columns[HEAD]), line_idx)) from e
+            if head_id < 0:
+                raise UDError("HEAD cannot be negative at line %d" % line_idx)
+
+            ud.words.append(UDWord(ud.tokens[-1], columns, is_multiword=False))
+
+    if sentence_start is not None:
+        raise UDError("The CoNLL-U file does not end with empty line")
+
+    return ud
+
+# Evaluate the gold and system treebanks (loaded using load_conllu).
+def evaluate(gold_ud, system_ud):
+    class Score:
+        def __init__(self, gold_total, system_total, correct, aligned_total=None):
+            self.correct = correct
+            self.gold_total = gold_total
+            self.system_total = system_total
+            self.aligned_total = aligned_total
+            self.precision = correct / system_total if system_total else 0.0
+            self.recall = correct / gold_total if gold_total else 0.0
+            self.f1 = 2 * correct / (system_total + gold_total) if system_total + gold_total else 0.0
+            self.aligned_accuracy = correct / aligned_total if aligned_total else aligned_total
+    class AlignmentWord:
+        def __init__(self, gold_word, system_word):
+            self.gold_word = gold_word
+            self.system_word = system_word
+    class Alignment:
+        def __init__(self, gold_words, system_words):
+            self.gold_words = gold_words
+            self.system_words = system_words
+            self.matched_words = []
+            self.matched_words_map = {}
+        def append_aligned_words(self, gold_word, system_word):
+            self.matched_words.append(AlignmentWord(gold_word, system_word))
+            self.matched_words_map[system_word] = gold_word
+
+    def spans_score(gold_spans, system_spans):
+        correct, gi, si = 0, 0, 0
+        while gi < len(gold_spans) and si < len(system_spans):
+            if system_spans[si].start < gold_spans[gi].start:
+                si += 1
+            elif gold_spans[gi].start < system_spans[si].start:
+                gi += 1
+            else:
+                correct += gold_spans[gi].end == system_spans[si].end
+                si += 1
+                gi += 1
+
+        return Score(len(gold_spans), len(system_spans), correct)
+
+    def alignment_score(alignment, key_fn=None, filter_fn=None):
+        if filter_fn is not None:
+            gold = sum(1 for gold in alignment.gold_words if filter_fn(gold))
+            system = sum(1 for system in alignment.system_words if filter_fn(system))
+            aligned = sum(1 for word in alignment.matched_words if filter_fn(word.gold_word))
+        else:
+            gold = len(alignment.gold_words)
+            system = len(alignment.system_words)
+            aligned = len(alignment.matched_words)
+
+        if key_fn is None:
+            # Return score for whole aligned words
+            return Score(gold, system, aligned)
+
+        def gold_aligned_gold(word):
+            return word
+        def gold_aligned_system(word):
+            return alignment.matched_words_map.get(word, 'NotAligned') if word is not None else None
+        correct = 0
+        for words in alignment.matched_words:
+            if filter_fn is None or filter_fn(words.gold_word):
+                if key_fn(words.gold_word, gold_aligned_gold) == key_fn(words.system_word, gold_aligned_system):
+                    correct += 1
+
+        return Score(gold, system, correct, aligned)
+
+    def enhanced_alignment_score(alignment, EULAS):
+        # count all matching enhanced deprels in gold, system GB
+        # gold and system = sum of gold and predicted deps
+        # parents are pointers to word object, make sure to compare system parent with aligned word in gold in cases where
+        # tokenization introduces mismatches in number of words per sentence.
+        gold = 0
+        for gold_word in alignment.gold_words :
+            gold += len(gold_word.columns[DEPS])
+        system = 0
+        for system_word in alignment.system_words :
+            system += len(system_word.columns[DEPS])
+        correct = 0
+        for words in alignment.matched_words:
+            gold_deps = words.gold_word.columns[DEPS]
+            system_deps = words.system_word.columns[DEPS]
+            for (parent, dep) in gold_deps :
+                eulas_dep = [d.split(':')[0] for d in dep]
+                for (sparent, sdep) in system_deps:
+                    eulas_sdep = [d.split(':')[0] for d in sdep]
+                    if dep == sdep or ( eulas_dep == eulas_sdep and EULAS ) :
+                        if parent == alignment.matched_words_map.get(sparent, 'NotAligned') :
+                            correct += 1
+                        elif (parent == 0 and sparent == 0) :  # cases where parent is root
+                            correct += 1
+        return Score(gold, system, correct)
+
+    def beyond_end(words, i, multiword_span_end):
+        if i >= len(words):
+            return True
+        if words[i].is_multiword:
+            return words[i].span.start >= multiword_span_end
+        return words[i].span.end > multiword_span_end
+
+    def extend_end(word, multiword_span_end):
+        if word.is_multiword and word.span.end > multiword_span_end:
+            return word.span.end
+        return multiword_span_end
+
+    def find_multiword_span(gold_words, system_words, gi, si):
+        # We know gold_words[gi].is_multiword or system_words[si].is_multiword.
+        # Find the start of the multiword span (gs, ss), so the multiword span is minimal.
+        # Initialize multiword_span_end characters index.
+        if gold_words[gi].is_multiword:
+            multiword_span_end = gold_words[gi].span.end
+            if not system_words[si].is_multiword and system_words[si].span.start < gold_words[gi].span.start:
+                si += 1
+        else: # if system_words[si].is_multiword
+            multiword_span_end = system_words[si].span.end
+            if not gold_words[gi].is_multiword and gold_words[gi].span.start < system_words[si].span.start:
+                gi += 1
+        gs, ss = gi, si
+
+        # Find the end of the multiword span
+        # (so both gi and si are pointing to the word following the multiword span end).
+        while not beyond_end(gold_words, gi, multiword_span_end) or \
+              not beyond_end(system_words, si, multiword_span_end):
+            if gi < len(gold_words) and (si >= len(system_words) or
+                                         gold_words[gi].span.start <= system_words[si].span.start):
+                multiword_span_end = extend_end(gold_words[gi], multiword_span_end)
+                gi += 1
+            else:
+                multiword_span_end = extend_end(system_words[si], multiword_span_end)
+                si += 1
+        return gs, ss, gi, si
+
+    def compute_lcs(gold_words, system_words, gi, si, gs, ss):
+        lcs = [[0] * (si - ss) for i in range(gi - gs)]
+        for g in reversed(range(gi - gs)):
+            for s in reversed(range(si - ss)):
+                if gold_words[gs + g].columns[FORM].lower() == system_words[ss + s].columns[FORM].lower():
+                    lcs[g][s] = 1 + (lcs[g+1][s+1] if g+1 < gi-gs and s+1 < si-ss else 0)
+                lcs[g][s] = max(lcs[g][s], lcs[g+1][s] if g+1 < gi-gs else 0)
+                lcs[g][s] = max(lcs[g][s], lcs[g][s+1] if s+1 < si-ss else 0)
+        return lcs
+
+    def align_words(gold_words, system_words):
+        alignment = Alignment(gold_words, system_words)
+
+        gi, si = 0, 0
+        while gi < len(gold_words) and si < len(system_words):
+            if gold_words[gi].is_multiword or system_words[si].is_multiword:
+                # A: Multi-word tokens => align via LCS within the whole "multiword span".
+                gs, ss, gi, si = find_multiword_span(gold_words, system_words, gi, si)
+
+                if si > ss and gi > gs:
+                    lcs = compute_lcs(gold_words, system_words, gi, si, gs, ss)
+
+                    # Store aligned words
+                    s, g = 0, 0
+                    while g < gi - gs and s < si - ss:
+                        if gold_words[gs + g].columns[FORM].lower() == system_words[ss + s].columns[FORM].lower():
+                            alignment.append_aligned_words(gold_words[gs+g], system_words[ss+s])
+                            g += 1
+                            s += 1
+                        elif lcs[g][s] == (lcs[g+1][s] if g+1 < gi-gs else 0):
+                            g += 1
+                        else:
+                            s += 1
+            else:
+                # B: No multi-word token => align according to spans.
+                if (gold_words[gi].span.start, gold_words[gi].span.end) == (system_words[si].span.start, system_words[si].span.end):
+                    alignment.append_aligned_words(gold_words[gi], system_words[si])
+                    gi += 1
+                    si += 1
+                elif gold_words[gi].span.start <= system_words[si].span.start:
+                    gi += 1
+                else:
+                    si += 1
+
+        return alignment
+
+    # Check that the underlying character sequences match.
+    if gold_ud.characters != system_ud.characters:
+        # Identify the surrounding tokens and line numbers so the error is easier to debug.
+        index = 0
+        while index < len(gold_ud.characters) and index < len(system_ud.characters) and \
+                gold_ud.characters[index] == system_ud.characters[index]:
+            index += 1
+        gtindex = 0
+        while gtindex < len(gold_ud.tokens) and gold_ud.tokens[gtindex].end-1 < index:
+            gtindex += 1
+        stindex = 0
+        while stindex < len(system_ud.tokens) and system_ud.tokens[stindex].end-1 < index:
+            stindex += 1
+        gtokenreport = "The error occurs right at the beginning of the two files.\n"
+        stokenreport = ""
+        if gtindex > 0:
+            nprev = 10 if gtindex >= 10 else gtindex
+            nnext = 10 if gtindex + 10 <= len(gold_ud.tokens) else len(gold_ud.tokens) - gtindex
+            nfirst = gtindex - nprev
+            prevtokens = ' '.join([''.join(gold_ud.characters[t.start:t.end]) for t in gold_ud.tokens[nfirst:gtindex]])
+            nexttokens = ' '.join([''.join(gold_ud.characters[t.start:t.end]) for t in gold_ud.tokens[gtindex:gtindex + nnext]])
+            gtokenreport = "File '{}':\n".format(gold_ud.path)
+            gtokenreport += "  Token no. {} on line no. {} is the last one with all characters reproduced in the other file.\n".format(gtindex, gold_ud.tokens[gtindex-1].line)
+            gtokenreport += "  The previous {} tokens are '{}'.\n".format(nprev, prevtokens)
+            gtokenreport += "  The next {} tokens are '{}'.\n".format(nnext, nexttokens)
+        if stindex > 0:
+            nprev = 10 if stindex >= 10 else stindex
+            nnext = 10 if stindex + 10 <= len(system_ud.tokens) else len(system_ud.tokens) - stindex
+            nfirst = stindex - nprev
+            prevtokens = ' '.join([''.join(system_ud.characters[t.start:t.end]) for t in system_ud.tokens[nfirst:stindex]])
+            nexttokens = ' '.join([''.join(system_ud.characters[t.start:t.end]) for t in system_ud.tokens[stindex:stindex + nnext]])
+            stokenreport = "File '{}':\n".format(system_ud.path)
+            stokenreport += "  Token no. {} on line no. {} is the last one with all characters reproduced in the other file.\n".format(stindex, system_ud.tokens[stindex-1].line)
+            stokenreport += "  The previous {} tokens are '{}'.\n".format(nprev, prevtokens)
+            stokenreport += "  The next {} tokens are '{}'.\n".format(nnext, nexttokens)
+        raise UDError(
+            "The concatenation of tokens in gold file and in system file differ!\n" + gtokenreport + stokenreport +
+            "First 20 differing characters in gold file: '{}' and system file: '{}'".format(
+                "".join(map(_encode, gold_ud.characters[index:index + 20])),
+                "".join(map(_encode, system_ud.characters[index:index + 20]))
+            )
+        )
+
+    # Align words
+    alignment = align_words(gold_ud.words, system_ud.words)
+
+    # Compute the F1-scores
+    return {
+        "Tokens": spans_score(gold_ud.tokens, system_ud.tokens),
+        "Sentences": spans_score(gold_ud.sentences, system_ud.sentences),
+        "Words": alignment_score(alignment),
+        "UPOS": alignment_score(alignment, lambda w, _: w.columns[UPOS]),
+        "XPOS": alignment_score(alignment, lambda w, _: w.columns[XPOS]),
+        "UFeats": alignment_score(alignment, lambda w, _: w.columns[FEATS]),
+        "AllTags": alignment_score(alignment, lambda w, _: (w.columns[UPOS], w.columns[XPOS], w.columns[FEATS])),
+        "Lemmas": alignment_score(alignment, lambda w, ga: w.columns[LEMMA] if ga(w).columns[LEMMA] != "_" else "_"),
+        "UAS": alignment_score(alignment, lambda w, ga: ga(w.parent)),
+        "LAS": alignment_score(alignment, lambda w, ga: (ga(w.parent), w.columns[DEPREL])),
+        "ELAS": enhanced_alignment_score(alignment, 0),
+        "EULAS": enhanced_alignment_score(alignment, 1),
+        "CLAS": alignment_score(alignment, lambda w, ga: (ga(w.parent), w.columns[DEPREL]),
+                                filter_fn=lambda w: w.is_content_deprel),
+        "MLAS": alignment_score(alignment, lambda w, ga: (ga(w.parent), w.columns[DEPREL], w.columns[UPOS], w.columns[FEATS],
+                                                         [(ga(c), c.columns[DEPREL], c.columns[UPOS], c.columns[FEATS])
+                                                          for c in w.functional_children]),
+                                filter_fn=lambda w: w.is_content_deprel),
+        "BLEX": alignment_score(alignment, lambda w, ga: (ga(w.parent), w.columns[DEPREL],
+                                                          w.columns[LEMMA] if ga(w).columns[LEMMA] != "_" else "_"),
+                                filter_fn=lambda w: w.is_content_deprel),
+    }
+
+def load_conllu_file(path, treebank_type=None):
+    if treebank_type is None:
+        treebank_type = {}
+    _file = open(path, mode="r", **({"encoding": "utf-8"} if sys.version_info >= (3, 0) else {}))
+    return load_conllu(_file, path, treebank_type)
+
+def evaluate_wrapper(args):
+    treebank_type = {}
+    enhancements = list(args.enhancements)
+    treebank_type['no_gapping'] = 1 if '1' in enhancements else 0
+    treebank_type['no_shared_parents_in_coordination'] = 1 if '2' in enhancements else 0
+    treebank_type['no_shared_dependents_in_coordination'] = 1 if '3' in enhancements else 0
+    treebank_type['no_control'] = 1 if '4' in enhancements else 0
+    treebank_type['no_external_arguments_of_relative_clauses'] = 1 if '5' in enhancements else 0
+    treebank_type['no_case_info'] = 1 if '6' in enhancements else 0
+    treebank_type['no_empty_nodes'] = args.no_empty_nodes
+    treebank_type['multiple_roots_okay'] = args.multiple_roots_okay
+
+    # Load CoNLL-U files
+    gold_ud = load_conllu_file(args.gold_file, treebank_type)
+    system_ud = load_conllu_file(args.system_file, treebank_type)
+    return evaluate(gold_ud, system_ud)
+
+def build_evaluation_table(evaluation, verbose, counts, enhanced):
+    text = []
+
+    # Print the evaluation
+    if not verbose and not counts:
+        text.append("LAS F1 Score: {:.2f}".format(100 * evaluation["LAS"].f1))
+        text.append("MLAS Score: {:.2f}".format(100 * evaluation["MLAS"].f1))
+        text.append("BLEX Score: {:.2f}".format(100 * evaluation["BLEX"].f1))
+        if enhanced:
+            text.append("ELAS F1 Score: {:.2f}".format(100 * evaluation["ELAS"].f1))
+            text.append("EULAS F1 Score: {:.2f}".format(100 * evaluation["EULAS"].f1))
+    else:
+        if counts:
+            text.append("Metric     | Correct   |      Gold | Predicted | Aligned")
+        else:
+            text.append("Metric     | Precision |    Recall |  F1 Score | AligndAcc")
+        text.append("-----------+-----------+-----------+-----------+-----------")
+        metrics = ["Tokens", "Sentences", "Words", "UPOS", "XPOS", "UFeats", "AllTags", "Lemmas", "UAS", "LAS", "CLAS", "MLAS", "BLEX"]
+        if enhanced:
+            metrics += ["ELAS", "EULAS"]
+        for metric in metrics:
+            if counts:
+                text.append("{:11}|{:10} |{:10} |{:10} |{:10}".format(
+                    metric,
+                    evaluation[metric].correct,
+                    evaluation[metric].gold_total,
+                    evaluation[metric].system_total,
+                    evaluation[metric].aligned_total or (evaluation[metric].correct if metric == "Words" else "")
+                ))
+            else:
+                text.append("{:11}|{:10.2f} |{:10.2f} |{:10.2f} |{}".format(
+                    metric,
+                    100 * evaluation[metric].precision,
+                    100 * evaluation[metric].recall,
+                    100 * evaluation[metric].f1,
+                    "{:10.2f}".format(100 * evaluation[metric].aligned_accuracy) if evaluation[metric].aligned_accuracy is not None else ""
+                ))
+
+    return "\n".join(text)
+
+def main():
+    # Parse arguments
+    parser = argparse.ArgumentParser()
+    parser.add_argument('gold_file', type=str,
+                        help='Name of the CoNLL-U file with the gold data.')
+    parser.add_argument('system_file', type=str,
+                        help='Name of the CoNLL-U file with the predicted data.')
+    parser.add_argument('--verbose', '-v', default=False, action='store_true',
+                        help='Print all metrics.')
+    parser.add_argument('--counts', '-c', default=False, action='store_true',
+                        help='Print raw counts of correct/gold/system/aligned words instead of precision/recall/F1 for all metrics.')
+    parser.add_argument('--no-enhanced', dest='enhanced', action='store_false', default=True,
+                        help='Turn off evaluation of enhanced dependencies.')
+    parser.add_argument('--enhancements', type=str, default='0',
+                        help='Level of enhancements in the gold data (see guidelines) 0=all (default), 1=no gapping, 2=no shared parents, 3=no shared dependents 4=no control, 5=no external arguments, 6=no lemma info, combinations: 12=both 1 and 2 apply, etc.')
+    parser.add_argument('--no-empty-nodes', default=False,
+                        help='Empty nodes have been collapsed (needed to correctly evaluate enhanced/gapping). Raise exception if an empty node is encountered.')
+    parser.add_argument('--multiple-roots-okay', default=False, action='store_true',
+                        help='A single sentence can have multiple nodes with HEAD=0.')
+    args = parser.parse_args()
+
+    # Evaluate
+    evaluation = evaluate_wrapper(args)
+    results = build_evaluation_table(evaluation, args.verbose, args.counts, args.enhanced)
+    print(results)
+
+if __name__ == "__main__":
+    main()
+
+# Tests, which can be executed with `python -m unittest conll18_ud_eval`.
+class TestAlignment(unittest.TestCase):
+    @staticmethod
+    def _load_words(words):
+        """Prepare fake CoNLL-U files with fake HEAD to prevent multiple roots errors."""
+        lines, num_words = [], 0
+        for w in words:
+            parts = w.split(" ")
+            if len(parts) == 1:
+                num_words += 1
+                lines.append("{}\t{}\t_\t_\t_\t_\t{}\t_\t_\t_".format(num_words, parts[0], int(num_words>1)))
+            else:
+                lines.append("{}-{}\t{}\t_\t_\t_\t_\t_\t_\t_\t_".format(num_words + 1, num_words + len(parts) - 1, parts[0]))
+                for part in parts[1:]:
+                    num_words += 1
+                    lines.append("{}\t{}\t_\t_\t_\t_\t{}\t_\t_\t_".format(num_words, part, int(num_words>1)))
+        return load_conllu((io.StringIO if sys.version_info >= (3, 0) else io.BytesIO)("\n".join(lines+["\n"])))
+
+    def _test_exception(self, gold, system):
+        self.assertRaises(UDError, evaluate, self._load_words(gold), self._load_words(system))
+
+    def _test_ok(self, gold, system, correct):
+        metrics = evaluate(self._load_words(gold), self._load_words(system))
+        gold_words = sum((max(1, len(word.split(" ")) - 1) for word in gold))
+        system_words = sum((max(1, len(word.split(" ")) - 1) for word in system))
+        self.assertEqual((metrics["Words"].precision, metrics["Words"].recall, metrics["Words"].f1),
+                         (correct / system_words, correct / gold_words, 2 * correct / (gold_words + system_words)))
+
+    def test_exception(self):
+        self._test_exception(["a"], ["b"])
+
+    def test_equal(self):
+        self._test_ok(["a"], ["a"], 1)
+        self._test_ok(["a", "b", "c"], ["a", "b", "c"], 3)
+
+    def test_equal_with_multiword(self):
+        self._test_ok(["abc a b c"], ["a", "b", "c"], 3)
+        self._test_ok(["a", "bc b c", "d"], ["a", "b", "c", "d"], 4)
+        self._test_ok(["abcd a b c d"], ["ab a b", "cd c d"], 4)
+        self._test_ok(["abc a b c", "de d e"], ["a", "bcd b c d", "e"], 5)
+
+    def test_alignment(self):
+        self._test_ok(["abcd"], ["a", "b", "c", "d"], 0)
+        self._test_ok(["abc", "d"], ["a", "b", "c", "d"], 1)
+        self._test_ok(["a", "bc", "d"], ["a", "b", "c", "d"], 2)
+        self._test_ok(["a", "bc b c", "d"], ["a", "b", "cd"], 2)
+        self._test_ok(["abc a BX c", "def d EX f"], ["ab a b", "cd c d", "ef e f"], 4)
+        self._test_ok(["ab a b", "cd bc d"], ["a", "bc", "d"], 2)
+        self._test_ok(["a", "bc b c", "d"], ["ab AX BX", "cd CX a"], 1)

stanza/stanza/utils/helper_func.py

@@ -0,0 +1,38 @@
+def make_table(header, content, column_width=None):
+    '''
+    Input:
+    header -> List[str]: table header
+    content -> List[List[str]]: table content
+    column_width -> int: table column width; set to None for dynamically calculated widths
+    
+    Output:
+    table_str -> str: well-formatted string for the table
+    '''
+    table_str = ''
+    len_column, len_row = len(header), len(content) + 1
+    if column_width is None:
+        # dynamically decide column widths
+        lens = [[len(str(h)) for h in header]]
+        lens += [[len(str(x)) for x in row] for row in content]
+        column_widths = [max(c)+3 for c in zip(*lens)]
+    else:
+        column_widths = [column_width] * len_column
+    
+    table_str += '=' * (sum(column_widths) + 1) + '\n'
+    
+    table_str += '|'
+    for i, item in enumerate(header):
+        table_str += ' ' + str(item).ljust(column_widths[i] - 2) + '|'
+    table_str += '\n'
+    
+    table_str += '-' * (sum(column_widths) + 1) + '\n'
+    
+    for line in content:
+        table_str += '|'
+        for i, item in enumerate(line):
+            table_str += ' ' + str(item).ljust(column_widths[i] - 2) + '|'
+        table_str += '\n'
+    
+    table_str += '=' * (sum(column_widths) + 1) + '\n'
+    
+    return table_str