personal_site/markdown_generator/publications.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Publications markdown generator for academicpages\n",
    "\n",
    "Takes a TSV of publications with metadata and converts them for use with [academicpages.github.io](academicpages.github.io). This is an interactive Jupyter notebook ([see more info here](http://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html)). The core python code is also in `publications.py`. Run either from the `markdown_generator` folder after replacing `publications.tsv` with one containing your data.\n",
    "\n",
    "TODO: Make this work with BibTex and other databases of citations, rather than Stuart's non-standard TSV format and citation style.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data format\n",
    "\n",
    "The TSV needs to have the following columns: pub_date, title, venue, excerpt, citation, site_url, and paper_url, with a header at the top. \n",
    "\n",
    "- `excerpt` and `paper_url` can be blank, but the others must have values. \n",
    "- `pub_date` must be formatted as YYYY-MM-DD.\n",
    "- `url_slug` will be the descriptive part of the .md file and the permalink URL for the page about the paper. The .md file will be `YYYY-MM-DD-[url_slug].md` and the permalink will be `https://[yourdomain]/publications/YYYY-MM-DD-[url_slug]`\n",
    "\n",
    "This is how the raw file looks (it doesn't look pretty, use a spreadsheet or other program to edit and create)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pub_date\ttitle\tvenue\texcerpt\tcitation\turl_slug\tpaper_url\r\n",
      "2009-10-01\tPaper Title Number 1\tJournal 1\tThis paper is about the number 1. The number 2 is left for future work.\tYour Name, You. (2009). \"Paper Title Number 1.\" <i>Journal 1</i>. 1(1).\tpaper-title-number-1\thttp://academicpages.github.io/files/paper1.pdf\r\n",
      "2010-10-01\tPaper Title Number 2\tJournal 1\tThis paper is about the number 2. The number 3 is left for future work.\tYour Name, You. (2010). \"Paper Title Number 2.\" <i>Journal 1</i>. 1(2).\tpaper-title-number-2\thttp://academicpages.github.io/files/paper2.pdf\r\n",
      "2015-10-01\tPaper Title Number 3\tJournal 1\tThis paper is about the number 3. The number 4 is left for future work.\tYour Name, You. (2015). \"Paper Title Number 3.\" <i>Journal 1</i>. 1(3).\tpaper-title-number-3\thttp://academicpages.github.io/files/paper3.pdf"
     ]
    }
   ],
   "source": [
    "!cat publications.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import pandas\n",
    "\n",
    "We are using the very handy pandas library for dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import TSV\n",
    "\n",
    "Pandas makes this easy with the read_csv function. We are using a TSV, so we specify the separator as a tab, or `\\t`.\n",
    "\n",
    "I found it important to put this data in a tab-separated values format, because there are a lot of commas in this kind of data and comma-separated values can get messed up. However, you can modify the import statement, as pandas also has read_excel(), read_json(), and others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pub_date</th>\n",
       "      <th>title</th>\n",
       "      <th>venue</th>\n",
       "      <th>excerpt</th>\n",
       "      <th>citation</th>\n",
       "      <th>url_slug</th>\n",
       "      <th>paper_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2009-10-01</td>\n",
       "      <td>Paper Title Number 1</td>\n",
       "      <td>Journal 1</td>\n",
       "      <td>This paper is about the number 1. The number 2...</td>\n",
       "      <td>Your Name, You. (2009). \"Paper Title Number 1....</td>\n",
       "      <td>paper-title-number-1</td>\n",
       "      <td>http://academicpages.github.io/files/paper1.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-10-01</td>\n",
       "      <td>Paper Title Number 2</td>\n",
       "      <td>Journal 1</td>\n",
       "      <td>This paper is about the number 2. The number 3...</td>\n",
       "      <td>Your Name, You. (2010). \"Paper Title Number 2....</td>\n",
       "      <td>paper-title-number-2</td>\n",
       "      <td>http://academicpages.github.io/files/paper2.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-10-01</td>\n",
       "      <td>Paper Title Number 3</td>\n",
       "      <td>Journal 1</td>\n",
       "      <td>This paper is about the number 3. The number 4...</td>\n",
       "      <td>Your Name, You. (2015). \"Paper Title Number 3....</td>\n",
       "      <td>paper-title-number-3</td>\n",
       "      <td>http://academicpages.github.io/files/paper3.pdf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     pub_date                 title      venue  \\\n",
       "0  2009-10-01  Paper Title Number 1  Journal 1   \n",
       "1  2010-10-01  Paper Title Number 2  Journal 1   \n",
       "2  2015-10-01  Paper Title Number 3  Journal 1   \n",
       "\n",
       "                                             excerpt  \\\n",
       "0  This paper is about the number 1. The number 2...   \n",
       "1  This paper is about the number 2. The number 3...   \n",
       "2  This paper is about the number 3. The number 4...   \n",
       "\n",
       "                                            citation              url_slug  \\\n",
       "0  Your Name, You. (2009). \"Paper Title Number 1....  paper-title-number-1   \n",
       "1  Your Name, You. (2010). \"Paper Title Number 2....  paper-title-number-2   \n",
       "2  Your Name, You. (2015). \"Paper Title Number 3....  paper-title-number-3   \n",
       "\n",
       "                                         paper_url  \n",
       "0  http://academicpages.github.io/files/paper1.pdf  \n",
       "1  http://academicpages.github.io/files/paper2.pdf  \n",
       "2  http://academicpages.github.io/files/paper3.pdf  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "publications = pd.read_csv(\"publications.tsv\", sep=\"\\t\", header=0)\n",
    "publications\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Escape special characters\n",
    "\n",
    "YAML is very picky about how it takes a valid string, so we are replacing single and double quotes (and ampersands) with their HTML encoded equivilents. This makes them look not so readable in raw format, but they are parsed and rendered nicely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "html_escape_table = {\n",
    "    \"&\": \"&amp;\",\n",
    "    '\"': \"&quot;\",\n",
    "    \"'\": \"&apos;\"\n",
    "    }\n",
    "\n",
    "def html_escape(text):\n",
    "    \"\"\"Produce entities within text.\"\"\"\n",
    "    return \"\".join(html_escape_table.get(c,c) for c in text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the markdown files\n",
    "\n",
    "This is where the heavy lifting is done. This loops through all the rows in the TSV dataframe, then starts to concatentate a big string (```md```) that contains the markdown for each type. It does the YAML metadata first, then does the description for the individual page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "for row, item in publications.iterrows():\n",
    "    \n",
    "    md_filename = str(item.pub_date) + \"-\" + item.url_slug + \".md\"\n",
    "    html_filename = str(item.pub_date) + \"-\" + item.url_slug\n",
    "    year = item.pub_date[:4]\n",
    "    \n",
    "    ## YAML variables\n",
    "    \n",
    "    md = \"---\\ntitle: \\\"\"   + item.title + '\"\\n'\n",
    "    \n",
    "    md += \"\"\"collection: publications\"\"\"\n",
    "    \n",
    "    md += \"\"\"\\npermalink: /publication/\"\"\" + html_filename\n",
    "    \n",
    "    if len(str(item.excerpt)) > 5:\n",
    "        md += \"\\nexcerpt: '\" + html_escape(item.excerpt) + \"'\"\n",
    "    \n",
    "    md += \"\\ndate: \" + str(item.pub_date) \n",
    "    \n",
    "    md += \"\\nvenue: '\" + html_escape(item.venue) + \"'\"\n",
    "    \n",
    "    if len(str(item.paper_url)) > 5:\n",
    "        md += \"\\npaperurl: '\" + item.paper_url + \"'\"\n",
    "    \n",
    "    md += \"\\ncitation: '\" + html_escape(item.citation) + \"'\"\n",
    "    \n",
    "    md += \"\\n---\"\n",
    "    \n",
    "    ## Markdown description for individual page\n",
    "        \n",
    "    if len(str(item.excerpt)) > 5:\n",
    "        md += \"\\n\" + html_escape(item.excerpt) + \"\\n\"\n",
    "    \n",
    "    if len(str(item.paper_url)) > 5:\n",
    "        md += \"\\n[Download paper here](\" + item.paper_url + \")\\n\" \n",
    "        \n",
    "    md += \"\\nRecommended citation: \" + item.citation\n",
    "    \n",
    "    md_filename = os.path.basename(md_filename)\n",
    "       \n",
    "    with open(\"../_publications/\" + md_filename, 'w') as f:\n",
    "        f.write(md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These files are in the publications directory, one directory below where we're working from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2009-10-01-paper-title-number-1.md  2015-10-01-paper-title-number-3.md\r\n",
      "2010-10-01-paper-title-number-2.md\r\n"
     ]
    }
   ],
   "source": [
    "!ls ../_publications/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\r\n",
      "title: \"Paper Title Number 1\"\r\n",
      "collection: publications\r\n",
      "permalink: /publication/2009-10-01-paper-title-number-1\r\n",
      "excerpt: 'This paper is about the number 1. The number 2 is left for future work.'\r\n",
      "date: 2009-10-01\r\n",
      "venue: 'Journal 1'\r\n",
      "paperurl: 'http://academicpages.github.io/files/paper1.pdf'\r\n",
      "citation: 'Your Name, You. (2009). &quot;Paper Title Number 1.&quot; <i>Journal 1</i>. 1(1).'\r\n",
      "---\r\n",
      "This paper is about the number 1. The number 2 is left for future work.\r\n",
      "\r\n",
      "[Download paper here](http://academicpages.github.io/files/paper1.pdf)\r\n",
      "\r\n",
      "Recommended citation: Your Name, You. (2009). \"Paper Title Number 1.\" <i>Journal 1</i>. 1(1)."
     ]
    }
   ],
   "source": [
    "!cat ../_publications/2009-10-01-paper-title-number-1.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}