tom/personal_site
1
0
Fork 0
mirror of https://github.com/TomHodson/tomhodson.github.com.git synced 2025-06-26 10:01:18 +02:00
Code Issues Packages Projects Releases Wiki Activity
personal_site/markdown_generator/talks.ipynb

381 lines
14 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"# Talks markdown generator for academicpages\n",
"\n",
"Takes a TSV of talks with metadata and converts them for use with [academicpages.github.io](academicpages.github.io). This is an interactive Jupyter notebook ([see more info here](http://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html)). The core python code is also in `talks.py`. Run either from the `markdown_generator` folder after replacing `talks.tsv` with one containing your data.\n",
"\n",
"TODO: Make this work with BibTex and other databases, rather than Stuart's non-standard TSV format and citation style."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import os"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data format\n",
"\n",
"The TSV needs to have the following columns: title, type, url_slug, venue, date, location, talk_url, description, with a header at the top. Many of these fields can be blank, but the columns must be in the TSV.\n",
"\n",
"- Fields that cannot be blank: `title`, `url_slug`, `date`. All else can be blank. `type` defaults to \"Talk\" \n",
"- `date` must be formatted as YYYY-MM-DD.\n",
"- `url_slug` will be the descriptive part of the .md file and the permalink URL for the page about the paper. \n",
" - The .md file will be `YYYY-MM-DD-[url_slug].md` and the permalink will be `https://[yourdomain]/talks/YYYY-MM-DD-[url_slug]`\n",
" - The combination of `url_slug` and `date` must be unique, as it will be the basis for your filenames\n",
"\n",
"This is how the raw file looks (it doesn't look pretty, use a spreadsheet or other program to edit and create)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"title\ttype\turl_slug\tvenue\tdate\tlocation\ttalk_url\tdescription\r\n",
"Talk 1 on Relevant Topic in Your Field\tTalk\ttalk-1\tUC San Francisco, Department of Testing\t2012-03-01\tSan Francisco, California\t\tThis is a description of your talk, which is a markdown files that can be all markdown-ified like any other post. Yay markdown!\r\n",
"Tutorial 1 on Relevant Topic in Your Field\tTutorial\ttutorial-1\tUC-Berkeley Institute for Testing Science\t2013-03-01\tBerkeley CA, USA\thttp://exampleurl.com\tThis is a description of your tutorial, note the different field in type. This is a markdown files that can be all markdown-ified like any other post. Yay markdown!\r\n",
"Talk 2 on Relevant Topic in Your Field\tTalk\ttalk-2\tLondon School of Testing\t2014-02-01\tLondon, UK\thttp://example2.com\tThis is a description of your talk, which is a markdown files that can be all markdown-ified like any other post. Yay markdown!\r\n",
"Conference Proceeding talk 3 on Relevant Topic in Your Field\tConference proceedings talk\ttalk-3\tTesting Institute of America 2014 Annual Conference\t2014-03-01\tLos Angeles, CA\t\tThis is a description of your conference proceedings talk, note the different field in type. You can put anything in this field."
]
}
],
"source": [
"!cat talks.tsv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import TSV\n",
"\n",
"Pandas makes this easy with the read_csv function. We are using a TSV, so we specify the separator as a tab, or `\\t`.\n",
"\n",
"I found it important to put this data in a tab-separated values format, because there are a lot of commas in this kind of data and comma-separated values can get messed up. However, you can modify the import statement, as pandas also has read_excel(), read_json(), and others."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>type</th>\n",
" <th>url_slug</th>\n",
" <th>venue</th>\n",
" <th>date</th>\n",
" <th>location</th>\n",
" <th>talk_url</th>\n",
" <th>description</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Talk 1 on Relevant Topic in Your Field</td>\n",
" <td>Talk</td>\n",
" <td>talk-1</td>\n",
" <td>UC San Francisco, Department of Testing</td>\n",
" <td>2012-03-01</td>\n",
" <td>San Francisco, California</td>\n",
" <td>NaN</td>\n",
" <td>This is a description of your talk, which is a...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Tutorial 1 on Relevant Topic in Your Field</td>\n",
" <td>Tutorial</td>\n",
" <td>tutorial-1</td>\n",
" <td>UC-Berkeley Institute for Testing Science</td>\n",
" <td>2013-03-01</td>\n",
" <td>Berkeley CA, USA</td>\n",
" <td>http://exampleurl.com</td>\n",
" <td>This is a description of your tutorial, note t...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Talk 2 on Relevant Topic in Your Field</td>\n",
" <td>Talk</td>\n",
" <td>talk-2</td>\n",
" <td>London School of Testing</td>\n",
" <td>2014-02-01</td>\n",
" <td>London, UK</td>\n",
" <td>http://example2.com</td>\n",
" <td>This is a description of your talk, which is a...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Conference Proceeding talk 3 on Relevant Topic...</td>\n",
" <td>Conference proceedings talk</td>\n",
" <td>talk-3</td>\n",
" <td>Testing Institute of America 2014 Annual Confe...</td>\n",
" <td>2014-03-01</td>\n",
" <td>Los Angeles, CA</td>\n",
" <td>NaN</td>\n",
" <td>This is a description of your conference proce...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" title \\\n",
"0 Talk 1 on Relevant Topic in Your Field \n",
"1 Tutorial 1 on Relevant Topic in Your Field \n",
"2 Talk 2 on Relevant Topic in Your Field \n",
"3 Conference Proceeding talk 3 on Relevant Topic... \n",
"\n",
" type url_slug \\\n",
"0 Talk talk-1 \n",
"1 Tutorial tutorial-1 \n",
"2 Talk talk-2 \n",
"3 Conference proceedings talk talk-3 \n",
"\n",
" venue date \\\n",
"0 UC San Francisco, Department of Testing 2012-03-01 \n",
"1 UC-Berkeley Institute for Testing Science 2013-03-01 \n",
"2 London School of Testing 2014-02-01 \n",
"3 Testing Institute of America 2014 Annual Confe... 2014-03-01 \n",
"\n",
" location talk_url \\\n",
"0 San Francisco, California NaN \n",
"1 Berkeley CA, USA http://exampleurl.com \n",
"2 London, UK http://example2.com \n",
"3 Los Angeles, CA NaN \n",
"\n",
" description \n",
"0 This is a description of your talk, which is a... \n",
"1 This is a description of your tutorial, note t... \n",
"2 This is a description of your talk, which is a... \n",
"3 This is a description of your conference proce... "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"talks = pd.read_csv(\"talks.tsv\", sep=\"\\t\", header=0)\n",
"talks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Escape special characters\n",
"\n",
"YAML is very picky about how it takes a valid string, so we are replacing single and double quotes (and ampersands) with their HTML encoded equivilents. This makes them look not so readable in raw format, but they are parsed and rendered nicely."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"html_escape_table = {\n",
" \"&\": \"&amp;\",\n",
" '\"': \"&quot;\",\n",
" \"'\": \"&apos;\"\n",
" }\n",
"\n",
"def html_escape(text):\n",
" if type(text) is str:\n",
" return \"\".join(html_escape_table.get(c,c) for c in text)\n",
" else:\n",
" return \"False\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating the markdown files\n",
"\n",
"This is where the heavy lifting is done. This loops through all the rows in the TSV dataframe, then starts to concatentate a big string (```md```) that contains the markdown for each type. It does the YAML metadata first, then does the description for the individual page."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"loc_dict = {}\n",
"\n",
"for row, item in talks.iterrows():\n",
" \n",
" md_filename = str(item.date) + \"-\" + item.url_slug + \".md\"\n",
" html_filename = str(item.date) + \"-\" + item.url_slug \n",
" year = item.date[:4]\n",
" \n",
" md = \"---\\ntitle: \\\"\" + item.title + '\"\\n'\n",
" md += \"collection: talks\" + \"\\n\"\n",
" \n",
" if len(str(item.type)) > 3:\n",
" md += 'type: \"' + item.type + '\"\\n'\n",
" else:\n",
" md += 'type: \"Talk\"\\n'\n",
" \n",
" md += \"permalink: /talks/\" + html_filename + \"\\n\"\n",
" \n",
" if len(str(item.venue)) > 3:\n",
" md += 'venue: \"' + item.venue + '\"\\n'\n",
" \n",
" if len(str(item.location)) > 3:\n",
" md += \"date: \" + str(item.date) + \"\\n\"\n",
" \n",
" if len(str(item.location)) > 3:\n",
" md += 'location: \"' + str(item.location) + '\"\\n'\n",
" \n",
" md += \"---\\n\"\n",
" \n",
" \n",
" if len(str(item.talk_url)) > 3:\n",
" md += \"\\n[More information here](\" + item.talk_url + \")\\n\" \n",
" \n",
" \n",
" if len(str(item.description)) > 3:\n",
" md += \"\\n\" + html_escape(item.description) + \"\\n\"\n",
" \n",
" \n",
" md_filename = os.path.basename(md_filename)\n",
" #print(md)\n",
" \n",
" with open(\"../_talks/\" + md_filename, 'w') as f:\n",
" f.write(md)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These files are in the talks directory, one directory below where we're working from."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2012-03-01-talk-1.md\t 2014-02-01-talk-2.md\r\n",
"2013-03-01-tutorial-1.md 2014-03-01-talk-3.md\r\n"
]
}
],
"source": [
"!ls ../_talks"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"---\r\n",
"title: \"Tutorial 1 on Relevant Topic in Your Field\"\r\n",
"collection: talks\r\n",
"type: \"Tutorial\"\r\n",
"permalink: /talks/2013-03-01-tutorial-1\r\n",
"venue: \"UC-Berkeley Institute for Testing Science\"\r\n",
"date: 2013-03-01\r\n",
"location: \"Berkeley CA, USA\"\r\n",
"---\r\n",
"\r\n",
"[More information here](http://exampleurl.com)\r\n",
"\r\n",
"This is a description of your tutorial, note the different field in type. This is a markdown files that can be all markdown-ified like any other post. Yay markdown!\r\n"
]
}
],
"source": [
"!cat ../_talks/2013-03-01-tutorial-1.md"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in New Issue View Git Blame Copy Permalink