From the Firehose

Moving on

After being a professor for 10 years, I have decided to hang up my whiteboard markers and move on from academia. This was a tough decision that I struggled with for a while, but now as I am actually doing it, it feels right. Many of you I have already told, others are still hearing about it, so I thought I'd make a proper post about it.

A lot of my reasons are family-related. We are a family with young kids, and want to be closer to grandparents in the next decade. Having three young boys in a town without family around was harder than we thought it would be. A lot of the move also has to do with timing. Given the place I was at ASU, I wanted to make a decision to either ramp up or ramp down. Based on many discussions with my wife, we're moving on.

I'm not sure what I'm doing next, but I know where we're doing it. We're moving North of the Bay Area, to Petaluma, CA, where I hope to start a business centered around the things I found the most joy in as a Professor -- helping others make their projects work -- through technology, design, and robotics. I'm going to spend some time talking to folks; I need to learn a lot -- how to get started, what to spend my time on, and who to spend it with. I would really like to talk with you, too.

I am wrapping up this semester with a lot of nostalgia. I want to thank everyone at ASU for supporting me, first as a young professor who needed a lot of care and feeding, and more recently as a colleague, peer, and even mentor. I have worked with wonderful folks at my two academic homes (the Polytechnic School and then MSN), and I wish you all the best. Thanks especially to Binil Starly for understanding my situation and working with me in good faith. It made this last semester go well.

You haven't heard the last of me. I'm sure I'll be posting more on my website and sharing on LinkedIn, so get your "mute" button ready. And, even though I haven't called you out by name, thanks for investing your time in me. I've really valued the path I took and am glad you were a part of it -- it hasn't been wasted.

Sincerely,

Dan Aukes

I'm making a new website.

Background

A couple months ago I wrote about why I liked Markdown so much. Even though I have been using static site generators for over 10 years, and even though they do great, they haven't always satisfied every need that I have. They still produce a static site, and they haven't always delivered the stability that I need with some of the features you'd think they'd promise.

Static Site Generators

The whole idea of a static site generator is that you focus on the content rather than the formatting. Static site generators take in text files -- usually markdown -- which is little more than a text file with very light formatting guidelines baked in, and merge it with a template and theme of your choosing to produce a pretty, content-focused website. For me this was a game changer because it meant that I could twiddle with the details for a finite amount of time and then get down to writing. For me, fiddling with formatting always was a useful procrastination tool, and eliminating it from the writing process was key to keeping me focused longer.

Because static site generators depend mostly on markdown as the user-generated input for a new page, they have traditionally been oriented towards blogs and personal websites... anywhere where you can apply a consistent theme to every page and any customization can be provided by relatively straightforward tags, variables, or by organizing your content into themed directory structures where each directory might have a slightly different theme applied to it. If you want custom pages you need to fall back on HTML, and the theme and template of your choosing to produce anything that falls out of that norm. Therefore, one of the challenges in creating a static site is selecting a theme that matches your needs. Often you have to start with someone else's theme and then heavily modify it once you realize it's missing a key functionality.

Jekyll: The OG SSG

Over the years I have tried several different static site generators. First Jekyll, and then Hugo, and more recently MKDocs. Jekyll was a great entry-level static site generator written in Ruby that allowed me to apply what little I knew about web development to quickly get a small site up and running. Because I was using GitHub to host my site, I could start with a relatively simple project, and not worry about the build environment because it was included within GitHub. If you wanted to embed some customization into the page, you could embed it using templating logic that you could directly include into the HTML theme. The site would then use those tags to generate custom content on your page based on input data that either came from the markdown header or other external files. I was using Hugo before much of the templating could be subdivided and obfuscated away so I was quite used to a monolithic project that mixed both my content and my format.

Over the years, as my site grew, however, Jekyll started to struggle to process the hundreds of pages, many of which included nested for loops and other complex templating logic that had started to outgrow its build tool. It would take five to ten minutes to generate the site after a small change, and if I happened to make a mistake, it would be quite a long time before I could iteratively guess and check my way through a fix. I'm sure this is because my use of the templating engine grew and the complexity of the different kinds of pages I was generating grew with it. Something that I had liked at the beginning, which was the inclusion of templating engine tags directly in the HTML and the project I was working on, slowly became a chain around my neck that was dragging me down, making my project more cumbersome and breakable.

Another issue that began to crop up was as I became more and more dependent on plugins and options that were custom to specific package versions, I had to create and maintain my own specific Jekyll build environment. Installing Ruby on Rails and all of the packages that I needed started to become more arduous as versions began conflicting with each other, died, or upgraded. My knowledge of Ruby and package management within the Ruby ecosystem had always been quite limited, and I never invested enough time to become an expert in it. But I also suspect that it also became more complicated as the years went by. By the end of my time using Jekyll, I had needed to create and maintain a custom Docker environment for each site, just to be able to build each website. This was a big deal at the time because I didn't really know Docker either.

Hugo: The promise of speed

So I switched over to Hugo because I had heard so many good things about compile speed. Indeed, it was fast. Hugo also advocated the division between content and theme, which I really appreciated as I had come to fear the complexity of my own project. I jumped in headfirst and converted many of my old Jekyll websites into Hugo websites.

And yet, some small niggling issues kept me from loving it. Hugo's documentation is vague at best. Even though they may have quite complete documentation covering every function and variable, it's more of a list of functions rather than an exhaustive set of examples, and unless you are already an expert at it, some of their explanations for how to use specific templating functions could be interpreted a number of ways. Understanding Hugo's limits became an iterative testing and checking process, and one that was not fixed by a standard, but constantly evolving with each version.

With documentation not so helpful, I struggled to find solutions to the kinds of problems I was looking for. A lot of solutions in their forum were from versions long ago that no longer applied, and no one seemed to be asking the questions that I was. Whether it was or wasn't, it felt like a small community. And very much like Jekyll, the time I spent developing the website had been at a very specific point in time. Yet, later, as I maintained it, and as the slow march of versions went up and up, my sites would suddenly stop building on the latest version of Hugo.

So here I am chaining myself to specific tool sets that are highly dependent on the combination of versions that I happened to use at the time. They are complex to learn, and yet still complex to maintain. Yes, I could make daily changes for a year or two without event, but sooner or later the system would break down, and I would have to go digging into the guts of my theme, my HTML or cycle through versions of the static site generator itself itself to understand what had changed and what I had to update to make my site build again. A lot of that long-term payoff promised by the idea of static site generation was left unrealized.

Aside: MKDocs is great

I do want to give credit to MKDocs, however. This handy tool has been the answer to many of my needs coming from the other two tools, and it solved it in a way that I didn't expect. No, it is not as customizable at first glance. But it is a batteries-included experience, especially if you use the material theme. The installation process is a lot simpler, especially since I am a python nerd and know how to work with Python environments. But more importantly, even though I've been using it for about five years, it seems to have remained much more consistent in producing buildable sites that don't need my intervention to continue working. This is because I rely less on a complex web of external plugins. The core functionality of MKDocs produces very usable sites with many different built-in plugins that are maintained by the MKDocs team. The one exception is the material theme. With this one theme, all of my other needs are generally supplied.

Realizing some gains

There had been another payoff. I had switched my life over to Markdown and it was now my medium of exchange. I have now used markdown for over 10 years now in all sorts of situations: for documentation, course material generation, and websites. I have also used it to generate my textbooks to generate slideshows. And the coolest part is that I can use the same material with minimal changes across vastly different formats and maintain a source code repository that has to change very little to be used in those different contexts. In conjunction with pandoc, hugo, latex, MKdocs, and other similar tools, markdown is now the way I compose and store the things that I write... And I am not about to change that.

Looking forward

So now what do I do? Looking forward, I am hoping to need more functionality then a static site generator . I want my new website to look unique as well. This leads me away from static site generation and MKdocs, even though it is great for what it does. But I wouldn't like to abandon the mental approach of writing in Markdown.

Now that I've set up the problem, follow my next post to find out where I am headed, what tools I've chosen so far, and some of my rationale. This is available in next week's post.

So what is the new website going to be?

Last week I discussed some of my experiences with various static site generators, and my motivation for moving away from them. This week, I discuss where I am headed.

Considerations

First, I have to consider the fact that I love Python and am very comfortable in it. There are plenty of Python-based web frameworks, including Flask, Fast API, and Django. I really haven't used any of them, though.

Second, I have to consider some of my potential use cases. What do I want to do with my website?

• I still want to be able to write in Markdown.
• I potentially want a different user experience for someone who is logged in versus someone who isn't, or for someone who is a subscriber or is not.
• I also want to play with different kinds of interactions. This might include JavaScript based plots, figures, games, or quizzes. It might also include the ability to interact with virtual terminals, or coding environments such as a Python/Jupiter notebook like experience.
• Perhaps someday I might want a payment portal, sell a product, or perhaps a subscription.
• I either want a toolset that I have more control over, in terms of versioning, access to and knowledge of the source code... Or the exact opposite, with a batteries included experience that I don't have to think about at all.
• I want to be able to start small and grow the site as my needs grow and my knowledge of what I need becomes more concrete.
• I'd like to be able to host the site on my own computer during development, but also deploy it to a third-party server without needing a PhD in IT.
• I want it to be able to play nice on the back end with other Python tools, because I'm such a heavy Python user. It has to be a Python environment for me.

My choices so far

So what are some of the things that I've selected?

Django for the framework

I'm going to go with Django because it has a very long track record. I also know that I'm not relatively strong in database management, and Django automates the creation and maintenance of the basic data types of your website.

Django also automates the creation of the administration side of your website, which I have found really appealing. I like how, with minimal effort, Im able to create a new data type and add or remove new instances of that type all from the web interface.

Docker for portability

I'm going to create my Django site within Docker. Since my early days of trying to use Docker and Jekyll together, I've become a bit more of an expert in how to create and maintain custom Docker images. I've also come to appreciate the ability of Docker to be tied to specific image versions, which would help me control when or where I upgrade specific parts of my site. Thirdly, Docker Compose files are really good at setting up complex interdependencies that, once working, don't seem to break that often. I have certainly had gripes relating to some of Docker's idiosyncrasies, but I've gotten over many of them just by digging into complex corner cases and figuring out how it works.

Docker will also allow me to create a working instance on my local computer that mirrors the final version that I deploy on a third-party site. Hopefully this will make me less dependent on whatever third-party host I select.

Markdown at the gooey center

Core to my website update would be the inclusion of markdown functionality.

Django does not play with markdown at all, and though I know that there are some django derivatives out there that allow you to combine the two, my worry is that, given some of my other needs, they won't do exactly what I want. So for now I'm going to use the same Markdown to HTML conversion tool that MkDocs uses, which is the python-markdown library. More on how I solved that problem in an upcoming article.

Other needs: functional code blocks

In terms of website functionality, I think my most important functional addition is going to be how to render the embedded code within my markdown, and control its presentation. I would like code to have the option to be numbered, to be copyable, and to be able to control the formatting and color of my code separately from the rest of the page.

Search -- an open question

I'm going to want to include some sort of search tool. After including it in the previous iteration of my website, I found it to be invaluable for finding my own content. I know that Django provides an easy way to search, and I will try that, but I appreciated the fuzzy nature of my previous search tool and may just keep that for now. That means that instead of implementing search, I will need to implement some sort of JSON view of my site

Database selection: MariaDB for now

In terms of database, I'm going for now with MariaDB, and SQLite for internal development. I don't work with databases directly that often, but I will manage if I can select a database that is well documented and has a large community of people who have had similar problems as me. Picking a database that also has lots of support tools such as MySQL derivatives is probably a smart choice too.

Styling: Bootstrap for continuity

In terms of CSS frameworks, I am sheepishly going to stick with Bootstrap, though I will be upgrading to Bootstrap 5. Right now I don't want to learn something new in CSS. I just want something that I sort of understand and can deploy quickly.

Other Considerations

In terms of other functionality, coming from academia, I will still probably want some sort of math environment like MathJax. I might want some sort of formal referencing system that works with a citation manager, but I can forgo that for now. I also hope to include a javascript library for image galleries, and for mermaid (a handy javascript-based chart generation tool)

Django Defense

Why did I pick Django? That decision-making process seems to have been glossed over, but for me one of the biggest selling points of Django is the vast trove of documentation, both official and around the web. For many of the problems I've been having, it seems someone has had that problem before, which gives me hope that the community is large enough to sustain a relatively robust community knowledge. Django has been around for so long that there are also verifiable experts on it who have made it their career, their textbooks, and I don't think that I would really get stuck because it's in Python.

But the real selling point was that about five years ago I walked through a relatively straightforward tutorial of how to build a library's website in Django from the Mozilla Developers Network website. I found this walkthrough detailed and helpful. It revealed how flexible Django was and yet how simple it was to build and maintain something quite complex. For me at the time, and still now, it revealed Django to be a good blend between expert tool and yet still a good fit for a novice. For someone who wants to be able to iterate quickly and simply, but if need be, get under the hood and mess around with things, I think this is a good balance. Let's see if I eat my words.